Stanwang1210

Did you initialize lora parameters without fine-tuning?

Did you correctly set the `--token_type` to `whisper_multilingual` ? See[ here](https://github.com/espnet/espnet/blob/435803c90defade8a76a9e51769b65c80ea98b1b/egs2/aishell/asr1/run_whisper_finetune.sh#L28C5-L28C38) From the figure you provide, the Chinese transcriptions look reasonable. Therefore, I doubt that there is something wrong with...

Sorry for the late reply. Could you please check whether the parameters in your checkpoint align with the original whisper checkpoint? So that we can tell it's not related to...

This PR is only to support the option of enabling `lora.mark_only_lora_as_trainable`. It is related to the dependency of espnetez. For the issue issue #5721, we will create another PR to...

@simpleoier First, your understanding toward `lora_only` is correct. However, I think that it's better to use `freeze_param` option to control which groups of parameters should be frozen or not. Simply...

So, do we have any further problems about this PR?

Sorry for the late reply. This bug occurs due to the attention implementation in s3prl frontend. Specifically, it uses `torch.nn.functional.multi_head_attention_forward.`, which only uses the q_proj.weight. So currently, please avoid using...

I encounter the same issue. Also an 8-sample loss for 1 second audio. Does anyone solve this issue without hard-coding ?

Thanks for you help Can't wait for the next Wednesday lol

The error seems to be unrelated to my PR ``` =========================== short test summary info ============================ FAILED test/espnet2/text/test_phoneme_tokenizer.py::test_text2tokens[g2p_en] - LookupError: ``` Can someone help me to fix this?