ERROR of Flash_attn when finetuning with Deepspeed

[VVNMA]

I tried to finetune Vicuna using my own data with deepspeed, however, I met the following error:

I tried to solve this error by changing torch and deepspeed version, I tried deepspeed0.8.2, deepspeed0.8.3, torch 1.13.1, torch2.0.0, but the error were the same.

Does anyone else meet the same error or you know how to solve this problem?

[kungfu-eric]

Removing bfloat and fp32 flag and adding fp16 flag should get further

[zhisbug]

@kungfu-eric Yep -- hope @kungfu-eric 's solution helps. Feel free to re-open if there is still an issue.