zakinp

Why does the 'loss=nan' appear when finetuning model using fp16 or bf16 mixed precision in 'fine_tune.py' ?

2

Same as title. This problem has existed since the v9 version. It is related to the GPU type. It appears in 3090, A40, but doesn't appear in A100.

TypeError: int() argument must be a string, a bytes-like object or a number, not 'list'

6

i got this problem when using "diffusers_trainer.py": Exception caught on rank 0 at step 10, saving checkpoint... int() argument must be a string, a bytes-like object or a number, not...