centerformer icon indicating copy to clipboard operation
centerformer copied to clipboard

Is it correct that Nan appears in the loss?

Open cjfghddl0206 opened this issue 2 years ago • 5 comments

I ran the code as written on github. However, after a certain point, the loss is all Nan. I think it's a loss of the dataset, so I recreated the pkl file with create_data.py, but Nan comes out as it is. Is it correct to run the training to the end even if Nan comes out?

cjfghddl0206 avatar May 11 '23 05:05 cjfghddl0206

maybe you need to change the batch size, I met this situation as well.

DezeZhao avatar May 11 '23 09:05 DezeZhao

배치 크기를 변경해야 할 수도 있습니다. 이 상황도 만났습니다.

Did you change the batch size from how many to how many?

cjfghddl0206 avatar May 11 '23 11:05 cjfghddl0206

8 to 4

DezeZhao avatar May 11 '23 11:05 DezeZhao

8 to 4

Where should I edit it? I can't see the batch size 8 part.

cjfghddl0206 avatar May 11 '23 12:05 cjfghddl0206

8 to 4

Where should I edit it? I can't see the batch size 8 part.

samples_per_gpu

zhaowenZhou avatar May 18 '23 10:05 zhaowenZhou