Support Zero-3?

Open WF0511 opened this issue 1 year ago • 1 comments

I used accelerate launch with ZERO-3 to run train.llama.infini.noclm.1Mseq.sh. But I got this: RuntimeError: Function 'LinearFunctionForZeroStage3Backward' returned nan values in its 0th output

Jun 25 '24 06:06 WF0511

I have this question too

Jul 03 '24 05:07 Liuzirui666