effusiveperiscope comments

Repositories
Issues
Comments

Results 4 comments of


                                            effusiveperiscope

train_second.py model.decoder error (output tensor is nan)

I have experienced this before in a few situations: - actual model parameters are not being loaded from the checkpoint (there is some weird naming error involving "module" prefix between...

Extremely weird DDP issue for train_second.py

Has anyone here tried the fix by @stevenhillis?

Extremely weird DDP issue for train_second.py

> > Has anyone here tried the fix by @stevenhillis? > > where is the fix? broadcast_buffers=False in DistributedDataParallelKwargs. Seems to work OK if you remove the isnan() check (I...

Extremely weird DDP issue for train_second.py

Doesn't seem to work with slmadv training though.