igor0 issues

Repositories
Issues
Comments

Results 1 issues of


                                            igor0

Fine-tuning GPT-NeoX doesn't work (for many scenarios) with the 16-bit stage-0 optimizer

**TL,DR:** The non-ZeRO ("stage 0") optimizer in DeepSpeed makes fragile assumptions about the optimizer state in the checkpoint, even when ``finetune: true`` configuration parameter is set. A mitigating factor is...

bug