Leo

Assertion Error when Setting pipe_parallel_size or model_parallel_size in GPT-NeoX

3

Hello, I am encountering an issue with the GPT-NeoX library. When I set either `pipe_parallel_size` or `model_parallel_size` to 2, I get the following assertion error: ``` [2024-07-11 06:28:59,215] [INFO] [real_accelerator.py:203:get_accelerator]...

Hello,  I've recently trained a model using GPT-NeoX and packed the checkpoint global_step1000 into a pytorch_model.bin file using the zero_to_fp32.py script. However, I'm having trouble figuring out how to...