leon

https://github.com/EleutherAI/gpt-neox/blob/2534e3d76e320aba095894e7dc2a4b416a1ac8df/tools/convert_module_to_hf.py#L199 My understanding is that there should only be one set of layernorm parameters per layer, which should be the same across different mp ranks. Am I misunderstanding this? Please...