czyysx

How to run GPipe in distributed manner?

8

I was able to run GPipe with lm.one_billion_wds.OneBWdsGPipeTransformerWPM in a single node with multiple GPUs. However, I am a little confused about how to run GPipe with multiple nodes (or...