upwindflys

Results 9 comments of upwindflys

Horovod0.24.2 +pytorch 1.10.2 throw unexpected fault

> I guess you should run it with command "horovodrun -np 2 -H localhost:2 python pytorch_mnist.py" ? Thanks.I have modified the spelling error.

Horovod0.24.2 +pytorch 1.10.2 throw unexpected fault

I found a solution that reinstall horovod == 0.22.Is there any other tips?

Hanging when running transformer in distributed setting

@jinliangwei Thanks for your solution ,but I encountered an oom error,in the meanwhile,free memory is enough,tensorflow and tensor2tensor are both 1.8,Can you offer some suggestions?

Hanging when running transformer in distributed setting

@jinliangwei thanks,but that doesn't seem to be this problem,thanks anyway.

Hanging when running transformer in distributed setting

@jinliangwei sorry to ask you a question again.What kind of physical environment do you use?such as the cuDNN CUDA version?

[BUG]: run_gemini.sh RuntimeError: Error building extension 'fused_optim'

same problem

[BUG]: How can i save checkpoint trained by the way of GeminiDDP

Thank you for the reply. But I'm not sure whether this method ```model_to_save = model.module if hasattr(model, 'module') else model``` is a way of ```get_static_torch_model ```? this method's checkpoint is...

Expected performance

@axiniu I met the same problem,Has the problem been solved now?Thanks a lot

ValueError: Incompatible shapes between op input and calculated input gradient.

@AbelardLiu Thanks a lot,Fix works for me.But why out.set_shape does not work,I feel confused.