DeepSpeed icon indicating copy to clipboard operation
DeepSpeed copied to clipboard

[Question] why are overlap and contiguous grads meaningless in stage 1 and are ignored

Open woolpeeker opened this issue 3 years ago • 0 comments

https://github.com/microsoft/DeepSpeed/blob/80f94c10c552ec79473775adb8902b210656ed76/deepspeed/runtime/engine.py#L1384

I wonder why we cannot use overlap_comm in zero1 to reduce more latency? Appreciate any reply.

woolpeeker avatar Sep 06 '22 16:09 woolpeeker