Megatron-DeepSpeed icon indicating copy to clipboard operation
Megatron-DeepSpeed copied to clipboard
verified publisher icon bigscience-workshop

The difference between zero-3 and megatron with zero-2

Open nicosouth opened this issue 2 years ago • 0 comments

hi, I looked up a lot of information.

but I still don't understand the difference between zero-3 and megatron with zero-2.

they all split the model.

nicosouth avatar Aug 25 '23 04:08 nicosouth