oslo icon indicating copy to clipboard operation
oslo copied to clipboard
verified publisher icon EleutherAI

Integration ZeroDDP and ShardedModelv2 from colossal AI

Open dongsungkim opened this issue 3 years ago • 1 comments

Describe a TODO feature

There are two version of Zero support from ZeroDDP and ShardedModelv2

  1. check the possibility to merge two into one
  2. Otherwise, it has a function to choose one of them based on the flag (not shown to users directly)

Assignees

  • Dongsung and Hyen

dongsungkim avatar Nov 06 '22 11:11 dongsungkim

https://github.com/hpcaitech/ColossalAI/blob/6e51d296f07c0ad34d7f85cf9a70d4ceee15ede7/colossalai/zero/sharded_optim/low_level_optim.py#L29

There is updates about Zero-DP in Colossal AI. Zero 1 -> Low_level_optimizer Zero 2 -> Low_level_optimizer (partitioned gradient =True) Zero 3 -> Sharded optimizer v2, sharded modelV2

hyeinhyun avatar Nov 22 '22 12:11 hyeinhyun