DeepSpeed icon indicating copy to clipboard operation
DeepSpeed copied to clipboard
verified publisher icon microsoft

add sharded checkpoint loading for AutoTP path to reduce the peak mem…

Open sywangyi opened this issue 2 years ago • 2 comments

…ory in initialization stage

sywangyi avatar Mar 27 '23 02:03 sywangyi

@delock @yao-matrix

sywangyi avatar Mar 27 '23 02:03 sywangyi

Hi，@molly-smith, this PR is meant to reduce the host memory per Rank, support shard loading in AutoTP path, same with shard loading in kernel injection path.

sywangyi avatar Apr 04 '23 01:04 sywangyi