DeepSpeed icon indicating copy to clipboard operation
DeepSpeed copied to clipboard
verified publisher icon microsoft

[REQUEST] Examples for tensor parallelism and pipeline parallelism

Open x54-729 opened this issue 2 years ago • 1 comments

I noticed there are some settings about tensor parallelism in DeepSpeedEngine and PipielineEngine. Can you please provide us with some examples of combinig tensor parallelism with pipeline parallelism? I would be appreciate if there is a clear example to follow.

x54-729 avatar May 06 '23 06:05 x54-729

+1. I am also confusing how to train a model with both tensor parallelism and pipeline parallelism in deepspeed.

chenyaofo avatar Jun 08 '23 03:06 chenyaofo

Great question. It would be even better if it could be combined with DDP.

fwyc0573 avatar Jan 08 '24 10:01 fwyc0573