ColossalAI icon indicating copy to clipboard operation
ColossalAI copied to clipboard
verified publisher icon hpcaitech

[FEATURE]: Is hybrid parallelism supported in GPT demo?

Open nrailg opened this issue 2 years ago • 2 comments

Describe the feature

I found only DP and ZeRO strategy supports in ColossalAI/applications/Chat/examples, is hybrid parallelism (like PP / Megatron) supported?

nrailg avatar Apr 06 '23 14:04 nrailg

If you are looking for examples of training GPT2, there is one solution with our interesting auto-parallelism tool.

JThh avatar Apr 06 '23 18:04 JThh

Hi @nrailgun DP+ZERO is a simple way and enough for most cases. Examples of hybrid parallelism for large models will be provided soon. Thanks.

binmakeswell avatar Apr 17 '23 06:04 binmakeswell