OpenRLHF icon indicating copy to clipboard operation
OpenRLHF copied to clipboard
Published 1 month ago • verified publisher icon OpenRLHF

DeepSpeed AutoTP with Hybrid Engine

Open hijkzzz opened this issue 5 months ago • 2 comments

Please pip install git+https://github.com/deepspeedai/DeepSpeed.git which includes https://github.com/deepspeedai/DeepSpeed/commit/bc8c0db3b46b939cd3e96b0673d5964a676e4c52 and uses OpenRLHF 0.8.10.

We found that DeepSpeed AutoTP + ZeRO1 is more than 2× faster than ZeRO3.

hijkzzz avatar Aug 21 '25 01:08 hijkzzz

这么快

gzoftju avatar Aug 21 '25 10:08 gzoftju

请问通过哪些参数开启呢？通过更新deepspeed版本和设置--zero_stage 1 吗

YSQ-boop avatar Sep 04 '25 15:09 YSQ-boop