kang sheng comments

Results 19 comments of


                                            kang sheng

Add DDP token averaging for equivalent non-parallel training similar to #34191

Thanks, I already created the pull request and tested on my machine.

Fail to save megatron distributed checkpoints when using megatron as a back-end, and distributed checkpoint as actor model

You can use mbrige to save transformers' type checkpoint. You can refer to this script:https://github.com/volcengine/verl/blob/main/examples/grpo_trainer/run_qwen3-235b_megatron_96gb.sh#L96.

Fail to save megatron distributed checkpoints when using megatron as a back-end, and distributed checkpoint as actor model

It's related to mbridge. Maybe @ISEEKYAN can help.

Fail to save megatron distributed checkpoints when using megatron as a back-end, and distributed checkpoint as actor model

cc @ETOgaosion

Qwen3-30B-A3B OOM with GRPO on 4x8H200 141G

Can you test the recommanded script? https://github.com/volcengine/verl/blob/main/examples/grpo_trainer/run_qwen3moe-30b_megatron_96gb.sh

qwen3-235B-moe 在4台H100上能否训练成功

The script can also run on 4 H100 nodes, provided each node has sufficient CPU memory (>1.5 TB). We look forward to your feedback.

qwen3-235B-moe 在4台H100上能否训练成功

The `actor_rollout_ref.rollout.gpu_memory_utilization` is too high in your script. Please set to a lower value and test again. Maybe 0.7?

qwen3-235B-moe 在4台H100上能否训练成功

@ISEEKYAN @ETOgaosion Hello, do you have any ideas about this error?

qwen3-235B-moe 在4台H100上能否训练成功

@XQZZK Can you share with us what specific method you use/take?