Wu Houming

icon company Zhejiang University icon location Hangzhou,China icon location I'm Houming Wu, a Ph.D. student at Zhejiang University advised by Professor Ling Chen. My research focuses on systems for parallel and distributed training.

Results 2 issues of Wu Houming

Pipeline Schedule confused

1 comment

When I run “torchrun --rdzv-backend=c10d --rdzv-endpoint=localhost:29500 --nnodes=1 --nproc-per-node=4 test_pipeline_schedule.py --schedules gpipe”，I got the following outputs: ```shell [2023-12-03 08:40:53,722] torch.distributed.run: [WARNING] [2023-12-03 08:40:53,722] torch.distributed.run: [WARNING] ***************************************** [2023-12-03 08:40:53,722] torch.distributed.run: [WARNING] Setting...

NCCL stuck

6 comment

I tried to change the slurm script (i.e., prof_steps.sh) to torchrun and ran it directly, but encountered a stuck issue with NCCL as collective_backend. The torchran script is as follows:...