fzuo1230

Unable to save checkpoint during multi-GPU training

3

Hello, I am training wan-2.1-i2v-14b on 8-card A800, but when saving the checkpoint, I get the following error: Training failed: [Errno 17] File exists: '/mmu_mllm_hdd_2/zuofei/VideoTuna/results/train/train_wanvideo_i2v_fullft_20250612153116/checkpoints/flow' [rank1]: Traceback (most recent call...

hello, i was training wan2.1-14b-i2v-720p, but i found that i get two loss print after one step: Average Epoch time: 231.10 seconds Average Peak memory 68550.57MiB Epoch 0: 1%|▏ |...