Shangpeng
Shangpeng
Ape base
## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested....
## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested....
**Is your feature request related to a problem? Please describe.** In LLM CPT/SFT distributed training, each rank independently loads the data into CPU memory. This leads to 8x CPU memory...