Shangpeng

Results 3 issues of Shangpeng

## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested....

## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested....

**Is your feature request related to a problem? Please describe.** In LLM CPT/SFT distributed training, each rank independently loads the data into CPU memory. This leads to 8x CPU memory...

NLP
community-request