Wang, Yi

oneccl_bindings_for_pytorch 1.12.0 prebuilt wheel does not work with …

2

…PyTorch 1.12.1 raise error in this condition and update the doc Signed-off-by: Wang, Yi A Fixes # (issue) torch 1.12.1 does not work with intel ccl 1.12.0 ## Before submitting...

torch 1.12.1 does not work with intel ccl 1.12.0

4

Signed-off-by: Wang, Yi A

fix AutoTP in deepspeed could not work for bloom

5

# What does this PR do? Fixes # (issue) fix AutoTP in deepspeed could not work for bloom ## Before submitting - [ ] This PR fixes a typo or...

add low_cpu_mem_usage option in run_clm.py example which will benefit…

2

… LLM loading add low_cpu_mem_usage option in run clm example, set the option True will help reduce peak memory and loading time in LLM finetune and inference.

Update automatic-tensor-parallelism.md

1

remove bloom from unsupported Models

add sharded checkpoint loading for AutoTP path to reduce the peak mem…

2

…ory in initialization stage

do not use self.device. In FSDP cpu offload mode. self.device is "CPU…

2

…" instead of "cuda" and there's error like "Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:1"

add accelerate example for DDP and FSDP in sequence classification fo…

1

…r non-lora case