ColossalAI
ColossalAI copied to clipboard
[autoparallel] Add alpha-beta estimation
This is the first version of alpha-beta estimation.
What can it do It aims to return device ids and the corresponding alpha-beta for a given logical device mesh.
How to use
cd ~/ColossalAI/tests/test_auto_parallel/test_tensor_shard/test_device
export CUDA_VISIBLE_DEVICES=0,1,2,3
python test_alpha_beta.py
Plan
- [x] Get 1D alpha-beta
- [x] Get 2D alpha-beta for 4 devices
- [ ] Get 2D alpha-beta for 8 devices
Pitfalls The latency may have a large variance (~30%) caused by the framework.
Please improve your code based on https://www.codefactor.io/repository/github/hpcaitech/colossalai/pull/1791.