ColossalAI icon indicating copy to clipboard operation
ColossalAI copied to clipboard

[autoparallel] Add alpha-beta estimation

Open zhang677 opened this issue 3 years ago • 1 comments

This is the first version of alpha-beta estimation.

What can it do It aims to return device ids and the corresponding alpha-beta for a given logical device mesh.

How to use

cd ~/ColossalAI/tests/test_auto_parallel/test_tensor_shard/test_device
export CUDA_VISIBLE_DEVICES=0,1,2,3
python test_alpha_beta.py

Plan

  • [x] Get 1D alpha-beta
  • [x] Get 2D alpha-beta for 4 devices
  • [ ] Get 2D alpha-beta for 8 devices

Pitfalls The latency may have a large variance (~30%) caused by the framework.

zhang677 avatar Nov 02 '22 09:11 zhang677

Please improve your code based on https://www.codefactor.io/repository/github/hpcaitech/colossalai/pull/1791.

FrankLeeeee avatar Nov 03 '22 04:11 FrankLeeeee