V0.6.0 Release Plan

[cp5555]

Release Manager

@cp5555

Endgame

• [x] Code freeze: August 22nd
• [x] Bug Bash date: August 22nd
• [ ] Release date: September 1st

Main Features

SuperBench Improvement

1. • [x] Support running on host directly without Docker (#358, #362)
2. • [x] Support running sb command inside docker image (#356)
3. • [x] Support ROCm 5.1.1 (#353, #354)
4. • [x] Support ROCm 5.1.3 (#361)
5. • [x] Fix bugs in data diagnosis (#355)
6. • [x] Fix cmake and build issues (#360)
7. • [x] Support automatic configuration yaml selection on Azure VM (#365)
8. • [x] Refine error message when GPU is not detected. (#368)
9. • [x] Add return code for Timeout (#383)
10. • [x] Update Dockerfile for NCCL/RCCL version, tag name, and verbose output. (#371)
11. • [x] Support node_num=1 in mpi mode (#372)
12. • [x] Update Python setup for require packages (#387)
13. • [x] Enhance parameter parsing to allow spaces in value (#397)
14. • [x] Support NO_COLOR for SuperBench output (#404)

Micro-benchmark Improvement

1. • [x] Fix issues in ib loopback benchmark (#369)
2. • [x] Fix stability issue in ib loopback benchmark (#386)

Distributed Benchmark Improvement

1. • [x] Pair-wise IB benchmark (#363)
2. • [x] Bug Fix in IB benchmark (#370, #375, #377, #396)
3. • [x] Topology-aware IB benchmark (#373, #381)

Data Diagnosis & Analysis

1. • [x] Add failure check function in data_diagnosis.py (#378)
2. • [x] Support Json and Jsonl in Diagnosis. (#388)
3. • [x] Add support to store values of metrics in data diagnosis. (#392, #399)
4. • [x] Support exit code of sb result diagnosis (#403)

Backlog

Inference Benchmark Improvement

1. Support VGG, LSTM, and GPT-2 small in TensorRT Inference Backend
2. Support VGG, LSTM, and GPT-2 small in ORT Inference Backend
3. Support more TensorRT parameters (Related to #366)

Data Diagnosis & Analysis

1. Support boxplot and outlier analysis

Document

1. Metric Reasoning Doc