[CK2/CM] Development roadmap for 2022

[gfursin]

[gfursin]

Main CM phases for 2022:

• [x] Develop and stabilize CM core to treat R&D projects as a database of components and automations (completed: 20220419 - CK2/CM v0.7.7)
• [x] Document CM core and provide a few tutorials
• [ ] Develop CK2 scripts to detect and/or install all ML artifacts (platform description, OS scripts, ML frameworks, models, data sets, libraries, pre-/post-processing scripts, benchmarks, etc) (on-going): https://github.com/mlcommons/ck/issues/245
  • [x] Support for TF
  • [x] Support for ONNX
  • [ ] Support for TVM
  • [x] Support for PyTorch
  • [ ] Support for TFLite
• [x] Convert our old CK-based MLPerf inference vision benchmark automations using CM
• [ ] Implement CM-based "experiment" automation (record/reproduce/compare/visualize)
• [ ] Implement CM-based MLPerf Design Space Explorer for open division submissions
• [ ] Implement CM-based MLCubes for MLPerf DSE
• [ ] Convert all MLPerf inference reference benchmarks to CM

[gfursin]

Since we have stabilized the CM (CK2) framework and added CM scripts to automate ML Systems benchmarking, we can now work on decomposing MLPerf inference benchmark into these CM scripts and automating the submission: https://github.com/mlcommons/ck/issues/261 .

[gfursin]

After validating the new CM automation workflow for MLPerf at the Student Cluster Competition, we close this ticket and move to the updated roadmap: https://github.com/mlcommons/ck/issues/536 .