inference icon indicating copy to clipboard operation
inference copied to clipboard
verified publisher icon mlcommons

Need to fix TEST01 performance check

Open arjunsuresh opened this issue 1 year ago • 2 comments

This performance checker file for TEST01 has two main issues

  1. It has a lot of redundant code
  2. Regular expressions are used to extract performance value from the summary log instead of using mlperf log parser to extract the official performance value from the mlperf log detail like done by the submission checker. This also means, early stopping is not respected by the TEST01 check.

Since, the code freeze is in place, this can be a v5.0 postmortem item.

arjunsuresh avatar Feb 01 '25 00:02 arjunsuresh

PR exists but needs cleaned up. Arjun to do so this next week.

swasson488 avatar May 06 '25 17:05 swasson488

PR is here: https://github.com/mlcommons/inference/pull/2073/files

arjunsuresh avatar May 13 '25 15:05 arjunsuresh

Closing this issue since the PR has been merged.

hanyunfan avatar Oct 06 '25 16:10 hanyunfan