Elias Mizan

I was aware we had this doc, but we have not been maintaining it: https://docs.google.com/document/d/1VOwh8OS4W2Ev9GkO7o_fOruL4ODECsE3/edit#heading=h.gjdgxs

In the 2.1 round a submitter failed to provide any source code and the submission got accepted. This should not happen as source code is critical to the review process....

This has been motivated by the recurring problem that OEMs using code of another submitter do not bother to change the submission_org, submission_platform, submission_division, and submission_status in their logs. This...

Opening this per the discussion we had in the meeting. Setting this as a 2.1 issue. We need to see all the packages needed by the logging infra (python, numpy,...

Use original RCP for RCP test if batch size and ALL hparams of submission match the RCP, otherwise used pruned RCPs.

1

This is a comment made by the training committee, related to issue #206. If the submission hyperparams match exactly to an RCP in the database, then we should compare against...

Address 1.1 review period issues

2

The 1.1 review period was problematic. From the perspective of someone who has worked on the logger, we were too lenient and a lot of submissions that failed compliance (single-file,...

Add hparams and compliance checks for training and eval samples for all benchmarks

3

According to issue https://github.com/mlcommons/submission_training_1.0/issues/39, the number of training samples is 117266. Many submissions do hardcode this value, even though the reference does not. On that specific issue the submission in...

A list of ML Perf log constants is listed in file https://github.com/mlcommons/logging/blob/master/mlperf_logging/mllog/constants.py. Right now the list of constants is at least incomplete and we should figure out if and how...

Seed checker gets confused when handling multiple implementations

8

There is a bug in the seed checker: On this line we see that https://github.com/mlcommons/logging/blob/9ede9c6f2d1c8e6c02b6442ee15c64b29c8ecebb/mlperf_logging/package_checker/package_checker.py#L123 the seed checker checks all source directories under implementation. So if we have two implementations,...

New directory structure proposal.

2

Here is a proposal by Peter to change the submission directory structure. The first objective is to have inference and training submission directories as similar as possible. The second is...