Michal Szutenberg
Michal Szutenberg
According to https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#57-system_desc_idjson-metadata it's required to provide field host_processor_vcpu_count. I see that relevant checks are missing in https://github.com/mlcommons/inference/blob/master/tools/submission/submission_checker.py so the submitter get `submission_checker.py:3240 WARNING] , field host_processor_vcpu_count is unknown`. My...
We found that [evaluation.py](https://github.com/mlcommons/inference/blob/master/language/gpt-j/evaluation.py) is not deterministic. I narrowed down to small and fast reproducer using 100 examples which are already decoded. Reproducer code: ```python import numpy as np import...
I have analyzed [download_cnndm.py](https://github.com/mlcommons/inference/blob/master/language/gpt-j/download_cnndm.py) and [dataset.py](https://github.com/mlcommons/inference/blob/master/language/gpt-j/dataset.py): * validation dataset is used, not test. Why? * there are 13368 examples which is expected ( (https://huggingface.co/datasets/cnn_dailymail#data-splits](https://huggingface.co/datasets/cnn_dailymail#data-splits) ) * Problem A: 399 examples...
The script [https://github.com/mlcommons/inference/tree/master/language/gpt-j](https://github.com/mlcommons/inference/tree/master/language/gpt-j) refers to [https://github.com/badhri-intel/inference](https://github.com/badhri-intel/inference) - it asks to clone this repository instead of using mlcommons/inference. I think it's a mistake and we shouldn't refer to external github repositories...
Proposed fix for https://github.com/mlcommons/inference/issues/1631
It ends up that we have even three identical directories (see NVIDIA submission) which does not help with the review and understanding the repo by someone not involved in the...