inference icon indicating copy to clipboard operation
inference copied to clipboard

host_processor_vcpu_count in <system_desc_id>.json

Open szutenberg opened this issue 1 year ago • 2 comments

According to https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#57-system_desc_idjson-metadata it's required to provide field host_processor_vcpu_count.

I see that relevant checks are missing in https://github.com/mlcommons/inference/blob/master/tools/submission/submission_checker.py so the submitter get submission_checker.py:3240 WARNING] <redacted>, field host_processor_vcpu_count is unknown.

My proposal is to remove host_processor_vcpu_count from the submission rules because this rule is not enforced anyway. In inference_results_v3.1 this field is present only in closed/Intel-HabanaLabs/systems/HLS-Gaudi2-PT.json.

szutenberg avatar Feb 20 '24 15:02 szutenberg

If we provide only host_processor_vcpu_count instead of host_processor_core_count then submission checker fails:

submission_checker.py:3220 ERROR] <REDACTED>, field host_processor_core_count is missing
submission_checker.py:3240 WARNING] <REDACTED>, field host_processor_vcpu_count is unknown

szutenberg avatar Feb 20 '24 15:02 szutenberg

A related issue: https://github.com/mlcommons/inference/issues/1351

arjunsuresh avatar Feb 20 '24 16:02 arjunsuresh