algorithmic-efficiency icon indicating copy to clipboard operation
algorithmic-efficiency copied to clipboard

Add tests for scoring code

Open priyakasimbeg opened this issue 2 years ago • 0 comments

Add unit and integration tests to test the following requirements:

In both strict=False and strict=True, to receive a finite score for a workload a submission must:

  • Reach the validation target on the fixed workload within the maximum runtime.
  • Reach the validation target fixed workload within 4x of the fastest submission.

In strict=True, to receive a finite score for a workload a submission must:

  • Reach the validation target for at least 3/5 studies (the median).
  • Take the best over 5 trials per study.
  • Reach the validation target on the held-out workload (corresponding to the fixed workload) within the maximum runtime.
  • Reach the validation target on the held-out workload (corresponding to the fixed workload) within 4x of the fastest submission.

priyakasimbeg avatar Feb 01 '24 19:02 priyakasimbeg