algorithmic-efficiency
algorithmic-efficiency copied to clipboard
Add tests for scoring code
Add unit and integration tests to test the following requirements:
In both strict=False and strict=True, to receive a finite score for a workload a submission must:
- Reach the validation target on the fixed workload within the maximum runtime.
- Reach the validation target fixed workload within 4x of the fastest submission.
In strict=True, to receive a finite score for a workload a submission must:
- Reach the validation target for at least 3/5 studies (the median).
- Take the best over 5 trials per study.
- Reach the validation target on the held-out workload (corresponding to the fixed workload) within the maximum runtime.
- Reach the validation target on the held-out workload (corresponding to the fixed workload) within 4x of the fastest submission.