sotabench-api
sotabench-api copied to clipboard
Run hash should give a warning for empty/all-zero inputs
In some cases the first batch might be too small, or simply contain no interesting predictions (eg empty or all zeros).
Run hash calculation should catch this, produce a warning and not try to retrieve the cached results.
This might be an issue for the frontends (sotabencheval, torchbench) and is task-specific.
I've solved for COCO in sotabencheval by doing the following:
- Check whether all the bboxs in a list of dicts are empty
- If they are all empty, then don't calculate a batch hash (wait until the first addition that has detections)