sotabench-api icon indicating copy to clipboard operation
sotabench-api copied to clipboard

Run hash should give a warning for empty/all-zero inputs

Open rstojnic opened this issue 6 years ago • 1 comments

In some cases the first batch might be too small, or simply contain no interesting predictions (eg empty or all zeros).

Run hash calculation should catch this, produce a warning and not try to retrieve the cached results.

rstojnic avatar Sep 30 '19 10:09 rstojnic

This might be an issue for the frontends (sotabencheval, torchbench) and is task-specific.

I've solved for COCO in sotabencheval by doing the following:

  • Check whether all the bboxs in a list of dicts are empty
  • If they are all empty, then don't calculate a batch hash (wait until the first addition that has detections)

RJT1990 avatar Sep 30 '19 10:09 RJT1990