keepsake icon indicating copy to clipboard operation
keepsake copied to clipboard

Better validation for --filter string

Open andreasjansson opened this issue 5 years ago • 2 comments

We should validate the names you put in a filter string, e.g. replicate ls --filter "foo<0.2", where the name is foo.

andreasjansson avatar Nov 12 '20 22:11 andreasjansson

What kind of additional validation are you thinking? Currently the output when a filter is invalid is

 Failed to parse filter: "98".
   │ 
   │ Filters must be in the format "<name> <operator> <value>",
   │ where <operator> can be
   │   "=" (equal),
   │   "!=" (not equal),
   │   "<" (less than),
   │   "<=" (less than or equal),
   │   ">" (greater than), or
   │   ">=" (greater than or equal)

aaronleesmith avatar Nov 23 '20 15:11 aaronleesmith

@aaronleesmith Currently we don't validate the <name> component of the filter string. For example, if foo is a non-existing parameter or metric for a set of experiments, we return results anyway:

$ replicate ls --filter="foo > 0"
No experiments found

$ replicate ls --filter="foo != 0"
EXPERIMENT  STARTED     STATUS   LEARNING_RATE  LATEST CHECKPOINT  LOSS      BEST CHECKPOINT    LOSS
f34b197     2020-11-13  stopped  0.01           979ca4b (step 45)  0.34695   979ca4b (step 45)  0.34695   
93dab9f     2020-11-13  stopped  0.2            d6817db (step 61)  0.061989  d6817db (step 61)  0.061989  

Instead we should output an error along the lines of Invalid filter: "foo" not a parameter or metric for any of the experiments in this project.

andreasjansson avatar Nov 23 '20 20:11 andreasjansson