keepsake
keepsake copied to clipboard
Better validation for --filter string
We should validate the names you put in a filter string, e.g. replicate ls --filter "foo<0.2", where the name is foo.
What kind of additional validation are you thinking? Currently the output when a filter is invalid is
Failed to parse filter: "98".
│
│ Filters must be in the format "<name> <operator> <value>",
│ where <operator> can be
│ "=" (equal),
│ "!=" (not equal),
│ "<" (less than),
│ "<=" (less than or equal),
│ ">" (greater than), or
│ ">=" (greater than or equal)
@aaronleesmith Currently we don't validate the <name> component of the filter string. For example, if foo is a non-existing parameter or metric for a set of experiments, we return results anyway:
$ replicate ls --filter="foo > 0"
No experiments found
$ replicate ls --filter="foo != 0"
EXPERIMENT STARTED STATUS LEARNING_RATE LATEST CHECKPOINT LOSS BEST CHECKPOINT LOSS
f34b197 2020-11-13 stopped 0.01 979ca4b (step 45) 0.34695 979ca4b (step 45) 0.34695
93dab9f 2020-11-13 stopped 0.2 d6817db (step 61) 0.061989 d6817db (step 61) 0.061989
Instead we should output an error along the lines of Invalid filter: "foo" not a parameter or metric for any of the experiments in this project.