Mike

Just adding a params key to the evaluator that takes a dict should work. If you want to keep it clean you could use a [TypedDict](https://www.python.org/dev/peps/pep-0589/) (it's available in [mypy_extensions](https://github.com/python/mypy_extensions))...

I'm trying to hack together a human + model in the loop dataset management / annotation tool (for computer vision and nlp). It includes: 1. JSON based REST API (django...