dask
WIP: Add unified test suite infrastructure
Closes #8.
The idea is that there is a lot of tests that we would want to run for all the FooCluster classes. At the moment there is some duplication in test_sge.py, test_slurm.py, etc ... (e.g. test_basic, test_adaptive, ...) and some cluster-generic functionality are only tested for some clusters (e.g. only test_pbs.py has some tests for scaling with "grouped workers" e.g. test_scale_grouped).
This is quite WIP at the moment but if you have some early feed-back about the approach don't hesitate! The main thing to look at is test/test_cluster.py. The way it currently works is you add a check_whatever(cluster, client) function in tests/test_cluster.py and there is a parametrized test function at the end that will run the tests for all the check_* functions and all the FooCluster classes.
This is somewhat influenced by how the "common tests" are structured in scikit-learn (this tests some common properties of all the scikit-learn estimators, for example that fitting twice with the same data gives the same result) and there may be a more clever way of doing it.
A few things I want to look at still (besides cleaning up all the TODO):
- [ ] it would be great to differentiate
checkfunctions that need a real cluster (basically that need to do.scale). This could be based on a naming convention e.g.check_real_clustervscheckor be in different filestest_real_cluster.pyvstest_cluster.py(but then there would be some duplication). Maybe a clever pytest trick would do it (can we add pytest marks to normal functions and detect that) or even use a simple decorator that adds an attributed to the function and we can test this attribute in the parmetrizedtest. - [ ] some documentation in the module and/or in a contributing guide
- [ ] potentially other things that I will add later
In principle this seems fine to me. Thank you for paying attention to the test suite. I agree that it could be better unified.
