Reworking the tests

[f-hafner]

This issue tracks my work on reworking the tests of the community codes, following discussion with @LourensVeen (please correct where I'm wrong). I will keep it updated as I learn new things.

There are a bunch of related issues (list is probably not complete):

• https://github.com/amusecode/amuse/issues/504
• https://github.com/amusecode/amuse/issues/458

@LourensVeen has already copied the tests into each of the community codes (but not deleted the old ones IIUC). I will work on the test files inside the community codes.

Our intention is to

• make the tests more maintainable and more modular where possible
• use pytest built-in methods for assertions where possible (ie, pytest.approx, testdir)
• consider replacing parts or all of the infrastructure in amusetest.py with pytest fixtures

We will not improve the quality of the tests; this needs to be done by someone with domain knowledge.

I have started on reworking the tests for bhtree. From this work, in the medium term, it'll be necessary that

1. Tests should run with and without MPI. I don't understand exactly what the TestWithMPI class does on my local machine, but tests pass.
2. Tests should run on different hardware (CPU, GPU)
3. the AmuseTestCase can handle units, and I believe we'll have to port some of this functionality to pytest (which I have started)

For now, I will focus on using pytest, refactoring the tests and running them on my machine. At some point we should think about how we'll expand the tests for the above requirements.

I suggest to have an entire dev branch for this issue (on my own fork), but have smaller pull requests (for instance, one PR per community code) that we merge into this dev branch.

Further

• remove all plotting in tests

Pull requests per community code

• [ ] bhtree
• [x] generic n-body tests

[LourensVeen]

Small correction: with the new build system we are building everything with MPI, always. So there's no non-MPI version that needs to be tested. For point 2., the more general case is that each test should be run for each package. For Huayno for example, that means single-threaded, openmp and opencl.

Agreed on everything else, good plan!

[f-hafner]

Starting a list here on things we have encountered but are not fixing for the moment

• We're moving functionality from src/tests/amusetest.py into src/amuse/support/testing/. Our plan is to rework the tests for the community codes, and gradually adopt the testing methodology defined in src/amuse/support/testing/. However, the amuse framework itself also depends on amusetest.py, and we do not plan on changing this. So, amusetest.py is kept for compatibility.

[LourensVeen]

Link to partial work: https://github.com/f-hafner/amuse/tree/issue-1129-reworking-tests