Establish testing of containerized apps

[yarikoptic]

Could probably be

• regression tests
  • eventually some more advanced "testkraut"-like tests which annotate various steps in the execution (may be reproman's ability to trace into containers could be of help here, although ATM we do not react for every sub-command execution there)
• per family generic tests (e.g. to verify that all bids-apps or boutiques have consistent interface)

Existing tests (not that many! ;) )

• For Flywheel gears generated by gearificator, ideally should just reuse those tests defined under tests/ subdirectories of https://github.com/yarikoptic/gearificated-nipype/tree/master/gears/nipype
• For some boutiques -- could come from boutiques specs
• Some containers might have tests defined "internally" (or executed at package/container build time). E.g. SCI-F Singularity framework allows for them.

So it seems that we need a testing framework which for image(s) would list a set of "tests drivers" to pickup/execute tests, so we could point to various tests specifications and get them executed.

• IIRC boutiques testing implementation had some nice "integration" within pytest but I've not tried it yet
• good old testkraut by @mih defines tests specification (see testcase.py with examples under localtests and populates tests for nose (or pytest!) to pick them up -- so something to base things on possibly

NB with `pytest` it is possible to execute individual ones! (nose relies on import, fails)

(git)hopa:~exppsy/testkraut[master]git
$> python -m pytest -s -v testkraut/tests/test_localtests.py::LocalDogFoodTests::test_check_assertions                                  
================================================================= test session starts ==================================================================
platform linux2 -- Python 2.7.14+, pytest-3.10.1, py-1.7.0, pluggy-0.8.0 -- /home/yoh/deb/gits/pkg-exppsy/testkraut/venvs/dev/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/yoh/deb/gits/pkg-exppsy/testkraut/.hypothesis/examples')
rootdir: /home/yoh/deb/gits/pkg-exppsy/testkraut, inifile:
plugins: xdist-1.26.1, localserver-0.5.0, forked-1.0.1, cov-2.6.0, hypothesis-3.71.11, celery-4.2.1
collected 1 item                                                                                                                                       

testkraut/tests/test_localtests.py::LocalDogFoodTests::test_check_assertions PASSED

Attn @djarecka , @glatard

[glatard]

Indeed, Boutiques generates pytest tests from their JSON definition and runs them with pytest. For example, you can run bosh test tests_good.json where tests_good.json is https://raw.githubusercontent.com/boutiques/boutiques/master/tools/python/boutiques/schema/examples/tests_good.json

[yarikoptic]

Notes (to myself primarily):

• even if tests are bundled within the container, the ultimate testing scenario is to execute them "from outside", i.e. input data (where any) should be provided via bind mounts etc, options specified/passed through command line since the primary goal is to verify correct operation of the container as to be used in a present environment.
• Having said above, it might be nice to also be able to run "internal" tests within the container, again given the default bind mounts etc we would provide to see that there is no side effects of any kind from our way to execute the container