jMonkeyEngine
Create a screenshot based set of tests for jMonkeyEngine
As discussed at https://hub.jmonkeyengine.org/t/screenshot-based-jme-testing/47685/18 I will be creating a set of automated tests that can run as part of the build on standardised hardware. The purpose of this is to reduce the amount of human testing required to assure ourselves that a new JME version has not broken some graphics based behaviour (which is difficult to test under normal automated testing)
Objectives:
- Human readable html report including before and after screenshots for manual review
- Diff screenshot showing where in the image there is a change
- Easy (documented) mechanism for accepting changes
- Failed pipeline on test fail
- Ability for some tests to be marked as "non deterministic" and only triggering manual human review, not a failed test
- Ability for some tests to be marked as "known broken" and only triggering manual human review, not a failed test
(I don't think I have rights to assign myself to this issue, but for the avoidance of doubt I'm working on this)
It seems you already worked on that, but just a heads up that I did so, too, a few years ago. It could even run in docker using some CPU pipeline: https://github.com/MeFisto94/jme3-testing
@MeFisto94 Thank you for sharing that, thats interesting. Anything you think I need to watch out for from your previous experience? Did your docker version use cpu openGL rendering as well?