Setup and teardown handling in tests

[skrchnavy]

Motivation:

Good practice for testing is to define test that can be started without definition of order. So it require to have expected environment (fixture) before test starts.

Problem description:

Integration tests in PR #123 are run against DB and creates there some data. These data could be not proper for other tests, because developer of new test have to be aware of data created in other tests and also have to be able to run tests against empty database. This could be difficult and also a place for errors when not all situations properly handled.

Proposals:

• To divide tests into suites where each suite runs against own database, setup and teardown implemented on suite level
• To implement proper setup and teardown in each test

Some useful links

with solutions/proposal for similar issues:

• http://martinfowler.com/bliki/GivenWhenThen.html
• https://phpunit.de/manual/current/en/database.html
• http://davedevelopment.co.uk/2015/10/26/setting-up-a-database-fixture.html

[mlespiau]

Hi,

The approach I have in mind (and which I'm willing to implement) is to let each test run in parallel and create it's own data by using the GivenWhenThen gherkin pattern.

I've already done some research on how to implement this but there is no out-of-the-box solution. We need to:

• update test environment to allow starting/destroying multiple tests environments dynamically
• create a task that generates a list of all test to be executed
• generate a process that consumes the list of tests by creating a new environment, running the test, destroying the environment
• create a task that collects all the results and provides a unified report

Data fixtures per feature tag are an improvement from what we have so far, but in the long run they are a lot harder to maintain.