Linaro
test plan support
This ticket is to track progress of 'test plan' support in SQUAD. Test plan means list of tests that are expected to appear in the results. Once all expected tests are received the build should be considered 'finished' and notifications should be sent. It's up to discussion what to do with any extra results that might appear. When not all expected tests appear we should fall back to usual timeout approach.
https://github.com/Linaro/squad/issues/488 is another topic that discuss how to define finished builds
@mwasilew I think a "TestPlan" model could be linked to a one-to-one relationship with Project. A "TestPlan" could be a tuple like (suite/testName, env, enabled/disabled), where testName and env should always be unique. Here are my thoughts about the other considerations:
-
If we receive all the expected test results, then a notification can be sent out right away, as you mentioned. But, in case not all expected tests were received, then notification timeout is used as threshold to determine build finished state.
-
If we receive extra test results, we merge them to the TestPlan of that Project. If a test is unwanted in a TestPlan, a user could just removed it later on the UI.
How does that sound?
I would like test plan to be very explicit. So adding a test to plan implicitly should not happen. Test plan should not be mandatory. It can be enabled/disabled for Project. How do we share test plans between projects? Do we want to share? In case of LKFT for example there are 2 test plans for the whole group. So maybe it should not be 1-1 with Project?
Do you mean 1-1 with Group? I liked that. Maybe TesPlans could act like EmailTemplates, a group may have several test plans available, and they are shared among its projects. What do you think?
I also liked the idea of enabling/disabling test plans.
Wrt to test plan being explicit, I was thinking about an easy way to manage it, i.e. add/remove tests from/to the TestPlan. For instance, depending on the project, LKFT has over 26 thousand tests, and numbers vary along builds. I don't understand why that happens yet, though.
maybe 1 group -> many test plans is a better model.
Editing test plans is a biggest pain to solve IMHO. Enabling/disabling can be done on a project level. And I don't think we want to 'merge' test plans for a project (or maybe we do?). A feature of 'export existing tests as test plan' in some API sounds useful in this context. That would be used to seed the initial test plan. Than users should edit test plans by hand.
maybe 1 group -> many test plans is a better model.
Agreed
Exporting/importing would be the best way to go in order to do batch addition. Maybe a TestPlan json file would do the job. Also a way to manage test plans manually should be good.
Next week I'll think of something then open a PR so we can discuss it there. I think having a testplan improve the way we handle build states