StatCan
Plan Epic for Kubeflow Automated Testing
Jupyter-apis
For this repo, we already have Cypress setup and intergraded into our github workflow so that the tests run when we update the code. So for this repo, we would just need to define new test cases that we want covered, since we currently are only taking what upstream has given us, and it is very lacking in coverage. So these test scenarios could, but aren't limited to, test:
- the creation of every one of our types of notebooks (So test creating a R notebook, a SAS notebook, etc...)
- the different validation on our inputs to make sure that works as expected
- the creation of a protected b notebook
- various different configurations of volumes (so test new workspace volumes, attaching existing volumes as data volumes, etc...)
- various non-default values for our inputs (mainly thinking about not always using the default cpu and ram values; mixing it up a little)
- that the dashboard displays the correct data in the tables
- that the connect and delete buttons work on notebooks and volumes
- that the correct gets displayed when accessing the detail pages
These possibly don't have to all be single individual test cases and we can possibly combine some of them, for example testing the creation of a remote desktop image while also making sure that our inputs enforce their correct validators (we don't want to test the validators in every test case, that would just be way too much, but as long as it's covered once).
Nexts steps for this would be to possibly define exact test cases. We probably don't have to do 1 github issue per test case. But what we can do is define the test cases exactly in english so that afterwards its just a matter of converting the english instructions to cypress commands.
Kubeflow Central Dashboard
Central dashboard seems to have some frontend tests made with Jasmine. Although I am not familiar with Jasmine, it seems to just be another Javascript testing framework (described as being a DOM-less behavior-driven development framenwork). The way tests are written seems to be fairly similar to cypress.
I don't know why upstream is using different frameworks (Jasmine and Cypress) for their frontend testing. My theory is that is has something to do with Jasmine possibly being able to bypass the login and authorization session with it being a dom-less framework, but that is really just a theory.
We could probably look at normalizing our End to End testing and relace Jasmine with Cypress in central dashboard to be like with jupyter-apis.
Also, it seems like we don't currently run the central dashboard Jasmine frontend tests. There is a workflow for it but we aren't triggering it. We should probably enable that workflow so that the frontend tests run automatically when we make changes.
As for what to test, the only test currently implemented in Jasmine by upstream was to test the registration page, which is a feature that we don't have anymore. So that test can be removed and if we do implement Cypress for the central dashboard, that works out for us since we would be starting from scratch anyways.
What could be tested in central dashboard is:
- managing contributors
- the menu links
- the landing page when a user is not associated to a namespace.
Most of our logic is in the jupyter-apis repo, so there will definitely be less tests associated to central-dashboard
We have frameworks in place to test just the jupyter-apis component, and we have something in place that could test just the central-dashboard component. But we don't currently have something in place to test the entire package and making sure the connection between jupyter-apis and central-dashboard.
That isn't as much of a priority as testing those components on their own, but I am just thinking of if we could test the interaction between jupyter-apis and some common components found in central-dashboard like the namespace downdown (or if we eventually have a french/english toggle button).
If we proceed with this though, we would have to create a new repo or something since we might not be able to centralize those tests between those two components. Although, if we can have central-dashboard connect with dev jupyter-apis or something like that so that we can test the whole suite of components that we have, we might be able to write all those tests just in central-dashboard.