python-package-template icon indicating copy to clipboard operation
python-package-template copied to clipboard

Revisit CI testing strategies

Open behnazh opened this issue 4 years ago • 1 comments

We need to revisit how to trigger the CI checks based on the events (pull_request vs. push), branches (main, staging, and the rest), and matrices (e.g., OS and Python versions).

Potentially we could remove tox, but then it might make sense to enforce the CI tests on every push event for all the branches. Currently CI tests is only enabled for push to main and pull_request for other branches.

We also need to document the branching model and how it affects running the CI checks.

behnazh avatar Jan 16 '22 22:01 behnazh

Potentially we could remove tox, but then it might make sense to enforce the CI tests on every push event for all the branches.

I agree: we use tox to run pytest with coverage on a [py39, py310] matrix, but that matrix is already being expanded with OSs in PR #99. I think it makes little sense to run unit tests locally across different Python versions (which may not even be installed on the system) if we can do so more extensively in CI; it’s probably enough if unit tests pass locally in whatever venv the user works.

Given that, the full test matrix (three OS flavors times two Python versions) should run at least with every merge/push to the main branch. This repository also uses a staging branch into which PRs are merged, and a “release” is a merge from staging into main. So, merge/push to the staging branch should also run the full test matrix.

What about opening a PR though, and every push to an open PR — run the full test matrix every time?

jenstroeger avatar Jan 17 '22 04:01 jenstroeger