pytest-split icon indicating copy to clipboard operation
pytest-split copied to clipboard

Tests are incorrectly split when there are a lot of slow skipped tests

Open knedlsepp opened this issue 2 years ago • 0 comments

I'm running pytest-split on a test suite that has a lot of slow tests which are pytest.marked as slow and skipped by default. I can include all these tests by using a custom pytest --all-tests option. The issue I'm seeing now is that when I create a .test_durations file via pytest --all-tests --store-durations then the regular pytest --splits 4 ... run without the --all-tests will be unbalanced. It seems as though pytest-split is taking into account the runtime of the skipped tests and thus failing to produce a balanced split. A workaround is to store one .tests_durations file for pytest --all-tests and a separate file for the plain pytest, but I think ideally this should not be required.

knedlsepp avatar Sep 07 '23 07:09 knedlsepp