Are timeouts supposed to result in Error or Failure or both?

[kaddkaka]

I'm running a smaller testsuite and moste of the tests fail 10 seconds timeout because of slow setup code (currently residing inside the tests). Some of the timeouted tests result in Error some in Failure:

FAILED base/software/test_external_call.py::TestExtCall::test_can_call_external_lib[/local/path.slow-mode] - Failed: Timeout >10.0s
ERROR base/software/test_external_call.py::TestExtCall::test_can_call_external_lib[/local/path.fast-mode] - Failed: Timeout >10.0s

Why? Is this intentional?

If I increase my timeout the ERRORs seem to go away and all timeouted tests are listed as FAILUREs.

[RonnyPfannschmidt]

pytest considers exception/errors in setup/teardown an error (as that shouldnt happen) if it happens in a test, its as expected

i beleive timeout should transition to a new outcome type as a timeout is more similar to a xfail

[kaddkaka]

pytest considers exception/errors in setup/teardown an error (as that shouldnt happen) if it happens in a test, its as expected

i beleive timeout should transition to a new outcome type as a timeout is more similar to a xfail

That sounds reasonable and "Timeout" outcome sounds good. Is there work ongoing or another issue to link this one to?

[RonnyPfannschmidt]

Not yet

[flub]

On Fri 22 Mar 2024 at 02:48 -0700, Ronny Pfannschmidt wrote:

i beleive timeout should transition to a new outcome type as a timeout is more similar to a xfail

can plugins define new outcome types?

[The-Compiler]

I believe plugins can set any outcome as long as they handle it appropriately in a pytest_report_teststatus.