pytest-dev
raising an exception pytest_runtest_teardown causes all all further tests to fail
Raising an exception in pytest_runtest_teardown in one test will cause an exception in all subsequent tests. This is a regression from 5.2.4 and earlier releases. I use exceptions in pytest_runtest_teardown to assert resource cleanup for all tests regardless of other fixtures that are used. It's unusual to fail in this manner, but this makes it look like all tests are failing.
Here is the output from pytest 6.0.1, where it marks test_a as a failure in the expected fashion. However it marks test_b as a failure with assert colitem in self.stack. If there are more tests, all tests will be marked with the same failure message as test_b.
============================= test session starts ==============================
platform linux -- Python 3.8.2, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/colvin/pytest_repro
collected 2 items
test_exception.py .E.E [100%]
==================================== ERRORS ====================================
_________________________ ERROR at teardown of test_a __________________________
item = <Function test_a>, nextitem = <Function test_b>
def pytest_runtest_teardown(item, nextitem):
global raised
if not raised:
raised = True
> assert False, "fails raising a plain assert"
E AssertionError: fails raising a plain assert
E assert False
conftest.py:9: AssertionError
_________________________ ERROR at teardown of test_b __________________________
cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f83ab0d4940>
when = 'teardown'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
@classmethod
def from_call(
cls,
func: "Callable[[], _T]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: "Optional[Union[Type[BaseException], Tuple[Type[BaseException], ...]]]" = None,
) -> "CallInfo[_T]":
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result = func() # type: Optional[_T]
../pytest-venv/lib/python3.8/site-packages/_pytest/runner.py:294:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../pytest-venv/lib/python3.8/site-packages/_pytest/runner.py:247: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
../pytest-venv/lib/python3.8/site-packages/pluggy/hooks.py:286: in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
../pytest-venv/lib/python3.8/site-packages/pluggy/manager.py:93: in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
../pytest-venv/lib/python3.8/site-packages/pluggy/manager.py:84: in <lambda>
self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
../pytest-venv/lib/python3.8/site-packages/_pytest/runner.py:166: in pytest_runtest_teardown
item.session._setupstate.teardown_exact(item, nextitem)
../pytest-venv/lib/python3.8/site-packages/_pytest/runner.py:402: in teardown_exact
self._teardown_towards(needed_collectors)
../pytest-venv/lib/python3.8/site-packages/_pytest/runner.py:417: in _teardown_towards
raise exc
../pytest-venv/lib/python3.8/site-packages/_pytest/runner.py:410: in _teardown_towards
self._pop_and_teardown()
../pytest-venv/lib/python3.8/site-packages/_pytest/runner.py:370: in _pop_and_teardown
self._teardown_with_finalization(colitem)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <_pytest.runner.SetupState object at 0x7f83ab11fc70>
colitem = <Function test_b>
def _teardown_with_finalization(self, colitem) -> None:
self._callfinalizers(colitem)
colitem.teardown()
for colitem in self._finalizers:
> assert colitem in self.stack
E AssertionError
../pytest-venv/lib/python3.8/site-packages/_pytest/runner.py:391: AssertionError
=========================== short test summary info ============================
ERROR test_exception.py::test_a - AssertionError: fails raising a plain assert
ERROR test_exception.py::test_b - AssertionError
========================= 2 passed, 2 errors in 0.04s ==========================
Here's is the output from pytest 5.4.3
=============================== test session starts ================================
platform linux -- Python 3.8.2, pytest-5.4.3, py-1.9.0, pluggy-0.13.1
rootdir: /home/colvin/pytest_repro
collected 2 items
test_exception.py .E. [100%]
====================================== ERRORS ======================================
___________________________ ERROR at teardown of test_a ____________________________
item = <Function test_a>, nextitem = <Function test_b>
def pytest_runtest_teardown(item, nextitem):
global raised
if not raised:
raised = True
> assert False, "fails raising a plain assert"
E AssertionError: fails raising a plain assert
E assert False
conftest.py:9: AssertionError
============================= short test summary info ==============================
ERROR test_exception.py::test_a - AssertionError: fails raising a plain assert
============================ 2 passed, 1 error in 0.01s ============================
To reproduce, I use two files:
conftest.py
import pytest
raised = False
def pytest_runtest_teardown(item, nextitem):
global raised
if not raised:
raised = True
assert False, "fails raising a plain assert"
test_exception.py
def test_a():
assert True
def test_b():
assert True
pip list from python 3.8.2 on ubuntu 20.04
Package Version
-------------- -------
attrs 19.3.0
iniconfig 1.0.1
more-itertools 8.4.0
packaging 20.4
pip 20.0.2
pkg-resources 0.0.0
pluggy 0.13.1
py 1.9.0
pyparsing 2.4.7
pytest 6.0.1
setuptools 44.0.0
six 1.15.0
toml 0.10.1
wcwidth 0.2.5
Bisected to bb878a2b13508e88ace7718759065de56a63aac5 (runner: don't try to teardown previous items from pytest_runtest_setup) from #7368 - cc @bluetech
Thanks for the great report @torcolvin, and @The-Compiler for the bisection.
What happens here is:
- Your
pytest_runtest_teardownhook impl is registered last, and so runs first. - The hookimpl raises.
- Other hookimpls don't run.
pytest itself uses the pytest_runtest_teardown hook for some work, and assumes that it runs. Thus the scenario breaks it.
I consider it more of an issue with the user code than a regression in pytest, but maybe others will disagree.
In any case the simplest fix for you would be
@pytest.hookimpl(trylast=True)
def pytest_runtest_teardown(item, nextitem):
....
This makes your hookimpl run after pytest does its thing. This would have been a good idea even in previous pytest versions.
i think using pytest.hookimpl(trylast=True) could be a good idea in the use case I have this case, but if more than one plugin registers a teardown hook, we can end up right back where we started. I can see why this situation might be unavoidable, but maybe the exception assert colitem in self.stack could say something about this happening because a previous teardown raised an exception? Or maybe an exception raised here should terminate the test harness, since further results are invalidated?
I can see why this situation might be unavoidable, but maybe the exception assert colitem in self.stack could say something about this happening because a previous teardown raised an exception?
I think we can improve that exception message as you suggest.
FWIW, I used an autouse fixture for the same purpose in the past:
@pytest.fixture(autouse=True)
def ensure_cleanup():
yield
cleanup_all_resources()
Fixtures raising an exception are expected and pytest handles them correctly, but with hooks things are a bit more delicate.
@nicoddemus hi，I get the same error, but I don't use teardown_class. I annotate teardown_class in my code, and I get an assert colitem in self.stack error;
@allure.feature("vscode")
@allure.severity("blocker")
class TestVscodeNoteBookServer():
def setup_class(self):
self.vscode_args = get_arguments_from_yaml(module = "engineering-api", yaml_name = "VscodeNoteBookServer")
self.client = NotebookFunction(engineering_client = global_utils.get_value('client').engineering_client)
# def teardown_class(self):
# project_id = self.vscode_args['project_id']
# notebooks = self.client.ListNotebookServers(project_id)
# self.client.BatchDeleteNotebook(notebooks, "at-vscode")
@allure.title("vscode notebook")
@allure.severity("blocker")
@pytest.mark.run(order=1)
def test_vscode_notebook_create(self):
"""
my code
"""
pass
"""more case"""
@nicoddemus In addition, I have also tried this way of your example, and it is not without practical effect, My main test scenario needs to use multithreading, maybe pytest has multithreading support, but there is no way to completely resolve the assert colitem in self.stack error? This problem has troubled me for a long time. I tried to solve this problem, but I failed to solve it in many ways, which made me very sad
@pytest.fixture(scope="class")
def setup_and_teardown():
logging.info("Setting up...")
jupyter_args = get_arguments_from_yaml(module="engineering-api", yaml_name="JupyterNoteBookServer")
client = NotebookFunction(engineering_client=global_utils.get_value('client').engineering_client)
yield jupyter_args, client
logging.info("Tearing down...")
project_id = jupyter_args['project_id']
notebooks = client.ListNotebookServers(project_id)
client.BatchDeleteNotebook(notebooks, "at-jupyter")
@allure.feature("jupyter notebook")
@allure.severity("blocker")
class TestJupyterNoteBookServer():
# def setup_class(self):
# jupyter_args = get_arguments_from_yaml(module = "engineering-api", yaml_name = "JupyterNoteBookServer")
# client = NotebookFunction(engineering_client = global_utils.get_value('client').engineering_client)
# def teardown_class(self):
# project_id = jupyter_args['project_id']
# notebooks = client.ListNotebookServers(project_id)
# client.BatchDeleteNotebook(notebooks, "at-jupyter")
@allure.title("jupyter notebook")
@allure.severity("blocker")
@pytest.mark.run(order=1)
def test_jupyter_notebook_create(self, setup_and_teardown):
"""my code"""
pass
# other cases