pytest-mypy icon indicating copy to clipboard operation
pytest-mypy copied to clipboard
verified publisher icon realpython

`mypy-status` is not very useful if it triggers `-x`/`--exitfirst`

Open madduck opened this issue 8 months ago • 2 comments

The mypy plugin here falls over, complains about errors, but there aren't any…

% pytest -s -vvv --mypy
===================================== test session starts =====================================
platform linux -- Python 3.13.3, pytest-8.3.5, pluggy-1.5.0 -- /home/madduck/code/tc/tptools/.direnv/python-3.13.3/bin/python3
cachedir: .pytest_cache
rootdir: /home/madduck/code/tc/tptools
configfile: pyproject.toml
testpaths: tests, integration
plugins: anyio-4.9.0, cov-6.1.1, devtools-0.12.2, asyncio-0.26.0, subtests-0.14.1, mypy-1.0.1
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
collected 352 items

tests/__init__.py::mypy PASSED
tests/__init__.py::mypy-status FAILED

========================================== FAILURES ===========================================
________________________________________ test session _________________________________________
mypy exited with status 1.
-------------------------------------- Captured log call --------------------------------------
filelock DEBUG Attempting to acquire lock 140525569634576 on /home/madduck/.tmp/tmpw90y_v42.lock
filelock DEBUG Lock 140525569634576 acquired on /home/madduck/.tmp/tmpw90y_v42.lock
filelock DEBUG Attempting to release lock 140525569634576 on /home/madduck/.tmp/tmpw90y_v42.lock
filelock DEBUG Lock 140525569634576 released on /home/madduck/.tmp/tmpw90y_v42.lock
============================================ mypy =============================================
Found 4 errors in 1 file (checked 27 source files)
=================================== short test summary info ===================================
FAILED tests/__init__.py::mypy-status
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================= 1 failed, 1 passed in 0.81s =================================

However:

% mypy tests/__init__.py
Success: no issues found in 1 source file
% echo $?
0

madduck avatar Jun 04 '25 14:06 madduck

This is a duplicate of #120 (but the write-up below is more thorough/clear, so I think we'll keep this ticket going forward)...

tl;dr: There are errors in a file that was collected by Pytest, but the status check triggered -x which prevented those errors from showing up.

Here's an SSCCE:

$ tree 
.
├── Pipfile
├── Pipfile.lock
└── tests
    └── bad.py

1 directory, 3 files

$ pipenv run pytest --mypy --randomly-seed=4137011734 -vx tests/bad.py 
====================== test session starts =======================
platform linux -- Python 3.10.12, pytest-8.4.0, pluggy-1.6.0 -- /home/dtux/.local/share/virtualenvs/tmp.8zHtPjayyy-wB_JhAoz/bin/python
cachedir: .pytest_cache
Using --randomly-seed=4137011734
rootdir: /tmp/tmp.8zHtPjayyy
plugins: mypy-1.0.1, randomly-3.16.0
collected 2 items                                                

tests/bad.py::mypy-status FAILED                           [ 50%]

============================ FAILURES ============================
__________________________ test session __________________________
mypy exited with status 1.
============================== mypy ==============================
Found 1 error in 1 file (checked 1 source file)
==================== short test summary info =====================
FAILED tests/bad.py::mypy-status
!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!
======================= 1 failed in 0.28s ========================

It happens when all 3 of the following conditions are met:

  1. A module that was collected by Pytest contains a Mypy error. Without this condition (i.e. if there were an error in a file that was not collected), we would see the error in the terminal summary: 
    $ echo 'import bad' > tests/indirect.py
$ pipenv run pytest --mypy --randomly-seed=4137011734 -vx tests/indirect.py
====================== test session starts =======================
platform linux -- Python 3.10.12, pytest-8.4.0, pluggy-1.6.0 -- /home/dtux/.local/share/virtualenvs/tmp.8zHtPjayyy-wB_JhAoz/bin/python
cachedir: .pytest_cache
Using --randomly-seed=4137011734
rootdir: /tmp/tmp.8zHtPjayyy
plugins: mypy-1.0.1, randomly-3.16.0
collected 2 items                                                

tests/indirect.py::mypy PASSED                             [ 50%]
tests/indirect.py::mypy-status FAILED                      [100%]

============================ FAILURES ============================
__________________________ test session __________________________
mypy exited with status 1.
============================== mypy ==============================
tests/bad.py:1: error: Incompatible types in assignment (expression has type "str", variable has type "int")  [assignment]
Found 1 error in 1 file (checked 1 source file)
==================== short test summary info =====================
FAILED tests/indirect.py::mypy-status
!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!
================== 1 failed, 1 passed in 0.28s ===================
  2. The status check (i.e. MypyStatusItem) runs before any other failure. Without this condition, one of the files containing Mypy errors (or any other failed test) would have triggered condition 3. 
    $ pipenv run pytest --mypy --randomly-seed=4264782955 -vx tests/bad.py 
====================== test session starts =======================
platform linux -- Python 3.10.12, pytest-8.4.0, pluggy-1.6.0 -- /home/dtux/.local/share/virtualenvs/tmp.8zHtPjayyy-wB_JhAoz/bin/python
cachedir: .pytest_cache
Using --randomly-seed=4264782955
rootdir: /tmp/tmp.8zHtPjayyy
plugins: mypy-1.0.1, randomly-3.16.0
collected 2 items                                                

tests/bad.py::mypy FAILED                                  [ 50%]

============================ FAILURES ============================
______________________ [mypy] tests/bad.py _______________________
1: error: Incompatible types in assignment (expression has type "str", variable has type "int")  [assignment]
============================== mypy ==============================
Found 1 error in 1 file (checked 1 source file)
==================== short test summary info =====================
FAILED tests/bad.py::mypy
!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!
======================= 1 failed in 0.29s ========================
  3. -x (--exitfirst) prevents that failure from being rendered. Without this condition, all errors are reported: 
    $ pipenv run pytest --mypy --randomly-seed=4137011734 -v tests/bad.py 
====================== test session starts =======================
platform linux -- Python 3.10.12, pytest-8.4.0, pluggy-1.6.0 -- /home/dtux/.local/share/virtualenvs/tmp.8zHtPjayyy-wB_JhAoz/bin/python
cachedir: .pytest_cache
Using --randomly-seed=4137011734
rootdir: /tmp/tmp.8zHtPjayyy
plugins: mypy-1.0.1, randomly-3.16.0
collected 2 items                                                

tests/bad.py::mypy-status FAILED                           [ 50%]
tests/bad.py::mypy FAILED                                  [100%]

============================ FAILURES ============================
__________________________ test session __________________________
mypy exited with status 1.
______________________ [mypy] tests/bad.py _______________________
1: error: Incompatible types in assignment (expression has type "str", variable has type "int")  [assignment]
============================== mypy ==============================
Found 1 error in 1 file (checked 1 source file)
==================== short test summary info =====================
FAILED tests/bad.py::mypy-status
FAILED tests/bad.py::mypy
======================= 2 failed in 0.28s ========================

mypy-status fails on empty file

Almost certainly not... I see why it appears that way, though: tests/__init__.py::mypy-status FAILED The MypyStatusItem is generated by the first MypyFileItem to run: https://github.com/realpython/pytest-mypy/blob/1b3e931cb6ff00b47436455c69e3cd978b745479/src/pytest_mypy/init.py#L226-L235

So, all that "tests/__init__.py::mypy-status" really tells us is that the MypyFileItem for tests/__init__.py was the first to run. Aside from that, there is no correlation between the status check and the file it's reported with.

4 errors reported but there are none

The errors are still there, in one of the 27 files that were checked: Found 4 errors in 1 file (checked 27 source files) More specifically, they are in a file that was also collected by Pytest: testpaths: tests, integration

To see which ones, you'll need to run without -x.

% mypy tests/__init__.py
Success: no issues found in 1 source file

One other tip: we know the errors aren't in this file because the plugin told us so: tests/__init__.py::mypy PASSED

dmtucker avatar Jun 07 '25 17:06 dmtucker

Thank you for your excellent analysis, @dmtucker. Indeed, I have -x in my addopts…

madduck avatar Jun 08 '25 09:06 madduck