`mypy-status` is not very useful if it triggers `-x`/`--exitfirst`

[madduck]

The mypy plugin here falls over, complains about errors, but there aren't any…

% pytest -s -vvv --mypy
===================================== test session starts =====================================
platform linux -- Python 3.13.3, pytest-8.3.5, pluggy-1.5.0 -- /home/madduck/code/tc/tptools/.direnv/python-3.13.3/bin/python3
cachedir: .pytest_cache
rootdir: /home/madduck/code/tc/tptools
configfile: pyproject.toml
testpaths: tests, integration
plugins: anyio-4.9.0, cov-6.1.1, devtools-0.12.2, asyncio-0.26.0, subtests-0.14.1, mypy-1.0.1
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
collected 352 items

tests/__init__.py::mypy PASSED
tests/__init__.py::mypy-status FAILED

========================================== FAILURES ===========================================
________________________________________ test session _________________________________________
mypy exited with status 1.
-------------------------------------- Captured log call --------------------------------------
filelock DEBUG Attempting to acquire lock 140525569634576 on /home/madduck/.tmp/tmpw90y_v42.lock
filelock DEBUG Lock 140525569634576 acquired on /home/madduck/.tmp/tmpw90y_v42.lock
filelock DEBUG Attempting to release lock 140525569634576 on /home/madduck/.tmp/tmpw90y_v42.lock
filelock DEBUG Lock 140525569634576 released on /home/madduck/.tmp/tmpw90y_v42.lock
============================================ mypy =============================================
Found 4 errors in 1 file (checked 27 source files)
=================================== short test summary info ===================================
FAILED tests/__init__.py::mypy-status
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================= 1 failed, 1 passed in 0.81s =================================

However:

% mypy tests/__init__.py
Success: no issues found in 1 source file
% echo $?
0

[dmtucker]

This is a duplicate of #120 (but the write-up below is more thorough/clear, so I think we'll keep this ticket going forward)...

tl;dr: There are errors in a file that was collected by Pytest,
but the status check triggered -x
which prevented those errors from showing up.

Here's an SSCCE:

$ tree 
.
├── Pipfile
├── Pipfile.lock
└── tests
    └── bad.py

1 directory, 3 files

• Pipfile
• Pipfile.lock

• $ cat tests/bad.py 
  x: int = "5"

$ pipenv run pytest --mypy --randomly-seed=4137011734 -vx tests/bad.py 
====================== test session starts =======================
platform linux -- Python 3.10.12, pytest-8.4.0, pluggy-1.6.0 -- /home/dtux/.local/share/virtualenvs/tmp.8zHtPjayyy-wB_JhAoz/bin/python
cachedir: .pytest_cache
Using --randomly-seed=4137011734
rootdir: /tmp/tmp.8zHtPjayyy
plugins: mypy-1.0.1, randomly-3.16.0
collected 2 items                                                

tests/bad.py::mypy-status FAILED                           [ 50%]

============================ FAILURES ============================
__________________________ test session __________________________
mypy exited with status 1.
============================== mypy ==============================
Found 1 error in 1 file (checked 1 source file)
==================== short test summary info =====================
FAILED tests/bad.py::mypy-status
!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!
======================= 1 failed in 0.28s ========================

It happens when all 3 of the following conditions are met:

1. A module that was collected by Pytest contains a Mypy error.
   Without this condition (i.e. if there were an error in a file that was not collected),
   we would see the error in the terminal summary:

   $ echo 'import bad' > tests/indirect.py
   $ pipenv run pytest --mypy --randomly-seed=4137011734 -vx tests/indirect.py
   ====================== test session starts =======================
   platform linux -- Python 3.10.12, pytest-8.4.0, pluggy-1.6.0 -- /home/dtux/.local/share/virtualenvs/tmp.8zHtPjayyy-wB_JhAoz/bin/python
   cachedir: .pytest_cache
   Using --randomly-seed=4137011734
   rootdir: /tmp/tmp.8zHtPjayyy
   plugins: mypy-1.0.1, randomly-3.16.0
   collected 2 items                                                
    
   tests/indirect.py::mypy PASSED                             [ 50%]
   tests/indirect.py::mypy-status FAILED                      [100%]
    
   ============================ FAILURES ============================
   __________________________ test session __________________________
   mypy exited with status 1.
   ============================== mypy ==============================
   tests/bad.py:1: error: Incompatible types in assignment (expression has type "str", variable has type "int")  [assignment]
   Found 1 error in 1 file (checked 1 source file)
   ==================== short test summary info =====================
   FAILED tests/indirect.py::mypy-status
   !!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!
   ================== 1 failed, 1 passed in 0.28s ===================

2. The status check (i.e. MypyStatusItem) runs before any other failure.
   Without this condition, one of the files containing Mypy errors (or any other failed test) would have triggered condition 3.

   $ pipenv run pytest --mypy --randomly-seed=4264782955 -vx tests/bad.py 
   ====================== test session starts =======================
   platform linux -- Python 3.10.12, pytest-8.4.0, pluggy-1.6.0 -- /home/dtux/.local/share/virtualenvs/tmp.8zHtPjayyy-wB_JhAoz/bin/python
   cachedir: .pytest_cache
   Using --randomly-seed=4264782955
   rootdir: /tmp/tmp.8zHtPjayyy
   plugins: mypy-1.0.1, randomly-3.16.0
   collected 2 items                                                
    
   tests/bad.py::mypy FAILED                                  [ 50%]
   
   ============================ FAILURES ============================
   ______________________ [mypy] tests/bad.py _______________________
   1: error: Incompatible types in assignment (expression has type "str", variable has type "int")  [assignment]
   ============================== mypy ==============================
   Found 1 error in 1 file (checked 1 source file)
   ==================== short test summary info =====================
   FAILED tests/bad.py::mypy
   !!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!
   ======================= 1 failed in 0.29s ========================

3. -x (--exitfirst) prevents that failure from being rendered.
   Without this condition, all errors are reported:

   $ pipenv run pytest --mypy --randomly-seed=4137011734 -v tests/bad.py 
   ====================== test session starts =======================
   platform linux -- Python 3.10.12, pytest-8.4.0, pluggy-1.6.0 -- /home/dtux/.local/share/virtualenvs/tmp.8zHtPjayyy-wB_JhAoz/bin/python
   cachedir: .pytest_cache
   Using --randomly-seed=4137011734
   rootdir: /tmp/tmp.8zHtPjayyy
   plugins: mypy-1.0.1, randomly-3.16.0
   collected 2 items                                                
    
   tests/bad.py::mypy-status FAILED                           [ 50%]
   tests/bad.py::mypy FAILED                                  [100%]
    
   ============================ FAILURES ============================
   __________________________ test session __________________________
   mypy exited with status 1.
   ______________________ [mypy] tests/bad.py _______________________
   1: error: Incompatible types in assignment (expression has type "str", variable has type "int")  [assignment]
   ============================== mypy ==============================
   Found 1 error in 1 file (checked 1 source file)
   ==================== short test summary info =====================
   FAILED tests/bad.py::mypy-status
   FAILED tests/bad.py::mypy
   ======================= 2 failed in 0.28s ========================

mypy-status fails on empty file

Almost certainly not... I see why it appears that way, though: tests/__init__.py::mypy-status FAILED
The MypyStatusItem is generated by the first MypyFileItem to run:

pytest-mypy/src/pytest_mypy/__init__.py

Lines 226 to 235 in 1b3e931

	# Since mypy might check files that were not collected,
	# pytest could pass even though mypy failed!
	# To prevent that, add an explicit check for the mypy exit status.
	if not self.session.config.option.mypy_no_status_check and not any(
	isinstance(item, MypyStatusItem) for item in self.session.items
	):
	yield MypyStatusItem.from_parent(
	parent=self,
	name=nodeid_name + "-status",
	)

So, all that "tests/__init__.py::mypy-status" really tells us is that the MypyFileItem for tests/__init__.py was the first to run. Aside from that, there is no correlation between the status check and the file it's reported with.

4 errors reported but there are none

The errors are still there, in one of the 27 files that were checked:
Found 4 errors in 1 file (checked 27 source files)
More specifically, they are in a file that was also collected by Pytest:
testpaths: tests, integration

To see which ones, you'll need to run without -x.

% mypy tests/__init__.py
Success: no issues found in 1 source file

One other tip: we know the errors aren't in this file because the plugin told us so: tests/__init__.py::mypy PASSED

[madduck]

Thank you for your excellent analysis, @dmtucker. Indeed, I have -x in my addopts…