Repository navigation
Inaccurate Zuban benchmarks #160
Description
There are a few benchmarks where Zuban is much faqster than any other type checker (sometimes 80x, which seems unlikely). I had codex invest why that is and it found a few issues with the benchmark setup:
| Package | Published Zuban / ty | What the reproduction showed |
|---|---|---|
| Salt | 0.051 / 7.308 s | Salt’s mypy config excludes salt. Zuban checked zero files and exited 2. With a neutral config and Python 3.12, Zuban checked 960 files in 5.16 s versus 6.85 s for ty. A genuine Zuban win, much smaller than 143×. Salt config |
| SQLAlchemy | 0.186 / 2.529 s | Zuban panicked on strict = true in a module override and exited 101. With a neutral config, the 258-file check took 1.94 s for Zuban and 2.84 s for ty. SQLAlchemy override |
| Matplotlib | 0.488 / 5.062 s | Zuban checked 108 files with the project config and 263 with a neutral config. Its directory walker also skips .py implementations beside .pyi stubs. There were 79 such implementations, containing 52% of the target’s Python implementation lines. Given the identical list of 342 .py and .pyi files, Zuban took 4.52 s, ty 4.24 s. Matplotlib exclusions · Zuban file discovery |
| pandas | 4.571 / 20.415 s | pandas disables check_untyped_defs for its tests and clipboard module. Changing only that setting took Zuban from 5.43 s to 12.51 s, with the same 1,481 files. ty took 17.69 s locally. This explains a substantial part of the gap and shows why matching file counts alone is insufficient. pandas override |
Activity
sharkdp commentedon Nov 2, 2026
A few more findings. Those are not necessarily related to just Zuban, I mainly looked for (large) discrepancies in the number of checked files / lines of code:
setuptools
Has this exclusion list
mypy.ini. For mypy itself, this file does nothing, since it's overwritten with another config file in the benchmark harness. However, zuban parses this and therefore needs to check significantly fewer files. The remaining differences are due to vendored stubs formore_itertoolsin mypy:Dask
Similar problem with this exclusion list. Mypy doesn't see it because it's config is explicitly overwritten, but Zuban recognizes it:
SymPy
Same problem again, but a smaller difference. I'm mentioning it since those are entire examples which could pull in many transitive dependencies.
Airflow
airflow-core/src/airflow/_shared/contains symlinks. Pyrefly, mypy and zuban traverse them; pyright and ty do not, leading to this discrepancy:Numpy
Mypy and Zuban omit 150 .py implementations that have adjacent .pyi stubs. Other type checkers select/check both.
lolpack commentedon Nov 3, 2026
Hi @sharkdp, thanks for raising this. The mypy.ini overwrite issue for Zuban is a legit bug and I will fix it. I struggle a bit with the transitive dependency following discrepancies. The intent of the benchmark is a casual analysis across a number of packages when running the different checkers. Given the packages will change over time and the challenge in installing every package, I don’t want to make any guarantees that this will be perfectly accurate. With that said, the intention is to educate the user and provide some directional information. Maybe the dependency issue is a good teaching moment like the other pydantic issue you raised? Would a tooltip or other hover explaining the difference in checker behavior suffice in fairly explaining the difference?
sharkdp commentedon Nov 3, 2026
Thank you! I created a fix here: #164
I think the SQLAlchemy problem should also be fixed, where Zuban actually fails with exit code 101, which isn't currently reflected in the benchmark results (see #163)
Of course.
I think tooltips would help. And maybe some projects with severe discrepancies (following the bug fixes mentioned above) should be excluded from the statistical mean? Not sure.
lolpack commentedon Nov 5, 2026
Resolved in merged PRs.