[sharkdp]

Reject exit codes other than 0 (success) and 1 (diagnostics) for every type checker.

I checked the upstream exit-code implementations for Zuban, Pyright, Pyrefly, ty, and mypy. All of them use 0 for "no diagnostics", 1 for "diagnostics", and other exit codes for unrelated problems.

This is related to the discussion in #160 and addresses the zuban / SQLAlchemy problem. When merged in isolation, this PR will also affect the zuban run on Salt, since zuban currently exits with exit code 2 on that project (No Python files found to check). This seems like a good thing. Once we ignore the mypy configuration for zuban, the Salt project should be included again.

[Copilot]

🟡 Changes recommended

Existing tests cover only exit code 0, leaving failure handling unprotected across all five checkers and both capture modes; parametrized regression tests must verify that 0/1 succeed and 2, 101, and -9 return ok=False with the exit code in the error message.

1 open issue

• Test checker failure exit codes across all checkers and capture modes · New

What changed in this PR

This PR prevents failed type-checker runs from being recorded as valid benchmark timings, addressing the crash and incomplete-check cases described in issue #160.

Changes:

• Accept only exit codes 0 and 1 for all supported type checkers, including runs without captured output.
• Include rejected exit codes and available error context in failure messages.
• Clarify that exit-code checks always run, while output-based checks depend on captured output.

File	Description
typecheck_benchmark/​daily_runner.py	Applies exit-code validation across checkers and capture modes, and improves failure reporting.

_{Review effort: Balanced}

[lolpack]

@sharkdp do you mind addressing this? I will merge after.

[sharkdp]

Yes, I added a new test.