seedlint 🌱

A linter that catches reproducibility bugs in ML code before your reviewers (or your future self) do.

Your model scored 0.87 yesterday and 0.85 today, and you changed nothing. Somewhere in your code an RNG was never seeded, a train_test_split has no random_state, or cuDNN is silently picking a different convolution algorithm. seedlint finds those spots by static analysis — no imports, no GPU, no dependencies, just the Python AST.

$ seedlint examples/bad_training.py
examples/bad_training.py:14:1 SEED303 PYTHONHASHSEED set at runtime — hash randomization is fixed at interpreter startup, so this line has no effect on the current process
    hint: export PYTHONHASHSEED before launching Python (e.g. PYTHONHASHSEED=0 python train.py)
examples/bad_training.py:16:1 SEED301 cudnn.benchmark = True picks convolution algorithms at runtime, which is nondeterministic
    hint: set cudnn.benchmark = False (and cudnn.deterministic = True) when you need reproducibility
examples/bad_training.py:19:5 SEED002 numpy.random.randn() draws from a global RNG that is never seeded in this file
    hint: call np.random.seed(<int>) once at startup
examples/bad_training.py:22:36 SEED101 train_test_split() without random_state — the split changes on every run
    hint: pass random_state=<int> (or shuffle=False for an ordered split)
...

11 issue(s) (7 warning, 4 info) in 1 of 1 file(s)

Install

pip install git+https://github.com/nktykst/seedlint
# or, without installing:
uvx --from git+https://github.com/nktykst/seedlint seedlint .

Zero runtime dependencies — it is pure standard library.

Usage

seedlint .                       # lint a whole project
seedlint train.py src/           # specific files or directories
seedlint . --format json         # machine-readable output
seedlint . --format github       # GitHub Actions annotations
seedlint . --select SEED101      # only some rules
seedlint . --ignore SEED103,SEED104
seedlint --list-rules

Exit code is 1 when issues are found (use --exit-zero to disable), so it drops straight into CI:

- run: pip install git+https://github.com/nktykst/seedlint
- run: seedlint src/ --format github

Suppress a finding inline when the randomness is intentional:

explore = df.sample(frac=0.01)  # seedlint: ignore
split = train_test_split(X, y)  # seedlint: ignore[SEED101]

Rules

Code	Severity	Catches
SEED001	warning	random.* used but random.seed() never called in the file
SEED002	warning	np.random.* (global RNG) used but np.random.seed() never called
SEED003	warning	torch.rand/randn/randperm/… used but torch.manual_seed() never called
SEED004	info	np.random.default_rng() / RandomState() created without a seed
SEED101	warning	train_test_split() without random_state
SEED102	warning	Shuffling CV splitter (KFold(shuffle=True), ShuffleSplit, …) without random_state
SEED103	info	Stochastic sklearn estimator (RandomForest*, KMeans, TSNE, …) without random_state
SEED104	info	.sample(n=/frac=) without random_state (pandas-style sampling)
SEED201	warning	DataLoader(shuffle=True) without generator= in a file that never seeds torch
SEED202	info	DataLoader(num_workers>0) without worker_init_fn (NumPy/random state in workers)
SEED301	warning	torch.backends.cudnn.benchmark = True (nondeterministic conv algorithms)
SEED302	info	torch.backends.cudnn.deterministic = False set explicitly
SEED303	warning	os.environ["PYTHONHASHSEED"] = ... at runtime — it has no effect on the running process

seedlint understands import aliases (import numpy as np, from sklearn.model_selection import train_test_split as tts), and it knows about umbrella seeders: if a file calls set_seed(...), seed_everything(...), fix_seed(...) etc. — whether from transformers, Lightning, accelerate, or your own utils — the SEED001–003 rules stay quiet.

Design notes & limitations

• File-level heuristic. SEED001–003 fire per file: RNG use with no seeding call anywhere in the same file. A seed applied in another module (e.g. a main.py that seeds before importing train.py) is not visible; silence those cases with # seedlint: ignore or --ignore.
• Conservative by construction. Calls made through **kwargs are skipped, shuffle=False splits are recognized as deterministic, and the sklearn estimator list only contains estimators that are stochastic with default settings. When seedlint can't tell, it stays quiet.
• A linter, not a proof. A clean run does not guarantee bitwise reproducibility (GPU atomics, library versions, and data order all matter) — it catches the common, silent mistakes that account for most "why is my score different today?" moments.

Library use

from seedlint import check_source

for finding in check_source(open("train.py").read(), "train.py"):
    print(finding.code, finding.line, finding.message)

日本語での紹介

seedlint は機械学習コードの再現性バグを静的解析で検出するリンターです。 シード未設定の乱数生成、random_state の指定漏れ、 cudnn.benchmark = True などの非決定的なフラグ、実行時に設定しても効かない PYTHONHASHSEED などを、コードを実行せずに検出します。依存ライブラリはゼロ、 pip install してすぐ CI に組み込めます。

Development

git clone https://github.com/nktykst/seedlint && cd seedlint
uv sync
uv run pytest

Contributions welcome — especially new rules (JAX, TensorFlow, polars?) and false-positive reports.

License

MIT