Maintenance WP1: test-runner and tooling hygiene (pytest addopts, redundant linters, stale CLAUDE.md/README, committed artifact)

[kgdunn]

Work package 1 of 3 from the 2026-07-10 maintenance review

This issue is a complete, self-contained work prompt. Anyone (human or agent) should be able to execute it with no other context. Companion packages: WP2 (CI matrix + composite action) and WP3 (docs API reference gaps) are filed as separate issues and are independent of this one.

Background

A maintenance audit of the repo (v1.52.2) found low-risk configuration and documentation debt that misleads contributors and silently degrades CI. All items below were verified against the working tree on 2026-07-10.

Task A: fix pytest.ini addopts

Current addopts in pytest.ini (repo root) is:

addopts = --new-first
          --exitfirst
          -v
          -r=s
          -r=a
          -n auto
          --pdb

# The rest of the settings are specified in ".coveragerc", to prevent duplication/unexpected clashes in specification
    --cov=process_improve
    --cov-report=html
    --cov=process_improve
    --cov-report=term-missing:skip-covered
    --cov-config=.coveragerc
    --no-cov-on-fail
    --cov-branch
    --cov-fail-under 92

Problems, in order of severity:

1. --pdb is a default: a bare pytest drops into the interactive debugger on first failure. Worse, pytest-xdist disables parallel workers when --pdb is active, so -n auto is silently a no-op; the CI test job (which runs uv run pytest --cov-report=xml) has been running single-process.
2. --exitfirst as a default means a normal run never reports the full failure surface.
3. --cov=process_improve is duplicated.
4. -r=s -r=a is a nonstandard spelling; use -ra.
5. The coverage flags sit after a # comment line inside the multiline value; it parses today but is fragile.

Required end state for the addopts block (keep the markers block and everything else in the file unchanged):

addopts = --new-first
          -ra
          -n auto
          --cov=process_improve
          --cov-report=html
          --cov-report=term-missing:skip-covered
          --cov-config=.coveragerc
          --no-cov-on-fail
          --cov-branch
          --cov-fail-under 92

Notes: drop -v (log noise; anyone can pass it manually). Keep the explanatory ## comment lines above the block but update them so they only describe flags that are still present (remove the --pdb line, add a line noting that --pdb/--exitfirst/-v are deliberately not defaults and can be passed manually when debugging).

Follow-up edits required by this change:

• CLAUDE.md tells contributors to run pytest ... -o "addopts=" to escape the debugger defaults. After this fix a plain pytest works; Task D rewrites those commands anyway (same PR).
• Check .github/workflows/run-tests.yml: the test-under-dash-O job passes -p no:cacheprovider -o addopts= or similar overrides. Do not break it; it may keep its override (it deliberately runs without coverage), but confirm it still passes.

Task B: remove redundant lint/format tooling (ruff subsumes it)

The repo currently runs ruff (select = ["ALL"]) + ruff-format, and also flake8, isort, and a black config that nothing invokes. Remove the redundancy:

1. Delete the .flake8 file (repo root). Before deleting, note it enforced max-complexity = 18; confirm pyproject.toml ruff config has an equivalent (look for [tool.ruff.lint.mccabe] max-complexity; ruff's C901 default is 10 but the repo may ignore C901 entirely; if complexity checking would be lost and C901 is not in the ruff ignore list, add [tool.ruff.lint.mccabe] max-complexity = 18 to preserve behavior; if C901 findings appear, do not fix code in this PR, just keep the threshold at a level that passes, documenting it).
2. In .pre-commit-config.yaml:
   • Remove the flake8 hook (repo https://github.com/PyCQA/flake8).
   • Remove the isort hook (repo https://github.com/pre-commit/mirrors-isort, pinned to old v5.10.1). Ruff's I rules cover import sorting; verify I is not in the ruff ignore list in pyproject.toml (it is not, as of the audit).
   • Delete the stale comment line # Remove MyPy: conflicts in python 3.9 with pytz above the mirrors-mypy hook (the hook itself stays).
   • Keep: pre-commit-hooks, mirrors-mypy, blacken-docs (formats code blocks in docs, which ruff-format does not do), nbstripout, ruff-pre-commit (ruff-check + ruff-format).
3. In pyproject.toml: delete the entire [tool.black] section (the formatter is ruff-format; black is never invoked by the Makefile, pre-commit, or CI).
4. Run uv run pre-commit run --all-files and fix any fallout. Expected fallout: none, because ruff was already the effective gate. If ruff I-rule import-order differences from isort's historical output show up, let ruff-check with --fix settle them and commit the result.

Task C: remove committed build artifact

• git rm tools/check_estimator_audit_main.log (a committed log file).
• Add *.log to .gitignore if not already covered.

Task D: rewrite the stale sections of CLAUDE.md

The repo owner has explicitly approved this rewrite (normally CLAUDE.md edits require asking first, per the file's own footer; that footer policy stays in the file). Every item below was verified stale on 2026-07-10. Preserve the file's overall structure and preserve verbatim the policy sections: Versioning, Changelog, "Never push lock files", "Git & PR workflow", "Updating this file (CLAUDE.md)". Fix the factual sections:

1. Package structure: the tree shows a flat process_improve/ layout with 9 subpackages. Reality: src layout (src/process_improve/) containing subpackages multivariate, univariate, experiments, monitoring, batch, regression, bivariate, visualization, datasets, sensory, simulation, plus top-level modules recipes.py, mcp_server.py, config.py, tool_safety.py, tool_spec.py, _extras.py, _linalg.py, _random.py. Update the tree with one-line descriptions (read each module's docstring for accurate wording).
2. Header facts: "Python >= 3.10 (CI runs on 3.11)" is wrong; CI tests 3.10, 3.11, 3.12, and 3.13, with 3.13 as the primary version for lint/typecheck/coverage.
3. Version examples in the Versioning section use 1.3.1/1.3.0; update the examples to the current numbering (e.g. 1.52.2 -> 1.52.3) without changing the policy wording.
4. CI/CD section: cites .github/workflows/pythonpackage.yml, which does not exist. Actual workflows: run-tests.yml (ruff lint, blocking mypy typecheck, pytest matrix, python -O job), docs.yml (strict Sphinx build, Pages deploy), publish.yml (tag-gated PyPI), codeql.yml. Commands use uv, not pip: uv sync --dev --all-extras, uv run ruff check ., uv run pytest.
5. Code Quality section: "Formatter: black" is wrong; the formatter is ruff-format (and after Task B black config no longer exists).
6. Testing section: it assumes all multivariate tests live in tests/test_multivariate.py and prescribes -o "addopts=" on every command. Reality: multivariate tests are split across ~15 files (test_multivariate.py, test_multiblock_reference.py, test_multivariate_tsr.py, etc.); after Task A, plain pytest works and the -o "addopts=" incantations should be removed. Update the example commands, e.g. uv run pytest tests/test_multivariate.py -v and uv run pytest -k "pls".
7. "Adding New Methods to PCA/PLS" section: step 1 says methods live in process_improve/multivariate/methods.py. Reality: methods.py is a re-export shim; PCA lives in src/process_improve/multivariate/_pca.py, PLS in _pls.py, shared plotting in plots.py, limits in _limits.py. Update paths.

Task E: fix README.md drift

1. The "longer examples" pointer references process_improve/notebooks_examples/, which does not exist; point it at examples/.
2. The BibTeX citation block pins version = {v1.21.4}; either update to the current version being shipped by this PR or (better, drift-proof) reference the CITATION.cff file and drop the hardcoded version from the BibTeX. Choose the CITATION.cff pointer approach.
3. The install/dependency prose lists scipy as a core dependency; scipy is only transitive (via scikit-learn/statsmodels) and is not in [project] dependencies. Correct the list to match pyproject.toml.

Version bump and metadata (last commit of the PR)

• pyproject.toml: version = "1.52.2" -> "1.52.3" (PATCH; config/docs only). If main has moved past 1.52.2 by the time this is executed, bump PATCH relative to current main instead.
• CITATION.cff: set version: to the identical value and date-released: to the current date, in the same commit.
• No CHANGELOG entry and no heading roll: the repo owner explicitly decided this batch is internal-only and gets no changelog line (decision recorded 2026-07-10).

Constraints

• Branch: claude/process-improve-maintenance-review-0lbiru; one PR for this whole work package, opened ready-for-review, micro-committed (roughly one commit per task A-E plus the version bump).
• Never stage or push lock files (uv.lock etc.).
• No em-dash characters in any committed prose, commit message, or PR description (repo convention).

Verification (all must pass before the PR is done)

1. uv run ruff check . and uv run ruff format --check .
2. uv run pre-commit run --all-files
3. uv run pytest with no extra flags: must not enter pdb, must actually run parallel workers (xdist header shows N workers), must pass the 92 percent coverage gate.
4. uv run mypy . (or the exact CI invocation from run-tests.yml)
5. CI green on the PR, including the test-under-dash-O job.

[kgdunn]

Implemented in PR #451. Three adjacent problems surfaced during the work that are deliberately NOT fixed there (out of scope, need a maintainer decision):

1. The repo is not ruff-format clean and CI does not enforce formatting. uv run ruff format --check . with the dev environment's ruff (0.15.21 today, floor is >=0.11.0) would reformat 112 of 250 files, and the pinned pre-commit ruff (v0.15.6) agrees. CI's lint job only runs ruff check .; only the Makefile's lint target runs ruff format --check, so formatter drift accumulates invisibly as ruff versions advance. Options: (a) pin ruff to an exact version, run the one-time reformat as a dedicated mechanical PR, and add ruff format --check to the CI lint job; or (b) drop the format-check from the Makefile and accept format drift. Recommend (a).
2. Other pre-commit hooks have drifted too. A full pre-commit run --all-files also modifies files via trailing-whitespace/end-of-file-fixer (several markdown/config files), nbstripout (notebooks under docs/ retain outputs and lack cell ids), and blacken-docs (a few .rst code blocks). Same treatment as item 1: one mechanical cleanup PR, then the hooks keep it clean.
3. A pre-existing mypy error sits outside CI's gate: tests/test_omars_ilp.py:417 passes type as str where Factor expects FactorType. CI's mypy job only checks src/process_improve, so it never fires. Trivial fix, but widening the mypy gate to tests/ is the real question.

Also fixed in the PR while getting CI green: the macOS test_omars_ilp.py failure (pulp's bundled CBC binary is present but non-executable on the Apple Silicon runner; the module guard now probes an actual solve instead of only checking importability).

---

Generated by Claude Code