[luca-belli]

Summary

A guard against code changes at real scale, ahead of the refactors planned in #294 and #296. Tests only.

tests/safe_changes/full_run.py works from a saved run of the recommended SI profile: 2 user models (gpt-5.2, claude-opus-4-5) × 100 conversations, chatbot gpt-5.4-mini, judged by gpt-5.4, pooled.

• seed <saved run> <baseline> turns the saved run into a baseline without calling any model. vera pipeline runs with every user and chatbot turn answered from the saved transcripts and every judge question from the saved judge logs. Seeding is strict:

  • every history the code sends must equal the saved transcript up to that turn
  • every judge question must be the next one the log asked
  • every saved turn and answer must be used
  • each transcript is read back and must reformat to its file byte for byte, so a reply containing a chatbot: line can't be mis-split

  It records every call in full (as a sha256 key, so the file stays small) plus the outputs the current code produces.

• check <baseline> replays it with replay.py's matching: a call must match on everything sent (model, system prompt, full history or judge question), and every recorded reply must be used. Transcripts, judge TSVs, answers.tsv, results.csv, scores and run records must be identical. Paths, timestamps, the random filename prefix, and the run record's provenance and .sha256 are normalized away.

Why the baseline comes from the current code, not the saved run: the saved run came from July's code, so its own outputs may legitimately differ. The saved run supplies the model answers; the current code supplies the expected outputs. Seeding confirms the two agree on every turn and question.

What's committed:

• tests/fixtures/full_run/: a 10-conversation slice (--sample 5, 880 KB), checked by tests/unit/test_full_run_replay.py in CI
• .gitignore: an exception for that fixture, because the repo ignores every conversations/ and evaluations/ folder
• docs/safe-changes.md and CHANGELOG

The full 200-conversation baseline (~17 MB) stays local in output/full_run_baseline/ (gitignored).

Testing

• Full run: seeding all 200 conversations (9,559 calls) passes in 6 s, so today's code runs those conversations and walks the rubric exactly as the saved run did. The full check passes in 5 s.
• The 10-conversation slice seeds and checks clean.
• Deliberate breaks, both caught and then reverted:
  • sending only the last 20 turns of history: caught at the first chatbot call
  • adding 0.01 to the score formula: caught in every scores.json and scores_by_risk.json

I didn't run the pytest suite locally; CI runs it.

For review: the fixture puts 10 synthetic SI conversations (generated personas discussing suicide risk), plus the judge's answers and reasoning, into this public repo. publication_data/ is the precedent.

🤖 Generated with Claude Code

[luca-belli]

is there a command one can run that does both under the hood and skips the seeding if unnecessary?

[luca-belli]

Yes, as of faebbc2:

uv run python -m tests.safe_changes.full_run check output/full_run_baseline --saved output

With --saved, it seeds first only when the baseline's inputs changed: the baseline is missing, or the saved run is different from the one it was seeded from. To tell, each baseline now records a fingerprint (sha256) of the saved transcripts and judge answers in its settings.json. When it seeds, it prints why, then runs the check.

It never seeds because a check failed. Otherwise every code change would quietly become the new expected output, and the guard would never fire. When a change is meant to alter the outputs, re-seeding stays an explicit seed, so the diff shows up in the PR.

Tested all three cases: an existing baseline without a fingerprint gets seeded once, then just checks; a deliberate +0.01 to the score formula fails with 6 differences and does not re-seed; a missing baseline gets seeded from scratch.