[luca-belli]

Summary

PR C of config-and-manifest.md. Stacked on #290, which is stacked on #289. Merge in order, each with a merge commit, and retarget the next PR after each merge.

Every new run folder (generation, judging, pooling) now gets:

• config.json, the run record. It holds the resolved config with every default written out (the same content --print emits), plus:
  • format_version: 1
  • inputs: sha256 of each file the run reads (personas, context template, rubric files)
  • provenance: vera_version, the code's git_commit, and started_at
• config.json.sha256, a hash of everything except provenance and invocation.

New module: src/vera_mh/utils/run_record.py. create_run_root (#290) writes the record when it's given one.

Why it's built this way

Why now, and together with #289. #289 lets a config file leave behavior fields to their defaults. After that, the config file alone no longer pins a run down: if a default later changes, rerunning the same file quietly gives a different run. The design accepts that trade only because the run record writes every value out. Merging #289, #290 and this PR together means release/2.1 never has defaults without a record.

Why create_run_root writes it. #268's order of events puts writing config.json in the same step that creates the run folder. Every run folder comes into being through that one function, so no run can skip its record, and the AST tests from #290 keep it that way.

Why one record per run folder. vera generate -u a -u b makes two folders, one per user model. If both records named both models, --config <folder>/config.json would regenerate the sibling folder too. So each folder's record names only its own user model, and rerunning a folder repeats exactly that folder.

Why started_at is stamped late. The record is built in vera_cli before the domain runs. The domain then reads the clock once (run_started_at, #290) and uses that single reading for both the folder name and the record's started_at, via with_start_time. A test checks that the two match.

Why the hash leaves out provenance and invocation.

• provenance describes where the run came from, not what it did. A docs-only commit shouldn't make every run look like a different experiment.
• invocation (debug, sample) is how this execution behaved, which is the same split RunConfig already draws.
• max_concurrent still lives inside generation/judging, so it counts toward the hash until PR E moves it to an execution section. I left that move out to keep this PR to one change.

Why input checksums, and why a changed input stops a rerun. Editing a rubric or personas file changes what a run measures, so the checksums are part of the hash: two runs are never mistaken for the same experiment when their inputs differed. On rerun, --config accepts the record's extra keys (format_version, inputs, provenance), and a checksum that no longer matches is an error naming the file. Rerunning against an edited rubric under the old record's name would be a different experiment. A plain config has no inputs and is unaffected.

Why folders aren't checksummed. Conversations to judge and evaluations to pool are the output of an earlier run, which has its own record.

Why into is left out. It continues a folder that already exists, which can't be part of that folder's record (see RunConfig's docstring). A continued run keeps the record it started with and gets no new one.

Signature changes (flagged, per AGENTS.md). run_generation, run_for_user_models, run_judging, judge_conversations, and run_pooling each gain one optional run_record argument. Callers outside a vera command (tests, direct domain calls) pass nothing and write no record.

Architecture change (escalation): AD-18. ARCHITECTURE-SPINE.md's AD-18 said the config.json hash covers debug and sample. This PR follows the newer config-and-manifest.md decision and leaves them out, so AD-18 now says so and links that decision. The reason: debug only adds logging and sample only runs a subset, so neither changes how any conversation or evaluation is produced. This needs maintainer sign-off.

Not in this PR

• D: the target folder's git commit and uncommitted-edit check
• A: target name and version in the record
• E: the execution section
• The model version the provider actually reports. Per AD-18 it belongs in state.json, the runtime record, which comes after the job loop (docs: design draft — one job loop for generate and judge #266).
• The shared persona prompt template and reminder files aren't checksummed yet, because they aren't named in the resolved config.

Testing

tests/unit/utils/test_run_record.py (13 tests) covers:

• what the record holds
• the start time stamped on a copy, and matching the folder name through the real run_generation
• what the hash ignores and what changes it
• writing both files
• rerun checks for a changed or missing input
• one record per user-model folder
• no new record under --into
• a record resolving back to the exact run it records
• judging passing a judging-only record

The pipeline pooling fake now checks the pooled record names the same evaluations. Locally, unit (1025) and non-live integration (134) pass; ruff, pyright, and import-linter are clean. CI is started by hand because this PR is stacked.

🤖 Generated with Claude Code

[jgieringer]

Looks good. Love that commit info makes it into the run config.
Just a few questions, but nothing blocking!

[jgieringer]

Should we log if the run_record is None?

[luca-belli]

yeah that;a great question. maybe we should error out. would would it be None?

[luca-belli]

It was None in two cases:

1. Continuing a run with --into. vera_cli passes None on purpose, because that folder already has its config.json from when it started.
2. Direct calls into the domain (tests). Only this case could leave a new run folder without a config.json.

Every vera command already passes a record for a new run, so erroring out is safe. Done in 044ef03:

• create_run_root now requires the record.
• Starting a new generation, judging, or pooling run without one raises ValueError, naming build_run_record (require_run_record in utils/run_record.py).
• --into never creates a run folder, so continuing a run is unaffected.

Tests that start runs by calling the domain directly now pass tests/mocks/run_record.minimal_run_record(). That's about 10 call sites; the ones writing into an existing folder (output_folder=) are exempt. I chose an error over a log line because a log would still let folders without a record appear, and in practice it would mostly fire in tests.

[jgieringer]

yesss this is great

[jgieringer]

if record is none then we also through away the started_at, is that okay?

[luca-belli]

Good catch on how it reads, but the start time isn't lost. Two lines up, timestamp comes from started_at, so it's already in the evaluation folder's name. run_record is None only when judging is called outside a vera command (tests, direct calls into the domain). Those callers have no resolved config, so there's no config.json to write, and nothing to stamp the time into. Every vera judge / vera pipeline run passes a record, and --into never reaches this branch.

The conditional did make it look as though the time were dropped, so b61b987 moves the None check into with_start_time itself, with a docstring saying the above. This call site and the matching ones in generate/run.py and score/pool.py are now a plain with_start_time(run_record, started_at). There's also a test for the None case.