docs: design draft — the manifest, run config, and run record by luca-belli · Pull Request #277 · SpringCare/VERA-MH

luca-belli · 2026-10-19T22:40:05Z

Summary

Design draft for discussion, docs only: docs/design/config-and-manifest.md. Connected to #268 (config.json and its hash), #276 (where the run record lives), and #274 (what someone writing a target edits; the link resolves once #274 merges).

Three JSON files, three jobs, and what belongs in each:

Target manifest (data/<target>/manifest.json), what the evaluation is:
- Persona fields required only to generate, so judge-only targets work.
- The hardcoded SI risk-level column and order move here from scoring code.
- Adds name, description, version.
- rubric_manifest.json goes with the legacy scripts.
Run config (--config, configs/*.json), how to run it:
- Defaults allowed, so adding an option stops breaking every config (as user_language did in feat: let the user-side LLM write in another language #269).
- An execution section (max_concurrent, debug, sample, into) kept out of the run's hash.
- Placeholder model names rejected at load.
- One path rule.
Run record (config.json in the run folder, not written yet), what ran:
- The resolved config plus provenance.
- Input files by checksum, and the code version recorded but not hashed.
- The requested model hashed, the resolved model recorded.
- Re-runnable with --config <run>/config.json.

Decisions from review (in the doc): defaults allowed in config files, execution settings recorded but not hashed, manifest version as human text with checksums as the source of truth, model names checked with the provider at load, and uncommitted rubric edits stop a run unless --allow-uncommitted.

Implementation plan (in the doc): PRs A–G into release/2.1. A and B first, then C (the run record), then D, E, F; G any time.

Also in this PR: docs/roadmap.md and AGENTS.md now say the legacy scripts are no longer supported. Removing legacy/ becomes step 1 of the 2.1 plan, ahead of the safety net.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

luca-belli · 2026-10-19T22:45:25Z

+{
+  "name": "SI",
+  "description": "Suicidal ideation",
+  "version": "1.2",


i wonder if we should also link a commit hash to make ti real

Added: the run record keeps the human version and next to it git_commit (the last commit that changed the target folder) plus uncommitted_changes, so a score links to the exact rubric on GitHub.

luca-belli · 2026-10-19T22:45:57Z

+```
+
+- **Defaults allowed.** A config states what differs from the defaults. Reproducibility comes from the run record, which writes every value out, not from forcing every config file to.
+- **Execution settings in their own section:** `max_concurrent`, `debug`, `sample`, `into`. They are excluded from the run's hash, because they don't change the result.


Thanks. Kept as proposed.

luca-belli · 2026-10-19T22:46:31Z

+```json
+{
+  "format_version": 1,
+  "target": {"name": "SI", "version": "1.2", "files": {"rubric.tsv": "sha256:…", "personas.tsv": "sha256:…"}},


also add the hash of the commit of the rburnic, which might be what sha256 is meant to do

They complement each other: the per-file sha256 checksums are the source of truth (they also catch uncommitted edits) and are hashed; git_commit is for people, to open that exact rubric, and is recorded but not hashed. New open question: whether to refuse runs when the target has uncommitted changes.

luca-belli · 2026-10-19T22:47:27Z

+
+## Open questions
+
+1. **Defaults in config files.** The current rule (every field required) was chosen so a config file alone describes a run. With a run record that does that job, are defaults acceptable, or should configs stay complete?


defaults are acceptable, but they are explicitly written in the run record

Decided: defaults allowed in config files, every value written out explicitly in the run record.

luca-belli · 2026-10-19T22:47:54Z

+## Open questions
+
+1. **Defaults in config files.** The current rule (every field required) was chosen so a config file alone describes a run. With a run record that does that job, are defaults acceptable, or should configs stay complete?
+2. **Where execution settings go.** A separate `execution` section, or keep `max_concurrent` per stage and only exclude it from the hash?


would it be possible to add it, but not hash it, just in case it's useufl for the future?

Yes, that's now the design: the execution section is recorded in the run record but excluded from the hash.

luca-belli · 2026-10-19T22:48:03Z

+
+1. **Defaults in config files.** The current rule (every field required) was chosen so a config file alone describes a run. With a run record that does that job, are defaults acceptable, or should configs stay complete?
+2. **Where execution settings go.** A separate `execution` section, or keep `max_concurrent` per stage and only exclude it from the hash?
+3. **Manifest `version`.** Free text set by the rubric's authors, or derived from the files' checksums?


checksum are the source of truth, human text is more helpul

Decided: both. Free-text version in the manifest, plus checksums and the commit in the run record.

Written in exactly that way: checksums are the source of truth, the human text is for readability.

luca-belli · 2026-10-19T22:48:53Z

+1. **Defaults in config files.** The current rule (every field required) was chosen so a config file alone describes a run. With a run record that does that job, are defaults acceptable, or should configs stay complete?
+2. **Where execution settings go.** A separate `execution` section, or keep `max_concurrent` per stage and only exclude it from the hash?
+3. **Manifest `version`.** Free text set by the rubric's authors, or derived from the files' checksums?
+4. **Validating model names at load.** A provider lookup needs network and keys at resolve time; acceptable, or check only for obvious placeholders?


acceptable to avoid problems later

Decided: model names are looked up with the provider at load, before any run starts; --print skips the check.

…target's git commit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

luca-belli · 2026-10-22T21:16:18Z

+
+## Open questions
+
+1. **Uncommitted rubric edits.** When the target's files differ from their last commit, record `uncommitted_changes: true` and run anyway, or refuse to run unless an explicit flag allows it? Recording is enough for experiments; refusing protects published numbers.


refuse unless a specific flag is set

Recorded as decision 5: if the target's files differ from their last commit, vera refuses to start and names the changed files, unless --allow-uncommitted is given. The flag is an execution setting (recorded, not hashed), and the run record then has uncommitted_changes: true, so such a run is always identifiable. That closes the doc's open questions.

…less --allow-uncommitted Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… of 2.1 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: design draft for the manifest, run config, and run record

d9c004f

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

luca-belli commented Oct 19, 2026

View reviewed changes

luca-belli and others added 2 commits October 19, 2026 15:49

docs: config/manifest design — record decisions from review, add the …

dc697b9

…target's git commit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: config/manifest design — the score label comes from the manifest

cfe2841

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

luca-belli mentioned this pull request Oct 21, 2026

docs, ci: plan 2.1 on a release/2.1 branch #281

Merged

luca-belli commented Oct 22, 2026

View reviewed changes

docs: config/manifest design — uncommitted rubric edits stop a run un…

ef4bd3b

…less --allow-uncommitted Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

luca-belli marked this pull request as ready for review October 22, 2026 21:17

luca-belli and others added 2 commits October 22, 2026 14:24

Merge main into docs/config-manifest-design

a654000

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: #277 implementation plan, and the step-2 sequence in the roadmap

2f25fc7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

luca-belli requested a review from a team as a code owner October 22, 2026 21:25

docs: legacy scripts are no longer supported; removing them is step 1…

4ec11a9

… of 2.1 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

luca-belli mentioned this pull request Oct 26, 2026

chore: remove the legacy scripts and their tests #287

Merged

luca-belli merged commit 3794b6b into main Oct 26, 2026
3 checks passed


		## Open questions

		1. Defaults in config files. The current rule (every field required) was chosen so a config file alone describes a run. With a run record that does that job, are defaults acceptable, or should configs stay complete?


		## Open questions

		1. Uncommitted rubric edits. When the target's files differ from their last commit, record `uncommitted_changes: true` and run anyway, or refuse to run unless an explicit flag allows it? Recording is enough for experiments; refusing protects published numbers.

Conversation

luca-belli commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

luca-belli commented Oct 19, 2026 •

edited

Loading