From d9c004fb0e6a688281a82fe0118d917fe2ac59ac Mon Sep 17 00:00:00 2001 From: Luca Belli <129434630+sator-labs@users.noreply.github.com> Date: Mon, 19 Oct 2026 15:36:15 -0700 Subject: [PATCH 1/6] docs: design draft for the manifest, run config, and run record Co-Authored-By: Claude Opus 5.5 --- docs/design/config-and-manifest.md | 104 +++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100644 docs/design/config-and-manifest.md diff --git a/docs/design/config-and-manifest.md b/docs/design/config-and-manifest.md new file mode 100644 index 00000000..d855bcc9 --- /dev/null +++ b/docs/design/config-and-manifest.md @@ -0,0 +1,104 @@ +# The three JSON files: manifest, config, run record + +**Status:** draft for discussion. Connected to [run-lifecycle.md](./run-lifecycle.md) (`config.json` and its hash), [output-layout.md](./output-layout.md) (where the run record lives), and [create-an-evaluation.md](../create-an-evaluation.md) (what someone writing a target edits). + +## Three files, three jobs + +| File | Who writes it | What it says | Example | +|---|---|---|---| +| **Target manifest** | Whoever builds an evaluation | What the evaluation is: its personas, rubric, and prompts | `data/SI/manifest.json` | +| **Run config** | Whoever runs an evaluation | How to run it: which models, how many turns, which target | `configs/recommended-SI.json`, `--config` | +| **Run record** | `vera`, at the start of every run | Exactly what ran: the config with every default filled in, plus provenance | `config.json` in the run folder (not written yet) | + +The first two exist today and have grown separately; the third is planned in run-lifecycle.md. This doc settles what belongs in each, so that each fact has one home. + +## Problems today + +**Manifest** + +- **Judging-only targets aren't possible.** All five fields are required, and every file must exist, so a target meant only for judging real conversations still has to ship a personas file and a context template it never uses. +- **Scoring knobs live in code, not the manifest.** The risk-level column (`Short Current Suicide Risk Level`) and its order (`None`, `Low`, `High`, `Immediate`) are hardcoded in scoring, so any other target's risk breakdown comes back empty. `persona_annotation_columns` is the one scoring-related field that *is* in the manifest. +- **No identity.** There is no name, description, or version, so nothing records which edition of a rubric a run used; the folder name stands in. +- **A legacy copy.** `rubric_manifest.json` is a symlink kept for the old scripts. + +**Run config** + +- **Every behavior field is required.** That makes each config file a complete description, but adding any option breaks every existing config until it's edited: adding `generation.user_language` (#269) meant touching all four files in `configs/` and every doc example. +- **Execution settings mixed with run-defining ones.** `max_concurrent` sits in `generation` and `judging` next to `turns` and the models, though it changes how fast a run goes, not what it produces. Only `debug`, `sample`, and `into` are kept apart, in `InvocationConfig`. +- **Placeholders that only fail at run time.** `recommended-SI.json` ships with `"REPLACE-WITH-THE-MODEL-UNDER-TEST"` as the chatbot; it loads fine and every conversation then fails ("Unsupported model"). +- **Two ways to pass the same thing.** A config file, or `VERA_RUN_CONFIG` (what `--print` emits), with different path rules from CLI flags (config paths resolve against the repo root, CLI paths against the working directory). + +**Run record** + +- **Doesn't exist yet.** Nothing persists what a run actually used, so reproducing one means reconstructing it from folder names and logs. + +## Proposal + +### Target manifest: what the evaluation is + +```json +{ + "name": "SI", + "description": "Suicidal ideation", + "version": "1.2", + "rubric_file": "rubric.tsv", + "rubric_prompt_beginning_file": "rubric_prompt_beginning.txt", + "question_prompt_file": "question_prompt.txt", + "personas": ["personas.tsv"], + "persona_context_template_file": "persona_context_template.txt", + "persona_annotation_columns": ["Short Current Suicide Risk Level"], + "risk_level": {"column": "Short Current Suicide Risk Level", "order": ["None", "Low", "High", "Immediate"]} +} +``` + +- **Rubric fields required; persona fields required only to generate.** `vera judge` accepts a manifest without personas; `vera generate` and `vera pipeline` say which field is missing. +- **Scoring knobs move here from code:** the risk-level column and its order. (The roadmap's [Rubric-agnostic scoring](../roadmap.md#rubric-agnostic-scoring) later replaces the persona column with a rubric question; the manifest is where that pointer lives either way.) +- **`name`, `description`, `version`** identify the evaluation. The run record copies them, so a score always says which edition of a rubric produced it. +- `rubric_manifest.json` is removed with the legacy scripts. + +### Run config: how to run it + +```json +{ + "target": "SI", + "generation": { + "chatbot": {"name": "gpt-5.4"}, + "user": [{"name": "gpt-5.2"}, {"name": "claude-opus-4-5"}], + "turns": 30 + }, + "judging": {"models": [{"name": "gpt-5.4", "reasoning_effort": "low"}]}, + "execution": {"max_concurrent": 10} +} +``` + +- **Defaults allowed.** A config states what differs from the defaults. Reproducibility comes from the run record, which writes every value out, not from forcing every config file to. +- **Execution settings in their own section:** `max_concurrent`, `debug`, `sample`, `into`. They are excluded from the run's hash, because they don't change the result. +- **Placeholders rejected at load.** A model name that no provider recognizes fails before anything runs, naming the field. +- **One path rule:** paths in a config file resolve against the config file's own folder, like the manifest's. + +### Run record: what ran + +The run record is the run config **resolved**: every default filled in, every path absolute, every model ID as sent, plus a `provenance` section. + +```json +{ + "format_version": 1, + "target": {"name": "SI", "version": "1.2", "files": {"rubric.tsv": "sha256:…", "personas.tsv": "sha256:…"}}, + "generation": {"…": "every field, with defaults written out"}, + "judging": {"…": "…"}, + "provenance": {"vera_version": "2.1.0", "git_commit": "…", "started_at": "…", "models_resolved": {"gpt-5.4": "gpt-5.4-2026-08-01"}} +} +``` + +- **Hashed:** everything except `provenance` and the execution section. Same hash, same experiment (run-lifecycle.md). +- **Input files by checksum.** Editing a rubric or personas file changes the hash, so two runs are never mistaken for the same experiment when their inputs differed. +- **Code version recorded, not hashed,** so a docs-only commit doesn't make every run look like a different experiment. +- **Requested vs. resolved model:** the requested ID is part of the settings (hashed); what the provider reports back goes in `provenance` (not hashed). This settles the disagreement run-lifecycle.md found between architecture.md and the use-cases doc. +- **`--config /config.json` re-runs it.** A run record is a valid run config, so reproducing a run is one command. + +## Open questions + +1. **Defaults in config files.** The current rule (every field required) was chosen so a config file alone describes a run. With a run record that does that job, are defaults acceptable, or should configs stay complete? +2. **Where execution settings go.** A separate `execution` section, or keep `max_concurrent` per stage and only exclude it from the hash? +3. **Manifest `version`.** Free text set by the rubric's authors, or derived from the files' checksums? +4. **Validating model names at load.** A provider lookup needs network and keys at resolve time; acceptable, or check only for obvious placeholders? From dc697b907dc12737a3fc1a0a3e87ed77c13cb929 Mon Sep 17 00:00:00 2001 From: Luca Belli <129434630+sator-labs@users.noreply.github.com> Date: Mon, 19 Oct 2026 15:49:43 -0700 Subject: [PATCH 2/6] =?UTF-8?q?docs:=20config/manifest=20design=20?= =?UTF-8?q?=E2=80=94=20record=20decisions=20from=20review,=20add=20the=20t?= =?UTF-8?q?arget's=20git=20commit?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- docs/design/config-and-manifest.md | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/docs/design/config-and-manifest.md b/docs/design/config-and-manifest.md index d855bcc9..f898c08d 100644 --- a/docs/design/config-and-manifest.md +++ b/docs/design/config-and-manifest.md @@ -53,7 +53,7 @@ The first two exist today and have grown separately; the third is planned in run - **Rubric fields required; persona fields required only to generate.** `vera judge` accepts a manifest without personas; `vera generate` and `vera pipeline` say which field is missing. - **Scoring knobs move here from code:** the risk-level column and its order. (The roadmap's [Rubric-agnostic scoring](../roadmap.md#rubric-agnostic-scoring) later replaces the persona column with a rubric question; the manifest is where that pointer lives either way.) -- **`name`, `description`, `version`** identify the evaluation. The run record copies them, so a score always says which edition of a rubric produced it. +- **`name`, `description`, `version`** identify the evaluation for people. `version` is free text set by the rubric's authors ("1.2"); it is helpful but not trusted. The run record adds what is: the checksum of every file and the git commit the target folder was last changed in (see below), so a score always says exactly which edition of a rubric produced it. - `rubric_manifest.json` is removed with the legacy scripts. ### Run config: how to run it @@ -71,9 +71,9 @@ The first two exist today and have grown separately; the third is planned in run } ``` -- **Defaults allowed.** A config states what differs from the defaults. Reproducibility comes from the run record, which writes every value out, not from forcing every config file to. -- **Execution settings in their own section:** `max_concurrent`, `debug`, `sample`, `into`. They are excluded from the run's hash, because they don't change the result. -- **Placeholders rejected at load.** A model name that no provider recognizes fails before anything runs, naming the field. +- **Defaults allowed.** A config states what differs from the defaults, and the run record writes every value out explicitly, defaults included. Reproducibility comes from the run record, not from forcing every config file to be complete. +- **Execution settings in their own section:** `max_concurrent`, `debug`, `sample`, `into`. The run record writes them out like everything else, in case they're useful later, but they are excluded from the run's hash, because they don't change the result. +- **Model names checked at load.** Every model name is looked up with its provider before anything runs, so a placeholder or a typo fails in seconds, naming the field, instead of in every conversation. This needs network and keys when a run starts, which every real run needs anyway; `--print` skips the check. - **One path rule:** paths in a config file resolve against the config file's own folder, like the manifest's. ### Run record: what ran @@ -83,7 +83,11 @@ The run record is the run config **resolved**: every default filled in, every pa ```json { "format_version": 1, - "target": {"name": "SI", "version": "1.2", "files": {"rubric.tsv": "sha256:…", "personas.tsv": "sha256:…"}}, + "target": { + "name": "SI", "version": "1.2", + "git_commit": "65566744…", "uncommitted_changes": false, + "files": {"rubric.tsv": "sha256:…", "personas.tsv": "sha256:…"} + }, "generation": {"…": "every field, with defaults written out"}, "judging": {"…": "…"}, "provenance": {"vera_version": "2.1.0", "git_commit": "…", "started_at": "…", "models_resolved": {"gpt-5.4": "gpt-5.4-2026-08-01"}} @@ -91,14 +95,18 @@ The run record is the run config **resolved**: every default filled in, every pa ``` - **Hashed:** everything except `provenance` and the execution section. Same hash, same experiment (run-lifecycle.md). -- **Input files by checksum.** Editing a rubric or personas file changes the hash, so two runs are never mistaken for the same experiment when their inputs differed. +- **Input files by checksum, plus the commit.** Each file's checksum is the source of truth: editing a rubric or personas file changes the hash, so two runs are never mistaken for the same experiment when their inputs differed. Next to it, `git_commit` is the last commit that changed the target folder, so a reader can open that exact rubric on GitHub, and `uncommitted_changes` says whether the files on disk differed from that commit. The checksums are hashed; the commit is provenance and isn't. - **Code version recorded, not hashed,** so a docs-only commit doesn't make every run look like a different experiment. - **Requested vs. resolved model:** the requested ID is part of the settings (hashed); what the provider reports back goes in `provenance` (not hashed). This settles the disagreement run-lifecycle.md found between architecture.md and the use-cases doc. - **`--config /config.json` re-runs it.** A run record is a valid run config, so reproducing a run is one command. +## Decisions from review + +1. **Defaults are allowed in config files,** and the run record writes every value out explicitly. +2. **Execution settings are recorded but not hashed,** in their own section. +3. **The manifest `version` is human text, and checksums are the source of truth.** The run record holds both, plus the git commit of the target folder. +4. **Model names are validated with the provider at load,** to catch problems before a run spends anything. + ## Open questions -1. **Defaults in config files.** The current rule (every field required) was chosen so a config file alone describes a run. With a run record that does that job, are defaults acceptable, or should configs stay complete? -2. **Where execution settings go.** A separate `execution` section, or keep `max_concurrent` per stage and only exclude it from the hash? -3. **Manifest `version`.** Free text set by the rubric's authors, or derived from the files' checksums? -4. **Validating model names at load.** A provider lookup needs network and keys at resolve time; acceptable, or check only for obvious placeholders? +1. **Uncommitted rubric edits.** When the target's files differ from their last commit, record `uncommitted_changes: true` and run anyway, or refuse to run unless an explicit flag allows it? Recording is enough for experiments; refusing protects published numbers. From cfe2841b069ee63ab218c9f048735611898438a4 Mon Sep 17 00:00:00 2001 From: Luca Belli <129434630+sator-labs@users.noreply.github.com> Date: Mon, 19 Oct 2026 16:05:37 -0700 Subject: [PATCH 3/6] =?UTF-8?q?docs:=20config/manifest=20design=20?= =?UTF-8?q?=E2=80=94=20the=20score=20label=20comes=20from=20the=20manifest?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- docs/design/config-and-manifest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/config-and-manifest.md b/docs/design/config-and-manifest.md index f898c08d..c910afa3 100644 --- a/docs/design/config-and-manifest.md +++ b/docs/design/config-and-manifest.md @@ -52,7 +52,7 @@ The first two exist today and have grown separately; the third is planned in run ``` - **Rubric fields required; persona fields required only to generate.** `vera judge` accepts a manifest without personas; `vera generate` and `vera pipeline` say which field is missing. -- **Scoring knobs move here from code:** the risk-level column and its order. (The roadmap's [Rubric-agnostic scoring](../roadmap.md#rubric-agnostic-scoring) later replaces the persona column with a rubric question; the manifest is where that pointer lives either way.) +- **Scoring knobs move here from code:** the risk-level column and its order, and the label charts and CSV headers put on scores (today the constant `SCORE_NAME = "VERA-MH v1.2"` in `score/utils.py`), which becomes the manifest's `name` and `version`. Scoring reads them from the run record, which names the target that produced the results. (The roadmap's [Rubric-agnostic scoring](../roadmap.md#rubric-agnostic-scoring) later replaces the persona column with a rubric question; the manifest is where that pointer lives either way.) - **`name`, `description`, `version`** identify the evaluation for people. `version` is free text set by the rubric's authors ("1.2"); it is helpful but not trusted. The run record adds what is: the checksum of every file and the git commit the target folder was last changed in (see below), so a score always says exactly which edition of a rubric produced it. - `rubric_manifest.json` is removed with the legacy scripts. From ef4bd3bfda1540dbf2a077b23897a5e7f48010b2 Mon Sep 17 00:00:00 2001 From: Luca Belli <129434630+sator-labs@users.noreply.github.com> Date: Thu, 22 Oct 2026 14:17:39 -0700 Subject: [PATCH 4/6] =?UTF-8?q?docs:=20config/manifest=20design=20?= =?UTF-8?q?=E2=80=94=20uncommitted=20rubric=20edits=20stop=20a=20run=20unl?= =?UTF-8?q?ess=20--allow-uncommitted?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- docs/design/config-and-manifest.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/docs/design/config-and-manifest.md b/docs/design/config-and-manifest.md index c910afa3..6f1e8b2b 100644 --- a/docs/design/config-and-manifest.md +++ b/docs/design/config-and-manifest.md @@ -72,7 +72,7 @@ The first two exist today and have grown separately; the third is planned in run ``` - **Defaults allowed.** A config states what differs from the defaults, and the run record writes every value out explicitly, defaults included. Reproducibility comes from the run record, not from forcing every config file to be complete. -- **Execution settings in their own section:** `max_concurrent`, `debug`, `sample`, `into`. The run record writes them out like everything else, in case they're useful later, but they are excluded from the run's hash, because they don't change the result. +- **Execution settings in their own section:** `max_concurrent`, `debug`, `sample`, `into`, `allow_uncommitted`. The run record writes them out like everything else, in case they're useful later, but they are excluded from the run's hash, because they don't change the result. - **Model names checked at load.** Every model name is looked up with its provider before anything runs, so a placeholder or a typo fails in seconds, naming the field, instead of in every conversation. This needs network and keys when a run starts, which every real run needs anyway; `--print` skips the check. - **One path rule:** paths in a config file resolve against the config file's own folder, like the manifest's. @@ -95,7 +95,7 @@ The run record is the run config **resolved**: every default filled in, every pa ``` - **Hashed:** everything except `provenance` and the execution section. Same hash, same experiment (run-lifecycle.md). -- **Input files by checksum, plus the commit.** Each file's checksum is the source of truth: editing a rubric or personas file changes the hash, so two runs are never mistaken for the same experiment when their inputs differed. Next to it, `git_commit` is the last commit that changed the target folder, so a reader can open that exact rubric on GitHub, and `uncommitted_changes` says whether the files on disk differed from that commit. The checksums are hashed; the commit is provenance and isn't. +- **Input files by checksum, plus the commit.** Each file's checksum is the source of truth: editing a rubric or personas file changes the hash, so two runs are never mistaken for the same experiment when their inputs differed. Next to it, `git_commit` is the last commit that changed the target folder, so a reader can open that exact rubric on GitHub, and `uncommitted_changes` says whether the files on disk differed from that commit; a run only starts with it `true` when `--allow-uncommitted` was given (decision 5). The checksums are hashed; the commit is provenance and isn't. - **Code version recorded, not hashed,** so a docs-only commit doesn't make every run look like a different experiment. - **Requested vs. resolved model:** the requested ID is part of the settings (hashed); what the provider reports back goes in `provenance` (not hashed). This settles the disagreement run-lifecycle.md found between architecture.md and the use-cases doc. - **`--config /config.json` re-runs it.** A run record is a valid run config, so reproducing a run is one command. @@ -106,7 +106,4 @@ The run record is the run config **resolved**: every default filled in, every pa 2. **Execution settings are recorded but not hashed,** in their own section. 3. **The manifest `version` is human text, and checksums are the source of truth.** The run record holds both, plus the git commit of the target folder. 4. **Model names are validated with the provider at load,** to catch problems before a run spends anything. - -## Open questions - -1. **Uncommitted rubric edits.** When the target's files differ from their last commit, record `uncommitted_changes: true` and run anyway, or refuse to run unless an explicit flag allows it? Recording is enough for experiments; refusing protects published numbers. +5. **Uncommitted rubric edits stop a run.** If the target's files differ from their last commit, `vera` refuses to start and names the changed files, unless `--allow-uncommitted` is given. That flag is an execution setting: recorded, not hashed. The run record then has `uncommitted_changes: true`, so a run made that way is always identifiable. This protects published numbers from coming from a rubric nobody can look up. From 2f25fc755239e9207721e6b262a133294055bcce Mon Sep 17 00:00:00 2001 From: Luca Belli <129434630+sator-labs@users.noreply.github.com> Date: Thu, 22 Oct 2026 14:25:03 -0700 Subject: [PATCH 5/6] docs: #277 implementation plan, and the step-2 sequence in the roadmap Co-Authored-By: Claude Opus 5.5 --- docs/design/config-and-manifest.md | 17 +++++++++++++++++ docs/roadmap.md | 2 +- 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/docs/design/config-and-manifest.md b/docs/design/config-and-manifest.md index 6f1e8b2b..a65e0190 100644 --- a/docs/design/config-and-manifest.md +++ b/docs/design/config-and-manifest.md @@ -107,3 +107,20 @@ The run record is the run config **resolved**: every default filled in, every pa 3. **The manifest `version` is human text, and checksums are the source of truth.** The run record holds both, plus the git commit of the target folder. 4. **Model names are validated with the provider at load,** to catch problems before a run spends anything. 5. **Uncommitted rubric edits stop a run.** If the target's files differ from their last commit, `vera` refuses to start and names the changed files, unless `--allow-uncommitted` is given. That flag is an execution setting: recorded, not hashed. The run record then has `uncommitted_changes: true`, so a run made that way is always identifiable. This protects published numbers from coming from a rubric nobody can look up. + +## Implementation plan + +Each step is its own PR into `release/2.1`. None changes scores, so the [safety net](../safe-changes.md) stays green throughout. + +| | PR | What changes | Depends on | +|---|---|---|---| +| A | Manifest: judge-only targets and identity | Rubric fields stay required; persona fields are required only to generate. Optional `name`, `description`, `version`, validated. | — | +| B | Config defaults allowed | Config files may leave out behavior fields, which take the CLI defaults. | — | +| C | The run record | Each run folder gets `config.json`: the resolved config with every default written out, input-file checksums, and `provenance`; plus its `.sha256`. Written into today's run folders; [run-lifecycle.md](./run-lifecycle.md) later moves where it lives. | B | +| D | Uncommitted rubric edits stop a run | Refuse to start unless `--allow-uncommitted`; the record then says `uncommitted_changes: true`. | C | +| E | Execution section | `max_concurrent`, `debug`, `sample`, `into`, `allow_uncommitted` move to `execution`, outside the hash. The old `generation.max_concurrent` keeps working with a deprecation warning. | C | +| F | Scoring knobs from the manifest | The risk-level column and order, and the chart label (`SCORE_NAME`), come from the manifest, found through the run record. | C | +| G | Model names checked at load | Each model name is looked up with its provider before anything runs; `--print` skips it. | — | + +A and B first (independent), then C, then D, E, F. G can land any time. + diff --git a/docs/roadmap.md b/docs/roadmap.md index 8d08cf58..1a709ee3 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -23,7 +23,7 @@ Smaller items that belong to no phase are in the [Backlog](#backlog). | In parallel | A judge flow that decides each outcome once; needs the HFO rubric authors' answer on Q11 | [#267](https://github.com/SpringCare/VERA-MH/pull/267) (after step 1) | [Rubric-agnostic scoring](#rubric-agnostic-scoring) | - **Why this order.** Step 1 comes first because every later step is meant to leave scores unchanged, and it's what proves that. Step 2 defines the files and folders the job loop in step 3 reads and writes (its resume checks, the run roots it needs up front). Step 4's live display needs the events step 3 emits; its other half needs nothing. -- **Inside step 2:** first one function that creates run roots, one clock read per run, and the tests that enforce both, still writing today's folder names; then the run record; then the new layout and conversation files. #276 replaces #268's layout section, so the two are settled as one decision. +- **Inside step 2,** the two designs interleave: #277's manifest and config-defaults PRs (A, B); then the run-root function, one clock read, and their enforcement tests ([run-lifecycle.md](./design/run-lifecycle.md)); then the run record and what builds on it (#277's C, then D, E, F); then the new layout and conversation files (#268, #276). #277's model check at load (G) can land any time. Each design's PR list is in its "Implementation plan" section, for example [config-and-manifest.md](./design/config-and-manifest.md#implementation-plan). - **Removing `legacy/` last** keeps the legacy scripts working through the other steps on `release/2.1` too. Until then the [phase rules](#how-phases-work) still apply: legacy adapts and the domain doesn't bend. - **Smaller PRs go to `main`:** [#270](https://github.com/SpringCare/VERA-MH/pull/270) (tests only), [#278](https://github.com/SpringCare/VERA-MH/pull/278) (`vera` command), [#269](https://github.com/SpringCare/VERA-MH/pull/269) (user-side language). - **When every step has merged into `release/2.1`:** merge `main` into it one last time, move `[Unreleased]` to `v2.1.0` in the CHANGELOG, merge `release/2.1` into `main`, and tag that merge commit `v2.1.0`. From 4ec11a98f0c23b8455768c942ac4d959620020d2 Mon Sep 17 00:00:00 2001 From: Luca Belli <129434630+sator-labs@users.noreply.github.com> Date: Mon, 26 Oct 2026 11:44:24 -0700 Subject: [PATCH 6/6] docs: legacy scripts are no longer supported; removing them is step 1 of 2.1 Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 14 +++++--------- docs/roadmap.md | 27 +++++++++++++-------------- 2 files changed, 18 insertions(+), 23 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 775e30a5..010da653 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -45,15 +45,11 @@ cp .env.example .env # Add API keys (ANTHROPIC_API_KEY, OPENAI_API_KEY, et **Entry points:** `vera.py` (`generate`, `judge`, `score`, `pipeline`) is the only supported entry point. The old scripts live in [`legacy/`](legacy/); they are -deprecated but kept for now, and their removal is deferred (see -[docs/roadmap.md](docs/roadmap.md#deferred)). +no longer supported, and removing them is the first step of the 2.1 release +(see [docs/roadmap.md](docs/roadmap.md#release-21-plan)). -**Legacy adapts; the domain doesn't bend.** `legacy/` may import domain packages -and `utils/`, never the reverse. When a change alters a domain signature or a -`utils/` helper that a legacy script uses, put the old-shape adapter in `legacy/` -rather than keeping a compatibility stopgap in the domain. The parity tests -(`tests/integration/test_cli_parity.py`, `tests/integration/test_score_parity.py`) -must stay green. +**Don't keep legacy working.** Don't add adapters or compatibility stopgaps for +the legacy scripts, and don't shape a domain change around them. **Temporary experiments:** `tmp_tests/` (not committed). **Permanent tests:** `tests/`. @@ -146,7 +142,7 @@ uv run pyright pre-commit run --all-files ``` -The scripts in [`legacy/`](legacy/) (`generate.py`, `judge.py`, `score.py`, `run_pipeline.py`, `run_recommended_vera_pipeline.sh`) are deprecated and may be removed at any time; don't build on them, and never import them from `vera_cli/` or the domain packages. Their flags and `vera` equivalents are in [docs/legacy-scripts.md](docs/legacy-scripts.md). +The scripts in [`legacy/`](legacy/) (`generate.py`, `judge.py`, `score.py`, `run_pipeline.py`, `run_recommended_vera_pipeline.sh`) are no longer supported and are removed in 2.1; don't build on them, and never import them from `vera_cli/` or the domain packages. Their flags and `vera` equivalents are in [docs/legacy-scripts.md](docs/legacy-scripts.md). Use full provider model IDs (e.g. `claude-opus-4-5`, `gpt-5.4`) as in README. There are no aliases: `sonnet` or `gpt` alone won't resolve. Current defaults and the recommended models are in `llm_clients/config.py`. diff --git a/docs/roadmap.md b/docs/roadmap.md index 1a709ee3..eaa0a012 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -15,16 +15,16 @@ Smaller items that belong to no phase are in the [Backlog](#backlog). | Step | What | PRs | Phase | |---|---|---|---| -| 1. Safety net | Tests that prove a change didn't move scores: golden scoring, scripted judge, replay of a recorded run | [#270](https://github.com/SpringCare/VERA-MH/pull/270) (needs the replay recording) | | -| 2. Data contracts | What a run records (manifest, run config, run record), then naming, order of events, and folder layout, decided together | [#277](https://github.com/SpringCare/VERA-MH/pull/277), then [#268](https://github.com/SpringCare/VERA-MH/pull/268) + [#276](https://github.com/SpringCare/VERA-MH/pull/276) | [Traceability](#traceability) | -| 3. Engine | One job loop for generation and judging; the pipeline as a single job list | [#266](https://github.com/SpringCare/VERA-MH/pull/266) | [Shared engine](#shared-engine) | -| 4. Terminal output | Plain event lines, fail-fast, end summary, and prints moved to logs (any time); the live display (after step 3) | [#271](https://github.com/SpringCare/VERA-MH/pull/271) | [Backlog](#cli-and-run-experience) | -| 5. Remove `legacy/` | Delete the scripts, their parity tests, and [legacy-scripts.md](./legacy-scripts.md), as announced in v2.0.4 | — | [Deferred](#deferred) | -| In parallel | A judge flow that decides each outcome once; needs the HFO rubric authors' answer on Q11 | [#267](https://github.com/SpringCare/VERA-MH/pull/267) (after step 1) | [Rubric-agnostic scoring](#rubric-agnostic-scoring) | - -- **Why this order.** Step 1 comes first because every later step is meant to leave scores unchanged, and it's what proves that. Step 2 defines the files and folders the job loop in step 3 reads and writes (its resume checks, the run roots it needs up front). Step 4's live display needs the events step 3 emits; its other half needs nothing. -- **Inside step 2,** the two designs interleave: #277's manifest and config-defaults PRs (A, B); then the run-root function, one clock read, and their enforcement tests ([run-lifecycle.md](./design/run-lifecycle.md)); then the run record and what builds on it (#277's C, then D, E, F); then the new layout and conversation files (#268, #276). #277's model check at load (G) can land any time. Each design's PR list is in its "Implementation plan" section, for example [config-and-manifest.md](./design/config-and-manifest.md#implementation-plan). -- **Removing `legacy/` last** keeps the legacy scripts working through the other steps on `release/2.1` too. Until then the [phase rules](#how-phases-work) still apply: legacy adapts and the domain doesn't bend. +| 1. Remove `legacy/` | Delete the scripts, their parity tests, [legacy-scripts.md](./legacy-scripts.md), and the domain stopgaps that exist only for them, as announced in v2.0.4 | — | | +| 2. Safety net | Tests that prove a change didn't move scores: golden scoring, scripted judge, replay of a recorded run | [#270](https://github.com/SpringCare/VERA-MH/pull/270) (needs the replay recording) | | +| 3. Data contracts | What a run records (manifest, run config, run record), then naming, order of events, and folder layout, decided together | [#277](https://github.com/SpringCare/VERA-MH/pull/277), then [#268](https://github.com/SpringCare/VERA-MH/pull/268) + [#276](https://github.com/SpringCare/VERA-MH/pull/276) | [Traceability](#traceability) | +| 4. Engine | One job loop for generation and judging; the pipeline as a single job list | [#266](https://github.com/SpringCare/VERA-MH/pull/266) | [Shared engine](#shared-engine) | +| 5. Terminal output | Plain event lines, fail-fast, end summary, and prints moved to logs (any time); the live display (after step 4) | [#271](https://github.com/SpringCare/VERA-MH/pull/271) | [Backlog](#cli-and-run-experience) | +| In parallel | A judge flow that decides each outcome once; needs the HFO rubric authors' answer on Q11 | [#267](https://github.com/SpringCare/VERA-MH/pull/267) (after step 2) | [Rubric-agnostic scoring](#rubric-agnostic-scoring) | + +- **Legacy scripts are no longer supported.** From now on, no change needs to keep `legacy/` working, so removing it is step 1: every later step then changes one code path instead of two, with no old-shape adapters. Users who still need the scripts stay on 2.0.x (`main`) until 2.1 ships. +- **Why this order.** Step 2 comes before the rest because every later step is meant to leave scores unchanged, and it's what proves that. Step 3 defines the files and folders the job loop in step 4 reads and writes (its resume checks, the run roots it needs up front). Step 5's live display needs the events step 4 emits; its other half needs nothing. +- **Inside step 3,** the two designs interleave: #277's manifest and config-defaults PRs (A, B); then the run-root function, one clock read, and their enforcement tests ([run-lifecycle.md](./design/run-lifecycle.md)); then the run record and what builds on it (#277's C, then D, E, F); then the new layout and conversation files (#268, #276). #277's model check at load (G) can land any time. Each design's PR list is in its "Implementation plan" section, for example [config-and-manifest.md](./design/config-and-manifest.md#implementation-plan). - **Smaller PRs go to `main`:** [#270](https://github.com/SpringCare/VERA-MH/pull/270) (tests only), [#278](https://github.com/SpringCare/VERA-MH/pull/278) (`vera` command), [#269](https://github.com/SpringCare/VERA-MH/pull/269) (user-side language). - **When every step has merged into `release/2.1`:** merge `main` into it one last time, move `[Unreleased]` to `v2.1.0` in the CHANGELOG, merge `release/2.1` into `main`, and tag that merge commit `v2.1.0`. @@ -33,7 +33,7 @@ Smaller items that belong to no phase are in the [Backlog](#backlog). - **Phases have names, not numbers.** Code comments and docs cite them by name ("Traceability"), so reordering or inserting a phase never changes what an old reference means. - **Order is dependency order.** Each phase builds only on what earlier phases leave behind. Storage and OpenSpec are independent of the others and can run whenever a concrete need appears. - **"Done when" is the bar.** A phase isn't done because its changes landed; it's done when its Done-when holds. Every Done-when also includes updating `README.md`, `AGENTS.md`, and any other doc the phase makes stale. -- **Every phase keeps [`legacy/`](../legacy/) working.** Removing the legacy scripts is [deferred](#deferred). Until then, legacy adapts and the domain doesn't bend: `legacy/` may import domain packages and `utils/`, never the reverse, and when a phase changes a signature or helper a legacy script uses, the old-shape adapter moves into `legacy/` instead of staying in the domain. The parity tests (`tests/integration/test_cli_parity.py`, `test_score_parity.py`) must stay green. +- **No phase keeps [`legacy/`](../legacy/) working.** The legacy scripts are no longer supported, and removing them is step 1 of the [Release 2.1 plan](#release-21-plan). Don't add adapters or compatibility stopgaps for them. - **Planning a phase doesn't pre-clear it.** Each phase still needs maintainer sign-off before it starts (see [ESCALATE](./architecture.md#escalate-stop-and-ask)). - **Rollback:** a phase that ships broken is reverted through its own PRs. There is no feature-flag period carrying old and new behavior side by side. @@ -65,7 +65,7 @@ Scoring today assumes the SI rubric in several places. ### Traceability - **Persisted run records.** `utils/config_schema.py` formalizes the config shape into a stable interface (design doc required for later changes). Each run writes `config.json` once at start, plus `state.json` and a `.sha256` sidecar. The config records the checksum of every input file it uses (rubrics, personas, prompts), so a changed input fails loudly instead of silently. -- **Naming.** Rewrite `utils/naming.py` for the `c_`/`u_`/`j_` scheme. The `p_`/`j_` helpers the legacy scripts use (`legacy/judge.py`, `legacy/run_pipeline.py`) move into `legacy/`, which keeps writing the old layout. +- **Naming.** Rewrite `utils/naming.py` for the `c_`/`u_`/`j_` scheme. The `p_`/`j_` helpers only the legacy scripts used go with them. - **Run metadata from records, not folder names.** Pooling (`score/pool.py`) works out persona, agent, turns, and judge by parsing `p_*`/`j_*` names and folder nesting, so runs outside that layout become `unknown`. Read them from each run's `config.json` instead. - **Target-rooted output.** New runs write under `output//`, with pooled results under `c_/pooled/` (see [Data flow and artifacts](./architecture.md#data-flow-and-artifacts)). The target root attributes judging output on its own, which unblocks `vera judge --target all`: its only blocker was output attribution. - **Resume.** `--into` already resumes a run folder. Decide whether a `state.json`-backed `vera resume` still adds enough to build. @@ -77,7 +77,7 @@ Scoring today assumes the SI rubric in several places. ### Model spec boundary -- Generation and judging take `ModelSpec` directly. Delete the `run_for_user_models` / `_legacy_model_config` stopgaps in `generate/run.py`; `legacy/generate.py` builds its own `ModelSpec` instead. +- Generation and judging take `ModelSpec` directly. Delete the `run_for_user_models` / `_legacy_model_config` stopgaps in `generate/run.py`. - Per-model judge parameters: lift the "all judge models must share provider parameters" check in `utils/config_schema.py`. - Afterwards `generate` and `judge` describe models identically. @@ -155,7 +155,6 @@ Smaller items that belong to no phase. Each can ship as its own PR whenever it's ## Deferred -- **Removing `legacy/`.** Triggered by a tagged release whose CHANGELOG announces the removal; [v2.0.4](https://github.com/SpringCare/VERA-MH/releases/tag/v2.0.4) did. Then delete `legacy/`, the legacy side of the parity tests, and [legacy-scripts.md](./legacy-scripts.md). Scheduled as the last step of the [Release 2.1 plan](#release-21-plan). - **Per-persona system prompts.** Personas may need their own system prompts rather than one global prompt. The mechanism (a `system_prompt` field, a `system_prompt_file` pointer, or something else) is undecided; revisit when the persona schema is formalized, likely in [Traceability](#traceability). ## Completed