From 153b66905641fe99344f8cc0fee355e942ed49f9 Mon Sep 17 00:00:00 2001 From: Luca Belli <129434630+sator-labs@users.noreply.github.com> Date: Tue, 27 Oct 2026 12:33:55 -0700 Subject: [PATCH 1/2] =?UTF-8?q?docs:=20roadmap=20=E2=80=94=20full-run=20ch?= =?UTF-8?q?eck,=20then=20move=20and=20rename=20Role,=20before=20simplifyin?= =?UTF-8?q?g?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a first step to the pre-loop simplification: a replay of a saved full run (generation from transcripts, judging from judge logs, then scoring and pooling) as the guard for everything after it; then move Role to utils/role.py and rename it to USER/CHATBOT/JUDGE. Making first_speaker a Role is folded into phase 1(b), which rewrites the same functions. Code-only names are left to that rewrite; public output names to the layout and chart work. Co-Authored-By: Claude Opus 5.5 --- docs/roadmap.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/roadmap.md b/docs/roadmap.md index 6ca307f2..71c15f08 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -26,7 +26,13 @@ Smaller items that belong to no phase are in the [Backlog](#backlog). - **Why this order.** Step 2 comes before the rest because every later step is meant to leave scores unchanged, and it's what proves that. Step 3 defines the files and folders the job loop in step 4 reads and writes (its resume checks, the run roots it needs up front). Step 5's live display needs the events step 4 emits; its other half needs nothing. - **Inside step 3,** the two designs interleave: #277's manifest and config-defaults PRs (A, B); then the run-root function, one clock read, and their enforcement tests ([run-lifecycle.md](./design/run-lifecycle.md)); then the run record and what builds on it (#277's C, then D, E, F); then the new layout and conversation files (#268, #276). #277's model check at load (G) can land any time. Each design's PR list is in its "Implementation plan" section, for example [config-and-manifest.md](./design/config-and-manifest.md#implementation-plan). - **Before step 4, simplify each component** (waits for the step-3 stack, #289–#293, because it rewrites the same files). Each component reaches its work through layers that existed for the legacy scripts: generation goes `vera_cli` → `run_for_user_models` → `run_generation` → `ConversationRunner` → its pool → one conversation; judging goes `vera_cli` → `run_judging` → `judge_conversations` → `batch_evaluate_with_individual_judges` → its pool → one evaluation. Scoring is already one layer (`run_scoring`) and is the model. The target, from [#266](https://github.com/SpringCare/VERA-MH/pull/266), is the same shape everywhere: `/run.py` holds one `run_(config)` entry point, a module holds one unit of work, plus `utils.py` and `viz.py`. Two phases: - 1. **Before the loop, three behavior-preserving PRs.** (a) Delete legacy leftovers with no callers: `judge_single_conversation`, `build_single_conversation_judge_run_key`, `get_judge_logs_root`, the "legacy layout" log paths, and parameters every caller passes the same value for (`run_id`, `persona_names`, `verbose`). (b) Generation takes `GenerationConfig` directly: `run_for_user_models` and `run_generation` become one `run_generation(config, …)`, and `_legacy_model_config` is deleted ([Model spec boundary](#model-spec-boundary)). (c) Judging takes `JudgingConfig` directly: `run_judging`, `judge_conversations`, and `batch_evaluate_with_individual_judges` become one `run_judging(config, …)`, with one path for a new run versus `--into` instead of the `output_root` / `output_folder` split. None of these depend on the loop design, and they shrink step 4: swapping a component's pool then touches one function instead of four layers. + 0. **First, a full-run check, then one shared `Role`.** + - **Full-run check (tests only, first).** A saved run of the recommended SI profile (2 user models × 100 conversations, judged and pooled) is replayed with no model calls: generation answers each turn from the saved transcripts, the judge answers each rubric question from the saved judge logs, then scoring and pooling run as usual. The expected outputs (transcripts, judge TSVs, `results.csv`, scores) come from one run of the check on current `release/2.1`, not from the saved run itself, which older code produced; comparisons ignore absolute paths, timestamps, and log files. The full run stays local; a subset is committed so CI runs it. It guards everything below: it doesn't depend on prompt wording, but fails on any code change that alters how a conversation is run, judged, or scored. + - **Move `Role`** from `llm_clients/llm_interface.py` to `utils/role.py`, as [architecture.md](./architecture.md#layer-model) requires; import lines only. + - **Rename `Role`** to the public u/c/j vocabulary: `PERSONA`/`PROVIDER` become `USER`/`CHATBOT` (`"user"`, `"chatbot"`). The values appear only in log lines and LLM response metadata, not in transcripts (already `user:`/`chatbot:`) or scores. + + Code-only names (`persona_model_config`, `agent_model_config`, `llm1`/`llm2`) are not renamed here: phase 1 rewrites those functions and most of the names go with them. Public output names (`p_`/`a_` folder prefixes, `persona_model`/`agent_model` in `scores.json`) change with the new layout (#276) and the chart work (#286), each as a deliberate breaking change. + 1. **Before the loop, three behavior-preserving PRs.** (a) Delete legacy leftovers with no callers: `judge_single_conversation`, `build_single_conversation_judge_run_key`, `get_judge_logs_root`, the "legacy layout" log paths, and parameters every caller passes the same value for (`run_id`, `persona_names`, `verbose`). (b) Generation takes `GenerationConfig` directly: `run_for_user_models` and `run_generation` become one `run_generation(config, …)`, `_legacy_model_config` is deleted ([Model spec boundary](#model-spec-boundary)), and `first_speaker` reaches the domain as a `Role` instead of a bool (review question on #293). Doing that here rather than earlier avoids editing two functions that this step merges. (c) Judging takes `JudgingConfig` directly: `run_judging`, `judge_conversations`, and `batch_evaluate_with_individual_judges` become one `run_judging(config, …)`, with one path for a new run versus `--into` instead of the `output_root` / `output_folder` split. None of these depend on the loop design, and they shrink step 4: swapping a component's pool then touches one function instead of four layers. 2. **With the loop (step 4).** Splitting out "one conversation" (`generate/conversation.py`) and "one evaluation" (`judge/evaluation.py`) is the same work as defining #266's `Session` and `Evaluation` jobs and their resume checks, so it happens in the loop PRs, one command at a time: judge first, then generate, then the pipeline. Each deletes that component's `runner.py`. Not all at once: one PR changing structure and concurrency together would be the hardest to review, and the hardest to bisect if scores moved. @@ -75,7 +81,7 @@ Scoring today assumes the SI rubric in several places. - **Target-rooted output.** New runs write under `output//`, with pooled results under `c_/pooled/` (see [Data flow and artifacts](./architecture.md#data-flow-and-artifacts)). The target root attributes judging output on its own, which unblocks `vera judge --target all`: its only blocker was output attribution. - **Resume.** `--into` already resumes a run folder. Decide whether a `state.json`-backed `vera resume` still adds enough to build. - **Make `utils/` a real leaf, and enforce it.** [architecture.md](./architecture.md) says `utils/` imports no other project package, but `utils/conversation_utils.py` (for `Role`) and `utils/logging_utils.py` (for `LLMInterface`) import `llm_clients/`. Because importing any `utils` submodule runs `utils/__init__.py`, that coupling once made `import score` load the whole LLM stack; the Scoring split fixed the symptom with an empty `utils/__init__.py` and a narrower contract (`score` may not import `llm_clients`), not the cause. To discuss: - - where the two modules go: transcript writing to `generate/`, LLM response logging to `llm_clients/`, or `Role` moved into `utils/` so they can stay; + - where the two modules go: transcript writing to `generate/`, LLM response logging to `llm_clients/`, or `Role` moved into `utils/` so they can stay. `Role` moves into `utils/role.py` before step 4 (see the [Release 2.1 plan](#release-21-plan)), which settles it for `conversation_utils.py`; `logging_utils.py` still imports `LLMInterface`; - the import-linter contract that `utils/` imports no project package, which then replaces the narrower `score` one; - keeping a fresh-interpreter test of what each package actually loads (`tests/unit/score/test_import_isolation.py`), since import-linter reads import statements and can't see what an `__init__.py` pulls in. - **Acknowledged compatibility break.** Anything outside `vera.py` that parses `p_*`/`j_*` names directly (`spring_scripts/`, `distribute_files.py`, `score/compare.py`, notebooks, human-review tooling) stops auto-discovering new runs. Reading old data keeps working: `vera judge --conversations ` and `vera score -r ` take explicit paths and never derive meaning from folder names. Existing `output/` folders are left alone; there is no migration script. From e556448b141a69a2c42e40bf0443f3d856d0530b Mon Sep 17 00:00:00 2001 From: Luca Belli <129434630+sator-labs@users.noreply.github.com> Date: Tue, 27 Oct 2026 16:36:25 -0700 Subject: [PATCH 2/2] =?UTF-8?q?docs:=20roadmap=20=E2=80=94=20drop=20Role.J?= =?UTF-8?q?UDGE;=20constant=20parameters=20go=20with=20the=201(b)/(c)=20re?= =?UTF-8?q?writes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- docs/roadmap.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/roadmap.md b/docs/roadmap.md index 71c15f08..ea27d1ae 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -27,12 +27,13 @@ Smaller items that belong to no phase are in the [Backlog](#backlog). - **Inside step 3,** the two designs interleave: #277's manifest and config-defaults PRs (A, B); then the run-root function, one clock read, and their enforcement tests ([run-lifecycle.md](./design/run-lifecycle.md)); then the run record and what builds on it (#277's C, then D, E, F); then the new layout and conversation files (#268, #276). #277's model check at load (G) can land any time. Each design's PR list is in its "Implementation plan" section, for example [config-and-manifest.md](./design/config-and-manifest.md#implementation-plan). - **Before step 4, simplify each component** (waits for the step-3 stack, #289–#293, because it rewrites the same files). Each component reaches its work through layers that existed for the legacy scripts: generation goes `vera_cli` → `run_for_user_models` → `run_generation` → `ConversationRunner` → its pool → one conversation; judging goes `vera_cli` → `run_judging` → `judge_conversations` → `batch_evaluate_with_individual_judges` → its pool → one evaluation. Scoring is already one layer (`run_scoring`) and is the model. The target, from [#266](https://github.com/SpringCare/VERA-MH/pull/266), is the same shape everywhere: `/run.py` holds one `run_(config)` entry point, a module holds one unit of work, plus `utils.py` and `viz.py`. Two phases: 0. **First, a full-run check, then one shared `Role`.** - - **Full-run check (tests only, first).** A saved run of the recommended SI profile (2 user models × 100 conversations, judged and pooled) is replayed with no model calls: generation answers each turn from the saved transcripts, the judge answers each rubric question from the saved judge logs, then scoring and pooling run as usual. The expected outputs (transcripts, judge TSVs, `results.csv`, scores) come from one run of the check on current `release/2.1`, not from the saved run itself, which older code produced; comparisons ignore absolute paths, timestamps, and log files. The full run stays local; a subset is committed so CI runs it. It guards everything below: it doesn't depend on prompt wording, but fails on any code change that alters how a conversation is run, judged, or scored. + - **Full-run check (tests only, first).** A saved run of the recommended SI profile (2 user models × 100 conversations, judged and pooled) is replayed with no model calls: generation answers each turn from the saved transcripts, the judge answers each rubric question from the saved judge answers (`answers/by_question/`), then scoring and pooling run as usual. The expected outputs (transcripts, judge TSVs, `results.csv`, scores) come from one run of the check on current `release/2.1`, not from the saved run itself, which older code produced; comparisons ignore absolute paths, timestamps, and log files. The full run stays local; a subset is committed so CI runs it. It guards everything below: it doesn't depend on prompt wording, but fails on any code change that alters how a conversation is run, judged, or scored. - **Move `Role`** from `llm_clients/llm_interface.py` to `utils/role.py`, as [architecture.md](./architecture.md#layer-model) requires; import lines only. - - **Rename `Role`** to the public u/c/j vocabulary: `PERSONA`/`PROVIDER` become `USER`/`CHATBOT` (`"user"`, `"chatbot"`). The values appear only in log lines and LLM response metadata, not in transcripts (already `user:`/`chatbot:`) or scores. + - **Drop `Role.JUDGE`.** A judge is not in the conversation, so it gets no role (`role=None`); logs and response metadata still say `"judge"`. `Role` then names only who speaks. + - **Rename `Role`** to the public u/c/j vocabulary: `PERSONA`/`PROVIDER` become `USER`/`CHATBOT` (`"user"`, `"chatbot"`). The values appear only in log lines and LLM response metadata, not in transcripts (already `user:`/`chatbot:`) or scores. The transcript's early-termination line keeps saying `persona`/`provider`, since the judge reads transcripts. Code-only names (`persona_model_config`, `agent_model_config`, `llm1`/`llm2`) are not renamed here: phase 1 rewrites those functions and most of the names go with them. Public output names (`p_`/`a_` folder prefixes, `persona_model`/`agent_model` in `scores.json`) change with the new layout (#276) and the chart work (#286), each as a deliberate breaking change. - 1. **Before the loop, three behavior-preserving PRs.** (a) Delete legacy leftovers with no callers: `judge_single_conversation`, `build_single_conversation_judge_run_key`, `get_judge_logs_root`, the "legacy layout" log paths, and parameters every caller passes the same value for (`run_id`, `persona_names`, `verbose`). (b) Generation takes `GenerationConfig` directly: `run_for_user_models` and `run_generation` become one `run_generation(config, …)`, `_legacy_model_config` is deleted ([Model spec boundary](#model-spec-boundary)), and `first_speaker` reaches the domain as a `Role` instead of a bool (review question on #293). Doing that here rather than earlier avoids editing two functions that this step merges. (c) Judging takes `JudgingConfig` directly: `run_judging`, `judge_conversations`, and `batch_evaluate_with_individual_judges` become one `run_judging(config, …)`, with one path for a new run versus `--into` instead of the `output_root` / `output_folder` split. None of these depend on the loop design, and they shrink step 4: swapping a component's pool then touches one function instead of four layers. + 1. **Before the loop, three behavior-preserving PRs.** (a) Delete legacy leftovers with no callers: `judge_single_conversation`, `build_single_conversation_judge_run_key`, `get_judge_logs_root`, the "legacy layout" log paths, and `ConversationRunner._create_conversation_jobs`. (b) Generation takes `GenerationConfig` directly: `run_for_user_models` and `run_generation` become one `run_generation(config, …)`, `_legacy_model_config` is deleted ([Model spec boundary](#model-spec-boundary)), and `first_speaker` reaches the domain as a `Role` instead of a bool (review question on #293). Doing that here rather than earlier avoids editing two functions that this step merges. Parameters every caller passes the same value for go with the rewrite instead of in (a), so each signature changes once: `run_id` (production always passes `None`; `ConversationRunner` never reads it) `persona_names` (always `None`; only tests use the `name_list` filter in `load_prompts_from_csv`), and `verbose` (always `True`). (c) Judging takes `JudgingConfig` directly: `run_judging`, `judge_conversations`, and `batch_evaluate_with_individual_judges` become one `run_judging(config, …)`, with one path for a new run versus `--into` instead of the `output_root` / `output_folder` split, and `verbose` (always `True` there, effectively always `False` inside `LLMJudge`) is removed with its print branches. None of these depend on the loop design, and they shrink step 4: swapping a component's pool then touches one function instead of four layers. 2. **With the loop (step 4).** Splitting out "one conversation" (`generate/conversation.py`) and "one evaluation" (`judge/evaluation.py`) is the same work as defining #266's `Session` and `Evaluation` jobs and their resume checks, so it happens in the loop PRs, one command at a time: judge first, then generate, then the pipeline. Each deletes that component's `runner.py`. Not all at once: one PR changing structure and concurrency together would be the hardest to review, and the hardest to bisect if scores moved.