From 2b2bc26a5b82b6c2aff8303959aa0eca76d1c7b7 Mon Sep 17 00:00:00 2001 From: EllAchE <26192612+EllAchE@users.noreply.github.com> Date: Tue, 25 Aug 2026 10:48:49 -0400 Subject: [PATCH] S-135743: Require agents to declare evaluation substitutions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Evaluation runs can silently substitute Cicero's seeded demo entities for the data a scenario brief specifies — the evaluator's browse skill tells agents to continue with whatever data exists, and nothing required saying so. That leaves judges unable to tell whether agenda/widget evidence came through the CFP-review-accept chain or from a seeded stand-in. Require each completed scenario to declare any substitution in its observe notes and done summary, and direct judges to read those declarations, say when a verdict rests on a seeded stand-in, and flag undeclared seeded entities as an evidence defect. Co-Authored-By: Claude Opus 5 --- docs/handoff/sessionboard-eval.md | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/docs/handoff/sessionboard-eval.md b/docs/handoff/sessionboard-eval.md index b70e9487..e61172d1 100644 --- a/docs/handoff/sessionboard-eval.md +++ b/docs/handoff/sessionboard-eval.md @@ -79,12 +79,24 @@ For every unfinished scenario: 2. Drive the hosted product with `snapshot`, `click`, `fill`, `select`, `press`, `scroll`, `drag`, and `upload` as needed. 3. Save screenshots at meaningful states and record factual findings with `observe`. -4. Finish with `done`, including an honest `completed`, `blocked`, or `feature_not_found` outcome. +4. Declare every substitution before finishing. The evaluator's browse skill tells you to continue + with whatever data exists when the brief's sample data does not match the app, so runs can end up + exercising Cicero's seeded demo entities instead of the data the brief specifies — legitimately, + but never silently. Whenever a scenario proceeds against a seeded entity in place of the brief's + sample data, or in place of an entity an upstream scenario was supposed to create, record an + `observe` note naming what the brief specified, what was actually used, and why. +5. Finish with `done`, including an honest `completed`, `blocked`, or `feature_not_found` outcome. + A `completed` scenario that ran on substituted data must restate the substitution in its + summary; an undeclared substitution leaves the evidence ambiguous, so the scenario is not + honestly complete without it. Run browsing scenarios serially and in spec order. They intentionally share application state: CFP submissions feed review, accepted submissions feed agenda construction, and the published agenda -feeds public widgets. The evaluator also has one shared `.sbek-current-run` pointer and no scenario -claim lock, so parallel browser workers can corrupt the run or race the product state. +feeds public widgets. That chain is why substitutions must be declared per scenario: a screenshot of +a seeded session in the agenda scenario looks identical to one the CFP→review→accept chain produced, +and only the declaration tells a judge which data path the evidence actually proves. The evaluator +also has one shared `.sbek-current-run` pointer and no scenario claim lock, so parallel browser +workers can corrupt the run or race the product state. Re-run `plan` to see remaining coverage. Resume an interrupted API-path run with the command printed at the end of `run.log`; in harness mode, completed `evidence.json` files remain valid and only the @@ -101,7 +113,13 @@ pnpm run sbek -- judge-brief --area call-for-papers Read every selected screenshot and write the area judgement under `runs//judgements/`. Distinguish `not_found` (the product lacks it) from `cannot_judge` -(the evidence did not reach it). Judge areas may run in parallel because each writes a distinct +(the evidence did not reach it). Browsing agents are required to declare, in `observe` notes and +`done` summaries, any scenario that ran against seeded demo entities instead of the brief's sample +data or an upstream scenario's output — those declarations reach you through the rendered evidence, +so read them before crediting a scenario, and state in the judgement when a verdict rests on a +seeded stand-in rather than the scenario's own data path. Evidence showing seeded entities with no +declared substitution is ambiguous: flag it as an evidence defect rather than crediting the flow. +Judge areas may run in parallel because each writes a distinct file; scoring must have exactly one writer: ```bash