From 91295837c5ad698759b655fbdc3f629de03830ec Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Wed, 11 Nov 2026 12:11:06 -0500 Subject: [PATCH 1/7] feat(planning): run the plan reviewer only for risky plans Under plan_stress_test auto, /planning:plan dispatches the plan-reviewer subagent only when the blast radius is HIGH or CRITICAL, a stress-test trigger matches, or the user asks for a review; other plans skip it and say so in a Reviewer: line. reviewer, both and verification strict still force it; devils-advocate and verification light/off still skip it. A new eval case checks that a low-risk plan skips the reviewer. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VQb9Q4F46UumaUgXeUdpYZ --- .../feat-plan-reviewer-risk-gated-140d7eb6.md | 8 ++ docs/contracts/verification-level/README.md | 2 +- plugins/planning/README.md | 2 +- .../plan-low-risk-skips-reviewer/case.yaml | 4 + .../graders/no-reviewer-dispatch.md | 8 ++ .../graders/writes-skip-line.md | 5 + .../plan-low-risk-skips-reviewer/prompt.md | 11 ++ .../samples/no-reviewer-dispatch.json | 14 +++ .../samples/writes-skip-line.json | 26 +++++ .../plan-low-risk-skips-reviewer/scaffold.sh | 102 ++++++++++++++++++ plugins/planning/reference/config.md | 3 +- plugins/planning/schemas/planning.schema.json | 2 +- plugins/planning/skills/plan/SKILL.md | 23 ++-- .../plan/context/stress-test-triggers.md | 2 +- plugins/planning/skills/plan/evals/evals.json | 2 +- .../skills/plan/templates/checklist.md | 6 +- 16 files changed, 200 insertions(+), 20 deletions(-) create mode 100644 .changes/planning/feat-plan-reviewer-risk-gated-140d7eb6.md create mode 100644 plugins/planning/evals/plan-low-risk-skips-reviewer/case.yaml create mode 100644 plugins/planning/evals/plan-low-risk-skips-reviewer/graders/no-reviewer-dispatch.md create mode 100644 plugins/planning/evals/plan-low-risk-skips-reviewer/graders/writes-skip-line.md create mode 100644 plugins/planning/evals/plan-low-risk-skips-reviewer/prompt.md create mode 100644 plugins/planning/evals/plan-low-risk-skips-reviewer/samples/no-reviewer-dispatch.json create mode 100644 plugins/planning/evals/plan-low-risk-skips-reviewer/samples/writes-skip-line.json create mode 100755 plugins/planning/evals/plan-low-risk-skips-reviewer/scaffold.sh diff --git a/.changes/planning/feat-plan-reviewer-risk-gated-140d7eb6.md b/.changes/planning/feat-plan-reviewer-risk-gated-140d7eb6.md new file mode 100644 index 00000000000..992442afb1a --- /dev/null +++ b/.changes/planning/feat-plan-reviewer-risk-gated-140d7eb6.md @@ -0,0 +1,8 @@ +--- +bump: minor +--- + +### Changed + +- `/planning:plan` no longer dispatches the fresh-context `plan-reviewer` on every plan. Under the default `plan_stress_test: auto` it runs only when the blast radius is HIGH or CRITICAL, a stress-test trigger matches, or the user asks for a review; the blast radius is now assessed before that decision, and the Step 5 presentation states a skip and its reason (`Reviewer: skipped (blast radius LOW, no trigger)`). `plan_stress_test` `reviewer` and `both` and verification `strict` still always run it; `light` and `off` still skip it. Basis: Anthropic's [Prompting Claude Opus 5, "Task scope and over-verification"](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5#task-scope-and-over-verification): explicit verification steps and "use a subagent to verify" cause over-verification, and removing them reduces wasted tokens with no loss in quality. +- New eval case `plan-low-risk-skips-reviewer`: a LOW blast-radius draft with default settings dispatches no plan-reviewer and reports the skip line. diff --git a/docs/contracts/verification-level/README.md b/docs/contracts/verification-level/README.md index ee90497de72..5075cd72044 100644 --- a/docs/contracts/verification-level/README.md +++ b/docs/contracts/verification-level/README.md @@ -81,7 +81,7 @@ A skill that is itself the independent check when invoked does not read the leve | `/discovery:research`, `/discovery:explore`, `/discovery:trace-intent` | the sibling verifier (parent contract, "The sibling verifier, stated once"); `strict` widens research's grading to every row and claim | | `/discipline:do-your-research`, `/discipline:recheck-against-upstream`, `/discipline:sweep-all` | the fresh-context confirmation of a clean audit on load-bearing work (shared method, step 2), the fresh-context re-derivation (step 3) and the fan-out tiers' dispatch; `strict` takes do-your-research's fan-out tier at `full` depth | | `/education:quiz-me` | the fresh-context answer-key pass | -| `/planning:plan` | the plan-reviewer dispatch in Step 3 | +| `/planning:plan` | the plan-reviewer dispatch in Step 3; `strict` dispatches it for every plan, which `plan_stress_test`'s `auto` otherwise gives only to high-risk plans | | `/session-flow:orchestrate`, `/session-flow:workflow`, `/session-flow:retro` | imperative 3's verifier in a primed session; the fresh-context reviewer in pre-PR step 2, without which Stage 10 is not done; retro's single-pass analysis and scoring under `retro_lenses: 1` | | `/harness-config:audit-automation-gaps` | the fresh-context refusal gate (2.3), whose verdicts `light` lists as `unverified`, and the Phase 4 step 6 reviewer | | `/harness-config:audit-instructions` | the Phase C verifiers (per lane, B2 and I33) | diff --git a/plugins/planning/README.md b/plugins/planning/README.md index ede4d18739a..4f382f23685 100644 --- a/plugins/planning/README.md +++ b/plugins/planning/README.md @@ -20,7 +20,7 @@ where artifacts land in the consuming repo. | `/planning:design` | Design space | Explores types, contracts, module boundaries, and package topology through collaborative discussion rounds, producing capability-matrix / type-inventory / design-threads / topology artifacts; its `handoff` action delegates to `/planning:design-handoff`. Its `data`, `integration`, and `system` artifacts are typed: each carries a `scope` + `dialect` label and a fenced block in the dialect the `diagram_dialect` convention key resolves to (mermaid `erDiagram` or DBML for data; mermaid `sequenceDiagram` plus an OpenAPI 3.1 sketch for integration; a LikeC4 or C4-PlantUML container view for system, and no C4 view at all when that key is unset). `library` and `module` emit no typed artifact. | | `/planning:design-handoff` | Design→plan gate | Gates a finished design for `/planning:plan`. The gate is a binary check that every `design-threads.md` thread is RESOLVED, directional, or TAGGED-DEFERRED. Then it packages the plan-ready summary and resume prompt, or FAILs and routes back to `/planning:design`. On PASS and FAIL alike it also emits an advisory six-dimension coverage table (what, how, where, who, when, why) that never contributes to the verdict; uncovered dimensions ride along in the handoff summary and the resume prompt. | | `/planning:devils-advocate` | Adversarial review | Stress-tests plans via assumption extraction, evidence checks, failure scenarios, and operational-gotcha sweeps. Every finding evidence-backed, never generic warnings. An `incumbent` mode turns the same lens on the status quo: an Alternatives Sweep that stress-tests keeping an incumbent tool/approach against alternatives (native > official > vetted ladder, coupling priced, KEEP / MIGRATE / RESEARCH verdict), exploring the incumbent first-hand in a fresh sub-agent. | -| `/planning:plan` | Implementation plan | Produces a structured plan (goal, approach, test strategy, blast radius, parallelism analysis, tagged unilateral decisions) with a fresh-context `plan-reviewer` stress-test, on by default and set per mode by `plan_stress_test`, and a user approval gate, persisted to PLAN.md. | +| `/planning:plan` | Implementation plan | Produces a structured plan (goal, approach, test strategy, blast radius, parallelism analysis, tagged unilateral decisions) with a fresh-context `plan-reviewer` stress-test, run by default on high-risk plans and on request and set per mode by `plan_stress_test`, and a user approval gate, persisted to PLAN.md. | | `/planning:setup` | Configuration | `check` inspects the standards index read-only; `apply` interviews the consumer and bootstraps the standards index (idempotent; re-run to reconfigure). | The pipeline composes end-to-end. `wayfind` charts the fog upstream when an effort diff --git a/plugins/planning/evals/plan-low-risk-skips-reviewer/case.yaml b/plugins/planning/evals/plan-low-risk-skips-reviewer/case.yaml new file mode 100644 index 00000000000..e47697e60bf --- /dev/null +++ b/plugins/planning/evals/plan-low-risk-skips-reviewer/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-low-risk-skips-reviewer +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-low-risk-skips-reviewer/graders/no-reviewer-dispatch.md b/plugins/planning/evals/plan-low-risk-skips-reviewer/graders/no-reviewer-dispatch.md new file mode 100644 index 00000000000..95f685c2c18 --- /dev/null +++ b/plugins/planning/evals/plan-low-risk-skips-reviewer/graders/no-reviewer-dispatch.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Agent +input_match: 'plan-reviewer' +min: 0 +max: 0 +arm: both +--- diff --git a/plugins/planning/evals/plan-low-risk-skips-reviewer/graders/writes-skip-line.md b/plugins/planning/evals/plan-low-risk-skips-reviewer/graders/writes-skip-line.md new file mode 100644 index 00000000000..c8249bece92 --- /dev/null +++ b/plugins/planning/evals/plan-low-risk-skips-reviewer/graders/writes-skip-line.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: 'reviewer[*_`]*:[*_`\s]*skipped' +flags: i +--- diff --git a/plugins/planning/evals/plan-low-risk-skips-reviewer/prompt.md b/plugins/planning/evals/plan-low-risk-skips-reviewer/prompt.md new file mode 100644 index 00000000000..c373de19815 --- /dev/null +++ b/plugins/planning/evals/plan-low-risk-skips-reviewer/prompt.md @@ -0,0 +1,11 @@ +--- +description: "A LOW blast-radius, one-function draft with no stress-test trigger, in a repository with default settings, dispatches no plan-reviewer in Step 3; the reply that reports Step 3 carries the reviewer skip line" +tags: [plan, plan-reviewer, setting] +runs: 3 +max_turns: 60 +timeout_seconds: 1200 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "Resolves plan_stress_test to auto and verification to standard, assesses blast radius LOW with no trigger matching, dispatches no planning:plan-reviewer, and states Reviewer: skipped (blast radius LOW, no trigger) in the reply that reports Step 3" +--- + +/planning:plan Continue the draft at `.work/slugify-max-length/PLAN.md` from Step 3. The Brief and the design gate are settled; `.work/slugify-max-length/design/design-resolution.md` records the design early exit. Run Steps 3 and 3b as this skill and this repository's settings say, then report what each step did. Stop there: no Step 4 onward, and edit no code. diff --git a/plugins/planning/evals/plan-low-risk-skips-reviewer/samples/no-reviewer-dispatch.json b/plugins/planning/evals/plan-low-risk-skips-reviewer/samples/no-reviewer-dispatch.json new file mode 100644 index 00000000000..27a08770083 --- /dev/null +++ b/plugins/planning/evals/plan-low-risk-skips-reviewer/samples/no-reviewer-dispatch.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": [{"tool": "Read", "input": {"file_path": ".work/slugify-max-length/PLAN.md"}}], + "why": "no agent dispatched" + } + ], + "fail": [ + { + "answer": [{"tool": "Agent", "input": {"subagent_type": "planning:plan-reviewer", "description": "Review plan", "prompt": "Review .work/slugify-max-length/PLAN.md"}}], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-low-risk-skips-reviewer/samples/writes-skip-line.json b/plugins/planning/evals/plan-low-risk-skips-reviewer/samples/writes-skip-line.json new file mode 100644 index 00000000000..a1d4467eaa1 --- /dev/null +++ b/plugins/planning/evals/plan-low-risk-skips-reviewer/samples/writes-skip-line.json @@ -0,0 +1,26 @@ +{ + "pass": [ + { + "answer": "Blast radius: LOW\nReviewer: skipped (blast radius LOW, no trigger)\nStress-test needed: No", + "why": "the skip line beside the assessment" + }, + { + "answer": "- **Reviewer:** skipped (blast radius LOW, no trigger matched)", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "", + "why": "empty reply" + }, + { + "answer": "Reviewer: ran (blast radius LOW). Dispatched planning:plan-reviewer; no plan gaps found.", + "why": "the reviewer ran" + }, + { + "answer": "verification: skipped (light)", + "why": "a different skip reason" + } + ] +} diff --git a/plugins/planning/evals/plan-low-risk-skips-reviewer/scaffold.sh b/plugins/planning/evals/plan-low-risk-skips-reviewer/scaffold.sh new file mode 100755 index 00000000000..7e9fb5ebd7d --- /dev/null +++ b/plugins/planning/evals/plan-low-risk-skips-reviewer/scaffold.sh @@ -0,0 +1,102 @@ +#!/usr/bin/env bash +# A sandbox repository with no planning settings (verification standard, plan_stress_test auto), +# pushed to a local bare origin, holding a small textkit package with a test, a feature branch that +# changes it, and a settled one-function draft plan with LOW blast radius and no stress-test trigger. +set -euo pipefail + +git init -q -b main +mkdir -p textkit tests +printf '# textkit\n\nSmall text helpers. slugify lives in textkit/slug.py.\n' >README.md +: >textkit/__init__.py +cat >textkit/slug.py <<'PY' +def slugify(text): + return "-".join(text.lower().split()) +PY +cat >tests/test_slug.py <<'PY' +import unittest + +from textkit.slug import slugify + + +class SlugifyTest(unittest.TestCase): + def test_lowercases_and_hyphenates(self): + self.assertEqual(slugify("Hello World"), "hello-world") + + +if __name__ == "__main__": + unittest.main() +PY +printf '.work/\n__pycache__/\n' >.gitignore +git add . +git -c user.name=eval -c user.email=eval@example.invalid commit -q -m "scaffold textkit" +mkdir -p .work/origin.git +git init -q --bare -b main .work/origin.git +git remote add origin "$PWD/.work/origin.git" +git push -q origin main +git checkout -q -b feat/strip-punctuation +cat >textkit/slug.py <<'PY' +import re + + +def slugify(text): + words = re.sub(r"[^\w\s-]", "", text).lower().split() + return "-".join(words) +PY +git -c user.name=eval -c user.email=eval@example.invalid commit -q -am "feat: strip punctuation in slugify" + +# A settled draft plan and a design early exit, so the run reaches Step 3 without open questions. +mkdir -p .work/slugify-max-length/design +cat >.work/slugify-max-length/PLAN.md <<'MD' +# slugify max_length + +## Brief + +### Problem +Slugs from long titles make URLs too long for the share links that embed them. + +### Goal +`slugify` takes an optional `max_length` and returns a slug no longer than it, cut on a word boundary. + +### Acceptance criteria +- AC1: `slugify(text)` with no `max_length` returns what it returns today. +- AC2: `slugify(text, max_length=n)` drops whole trailing words so the result is at most `n` + characters and never ends in a hyphen; every hyphen in the output counts as a word boundary. +- AC3: When the first word alone is longer than `n`, the result is that word cut to `n` characters. +- AC4: `max_length` below 1 raises `ValueError`. + +### Out-of-scope +- Unicode transliteration. + +## Plan + +### Behavior +- B1: no limit keeps today's output (AC1). Delivered by Phase 1. +- B2: a limit cuts whole words, or the first word, to fit (AC2, AC3). Delivered by Phase 1. +- B3: a limit below 1 is refused (AC4). Delivered by Phase 1. + +### Phase 1: Add max_length [STATUS] +**Satisfies:** AC1, AC2, AC3, AC4 +**Delivers:** B1, B2, B3 + +- In `textkit/slug.py`, add `max_length=None`; after joining, when it is set, cut the slug at the + last hyphen at or before `max_length`, or at `max_length` when there is none. +- Raise `ValueError` when `max_length` is below 1. +- Add one test per criterion to `tests/test_slug.py`. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `textkit/slug.py` | modify | +| `tests/test_slug.py` | modify | +MD +# The phase heading takes its status tag here, outside the heredoc. +plan=.work/slugify-max-length/PLAN.md +text="$(cat "$plan")" +printf '%s\n' "${text//'[STATUS]'/[TODO]}" >"$plan" +cat >.work/slugify-max-length/design/design-resolution.md <<'MD' +outcome: early-exit +tier: C +reason: One optional parameter on one existing function; no new types. +MD diff --git a/plugins/planning/reference/config.md b/plugins/planning/reference/config.md index aac001d648c..b15852d2b99 100644 --- a/plugins/planning/reference/config.md +++ b/plugins/planning/reference/config.md @@ -166,7 +166,8 @@ layer. Its values: -- `auto`: the `plan-reviewer` agent always; `/planning:devils-advocate` in plan-review mode when +- `auto`: the `plan-reviewer` agent when Step 3b's blast radius is HIGH or CRITICAL, a trigger + matches, or the user asks for a review; `/planning:devils-advocate` in plan-review mode when Step 3b's blast radius is MEDIUM or higher or a trigger matches. - `reviewer`: the `plan-reviewer` agent only, whatever the blast radius. - `devils-advocate`: `/planning:devils-advocate` only, whatever the blast radius. diff --git a/plugins/planning/schemas/planning.schema.json b/plugins/planning/schemas/planning.schema.json index e067cdc23df..2a98bc58d2e 100644 --- a/plugins/planning/schemas/planning.schema.json +++ b/plugins/planning/schemas/planning.schema.json @@ -65,7 +65,7 @@ "default": 27 }, "plan_stress_test": { - "description": "Per mode of the draft, which stress-tests /planning:plan runs. auto: plan-reviewer always, /planning:devils-advocate when the blast-radius assessment calls for it. reviewer: plan-reviewer only. devils-advocate: /planning:devils-advocate only. both: both, whatever the blast radius. Only /planning:plan reads this key: a direct /planning:devils-advocate request ignores it and runs the mode its own arguments select.", + "description": "Per mode of the draft, which stress-tests /planning:plan runs. auto: plan-reviewer when the blast radius is HIGH or CRITICAL, a stress-test trigger matches, or the user asks for a review; /planning:devils-advocate when the blast-radius assessment calls for it. reviewer: plan-reviewer only. devils-advocate: /planning:devils-advocate only. both: both, whatever the blast radius. Only /planning:plan reads this key: a direct /planning:devils-advocate request ignores it and runs the mode its own arguments select.", "type": "object", "additionalProperties": false, "properties": { diff --git a/plugins/planning/skills/plan/SKILL.md b/plugins/planning/skills/plan/SKILL.md index 9d2ff7b9d70..cb0178b5e14 100644 --- a/plugins/planning/skills/plan/SKILL.md +++ b/plugins/planning/skills/plan/SKILL.md @@ -38,7 +38,7 @@ Every implementation starts with a clear design, evidence, a test strategy, and ## Emit checklist -For multi-step planning (almost always steps 1-5), copy `${CLAUDE_PLUGIN_ROOT}/skills/plan/templates/checklist.md` to `//plan-checklist.md` (default `.work/`) by reading the template with the Read tool and writing the copy with the Write tool, never `cp`, `sed` or a shell redirect from a plugin file, and tick each `- [ ]` as its step completes. Step 3 (fresh-context plan stress-test) and Step 5 (present for approval) are non-negotiable ticks, except that `plan_stress_test`'s `devils-advocate` value moves Step 3's tick to Step 4. Stress-test before presenting, except where the Step 3 verification level (`light` or `off`) skips it. +For multi-step planning (almost always steps 1-5), copy `${CLAUDE_PLUGIN_ROOT}/skills/plan/templates/checklist.md` to `//plan-checklist.md` (default `.work/`) by reading the template with the Read tool and writing the copy with the Write tool, never `cp`, `sed` or a shell redirect from a plugin file, and tick each `- [ ]` as its step completes. Step 5 (present for approval) is a non-negotiable tick. Step 3 is ticked once the reviewer rule (Gates) is applied: the reviewer ran, or its skip and reason are recorded. **Skip when:** the one-sentence skip (Purpose) applied, so no plan is written. On a mid-flight `review` replan, append a dated scope-change note and revise the PLAN phases instead; do not spawn a second checklist file. @@ -63,7 +63,7 @@ These hold after a compaction re-attach. Later steps say how to carry them out. **Verification.** These gates sit near the top so they stay inside the part of this skill that is re-attached after auto-compaction; `tests/reattach-slice.test.sh` stands in for that part with the first 20,000 bytes. How much is re-attached: [Skill content lifecycle](https://code.claude.com/docs/en/skills#skill-content-lifecycle), as of 2026-11-10; recheck when that section changes how much of a skill's invocation is re-attached, or stops describing a per-skill re-attach. -- **Reviewer.** Before assessing blast radius, dispatch a fresh-context plan-reviewer sub-agent for any plan you present, unless `plan_stress_test` (Step 3) sets `devils-advocate` for the draft's mode or the `verification` level (Step 3) is `light` or `off`. The producing thread does not self-attack the plan inline. A change skipped under the one-sentence rule (Purpose) has no plan to review. +- **Reviewer.** Before presenting, dispatch a fresh-context plan-reviewer sub-agent only when the first matching rule says so: the `verification` level (Step 3) `light` or `off` skips it; `plan_stress_test` (Step 3) `devils-advocate` for the draft's mode skips it; `reviewer` or `both`, or the `verification` level `strict`, runs it; under `auto` it runs when the Step 3b blast radius is HIGH or CRITICAL, any stress-test trigger matches, or the user asks for a plan review or stress-test (a `/planning:plan review` re-plan included), and is skipped otherwise. A skip is stated, never filled by the producing thread self-attacking the plan inline. A change skipped under the one-sentence rule (Purpose) has no plan to review. - **Hard-to-reverse decisions escalate EARLY** regardless of confidence. Below-bar judgment calls go to an interview round before the plan locks. - **Agent teams.** Route a phase to an agent team only when the parallel-safe workers must message each other and agent teams are enabled; otherwise use sub-agent workers or sequential. Disjoint file ownership between teammates is mandatory; for why, see [agent teams: avoid file conflicts](https://code.claude.com/docs/en/agent-teams#avoid-file-conflicts) (as of 2026-11-01; recheck when that section changes how teammates share files). @@ -102,7 +102,7 @@ The plan is a proposal, not a commitment. The user approves, modifies, or reject 1. The structured plan (Step 2, updated by Steps 3-4 if applicable) 2. Blast-radius assessment (Step 3b) -3. Stress-test summary (Step 4, if run), or "Skipped: blast radius LOW, no triggers matched", or "Skipped: `plan_stress_test` is `reviewer`" +3. Reviewer line (Step 3): `Reviewer: ran ()` or `Reviewer: skipped ()`, e.g. `Reviewer: skipped (blast radius LOW, no trigger)`; then the stress-test summary (Step 4, if run), or "Skipped: blast radius LOW, no triggers matched", or "Skipped: `plan_stress_test` is `reviewer`" 4. **Execution shape** (Step 4.5): the four-dimension table, parallelism shape AND per-phase routing table, with each `multi-shape` phase's attempt constraints and selection rule. Skipped for single-phase plans 5. **Displaced answers and new external effects** ("Plan changes after the Brief"). One row per change: `Q` or `none`, what the user said, what the plan now proposes, the new external effect, and the source (`reviewer fix`, `research update`, or `stress-test finding`). Omit the block when empty. The approval request names every row as needing its own reply 6. **Decisions made (gate-passed)** (Step 4.6). TABLE per [context/tag-decisions.md](context/tag-decisions.md) "Presentation contract": `Decision | What it changes in the plan | Basis (evidence) | Source`, one row per gate-passed `[EXEC-SHAPE]` / `[FALLBACK]` tag and per deferred Brief question the plan resolved (Source `deferred Q`), written for a cold reader (no session shorthand). Below-bar decisions never appear here. An empty section ("no unilateral decisions. Every PLAN item traces to brief") is valid @@ -227,29 +227,30 @@ For the removal step, invoke `/code-tidying:audit-dead-code` through the Skill t ### Step 3: Plan Stress-Test -**Which stress-tests run (`plan_stress_test`).** Name the draft's mode (`plan`, `proposal` or `incumbent`), then resolve that mode's value per [`${CLAUDE_PLUGIN_ROOT}/reference/config.md`](${CLAUDE_PLUGIN_ROOT}/reference/config.md) "Convention-home keys", which defines the modes and values (no `userConfig` layer; the default is `auto`). Report one line, e.g. `plan_stress_test.incumbent: both (docs/conventions/planning.yaml)`. `auto`, `reviewer` and `both` run this step; `devils-advocate` skips it and goes to Step 3b. +**Which stress-tests run (`plan_stress_test`).** Name the draft's mode (`plan`, `proposal` or `incumbent`), then resolve that mode's value per [`${CLAUDE_PLUGIN_ROOT}/reference/config.md`](${CLAUDE_PLUGIN_ROOT}/reference/config.md) "Convention-home keys", which defines the modes and values (no `userConfig` layer; the default is `auto`). Report one line, e.g. `plan_stress_test.incumbent: both (docs/conventions/planning.yaml)`. `reviewer` and `both` run this step; `auto` runs it when the reviewer rule (Gates) says so; `devils-advocate` skips it and goes to Step 3b. -**The verification level** decides what happens inside this step; `plan_stress_test` still decides whether it runs. Resolve it with `bash "${CLAUDE_PLUGIN_ROOT}/lib/read-verification.sh" --plugin planning --skill plan --file docs/conventions/planning.yaml` and report its line ([levels](https://github.com/melodic-software/claude-code-plugins/blob/main/docs/contracts/verification-level/README.md)). Under `light`, and `off`, which acts as `light` here, skip items 2 through 5, dispatch no reviewer and never stress-test the plan inline in its place: the reply that reports this step, and the Step 5 presentation, state `verification: skipped ()`, and the run goes to Step 3b, which no level governs. +**The verification level** decides what happens inside this step; the reviewer rule (Gates) decides whether it runs. Resolve it with `bash "${CLAUDE_PLUGIN_ROOT}/lib/read-verification.sh" --plugin planning --skill plan --file docs/conventions/planning.yaml` and report its line ([levels](https://github.com/melodic-software/claude-code-plugins/blob/main/docs/contracts/verification-level/README.md)). Under `light`, and `off`, which acts as `light` here, skip items 2 through 5, dispatch no reviewer and never stress-test the plan inline in its place: the reply that reports this step, and the Step 5 presentation, state `verification: skipped ()`, and the run goes to Step 3b, which no level governs. -Apply the reviewer rule under Planning Process before blast radius or presentation. The producing main thread MUST NOT self-attack the plan inline: the model that just wrote the plan rubber-stamps it. Where the plan is high-stakes and correlated blind spots are the risk, prefer a cross-vendor advisor **when one is enabled and set up** (e.g. the OpenAI Codex plugin, when its documented surface can take this artifact, invoked per its own docs), with the fresh-context plan-reviewer sub-agent as the stated fallback, never a route to a command that may not resolve (per `docs/plugin-philosophy.md` "Fresh-eyes checkpoints" in the marketplace repository). +**Whether the reviewer runs.** Apply the reviewer rule (Gates). Under `auto`, make the Step 3b blast-radius and trigger assessment first, before any dispatch; Step 3b then presents that assessment rather than redoing it. When the rule skips the reviewer, skip items 2 through 5, and the reply that reports this step, and the Step 5 presentation, state `Reviewer: skipped ()`, e.g. `Reviewer: skipped (blast radius LOW, no trigger)`, and a skip the verification level causes states `verification: skipped ()` as above; the run goes to Step 3b. The producing main thread MUST NOT self-attack the plan inline: the model that just wrote the plan rubber-stamps it. Where the plan is high-stakes and correlated blind spots are the risk, prefer a cross-vendor advisor **when one is enabled and set up** (e.g. the OpenAI Codex plugin, when its documented surface can take this artifact, invoked per its own docs), with the fresh-context plan-reviewer sub-agent as the stated fallback, never a route to a command that may not resolve (per `docs/plugin-philosophy.md` "Fresh-eyes checkpoints" in the marketplace repository). 1. Gather the plan draft + design artifacts (or `design-resolution.md`) + the Brief -2. **Surface the cost.** Tell the user, in one sentence, that a fresh-context plan review runs now whatever they answer: one bounded sub-agent (default `effort: medium`, capped turns). +2. **Surface the cost.** Tell the user, in one sentence, that a fresh-context plan review runs now and why (`Reviewer: ran ()`): one bounded sub-agent (default `effort: medium`, capped turns). 3. Dispatch the plugin's **`plan-reviewer`** agent (`agents/plan-reviewer.md`) with the prompt from [context/plan-reviewer.md](context/plan-reviewer.md). Do not substitute a generic read-only sub-agent, and pass no `effort` on the call: the agent definition carries the reviewer's bounded `effort` and `maxTurns` (the effort record in `agents/plan-reviewer.md`). A complete report ends with its `### Summary` counts, or is the literal `No plan gaps found.` for a clean pass; treat any other return as incomplete whether or not a partial marker is present (whether a `maxTurns` stop is marked depends on the client: the `maxTurns` record in `agents/plan-reviewer.md`): resume it (or re-dispatch with a narrowed brief); never treat a partial table as a clean pass. 4. **Verify reviewer findings** against the actual code/files before applying fixes. Sub-agent findings are synthesis, not ground truth 5. Fix every confirmed gap BEFORE proceeding; never present a plan with known gaps. A fix that displaces a user answer or adds an external effect follows "Plan changes after the Brief" below -Trivial single-file plans (3–5 bullets, no new types): the reviewer brief may be shortened to structural-integrity checks only; still dispatch fresh context, never inline self-critique. +When the reviewer runs on a trivial single-file plan (3–5 bullets, no new types), its brief may be shortened to structural-integrity checks only; still dispatch fresh context, never inline self-critique. ### Step 3b: Assess Blast Radius -Every plan gets a blast-radius check. Read [context/stress-test-triggers.md](context/stress-test-triggers.md) and assess whether this plan warrants a formal stress-test via `/planning:devils-advocate`. +Every plan gets a blast-radius check. Read [context/stress-test-triggers.md](context/stress-test-triggers.md) and assess whether this plan warrants a formal stress-test via `/planning:devils-advocate`; under `auto`, Step 3 already made this assessment, so present it. Present the assessment: ``` Blast radius: [LOW / MEDIUM / HIGH / CRITICAL] -Stress-test needed: [Yes — invoking /planning:devils-advocate / No — plan-reviewer sub-agent + research validation is sufficient] +Reviewer: [ran / skipped] () +Stress-test needed: [Yes — invoking /planning:devils-advocate / No — research validation is sufficient] Reason: <1-2 sentences> ``` @@ -262,7 +263,7 @@ Runs only when Step 3b sends the plan here; the deeper, formal version of Step 3 1. **Surface the cost.** Tell the user a formal `/planning:devils-advocate` run is starting in fresh context (another serial sub-agent, typically higher turn budget than Step 3) before dispatching. 2. **Dispatch `/planning:devils-advocate` via the Skill tool to a fresh-context sub-agent**. Under `plan_stress_test`'s `devils-advocate` or `both`, an `incumbent` draft runs the skill's `incumbent` mode, with `incumbent ` as its arguments; every other run is plan-review mode. Keep the brief short: findings table, not a narrative. Hand it the plan (plus the Brief, the interview ledger `//interview-checklist.md` when one exists, and any design artifacts), not your rationale for it. The producing main thread MUST NOT run the stress-test inline, for the same reason as Step 3. Verify its findings against the actual code/files before acting on them; sub-agent findings are synthesis, not ground truth - **Audit judgment (untested):** at MEDIUM or higher blast radius, Step 3 and Step 4 may dispatch against the same plan draft in one message when Step 4's criteria read off the draft rather than Step 3's findings. Document when used; default remains serial. + **Audit judgment (untested):** when both the reviewer and Step 4 run, Step 3 and Step 4 may dispatch against the same plan draft in one message when Step 4's criteria read off the draft rather than Step 3's findings. Document when used; default remains serial. 3. **Evaluate findings**. If `/planning:devils-advocate` produces CRITICAL or HIGH findings: - Run targeted research on the specific issues (invoke `/discovery:research` via the Skill tool if enabled, or the strongest research capability available) diff --git a/plugins/planning/skills/plan/context/stress-test-triggers.md b/plugins/planning/skills/plan/context/stress-test-triggers.md index d1df6f7601e..5442a69295d 100644 --- a/plugins/planning/skills/plan/context/stress-test-triggers.md +++ b/plugins/planning/skills/plan/context/stress-test-triggers.md @@ -1,6 +1,6 @@ # Stress-Test Trigger Criteria -When to invoke `/planning:devils-advocate` on a plan under `plan_stress_test`'s `auto`; the other values decide without these criteria (SKILL.md Step 3b). The goal is to catch plans that carry enough risk to warrant systematic adversarial review, without stress-testing every trivial change. +When to invoke `/planning:devils-advocate` on a plan under `plan_stress_test`'s `auto`, where a matching trigger, or a HIGH or CRITICAL blast radius, also runs the Step 3 `plan-reviewer`; the other values decide without these criteria (SKILL.md Step 3b). The goal is to catch plans that carry enough risk to warrant systematic adversarial review, without stress-testing every trivial change. ## Always stress-test when ANY of these match diff --git a/plugins/planning/skills/plan/evals/evals.json b/plugins/planning/skills/plan/evals/evals.json index 00771522b53..9c760eb320a 100644 --- a/plugins/planning/skills/plan/evals/evals.json +++ b/plugins/planning/skills/plan/evals/evals.json @@ -20,7 +20,7 @@ "id": 2, "name": "mandatory-fresh-context-stress-test", "prompt": "/planning:plan exploration and research are done, and I explicitly accept planning without a separate /design pass (no new types or modules). The task: add structured logging around every outbound HTTP call across the existing services — cross-cutting but mechanical. Formulate the plan and get it ready to present.", - "expected_output": "The design gate is satisfied by the explicit override, so planning proceeds. Before presenting the plan, the skill dispatches a fresh-context plan-reviewer sub-agent (Step 3) rather than self-critiquing the plan inline, and verifies the reviewer's findings against the actual code before applying fixes.", + "expected_output": "The design gate is satisfied by the explicit override, so planning proceeds. Cross-cutting logging matches a stress-test trigger, so before presenting the plan the skill dispatches a fresh-context plan-reviewer sub-agent (Step 3) rather than self-critiquing the plan inline, and verifies the reviewer's findings against the actual code before applying fixes.", "files": [], "expectations": [ "Output dispatches the plan-reviewer agent (or an equivalent fresh-context reviewer) rather than the authoring thread reviewing its own plan inline", diff --git a/plugins/planning/skills/plan/templates/checklist.md b/plugins/planning/skills/plan/templates/checklist.md index ee37f91bbe2..96c8d694a23 100644 --- a/plugins/planning/skills/plan/templates/checklist.md +++ b/plugins/planning/skills/plan/templates/checklist.md @@ -6,8 +6,8 @@ Copy into `//plan-checklist.md` (default `.work/`; the t - [ ] Step 1: Prerequisite check. Brief locked (PLAN.md Brief section exists OR equivalent crisp framing); exploration + research done or explicitly waived; design gate evaluated - [ ] Step 2: Formulate the plan. Phases with verifiable Sanity Checks per phase; estimate scope; identify parallelism -- [ ] Step 3: Plan stress-test. Resolve `plan_stress_test` for the draft's mode; unless it is `devils-advocate`, surface cost, then dispatch the plan-reviewer agent per context/plan-reviewer.md -- [ ] Step 3b: Assess blast radius (LOW / MEDIUM / HIGH / CRITICAL). Under `auto` this gates whether Step 4 runs +- [ ] Step 3: Plan stress-test. Resolve `plan_stress_test` for the draft's mode and apply the reviewer rule (SKILL.md Gates); when it runs the reviewer, surface cost, then dispatch the plan-reviewer agent per context/plan-reviewer.md; otherwise record `Reviewer: skipped ()` +- [ ] Step 3b: Assess blast radius (LOW / MEDIUM / HIGH / CRITICAL). Under `auto` it is assessed before Step 3's reviewer decision and gates both the reviewer and Step 4 - [ ] Step 4: Formal stress-test + research-iterate (CONDITIONAL on Step 3b). Invoke `/planning:devils-advocate` and targeted research on contested claims - [ ] Step 4.5: Execution-shape analysis (default ON for multi-phase plans). Emit scope-fencing tables + per-phase routing table - [ ] Step 4.6: Tag unilateral decisions. Flag any choice made without explicit user approval; interview below-bar decisions @@ -16,7 +16,7 @@ Copy into `//plan-checklist.md` (default `.work/`; the t ## Skip criteria -- Step 3 SKIPPED only under `plan_stress_test`'s `devils-advocate`, which runs Step 4 instead (stress-test before presenting) +- Step 3's reviewer SKIPPED under `plan_stress_test`'s `devils-advocate` (Step 4 runs instead), under verification `light` or `off`, and under `auto` when Step 3b's blast radius is LOW or MEDIUM, no trigger matches, verification is not `strict` and the user did not ask for a review; tick Step 3 with the skip and its reason - Step 4 SKIPPED under `plan_stress_test`'s `reviewer`, and under `auto` when Step 3b verdict = LOW blast radius and no trigger matches - Step 4.5 SKIPPED when the plan is single-phase (no parallelism axis, all-main-session default) - Step 4.6 SKIPPED when no unilateral decisions were made (user approved every choice during Q&A) From e5514044bb987e8457c2675100d46b3fd25a5763 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Wed, 11 Nov 2026 14:45:16 -0500 Subject: [PATCH 2/7] fix(planning): reassess blast radius after a reviewer fix changes the plan Under auto, Step 3 rates blast radius before dispatching the reviewer and Step 3b presents that rating. A reviewer fix can widen the plan, so Step 3b now reassesses when an item-5 fix changed the plan's files or scope. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VQb9Q4F46UumaUgXeUdpYZ --- plugins/planning/skills/plan/SKILL.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/plugins/planning/skills/plan/SKILL.md b/plugins/planning/skills/plan/SKILL.md index cb0178b5e14..c6cc81ea43c 100644 --- a/plugins/planning/skills/plan/SKILL.md +++ b/plugins/planning/skills/plan/SKILL.md @@ -231,7 +231,7 @@ For the removal step, invoke `/code-tidying:audit-dead-code` through the Skill t **The verification level** decides what happens inside this step; the reviewer rule (Gates) decides whether it runs. Resolve it with `bash "${CLAUDE_PLUGIN_ROOT}/lib/read-verification.sh" --plugin planning --skill plan --file docs/conventions/planning.yaml` and report its line ([levels](https://github.com/melodic-software/claude-code-plugins/blob/main/docs/contracts/verification-level/README.md)). Under `light`, and `off`, which acts as `light` here, skip items 2 through 5, dispatch no reviewer and never stress-test the plan inline in its place: the reply that reports this step, and the Step 5 presentation, state `verification: skipped ()`, and the run goes to Step 3b, which no level governs. -**Whether the reviewer runs.** Apply the reviewer rule (Gates). Under `auto`, make the Step 3b blast-radius and trigger assessment first, before any dispatch; Step 3b then presents that assessment rather than redoing it. When the rule skips the reviewer, skip items 2 through 5, and the reply that reports this step, and the Step 5 presentation, state `Reviewer: skipped ()`, e.g. `Reviewer: skipped (blast radius LOW, no trigger)`, and a skip the verification level causes states `verification: skipped ()` as above; the run goes to Step 3b. The producing main thread MUST NOT self-attack the plan inline: the model that just wrote the plan rubber-stamps it. Where the plan is high-stakes and correlated blind spots are the risk, prefer a cross-vendor advisor **when one is enabled and set up** (e.g. the OpenAI Codex plugin, when its documented surface can take this artifact, invoked per its own docs), with the fresh-context plan-reviewer sub-agent as the stated fallback, never a route to a command that may not resolve (per `docs/plugin-philosophy.md` "Fresh-eyes checkpoints" in the marketplace repository). +**Whether the reviewer runs.** Apply the reviewer rule (Gates). Under `auto`, make the Step 3b blast-radius and trigger assessment first, before any dispatch; Step 3b then presents that assessment rather than redoing it, unless a fix in item 5 changed the plan's files or scope. When the rule skips the reviewer, skip items 2 through 5, and the reply that reports this step, and the Step 5 presentation, state `Reviewer: skipped ()`, e.g. `Reviewer: skipped (blast radius LOW, no trigger)`, and a skip the verification level causes states `verification: skipped ()` as above; the run goes to Step 3b. The producing main thread MUST NOT self-attack the plan inline: the model that just wrote the plan rubber-stamps it. Where the plan is high-stakes and correlated blind spots are the risk, prefer a cross-vendor advisor **when one is enabled and set up** (e.g. the OpenAI Codex plugin, when its documented surface can take this artifact, invoked per its own docs), with the fresh-context plan-reviewer sub-agent as the stated fallback, never a route to a command that may not resolve (per `docs/plugin-philosophy.md` "Fresh-eyes checkpoints" in the marketplace repository). 1. Gather the plan draft + design artifacts (or `design-resolution.md`) + the Brief 2. **Surface the cost.** Tell the user, in one sentence, that a fresh-context plan review runs now and why (`Reviewer: ran ()`): one bounded sub-agent (default `effort: medium`, capped turns). @@ -243,7 +243,7 @@ When the reviewer runs on a trivial single-file plan (3–5 bullets, no new type ### Step 3b: Assess Blast Radius -Every plan gets a blast-radius check. Read [context/stress-test-triggers.md](context/stress-test-triggers.md) and assess whether this plan warrants a formal stress-test via `/planning:devils-advocate`; under `auto`, Step 3 already made this assessment, so present it. +Every plan gets a blast-radius check. Read [context/stress-test-triggers.md](context/stress-test-triggers.md) and assess whether this plan warrants a formal stress-test via `/planning:devils-advocate`; under `auto`, Step 3 already made this assessment, so present it, reassessing when a Step 3 fix changed the plan's files or scope. Present the assessment: From f707ab4c37df0c945f94fe22c7a6080ca7ffb738 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Wed, 11 Nov 2026 17:16:05 -0500 Subject: [PATCH 3/7] test(planning): add plan-reviewer-cut before/after eval cases Twelve plan requests (six rated LOW by design, six MEDIUM), each a seeded draft with two defects the plan-reviewer checks and the main thread's scripts do not, plus a must-not-flag guard and an arm-conformance check on reviewer dispatch. Frozen for the before/after measurement this PR is held on; fixtures/plan-reviewer-cut builds each arm and the degraded-reviewer control. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VQb9Q4F46UumaUgXeUdpYZ --- .../fixtures/plan-reviewer-cut/init-repo.sh | 23 +++ .../plan-reviewer-degraded-body.md.txt | 1 + .../plan-reviewer-degraded-prompt.md.txt | 17 ++ .../fixtures/plan-reviewer-cut/prepare-arm.sh | 59 ++++++ .../case.yaml | 4 + .../graders/behavior-is-a-claim.md | 9 + .../graders/blast-radius-low.md | 9 + .../graders/model-untouched.md | 10 ++ .../graders/reuses-slugify.md | 8 + .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/behavior-is-a-claim.json | 14 ++ .../samples/blast-radius-low.json | 22 +++ .../samples/model-untouched.json | 14 ++ .../samples/reuses-slugify.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 116 ++++++++++++ .../case.yaml | 4 + .../graders/ac3-built.md | 8 + .../graders/assumption-row.md | 8 + .../graders/blast-radius-low.md | 9 + .../graders/no-new-dependency.md | 10 ++ .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/ac3-built.json | 30 ++++ .../samples/assumption-row.json | 14 ++ .../samples/blast-radius-low.json | 22 +++ .../samples/no-new-dependency.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 105 +++++++++++ .../case.yaml | 4 + .../graders/appetite-fit-line.md | 8 + .../graders/assumption-row.md | 8 + .../graders/blast-radius-low.md | 9 + .../graders/no-dateutil.md | 10 ++ .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/appetite-fit-line.json | 14 ++ .../samples/assumption-row.json | 14 ++ .../samples/blast-radius-low.json | 22 +++ .../samples/no-dateutil.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 129 +++++++++++++ .../case.yaml | 4 + .../graders/blast-radius-low.md | 9 + .../graders/no-jitter.md | 10 ++ .../graders/real-ceiling-constant.md | 8 + .../graders/regression-test-named.md | 8 + .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/blast-radius-low.json | 22 +++ .../samples/no-jitter.json | 14 ++ .../samples/real-ceiling-constant.json | 14 ++ .../samples/regression-test-named.json | 22 +++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 97 ++++++++++ .../case.yaml | 4 + .../graders/blast-radius-low.md | 9 + .../graders/no-decimal-type.md | 9 + .../graders/out-of-scope-cut.md | 9 + .../graders/reviewer-not-dispatched.md | 9 + .../graders/sanity-check-command.md | 9 + .../prompt.md | 10 ++ .../samples/blast-radius-low.json | 22 +++ .../samples/no-decimal-type.json | 14 ++ .../samples/out-of-scope-cut.json | 22 +++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../samples/sanity-check-command.json | 14 ++ .../scaffold.sh | 88 +++++++++ .../case.yaml | 4 + .../graders/behavior-is-a-claim.md | 9 + .../graders/blast-radius-low.md | 9 + .../graders/missing-file-handled.md | 8 + .../graders/no-cache.md | 10 ++ .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/behavior-is-a-claim.json | 14 ++ .../samples/blast-radius-low.json | 22 +++ .../samples/missing-file-handled.json | 26 +++ .../samples/no-cache.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 105 +++++++++++ .../case.yaml | 4 + .../graders/ac3-sort-built.md | 8 + .../graders/appetite-fit-line.md | 8 + .../graders/blast-radius-medium.md | 9 + .../graders/no-new-dependency.md | 10 ++ .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/ac3-sort-built.json | 26 +++ .../samples/appetite-fit-line.json | 14 ++ .../samples/blast-radius-medium.json | 22 +++ .../samples/no-new-dependency.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 170 ++++++++++++++++++ .../case.yaml | 4 + .../graders/ac4-built.md | 8 + .../graders/assumption-row.md | 8 + .../graders/blast-radius-medium.md | 9 + .../graders/keeps-decimal-rounding.md | 10 ++ .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/ac4-built.json | 30 ++++ .../samples/assumption-row.json | 14 ++ .../samples/blast-radius-medium.json | 22 +++ .../samples/keeps-decimal-rounding.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 158 ++++++++++++++++ .../case.yaml | 4 + .../graders/assumption-row.md | 8 + .../graders/blast-radius-medium.md | 9 + .../graders/export-untouched.md | 10 ++ .../graders/regression-test-named.md | 8 + .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/assumption-row.json | 14 ++ .../samples/blast-radius-medium.json | 22 +++ .../samples/export-untouched.json | 14 ++ .../samples/regression-test-named.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 146 +++++++++++++++ .../case.yaml | 4 + .../graders/ac3-built.md | 8 + .../graders/behavior-is-a-claim.md | 9 + .../graders/blast-radius-medium.md | 9 + .../graders/keeps-outbox.md | 10 ++ .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/ac3-built.json | 30 ++++ .../samples/behavior-is-a-claim.json | 14 ++ .../samples/blast-radius-medium.json | 22 +++ .../samples/keeps-outbox.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 137 ++++++++++++++ .../case.yaml | 4 + .../graders/blast-radius-medium.md | 9 + .../graders/no-new-dependency.md | 10 ++ .../graders/out-of-scope-cut.md | 9 + .../graders/reuses-available.md | 8 + .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/blast-radius-medium.json | 22 +++ .../samples/no-new-dependency.json | 14 ++ .../samples/out-of-scope-cut.json | 22 +++ .../samples/reuses-available.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 125 +++++++++++++ .../case.yaml | 4 + .../graders/blast-radius-medium.md | 9 + .../graders/callers-untouched.md | 10 ++ .../graders/reuses-country-region.md | 8 + .../graders/reviewer-not-dispatched.md | 9 + .../graders/sanity-check-command.md | 9 + .../prompt.md | 10 ++ .../samples/blast-radius-medium.json | 22 +++ .../samples/callers-untouched.json | 14 ++ .../samples/reuses-country-region.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../samples/sanity-check-command.json | 14 ++ .../scaffold.sh | 146 +++++++++++++++ 160 files changed, 3836 insertions(+) create mode 100755 plugins/planning/evals/fixtures/plan-reviewer-cut/init-repo.sh create mode 100644 plugins/planning/evals/fixtures/plan-reviewer-cut/plan-reviewer-degraded-body.md.txt create mode 100644 plugins/planning/evals/fixtures/plan-reviewer-cut/plan-reviewer-degraded-prompt.md.txt create mode 100755 plugins/planning/evals/fixtures/plan-reviewer-cut/prepare-arm.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/behavior-is-a-claim.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/blast-radius-low.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/model-untouched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reuses-slugify.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/behavior-is-a-claim.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/blast-radius-low.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/model-untouched.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reuses-slugify.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/assumption-row.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/blast-radius-low.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/no-new-dependency.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/ac3-built.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/assumption-row.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/blast-radius-low.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/no-new-dependency.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/appetite-fit-line.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/assumption-row.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/blast-radius-low.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/no-dateutil.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/appetite-fit-line.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/assumption-row.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/blast-radius-low.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/no-dateutil.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-low-notes-since/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/blast-radius-low.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/no-jitter.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/real-ceiling-constant.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/blast-radius-low.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/no-jitter.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/real-ceiling-constant.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/blast-radius-low.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/no-decimal-type.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/sanity-check-command.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/blast-radius-low.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/no-decimal-type.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/out-of-scope-cut.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/reviewer-not-dispatched.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/sanity-check-command.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/behavior-is-a-claim.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/blast-radius-low.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/missing-file-handled.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/no-cache.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/behavior-is-a-claim.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/blast-radius-low.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/missing-file-handled.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/no-cache.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/ac3-sort-built.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/appetite-fit-line.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/blast-radius-medium.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/no-new-dependency.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/ac3-sort-built.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/appetite-fit-line.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/blast-radius-medium.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/no-new-dependency.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/ac4-built.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/assumption-row.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/blast-radius-medium.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/keeps-decimal-rounding.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/ac4-built.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/assumption-row.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/blast-radius-medium.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/keeps-decimal-rounding.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/assumption-row.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/blast-radius-medium.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/export-untouched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/assumption-row.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/blast-radius-medium.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/export-untouched.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/ac3-built.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/behavior-is-a-claim.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/blast-radius-medium.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/keeps-outbox.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/ac3-built.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/behavior-is-a-claim.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/blast-radius-medium.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/keeps-outbox.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/blast-radius-medium.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/no-new-dependency.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/out-of-scope-cut.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reuses-available.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/blast-radius-medium.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/no-new-dependency.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/out-of-scope-cut.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reuses-available.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/blast-radius-medium.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/callers-untouched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reuses-country-region.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/sanity-check-command.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/blast-radius-medium.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/callers-untouched.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reuses-country-region.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reviewer-not-dispatched.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/sanity-check-command.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/scaffold.sh diff --git a/plugins/planning/evals/fixtures/plan-reviewer-cut/init-repo.sh b/plugins/planning/evals/fixtures/plan-reviewer-cut/init-repo.sh new file mode 100755 index 00000000000..82f1752a667 --- /dev/null +++ b/plugins/planning/evals/fixtures/plan-reviewer-cut/init-repo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# Run by each plan-reviewer-cut-* scaffold after it writes the project files and the draft plan: +# commits the project on main, pushes it to a local bare origin, and moves to a feature branch. +# +# init-repo.sh +# +# Each draft's phase headings carry the placeholder [STATUS]; this script sets it to [TODO]. +set -euo pipefail + +branch="$1" +printf '.work/\n__pycache__/\n' >.gitignore +for plan in .work/*/PLAN.md; do + text="$(cat "$plan")" + printf '%s\n' "${text//'[STATUS]'/[TODO]}" >"$plan" +done +git init -q -b main +git add . +git -c user.name=dev -c user.email=dev@example.invalid commit -q -m "chore: import project" +mkdir -p .work/origin.git +git init -q --bare -b main .work/origin.git +git remote add origin "$PWD/.work/origin.git" +git push -q origin main +git checkout -q -b "$branch" diff --git a/plugins/planning/evals/fixtures/plan-reviewer-cut/plan-reviewer-degraded-body.md.txt b/plugins/planning/evals/fixtures/plan-reviewer-cut/plan-reviewer-degraded-body.md.txt new file mode 100644 index 00000000000..83426eda278 --- /dev/null +++ b/plugins/planning/evals/fixtures/plan-reviewer-cut/plan-reviewer-degraded-body.md.txt @@ -0,0 +1 @@ +You are a plan reviewer. Read the plan you are given and say what you think of it. diff --git a/plugins/planning/evals/fixtures/plan-reviewer-cut/plan-reviewer-degraded-prompt.md.txt b/plugins/planning/evals/fixtures/plan-reviewer-cut/plan-reviewer-degraded-prompt.md.txt new file mode 100644 index 00000000000..028fc7f4ff0 --- /dev/null +++ b/plugins/planning/evals/fixtures/plan-reviewer-cut/plan-reviewer-degraded-prompt.md.txt @@ -0,0 +1,17 @@ +# Plan Reviewer: Sub-Agent Dispatch + +Fresh-context plan review for `/planning:plan` Step 3. + +## Orchestrator inputs + +Attach the plan draft and the Brief. + +## Sub-agent prompt template + +```text +Here is a plan. Tell me what you think of it. +``` + +## After dispatch + +The main thread reads the reply, then proceeds to Step 3b (blast radius). diff --git a/plugins/planning/evals/fixtures/plan-reviewer-cut/prepare-arm.sh b/plugins/planning/evals/fixtures/plan-reviewer-cut/prepare-arm.sh new file mode 100755 index 00000000000..68d181d9488 --- /dev/null +++ b/plugins/planning/evals/fixtures/plan-reviewer-cut/prepare-arm.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# Writes one arm of the plan-reviewer-cut suite: a copy of the planning plugin at , outside any +# repository, holding this checkout's plan-reviewer-cut cases. `claude plugin eval` then runs it +# with --ablation none. +# +# prepare-arm.sh [--degrade-reviewer] +# +# is a commit of this repository (the before arm pins the PR's merge base). With +# --degrade-reviewer, the copy's plan-reviewer agent keeps its frontmatter and takes its body from +# plan-reviewer-degraded-body.md.txt, and the dispatch prompt (skills/plan/context/plan-reviewer.md) +# is replaced by plan-reviewer-degraded-prompt.md.txt: the suite's negative control. Both are +# replaced because the review axes and the report format live in the dispatch prompt. +# +# Exit: 0 written; 2 a usage error, or a dest that already exists. +set -euo pipefail + +usage() { + echo "usage: prepare-arm.sh [--degrade-reviewer]" >&2 + exit 2 +} + +[[ $# -ge 2 && $# -le 3 ]] || usage +ref="$1" +dest="$2" +degrade="${3:-}" +[[ -z "$degrade" || "$degrade" == "--degrade-reviewer" ]] || usage +if [[ -e "$dest" ]]; then + echo "prepare-arm.sh: $dest already exists" >&2 + exit 2 +fi + +parent="$(dirname "$dest")" +if [[ ! -d "$parent" ]] || git -C "$parent" rev-parse --git-dir >/dev/null 2>&1; then + echo "prepare-arm.sh: $dest must sit in an existing directory outside any git repository" >&2 + exit 2 +fi + +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +plugin="$(cd "$here/../../.." && pwd)" +repo="$(git -C "$plugin" rev-parse --show-toplevel)" +git -C "$repo" cat-file -e "$ref^{commit}" || usage + +mkdir -p "$dest" +git -C "$repo" archive "$ref" plugins/planning | tar -x -C "$dest" --strip-components=2 +rm -rf "${dest:?}/evals/results" +mkdir -p "$dest/evals/fixtures" +rm -rf "${dest:?}/evals/fixtures/plan-reviewer-cut" "$dest"/evals/plan-reviewer-cut-* +cp -R "$here" "$dest/evals/fixtures/plan-reviewer-cut" +cp -R "$plugin"/evals/plan-reviewer-cut-* "$dest/evals/" + +if [[ -n "$degrade" ]]; then + agent="$dest/agents/plan-reviewer.md" + frontmatter="$(awk 'NR == 1 || !done { print } NR > 1 && /^---$/ { done = 1 }' "$agent")" + printf '%s\n\n' "$frontmatter" >"$agent" + cat "$here/plan-reviewer-degraded-body.md.txt" >>"$agent" + cp "$here/plan-reviewer-degraded-prompt.md.txt" "$dest/skills/plan/context/plan-reviewer.md" +fi + +echo "$dest" diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/case.yaml b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/case.yaml new file mode 100644 index 00000000000..61e68ab1a79 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-low-blog-permalink +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/behavior-is-a-claim.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/behavior-is-a-claim.md new file mode 100644 index 00000000000..a3312120a58 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/behavior-is-a-claim.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: '^\s*[-*]\s*B\d+:\s*(?:Edit|Update|Modify|Change|Add)\b' +flags: mi +match: not_contains +target: + source: file + path: .work/post-permalink/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/blast-radius-low.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/blast-radius-low.md new file mode 100644 index 00000000000..4e4b39320a2 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/blast-radius-low.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b' +flags: i +target: + source: file + path: .work/post-permalink/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/model-untouched.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/model-untouched.md new file mode 100644 index 00000000000..2fe65e17da1 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/model-untouched.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: '^\|\s*`?blog/models\.py`?\s*\|\s*(?:modify|create|edit)' +flags: mi +match: not_contains +weight: 0.001 +target: + source: file + path: .work/post-permalink/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reuses-slugify.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reuses-slugify.md new file mode 100644 index 00000000000..21de46dc3d5 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reuses-slugify.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '\bslugify\b' +flags: i +target: + source: file + path: .work/post-permalink/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/prompt.md new file mode 100644 index 00000000000..398ae68fc21 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/prompt.md @@ -0,0 +1,10 @@ +--- +description: "LOW tier, no trigger: a draft adds post permalinks and writes its own slug helper although blog/text.py already has `slugify`, which AC2's accent folding needs (context/plan-reviewer.md: build what existing code already does). Also seeded: a Behavior item that names a file and a step instead of an observable claim (agents/plan-reviewer.md third check). hard: the duplicate is visible only by reading code the plan never cites. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] +runs: 3 +max_turns: 150 +timeout_seconds: 2700 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md reuses blog.text.slugify, rewrites the file-naming Behavior item as a claim, and does not change blog/models.py; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/post-permalink/PLAN.md`. Its Brief, its phases and the design early exit (`.work/post-permalink/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/behavior-is-a-claim.json b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/behavior-is-a-claim.json new file mode 100644 index 00000000000..e1e411ba17d --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/behavior-is-a-claim.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: The slug is lowercase ASCII words with accents folded (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/blast-radius-low.json new file mode 100644 index 00000000000..f1eb1c1bdd9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/blast-radius-low.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: LOW\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** LOW", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: MEDIUM\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/model-untouched.json b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/model-untouched.json new file mode 100644 index 00000000000..6f77d4bfdde --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/model-untouched.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n| `blog/models.py` | modify |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reuses-slugify.json b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reuses-slugify.json new file mode 100644 index 00000000000..ae7329ce5ef --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reuses-slugify.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: The slug is lowercase ASCII words with accents folded (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, import `slugify` from `blog.text`, which already folds accents.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{slugify(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..86031c3df49 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/post-permalink/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/post-permalink/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/scaffold.sh new file mode 100755 index 00000000000..e5ab7f6a532 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/scaffold.sh @@ -0,0 +1,116 @@ +#!/usr/bin/env bash +# A blog package with a shared slugify helper, and a draft plan for permalinks that writes its own. +set -euo pipefail + +mkdir -p .work/post-permalink/design blog tests +cat >README.md <<'EOF' +# blog + +Posts, listings and shared text helpers (blog/text.py). +EOF +: >blog/__init__.py +cat >blog/text.py <<'EOF' +"""Text helpers shared by the blog modules.""" +import re +import unicodedata + + +def slugify(title): + """Lowercase ASCII words joined by hyphens: 'Café Notes!' -> 'cafe-notes'.""" + ascii_title = unicodedata.normalize("NFKD", title).encode("ascii", "ignore").decode() + return "-".join(re.findall(r"[a-z0-9]+", ascii_title.lower())) +EOF +cat >blog/models.py <<'EOF' +from dataclasses import dataclass +from datetime import date + + +@dataclass +class Post: + title: str + body: str + published_on: date +EOF +cat >blog/posts.py <<'EOF' +"""Post listing helpers.""" + + +def recent(posts, limit=10): + return sorted(posts, key=lambda post: post.published_on, reverse=True)[:limit] +EOF +cat >tests/test_posts.py <<'EOF' +import unittest +from datetime import date + +from blog.models import Post +from blog.posts import recent + + +class RecentTest(unittest.TestCase): + def test_newest_first(self): + old = Post("a", "", date(2026, 1, 1)) + new = Post("b", "", date(2026, 2, 1)) + self.assertEqual(recent([old, new]), [new, old]) + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/post-permalink/PLAN.md <<'EOF' +# Post permalinks + +## Brief + +### Problem +Posts have no stable address: the site links to them by list position, so links break when a post is added. + +### Goal +Each post has a permalink built from its publication date and title. + +### Acceptance criteria +- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded. +- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`. +- AC3: Two posts with the same title in different months get different permalinks. + +### Out-of-scope +- Redirects from the old list-position links. +- Custom slugs chosen per post. + +## Design + +Early exit (tier C): One new function over an existing dataclass; no new types. + +## Plan + +### Behavior +- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1. +- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note | + +### Phase 1: Add permalink [STATUS] +**Satisfies:** AC1, AC2, AC3 +**Delivers:** B1, B2 + +- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`. +- Add `permalink(post)` returning `f"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/"`. +- In `tests/test_posts.py`, add one case per criterion. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `blog/posts.py` | modify | +| `tests/test_posts.py` | modify | +EOF +cat >.work/post-permalink/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: One new function over an existing dataclass; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/post-permalink diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/case.yaml b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/case.yaml new file mode 100644 index 00000000000..c786263e163 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-low-ledger-delimiter +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md new file mode 100644 index 00000000000..1bd8b345899 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '^###\s*Phase[\s\S]*?^\s*[-*]\s+[^\n]*(?:ValueError|unsupported delimiter|\brefus|\breject|\braise)' +flags: mi +target: + source: file + path: .work/export-delimiter/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/assumption-row.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/assumption-row.md new file mode 100644 index 00000000000..9ae87fcafc6 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/assumption-row.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '###\s*Load-bearing assumptions[^\n]*\n(?:(?!#)[^\n]*\n)*?\|[^\n]*(?:semicolon|separator|spreadsheet|import)' +flags: i +target: + source: file + path: .work/export-delimiter/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/blast-radius-low.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/blast-radius-low.md new file mode 100644 index 00000000000..05787b008a6 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/blast-radius-low.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b' +flags: i +target: + source: file + path: .work/export-delimiter/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/no-new-dependency.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/no-new-dependency.md new file mode 100644 index 00000000000..065d6e7ee20 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/no-new-dependency.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: '^\|\s*`?(?:requirements[\w.-]*\.txt|pyproject\.toml|setup\.cfg|Pipfile)`?\s*\|' +flags: mi +match: not_contains +weight: 0.001 +target: + source: file + path: .work/export-delimiter/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/prompt.md new file mode 100644 index 00000000000..4d81fed2834 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/prompt.md @@ -0,0 +1,10 @@ +--- +description: "LOW tier, no trigger: a two-file draft adds a delimiter option to a CSV export. Seeded: a Brief captured assumption with no Load-bearing row (agents/plan-reviewer.md second check), and AC3 claimed by Phase 1 that no work item builds (the Delivers check). hard: both seeds pass check-plan-outcome.sh, so only a reader who compares the Brief with the phases catches them. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] +runs: 3 +max_turns: 150 +timeout_seconds: 2700 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md carries a Load-bearing row for the semicolon-import assumption and a Phase 1 work item that refuses an unsupported delimiter with ValueError, and adds no dependency; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/export-delimiter/PLAN.md`. Its Brief, its phases and the design early exit (`.work/export-delimiter/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/ac3-built.json b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/ac3-built.json new file mode 100644 index 00000000000..141c0d9a301 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/ac3-built.json @@ -0,0 +1,30 @@ +{ + "pass": [ + { + "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- Raise `ValueError` when `delimiter` is not `,` or `;`, before `open`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Reject any delimiter outside `,` and `;` before opening the file.\n", + "why": "reject paraphrase" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Raise when `delimiter not in (\",\", \";\")`.\n", + "why": "raise paraphrase" + } + ], + "fail": [ + { + "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n", + "why": "the draft as seeded" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Pass `delimiter` to `csv.writer`.\n", + "why": "no validation step" + }, + { + "answer": "- B3: An unsupported delimiter is refused (AC3).\n### Phase 2: Next step [TODO]\n- Pass `delimiter` to `csv.writer`.\n", + "why": "only the Behavior item mentions it" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/assumption-row.json b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/assumption-row.json new file mode 100644 index 00000000000..8689092a92d --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/assumption-row.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Finance's spreadsheet imports semicolon-separated files with no hint line | Brief | Phase 1 Sanity Check | Moderate: flag and continue |\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/blast-radius-low.json new file mode 100644 index 00000000000..d15f46ec0d4 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/blast-radius-low.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: LOW\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** LOW", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: MEDIUM\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/no-new-dependency.json b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/no-new-dependency.json new file mode 100644 index 00000000000..abb4884c5d1 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/no-new-dependency.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n| `requirements.txt` | modify |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..492c1e86a57 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/export-delimiter/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/export-delimiter/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/scaffold.sh new file mode 100755 index 00000000000..2a2875cdcc5 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/scaffold.sh @@ -0,0 +1,105 @@ +#!/usr/bin/env bash +# A small ledger package whose CSV export takes no delimiter, and a draft plan to add one. +set -euo pipefail + +mkdir -p .work/export-delimiter/design ledger tests +cat >README.md <<'EOF' +# ledger + +Ledger rows and the CSV export finance opens in a spreadsheet. +EOF +: >ledger/__init__.py +cat >ledger/export.py <<'EOF' +"""Write ledger rows to CSV for the finance spreadsheet.""" +import csv + +HEADER = ["date", "account", "amount"] + + +def export_csv(rows, path): + with open(path, "w", newline="", encoding="utf-8") as handle: + writer = csv.writer(handle) + writer.writerow(HEADER) + for row in rows: + writer.writerow([row["date"], row["account"], row["amount"]]) +EOF +cat >tests/test_export.py <<'EOF' +import os +import tempfile +import unittest + +from ledger.export import export_csv + + +class ExportTest(unittest.TestCase): + def test_default_is_comma(self): + path = os.path.join(tempfile.mkdtemp(), "out.csv") + export_csv([{"date": "2026-03-01", "account": "4000", "amount": "12.50"}], path) + with open(path, encoding="utf-8") as handle: + self.assertEqual(handle.read().splitlines(), ["date,account,amount", "2026-03-01,4000,12.50"]) + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/export-delimiter/PLAN.md <<'EOF' +# Export delimiter + +## Brief + +### Problem +Finance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column. + +### Goal +`export_csv` takes an optional delimiter, so those staff get one value per column. + +### Acceptance criteria +- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today. +- AC2: `export_csv(rows, path, delimiter=";")` writes every row, header included, with semicolons between fields. +- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened. + +### Out-of-scope +- Changing the column set or order. +- Spreadsheet (.xlsx) output. + +### Captured assumptions +- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column. + +## Design + +Early exit (tier C): One optional parameter on one existing function; no new types. + +## Plan + +### Behavior +- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1. +- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1. +- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note | + +### Phase 1: Add a delimiter option [STATUS] +**Satisfies:** AC1, AC2, AC3 +**Delivers:** B1, B2, B3 + +- In `ledger/export.py`, add `delimiter=","` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`. +- In `tests/test_export.py`, add a case that writes two rows with `delimiter=";"` and reads the file back line by line. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `ledger/export.py` | modify | +| `tests/test_export.py` | modify | +EOF +cat >.work/export-delimiter/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: One optional parameter on one existing function; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/export-delimiter diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/case.yaml b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/case.yaml new file mode 100644 index 00000000000..018be92f519 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-low-notes-since +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/appetite-fit-line.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/appetite-fit-line.md new file mode 100644 index 00000000000..a778eb9cf9c --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/appetite-fit-line.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: 'appetite fit' +flags: i +target: + source: file + path: .work/notes-since/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/assumption-row.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/assumption-row.md new file mode 100644 index 00000000000..df70dbaece4 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/assumption-row.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '###\s*Load-bearing assumptions[^\n]*\n(?:(?!#)[^\n]*\n)*?\|[^\n]*(?:ISO|8601|YYYY|created|mobile|date format)' +flags: i +target: + source: file + path: .work/notes-since/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/blast-radius-low.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/blast-radius-low.md new file mode 100644 index 00000000000..320a897819e --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/blast-radius-low.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b' +flags: i +target: + source: file + path: .work/notes-since/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/no-dateutil.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/no-dateutil.md new file mode 100644 index 00000000000..1711b43e365 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/no-dateutil.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: 'dateutil' +flags: i +match: not_contains +weight: 0.001 +target: + source: file + path: .work/notes-since/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/prompt.md new file mode 100644 index 00000000000..9843dbf0488 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/prompt.md @@ -0,0 +1,10 @@ +--- +description: "LOW tier, no trigger: a draft adds a --since filter to a notes CLI. Seeded: the Brief records an appetite but the plan has no Appetite fit line (agents/plan-reviewer.md appetite check), and a Brief captured assumption about the date format has no Load-bearing row. hard: neither is a script criterion, and a run that resumes at the review step skips Step 2, where the main thread would write the line. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] +runs: 3 +max_turns: 150 +timeout_seconds: 2700 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md has an Appetite fit line and a Load-bearing row for the ISO date assumption, and adds no dateutil dependency; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/notes-since/PLAN.md`. Its Brief, its phases and the design early exit (`.work/notes-since/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/appetite-fit-line.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/appetite-fit-line.json new file mode 100644 index 00000000000..c69ee174d26 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/appetite-fit-line.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparseable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\nAppetite fit: one flag and one filter on one subcommand fits half a day.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparseable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/assumption-row.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/assumption-row.json new file mode 100644 index 00000000000..5141e3d6088 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/assumption-row.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparseable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Note `created` values are ISO 8601 dates | Brief | Phase 1 Sanity Check | Major: stop and replan |\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparseable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/blast-radius-low.json new file mode 100644 index 00000000000..236ee18df6a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/blast-radius-low.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: LOW\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** LOW", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparseable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: MEDIUM\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/no-dateutil.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/no-dateutil.json new file mode 100644 index 00000000000..4f6494b64c5 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/no-dateutil.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparseable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparseable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=dateutil.parser.parse`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..0760295f5a8 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/notes-since/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/notes-since/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/scaffold.sh new file mode 100755 index 00000000000..e50fa5c1cd5 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/scaffold.sh @@ -0,0 +1,129 @@ +#!/usr/bin/env bash +# A notes CLI whose list command has no date filter, and a draft plan to add --since. +set -euo pipefail + +mkdir -p .work/notes-since/design notes tests +cat >README.md <<'EOF' +# notes + +A small notes CLI: `python -m notes.cli list`. +EOF +: >notes/__init__.py +cat >notes/store.py <<'EOF' +"""Notes kept in notes.json as a list of {"created": "YYYY-MM-DD", "text": ...}.""" +import json +from pathlib import Path + +STORE = Path("notes.json") + + +def list_notes(path=STORE): + if not path.exists(): + return [] + return json.loads(path.read_text(encoding="utf-8")) +EOF +cat >notes/cli.py <<'EOF' +import argparse + +from notes import store + + +def main(argv=None): + parser = argparse.ArgumentParser(prog="notes") + sub = parser.add_subparsers(dest="command", required=True) + sub.add_parser("list") + args = parser.parse_args(argv) + if args.command == "list": + for note in store.list_notes(): + print(note["created"], note["text"]) + + +if __name__ == "__main__": + main() +EOF +cat >tests/test_cli.py <<'EOF' +import io +import unittest +from contextlib import redirect_stdout +from unittest import mock + +from notes import cli + + +class ListTest(unittest.TestCase): + def test_lists_all(self): + notes = [{"created": "2026-03-01", "text": "a"}] + out = io.StringIO() + with mock.patch("notes.store.list_notes", return_value=notes), redirect_stdout(out): + cli.main(["list"]) + self.assertEqual(out.getvalue(), "2026-03-01 a\n") + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/notes-since/PLAN.md <<'EOF' +# notes list --since + +## Brief + +### Problem +`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's. + +### Goal +`notes list --since DATE` shows only notes created on or after DATE. + +### Acceptance criteria +- AC1: `notes list` without `--since` prints what it prints today. +- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date. +- AC3: An unparseable `--since` value exits with status 2 and an error naming the flag. + +### Out-of-scope +- Relative dates such as "last week". +- An `--until` flag. + +### Work size +Appetite: half a day. + +### Captured assumptions +- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format. + +## Design + +Early exit (tier C): One flag on an existing subcommand; no new types. + +## Plan + +### Behavior +- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1. +- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1. +- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note | + +### Phase 1: Add --since [STATUS] +**Satisfies:** AC1, AC2, AC3 +**Delivers:** B1, B2, B3 + +- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`. +- Keep only notes whose `date.fromisoformat(note["created"])` is on or after the value. +- In `tests/test_cli.py`, add one case per criterion. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `notes/cli.py` | modify | +| `tests/test_cli.py` | modify | +EOF +cat >.work/notes-since/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: One flag on an existing subcommand; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/notes-since diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/case.yaml b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/case.yaml new file mode 100644 index 00000000000..c149fddff97 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-low-retry-backoff-cap +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/blast-radius-low.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/blast-radius-low.md new file mode 100644 index 00000000000..03849010240 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/blast-radius-low.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b' +flags: i +target: + source: file + path: .work/backoff-cap/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/no-jitter.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/no-jitter.md new file mode 100644 index 00000000000..ae714377cab --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/no-jitter.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: 'import random|random\.(?:uniform|random|randint)' +flags: i +match: not_contains +weight: 0.001 +target: + source: file + path: .work/backoff-cap/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/real-ceiling-constant.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/real-ceiling-constant.md new file mode 100644 index 00000000000..7c157b61ad0 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/real-ceiling-constant.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: 'MAX_BACKOFF_SECONDS' +flags: i +target: + source: file + path: .work/backoff-cap/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md new file mode 100644 index 00000000000..eeecbed8589 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '^## Plan[\s\S]*?(?:regression|\btest_\w+|\btests?\b)' +flags: mi +target: + source: file + path: .work/backoff-cap/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/prompt.md new file mode 100644 index 00000000000..fd54b683ee5 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/prompt.md @@ -0,0 +1,10 @@ +--- +description: "LOW tier, no trigger: a bug-fix draft caps retry backoff in one module. Seeded: it names a config constant `MAX_DELAY` that fetcher/config.py does not define (the ceiling is `MAX_BACKOFF_SECONDS`), and the bug fix names no regression test (context/plan-reviewer.md Design alignment). hard: the author believes its own constant name, and the mechanical gate checks neither. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] +runs: 3 +max_turns: 150 +timeout_seconds: 2700 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md uses MAX_BACKOFF_SECONDS for the ceiling and names a regression test for the uncapped delay, and adds no jitter; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/backoff-cap/PLAN.md`. Its Brief, its phases and the design early exit (`.work/backoff-cap/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/blast-radius-low.json new file mode 100644 index 00000000000..b111684f896 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/blast-radius-low.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: LOW\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** LOW", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: MEDIUM\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/no-jitter.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/no-jitter.json new file mode 100644 index 00000000000..ed62bec4877 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/no-jitter.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Add `random.uniform(0, 1)` seconds to each delay.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/real-ceiling-constant.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/real-ceiling-constant.json new file mode 100644 index 00000000000..9fa76210a60 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/real-ceiling-constant.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_BACKOFF_SECONDS)`; `MAX_BACKOFF_SECONDS` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json new file mode 100644 index 00000000000..34790649a43 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- In `tests/test_retry.py`, add a regression case: `backoff(8)` returns 30.0.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + }, + { + "answer": "## Plan\n- Add a unit test asserting `backoff(8) == 30.0`.\n", + "why": "test without the word regression" + } + ], + "fail": [ + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", + "why": "the draft as seeded" + }, + { + "answer": "## Plan\n- Bound the delay with `min`.\n", + "why": "no test" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..c2529d5c8b5 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/backoff-cap/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/backoff-cap/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/scaffold.sh new file mode 100755 index 00000000000..cebd79632a0 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/scaffold.sh @@ -0,0 +1,97 @@ +#!/usr/bin/env bash +# A fetcher package whose retry backoff grows past the configured ceiling, and a draft bug-fix plan. +set -euo pipefail + +mkdir -p .work/backoff-cap/design fetcher tests +cat >README.md <<'EOF' +# fetcher + +HTTP fetcher with retries. Tunables live in fetcher/config.py. +EOF +: >fetcher/__init__.py +cat >fetcher/config.py <<'EOF' +"""Tunables for the HTTP fetcher.""" +BASE_DELAY_SECONDS = 0.5 +MAX_BACKOFF_SECONDS = 30.0 +MAX_ATTEMPTS = 8 +EOF +cat >fetcher/retry.py <<'EOF' +"""Retry delays for failed fetches.""" +from fetcher import config + + +def backoff(attempt): + """Seconds to wait before retry number `attempt` (1-based).""" + return config.BASE_DELAY_SECONDS * 2 ** (attempt - 1) +EOF +cat >tests/test_retry.py <<'EOF' +import unittest + +from fetcher.retry import backoff + + +class BackoffTest(unittest.TestCase): + def test_doubles(self): + self.assertEqual([backoff(n) for n in (1, 2, 3)], [0.5, 1.0, 2.0]) + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/backoff-cap/PLAN.md <<'EOF' +# Cap retry backoff + +## Brief + +### Problem +With eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes. + +### Goal +No retry waits longer than the configured ceiling. + +### Acceptance criteria +- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling. +- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more. +- AC3: `backoff(0)` and negative attempts raise `ValueError`. + +### Out-of-scope +- Adding jitter. +- Changing the number of attempts. + +## Design + +Early exit (tier C): A bound inside one existing function; no new types. + +## Plan + +### Behavior +- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1. +- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1. +- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note | + +### Phase 1: Bound the delay [STATUS] +**Satisfies:** AC1, AC2, AC3 +**Delivers:** B1, B2, B3 + +- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling. +- Raise `ValueError` at the top of `backoff` when `attempt` is below 1. + +**Sanity Check:** `python -c "from fetcher.retry import backoff; assert backoff(8) <= 30"` exits 0. + +### Files affected +| File | Action | +|---|---| +| `fetcher/retry.py` | modify | +EOF +cat >.work/backoff-cap/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: A bound inside one existing function; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" fix/backoff-cap diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/case.yaml b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/case.yaml new file mode 100644 index 00000000000..1143281a83a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-low-temperature-decimals +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/blast-radius-low.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/blast-radius-low.md new file mode 100644 index 00000000000..de151c94dad --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/blast-radius-low.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b' +flags: i +target: + source: file + path: .work/temp-decimals/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/no-decimal-type.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/no-decimal-type.md new file mode 100644 index 00000000000..b726461ee3a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/no-decimal-type.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: '\bDecimal\b' +match: not_contains +weight: 0.001 +target: + source: file + path: .work/temp-decimals/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md new file mode 100644 index 00000000000..168467bfd74 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: '^\s*[-*]\s+(?![^\n]*\b(?:out[ -]of[ -]scope|not|no|never|without|leave|leaves|unchanged|drop\w*|cut|remov\w*|exclud\w*)\b)[^\n]*(?:fahrenheit|unit\s*=|°F)' +flags: mi +match: not_contains +target: + source: file + path: .work/temp-decimals/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/sanity-check-command.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/sanity-check-command.md new file mode 100644 index 00000000000..752cff52031 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/sanity-check-command.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'Sanity Check[^\n]*\b(?:looks? (?:right|correct|good|fine)|as expected|works (?:correctly|properly))' +flags: i +match: not_contains +target: + source: file + path: .work/temp-decimals/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/prompt.md new file mode 100644 index 00000000000..f8295b5dd5b --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/prompt.md @@ -0,0 +1,10 @@ +--- +description: "LOW tier, no trigger: a draft adds a decimals option to a temperature formatter. Seeded: a work item adds a unit switch the Brief's Out-of-scope excludes (context/plan-reviewer.md Plan mechanics), and the phase's Sanity Check is a vague `looks right` the script counts as present. routine for the out-of-scope seed, which Step 4.7's reading check also covers; hard for the vague Sanity Check. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] +runs: 3 +max_turns: 150 +timeout_seconds: 2700 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md drops the unit switch, replaces the vague Sanity Check with a command, and does not switch to Decimal; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/temp-decimals/PLAN.md`. Its Brief, its phases and the design early exit (`.work/temp-decimals/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/blast-radius-low.json new file mode 100644 index 00000000000..6e74cd6ff9a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/blast-radius-low.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: LOW\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** LOW", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, return `f\"{celsius:.{digits}f}°C\"`, and raise `ValueError` when it is below 0.\n- Add `unit=\"C\"` and convert to Fahrenheit when `unit=\"F\"`, so the panel can switch later.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** the formatted output looks right on the dashboard.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: MEDIUM\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/no-decimal-type.json b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/no-decimal-type.json new file mode 100644 index 00000000000..1f05a1b730a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/no-decimal-type.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, return `f\"{celsius:.{digits}f}°C\"`, and raise `ValueError` when it is below 0.\n- Add `unit=\"C\"` and convert to Fahrenheit when `unit=\"F\"`, so the panel can switch later.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** the formatted output looks right on the dashboard.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, quantize a `Decimal` of the value, and raise `ValueError` when it is below 0.\n- Add `unit=\"C\"` and convert to Fahrenheit when `unit=\"F\"`, so the panel can switch later.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** the formatted output looks right on the dashboard.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/out-of-scope-cut.json b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/out-of-scope-cut.json new file mode 100644 index 00000000000..a80fbe72be8 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/out-of-scope-cut.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, return `f\"{celsius:.{digits}f}°C\"`, and raise `ValueError` when it is below 0.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** the formatted output looks right on the dashboard.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + }, + { + "answer": "- Fahrenheit output dropped: out of scope.\n", + "why": "names it while cutting it" + } + ], + "fail": [ + { + "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, return `f\"{celsius:.{digits}f}°C\"`, and raise `ValueError` when it is below 0.\n- Add `unit=\"C\"` and convert to Fahrenheit when `unit=\"F\"`, so the panel can switch later.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** the formatted output looks right on the dashboard.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", + "why": "the draft as seeded" + }, + { + "answer": "- Support Fahrenheit through a `unit=` argument.\n", + "why": "paraphrased scope creep" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..614f6243110 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/temp-decimals/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/temp-decimals/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/sanity-check-command.json b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/sanity-check-command.json new file mode 100644 index 00000000000..2b927d5093a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/sanity-check-command.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, return `f\"{celsius:.{digits}f}°C\"`, and raise `ValueError` when it is below 0.\n- Add `unit=\"C\"` and convert to Fahrenheit when `unit=\"F\"`, so the panel can switch later.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, return `f\"{celsius:.{digits}f}°C\"`, and raise `ValueError` when it is below 0.\n- Add `unit=\"C\"` and convert to Fahrenheit when `unit=\"F\"`, so the panel can switch later.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** the formatted output looks right on the dashboard.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/scaffold.sh new file mode 100755 index 00000000000..a279cb20c6a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/scaffold.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash +# A weather formatter that rounds to whole degrees, and a draft plan to show decimals. +set -euo pipefail + +mkdir -p .work/temp-decimals/design tests weather +cat >README.md <<'EOF' +# weather + +Formatting helpers for the greenhouse dashboard. +EOF +: >weather/__init__.py +cat >weather/format.py <<'EOF' +def format_temp(celsius): + return f"{round(celsius)}°C" +EOF +cat >tests/test_format.py <<'EOF' +import unittest + +from weather.format import format_temp + + +class FormatTest(unittest.TestCase): + def test_whole_degrees(self): + self.assertEqual(format_temp(21.6), "22°C") + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/temp-decimals/PLAN.md <<'EOF' +# Temperature decimals + +## Brief + +### Problem +The greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift. + +### Goal +`format_temp` can show a fixed number of decimals. + +### Acceptance criteria +- AC1: `format_temp(c)` with no `digits` returns what it returns today. +- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`. +- AC3: A negative `digits` raises `ValueError`. + +### Out-of-scope +- Units other than Celsius. +- Localized decimal separators. + +## Design + +Early exit (tier C): One optional parameter on one existing function; no new types. + +## Plan + +### Behavior +- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1. +- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1. +- B3: A negative `digits` is refused (AC3). Delivered by Phase 1. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note | + +### Phase 1: Add digits [STATUS] +**Satisfies:** AC1, AC2, AC3 +**Delivers:** B1, B2, B3 + +- In `weather/format.py`, add `digits=None`; when set, return `f"{celsius:.{digits}f}°C"`, and raise `ValueError` when it is below 0. +- Add `unit="C"` and convert to Fahrenheit when `unit="F"`, so the panel can switch later. +- In `tests/test_format.py`, add one case per criterion. + +**Sanity Check:** the formatted output looks right on the dashboard. + +### Files affected +| File | Action | +|---|---| +| `weather/format.py` | modify | +| `tests/test_format.py` | modify | +EOF +cat >.work/temp-decimals/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: One optional parameter on one existing function; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/temp-decimals diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/case.yaml b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/case.yaml new file mode 100644 index 00000000000..6fb6a7a9488 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-low-till-price-lookup +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/behavior-is-a-claim.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/behavior-is-a-claim.md new file mode 100644 index 00000000000..da60f2a2ed1 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/behavior-is-a-claim.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: '^\s*[-*]\s*B\d+:\s*(?:Edit|Update|Modify|Change|Add)\b' +flags: mi +match: not_contains +target: + source: file + path: .work/price-lookup/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/blast-radius-low.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/blast-radius-low.md new file mode 100644 index 00000000000..c26a8c96bdf --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/blast-radius-low.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b' +flags: i +target: + source: file + path: .work/price-lookup/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/missing-file-handled.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/missing-file-handled.md new file mode 100644 index 00000000000..65c4f088af4 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/missing-file-handled.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: 'FileNotFoundError|OSError|\.exists\(\)|\.is_file\(\)' +flags: i +target: + source: file + path: .work/price-lookup/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/no-cache.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/no-cache.md new file mode 100644 index 00000000000..e2b0f063e2c --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/no-cache.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: 'lru_cache|functools\.cache|@cache\b' +flags: i +match: not_contains +weight: 0.001 +target: + source: file + path: .work/price-lookup/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/prompt.md new file mode 100644 index 00000000000..4edabc3a3e6 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/prompt.md @@ -0,0 +1,10 @@ +--- +description: "LOW tier, no trigger: a draft adds price_for and claims load_prices already returns an empty dict for a missing file; catalog/prices.py opens the file unguarded, so it raises FileNotFoundError (context/plan-reviewer.md: findings backed by a code path). Also seeded: a Behavior item that is a step of work, not a claim. hard: the false claim reads as settled fact and only a read of the code disproves it. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] +runs: 3 +max_turns: 150 +timeout_seconds: 2700 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md handles a missing prices file (FileNotFoundError or an existence check), rewrites the step-of-work Behavior item as a claim, and adds no cache; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/price-lookup/PLAN.md`. Its Brief, its phases and the design early exit (`.work/price-lookup/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/behavior-is-a-claim.json b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/behavior-is-a-claim.json new file mode 100644 index 00000000000..9a349712051 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/behavior-is-a-claim.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: An edit to the price file shows in the next lookup (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own.\n- Read the file on every call, so a staff edit shows at once.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own.\n- Read the file on every call, so a staff edit shows at once.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/blast-radius-low.json new file mode 100644 index 00000000000..00e7f5d22b2 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/blast-radius-low.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: LOW\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** LOW", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own.\n- Read the file on every call, so a staff edit shows at once.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: MEDIUM\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/missing-file-handled.json b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/missing-file-handled.json new file mode 100644 index 00000000000..5c37a9f6c2f --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/missing-file-handled.json @@ -0,0 +1,26 @@ +{ + "pass": [ + { + "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; catch `FileNotFoundError` from `load_prices` and return `None` (AC3).\n- Read the file on every call, so a staff edit shows at once.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + }, + { + "answer": "- Wrap `load_prices` in `try`/`except OSError` and return `None`.\n", + "why": "OSError paraphrase" + }, + { + "answer": "- Return `None` when `path.exists()` is false.\n", + "why": "existence check" + } + ], + "fail": [ + { + "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own.\n- Read the file on every call, so a staff edit shows at once.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", + "why": "the draft as seeded" + }, + { + "answer": "- `load_prices` already returns {} for a missing file.\n", + "why": "repeats the false claim" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/no-cache.json b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/no-cache.json new file mode 100644 index 00000000000..d379f2f5648 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/no-cache.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own.\n- Read the file on every call, so a staff edit shows at once.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own.\n- Wrap `load_prices` in `functools.lru_cache` so repeat lookups skip the disk.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..57a464b8eb9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/price-lookup/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/price-lookup/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/scaffold.sh new file mode 100755 index 00000000000..4122aa0bbdd --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/scaffold.sh @@ -0,0 +1,105 @@ +#!/usr/bin/env bash +# A catalog package whose price loader raises on a missing file, and a draft plan that says it does not. +set -euo pipefail + +mkdir -p .work/price-lookup/design catalog data tests +cat >README.md <<'EOF' +# catalog + +Price lookups for the till. Staff edit data/prices.json while the shop is open. +EOF +: >catalog/__init__.py +cat >catalog/prices.py <<'EOF' +"""Price lookups from data/prices.json, which staff edit while the shop is open.""" +import json +from pathlib import Path + +PRICES_PATH = Path("data/prices.json") + + +def load_prices(path=PRICES_PATH): + with open(path, encoding="utf-8") as handle: + return json.load(handle) +EOF +cat >data/prices.json <<'EOF' +{"A100": 3.5, "B200": 12.0} +EOF +cat >tests/test_prices.py <<'EOF' +import json +import os +import tempfile +import unittest + +from catalog.prices import load_prices + + +class LoadTest(unittest.TestCase): + def test_reads_json(self): + path = os.path.join(tempfile.mkdtemp(), "prices.json") + with open(path, "w", encoding="utf-8") as handle: + json.dump({"A100": 3.5}, handle) + self.assertEqual(load_prices(path), {"A100": 3.5}) + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/price-lookup/PLAN.md <<'EOF' +# Price lookup without a crash + +## Brief + +### Problem +The till crashes with a traceback when a product has no price, and staff restart it mid-sale. + +### Goal +`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price. + +### Acceptance criteria +- AC1: `price_for("A100")` returns the value from `data/prices.json`. +- AC2: An unknown SKU returns `None`. +- AC3: A missing price file returns `None`. +- AC4: An edit to `data/prices.json` shows in the next call without a restart. + +### Out-of-scope +- Editing prices from the till. + +## Design + +Early exit (tier C): One new function beside an existing loader; no new types. + +## Plan + +### Behavior +- B1: A known SKU returns its price (AC1). Delivered by Phase 1. +- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1. +- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note | + +### Phase 1: Add price_for [STATUS] +**Satisfies:** AC1, AC2, AC3, AC4 +**Delivers:** B1, B2, B3 + +- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own. +- Read the file on every call, so a staff edit shows at once. +- In `tests/test_prices.py`, add one case per criterion. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `catalog/prices.py` | modify | +| `tests/test_prices.py` | modify | +EOF +cat >.work/price-lookup/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: One new function beside an existing loader; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" fix/price-lookup diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/case.yaml b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/case.yaml new file mode 100644 index 00000000000..a32d574d24d --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-medium-catalogue-author-search +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/ac3-sort-built.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/ac3-sort-built.md new file mode 100644 index 00000000000..a1c4ed18e80 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/ac3-sort-built.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '^###\s*Phase[\s\S]*?^\s*[-*]\s+(?![^\n]*already sorts)[^\n]*\bsort' +flags: mi +target: + source: file + path: .work/author-search/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/appetite-fit-line.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/appetite-fit-line.md new file mode 100644 index 00000000000..4b8bbb0de9b --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/appetite-fit-line.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: 'appetite fit' +flags: i +target: + source: file + path: .work/author-search/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/blast-radius-medium.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/blast-radius-medium.md new file mode 100644 index 00000000000..55e0fe300ed --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/blast-radius-medium.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bMEDIUM\b' +flags: i +target: + source: file + path: .work/author-search/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/no-new-dependency.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/no-new-dependency.md new file mode 100644 index 00000000000..694d3bbc60c --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/no-new-dependency.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: '^\|\s*`?(?:requirements[\w.-]*\.txt|pyproject\.toml|setup\.cfg|Pipfile)`?\s*\||rapidfuzz|whoosh' +flags: mi +match: not_contains +weight: 0.001 +target: + source: file + path: .work/author-search/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/prompt.md new file mode 100644 index 00000000000..5307e940181 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/prompt.md @@ -0,0 +1,10 @@ +--- +description: "MEDIUM tier, no trigger: a five-file draft adds author search. Seeded: the Brief records an appetite and the plan has no Appetite fit line, and the plan says search_title already sorts by title, so AC3 needs no work, while library/search.py returns matches in catalogue order. hard: the sorting claim is false only on reading the code, and no script criterion covers either seed. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] +runs: 3 +max_turns: 200 +timeout_seconds: 3600 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md has an Appetite fit line and a work item that sorts author results by title, and adds no search dependency; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/author-search/PLAN.md`. Its Brief, its phases and the design early exit (`.work/author-search/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/ac3-sort-built.json b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/ac3-sort-built.json new file mode 100644 index 00000000000..dbcc4894fa7 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/ac3-sort-built.json @@ -0,0 +1,26 @@ +{ + "pass": [ + { + "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query, sorted by title (AC3); `search_title` returns catalogue order, so it does not sort for us.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Return `sorted(matches, key=lambda book: book[\"title\"])` from `search_author`.\n", + "why": "code paraphrase" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Sort author results by title before printing.\n", + "why": "prose paraphrase" + } + ], + "fail": [ + { + "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the draft as seeded" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- `search_title` already sorts by title, so nothing to do.\n", + "why": "repeats the false claim" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/appetite-fit-line.json b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/appetite-fit-line.json new file mode 100644 index 00000000000..ae7b1fd0d14 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/appetite-fit-line.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\nAppetite fit: one index map, one search function and one flag fit a day.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/blast-radius-medium.json b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/blast-radius-medium.json new file mode 100644 index 00000000000..0cbc3f56192 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/blast-radius-medium.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: MEDIUM\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** MEDIUM", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: HIGH\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/no-new-dependency.json b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/no-new-dependency.json new file mode 100644 index 00000000000..6357d9aea67 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/no-new-dependency.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n| `requirements.txt` | modify |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..bec7fc2e79a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/author-search/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/author-search/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/scaffold.sh new file mode 100755 index 00000000000..70d99c2ec2f --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/scaffold.sh @@ -0,0 +1,170 @@ +#!/usr/bin/env bash +# A library kiosk package that searches titles only, and a draft plan to add author search. +set -euo pipefail + +mkdir -p .work/author-search/design data library tests +cat >README.md <<'EOF' +# library + +The catalogue kiosk: `python -m library.cli search WORD`. +EOF +: >library/__init__.py +cat >data/books.json <<'EOF' +[{"title": "Beloved", "author": "Toni Morrison"}, {"title": "Jazz", "author": "Toni Morrison"}, {"title": "Emma", "author": "Jane Austen"}] +EOF +cat >library/catalogue.py <<'EOF' +import json +from pathlib import Path + + +def load_books(path=Path("data/books.json")): + return json.loads(path.read_text(encoding="utf-8")) +EOF +cat >library/index.py <<'EOF' +"""In-memory index of the catalogue.""" +from library.catalogue import load_books + + +def build(): + books = load_books() + by_word = {} + for book in books: + for word in book["title"].split(): + by_word.setdefault(word.lower(), []).append(book) + return {"by_word": by_word, "books": books} +EOF +cat >library/search.py <<'EOF' +def search_title(index, query): + """Books with a title word equal to the query, in catalogue order.""" + return list(index["by_word"].get(query.lower(), [])) +EOF +cat >library/cli.py <<'EOF' +import argparse + +from library import index, search + + +def main(argv=None): + parser = argparse.ArgumentParser(prog="library") + sub = parser.add_subparsers(dest="command", required=True) + find = sub.add_parser("search") + find.add_argument("query") + args = parser.parse_args(argv) + for book in search.search_title(index.build(), args.query): + print(book["title"], "-", book["author"]) + + +if __name__ == "__main__": + main() +EOF +cat >tests/test_index.py <<'EOF' +import unittest +from unittest import mock + +from library import index + + +class IndexTest(unittest.TestCase): + def test_title_words(self): + with mock.patch("library.index.load_books", return_value=[{"title": "Big Sky", "author": "A"}]): + self.assertIn("sky", index.build()["by_word"]) + + +if __name__ == "__main__": + unittest.main() +EOF +cat >tests/test_cli.py <<'EOF' +import io +import unittest +from contextlib import redirect_stdout +from unittest import mock + +from library import cli + + +class CliTest(unittest.TestCase): + def test_title_search(self): + books = [{"title": "Emma", "author": "Jane Austen"}] + out = io.StringIO() + with mock.patch("library.index.load_books", return_value=books), redirect_stdout(out): + cli.main(["search", "emma"]) + self.assertEqual(out.getvalue(), "Emma - Jane Austen\n") + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/author-search/PLAN.md <<'EOF' +# Search by author + +## Brief + +### Problem +Patrons ask at the desk for books by an author because the kiosk searches titles only. + +### Goal +The kiosk finds books by author name. + +### Acceptance criteria +- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case. +- AC2: Title search is unchanged. +- AC3: Author results are listed in title order. + +### Out-of-scope +- Fuzzy matching. +- Search by ISBN. + +### Work size +Appetite: one day. + +## Design + +Early exit (tier C): New functions beside existing ones; no new types. + +## Plan + +### Behavior +- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2. +- B2: Title search answers as before (AC2). Delivered by Phase 2. +- B3: Author results come in title order (AC3). Delivered by Phase 2. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue | + +### Phase 1: Index authors [STATUS] +**Satisfies:** AC1 +**Delivers:** B1 + +- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`. +- In `tests/test_index.py`, add a case for two books by one author. + +**Sanity Check:** `python -m unittest tests.test_index` exits 0. + +### Phase 2: Search and command [STATUS] +**Satisfies:** AC1, AC2, AC3 +**Delivers:** B1, B2, B3 + +- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way. +- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given. +- In `tests/test_cli.py`, add a case for `--author` and keep the title case. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `library/index.py` | modify | +| `library/search.py` | modify | +| `library/cli.py` | modify | +| `tests/test_index.py` | modify | +| `tests/test_cli.py` | modify | +EOF +cat >.work/author-search/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: New functions beside existing ones; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/author-search diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/case.yaml b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/case.yaml new file mode 100644 index 00000000000..1c90e506620 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-medium-invoice-discounts +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/ac4-built.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/ac4-built.md new file mode 100644 index 00000000000..fc00036f9f5 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/ac4-built.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '^###\s*Phase[\s\S]*?^\s*[-*]\s+[^\n]*(?:\brender|-15%|percentage)' +flags: mi +target: + source: file + path: .work/invoice-discounts/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/assumption-row.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/assumption-row.md new file mode 100644 index 00000000000..e7295a0e42a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/assumption-row.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '###\s*Load-bearing assumptions[^\n]*\n(?:(?!#)[^\n]*\n)*?\|[^\n]*(?:discount_pct|\b100\b|credit|between 0)' +flags: i +target: + source: file + path: .work/invoice-discounts/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/blast-radius-medium.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/blast-radius-medium.md new file mode 100644 index 00000000000..d35392a090f --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/blast-radius-medium.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bMEDIUM\b' +flags: i +target: + source: file + path: .work/invoice-discounts/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/keeps-decimal-rounding.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/keeps-decimal-rounding.md new file mode 100644 index 00000000000..aeb81e8c581 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/keeps-decimal-rounding.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: '\bfloat\(|round\([^)\n]*,\s*2\s*\)' +flags: i +match: not_contains +weight: 0.001 +target: + source: file + path: .work/invoice-discounts/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/prompt.md new file mode 100644 index 00000000000..6417de96f46 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/prompt.md @@ -0,0 +1,10 @@ +--- +description: "MEDIUM tier, no trigger: a three-phase-file draft applies a stored line discount to invoice totals. Seeded: Phase 2 claims AC4 and B3 (the invoice text shows the percentage) but no work item touches billing/render.py (agents/plan-reviewer.md Delivers check), and a Brief captured assumption has no Load-bearing row. hard: the script sees AC4 on a Satisfies line and passes it. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] +runs: 3 +max_turns: 200 +timeout_seconds: 3600 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md has a work item that changes the rendered invoice to show the percentage and a Load-bearing row for the 0-100 discount assumption, and keeps Decimal rounding through money.round_cents; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/invoice-discounts/PLAN.md`. Its Brief, its phases and the design early exit (`.work/invoice-discounts/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/ac4-built.json b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/ac4-built.json new file mode 100644 index 00000000000..bb5d29700c4 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/ac4-built.json @@ -0,0 +1,30 @@ +{ + "pass": [ + { + "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In `billing/render.py`, print `-%` beside each discounted line's total.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", + "why": "the plan with the seeded defect fixed" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Show `-15%` next to each discounted total in the text invoice.\n", + "why": "output paraphrase" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Update `render_text` to append the discount percentage.\n", + "why": "function name" + } + ], + "fail": [ + { + "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", + "why": "the draft as seeded" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Keep summing line totals.\n", + "why": "no render step" + }, + { + "answer": "- B3: The invoice text shows each line's discount percentage (AC4).\n### Phase 2: Next step [TODO]\n- Keep summing line totals.\n", + "why": "only the Behavior item mentions it" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/assumption-row.json b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/assumption-row.json new file mode 100644 index 00000000000..abdb285b281 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/assumption-row.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `discount_pct` is between 0 and 100 | Brief | Phase 1 Sanity Check | Major: stop and replan |\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/blast-radius-medium.json b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/blast-radius-medium.json new file mode 100644 index 00000000000..4d6e19c20a0 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/blast-radius-medium.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: MEDIUM\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** MEDIUM", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: HIGH\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/keeps-decimal-rounding.json b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/keeps-decimal-rounding.json new file mode 100644 index 00000000000..347c19e3386 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/keeps-decimal-rounding.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and round it with `round(float(total), 2)`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..958f82f90e5 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/invoice-discounts/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/invoice-discounts/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/scaffold.sh new file mode 100755 index 00000000000..3ecaaa1f6bc --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/scaffold.sh @@ -0,0 +1,158 @@ +#!/usr/bin/env bash +# A billing package that stores a line discount it never applies, and a draft plan to apply and show it. +set -euo pipefail + +mkdir -p .work/invoice-discounts/design billing tests +cat >README.md <<'EOF' +# billing + +Invoices for the order desk. Money rounding lives in billing/money.py. +EOF +: >billing/__init__.py +cat >billing/models.py <<'EOF' +from dataclasses import dataclass, field +from decimal import Decimal + + +@dataclass +class LineItem: + description: str + quantity: int + unit_price: Decimal + # Recorded by the order form; invoices do not apply it yet. + discount_pct: Decimal = Decimal("0") + + +@dataclass +class Invoice: + number: str + items: list = field(default_factory=list) +EOF +cat >billing/money.py <<'EOF' +from decimal import ROUND_HALF_UP, Decimal + + +def round_cents(amount): + return Decimal(amount).quantize(Decimal("0.01"), rounding=ROUND_HALF_UP) +EOF +cat >billing/pricing.py <<'EOF' +from billing import money + + +def line_total(item): + return money.round_cents(item.quantity * item.unit_price) +EOF +cat >billing/invoice.py <<'EOF' +from decimal import Decimal + +from billing import money, pricing + +TAX_RATE = Decimal("0.20") + + +def totals(invoice): + subtotal = sum((pricing.line_total(item) for item in invoice.items), Decimal("0")) + tax = money.round_cents(subtotal * TAX_RATE) + return {"subtotal": subtotal, "tax": tax, "total": subtotal + tax} +EOF +cat >billing/render.py <<'EOF' +from billing import invoice as invoice_totals +from billing import pricing + + +def render_text(invoice): + lines = ["Invoice " + invoice.number] + for item in invoice.items: + lines.append("%-30s %3d x %8s %10s" % (item.description, item.quantity, item.unit_price, pricing.line_total(item))) + sums = invoice_totals.totals(invoice) + lines.append("Subtotal %s Tax %s Total %s" % (sums["subtotal"], sums["tax"], sums["total"])) + return "\n".join(lines) +EOF +cat >tests/test_pricing.py <<'EOF' +import unittest +from decimal import Decimal + +from billing.models import LineItem +from billing.pricing import line_total + + +class LineTotalTest(unittest.TestCase): + def test_quantity_times_price(self): + self.assertEqual(line_total(LineItem("pens", 3, Decimal("10.00"))), Decimal("30.00")) + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/invoice-discounts/PLAN.md <<'EOF' +# Apply line discounts on invoices + +## Brief + +### Problem +Sales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand. + +### Goal +Invoices charge the discounted price and show the discount. + +### Acceptance criteria +- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50. +- AC2: The invoice subtotal and tax use the discounted line totals. +- AC3: An item with `discount_pct` 0 totals as today. +- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`. + +### Out-of-scope +- Invoice-level discounts. +- Changes to the order form. + +### Captured assumptions +- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100. + +## Design + +Early exit (tier C): Uses the existing `discount_pct` field; no new types or fields. + +## Plan + +### Behavior +- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1. +- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2. +- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue | + +### Phase 1: Discount line totals [STATUS] +**Satisfies:** AC1, AC3 +**Delivers:** B1 + +- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`. +- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent. + +**Sanity Check:** `python -m unittest tests.test_pricing` exits 0. + +### Phase 2: Carry discounted totals through the invoice [STATUS] +**Satisfies:** AC2, AC4 +**Delivers:** B2, B3 + +- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount. +- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `billing/pricing.py` | modify | +| `tests/test_pricing.py` | modify | +| `tests/test_invoice.py` | create | +EOF +cat >.work/invoice-discounts/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: Uses the existing `discount_pct` field; no new types or fields. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/invoice-discounts diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/case.yaml b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/case.yaml new file mode 100644 index 00000000000..c148f12cd52 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-medium-overnight-shifts +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/assumption-row.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/assumption-row.md new file mode 100644 index 00000000000..b2e16befeb4 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/assumption-row.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '###\s*Load-bearing assumptions[^\n]*\n(?:(?!#)[^\n]*\n)*?\|[^\n]*(?:noon|12:00|midday|overnight|night)' +flags: i +target: + source: file + path: .work/overnight-shifts/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/blast-radius-medium.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/blast-radius-medium.md new file mode 100644 index 00000000000..ed1d4adc9e5 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/blast-radius-medium.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bMEDIUM\b' +flags: i +target: + source: file + path: .work/overnight-shifts/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/export-untouched.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/export-untouched.md new file mode 100644 index 00000000000..e60a9c22946 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/export-untouched.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: '^\|\s*`?timesheet/export\.py`?' +flags: mi +match: not_contains +weight: 0.001 +target: + source: file + path: .work/overnight-shifts/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md new file mode 100644 index 00000000000..2d9159a4ea2 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '^## Plan[\s\S]*?(?:regression|\btest_\w+|\btests?\b)' +flags: mi +target: + source: file + path: .work/overnight-shifts/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/prompt.md new file mode 100644 index 00000000000..ea7550923de --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/prompt.md @@ -0,0 +1,10 @@ +--- +description: "MEDIUM tier, no trigger: a three-file bug-fix draft makes overnight shifts count forward to the next day. Seeded: the bug fix names no regression test (context/plan-reviewer.md Design alignment), and a Brief captured assumption has no Load-bearing row. hard: the Sanity Checks are real commands, so the plan looks verified while the bug has no test that would catch its return. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] +runs: 3 +max_turns: 200 +timeout_seconds: 3600 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md names a regression test for the 22:00-06:00 shift and has a Load-bearing row for the before-noon assumption, and does not touch timesheet/export.py; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/overnight-shifts/PLAN.md`. Its Brief, its phases and the design early exit (`.work/overnight-shifts/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/assumption-row.json b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/assumption-row.json new file mode 100644 index 00000000000..c33dea4e82a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/assumption-row.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Overnight shifts end before noon the next day | Brief | Phase 2 Sanity Check | Major: stop and replan |\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/blast-radius-medium.json b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/blast-radius-medium.json new file mode 100644 index 00000000000..74a0f7675ac --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/blast-radius-medium.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: MEDIUM\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** MEDIUM", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: HIGH\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/export-untouched.json b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/export-untouched.json new file mode 100644 index 00000000000..b2419860b7d --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/export-untouched.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n| `timesheet/export.py` | modify |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json new file mode 100644 index 00000000000..f417ff75c53 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n- In `tests/test_totals.py`, add a regression case: `22:00-06:00` totals 8.0.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..bd02cd5b58f --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/overnight-shifts/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/overnight-shifts/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/scaffold.sh new file mode 100755 index 00000000000..ee58d8924b1 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/scaffold.sh @@ -0,0 +1,146 @@ +#!/usr/bin/env bash +# A timesheet package that rejects or miscounts overnight shifts, and a draft bug-fix plan. +set -euo pipefail + +mkdir -p .work/overnight-shifts/design tests timesheet +cat >README.md <<'EOF' +# timesheet + +Parses shift lines and totals hours for payroll (timesheet/export.py writes the payroll file). +EOF +: >timesheet/__init__.py +cat >timesheet/models.py <<'EOF' +from dataclasses import dataclass +from datetime import date, time + + +@dataclass +class Shift: + day: date + start: time + end: time +EOF +cat >timesheet/parse.py <<'EOF' +from datetime import date, time + +from timesheet.models import Shift + + +def parse_line(line): + """'2026-03-02 09:00-17:00' -> Shift.""" + day_text, span = line.split() + start_text, end_text = span.split("-") + start, end = time.fromisoformat(start_text), time.fromisoformat(end_text) + if end <= start: + raise ValueError("end before start: " + line) + return Shift(date.fromisoformat(day_text), start, end) +EOF +cat >timesheet/totals.py <<'EOF' +from collections import defaultdict +from datetime import datetime + + +def hours(shift): + start = datetime.combine(shift.day, shift.start) + end = datetime.combine(shift.day, shift.end) + return (end - start).total_seconds() / 3600 + + +def weekly_hours(shifts): + week = defaultdict(float) + for shift in shifts: + week[shift.day] += hours(shift) + return dict(week) +EOF +cat >timesheet/export.py <<'EOF' +def fmt_hours(value): + return "%.2f" % max(value, 0.0) + + +def payroll_rows(week): + return [(day.isoformat(), fmt_hours(total)) for day, total in sorted(week.items())] +EOF +cat >tests/test_totals.py <<'EOF' +import unittest + +from timesheet.parse import parse_line +from timesheet.totals import hours + + +class HoursTest(unittest.TestCase): + def test_day_shift(self): + self.assertEqual(hours(parse_line("2026-03-02 09:00-17:00")), 8.0) + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/overnight-shifts/PLAN.md <<'EOF' +# Count overnight shifts + +## Brief + +### Problem +Night staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes. + +### Goal +A shift that ends at or before its start time counts as ending the next day. + +### Acceptance criteria +- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours. +- AC2: Day shifts total as today. +- AC3: `weekly_hours` counts an overnight shift on the day it starts. + +### Out-of-scope +- Shifts longer than 24 hours. +- Daylight-saving adjustments. + +### Captured assumptions +- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00. + +## Design + +Early exit (tier C): Changes existing functions; no new types. + +## Plan + +### Behavior +- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2. +- B2: Day shifts total as before (AC2). Delivered by Phase 2. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note | + +### Phase 1: Accept overnight lines [STATUS] +**Satisfies:** AC1 +**Delivers:** B1 + +- In `timesheet/parse.py`, drop the `end before start` rejection. +- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`. + +**Sanity Check:** `python -c "from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day"` exits 0. + +### Phase 2: Count the hours [STATUS] +**Satisfies:** AC1, AC2, AC3 +**Delivers:** B1, B2 + +- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`. + +**Sanity Check:** `python -c "from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0"` exits 0. + +### Files affected +| File | Action | +|---|---| +| `timesheet/parse.py` | modify | +| `timesheet/models.py` | modify | +| `timesheet/totals.py` | modify | +EOF +cat >.work/overnight-shifts/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: Changes existing functions; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" fix/overnight-shifts diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/case.yaml b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/case.yaml new file mode 100644 index 00000000000..754b72070ed --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-medium-quiet-hours +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/ac3-built.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/ac3-built.md new file mode 100644 index 00000000000..1703cbc852a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/ac3-built.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '^###\s*Phase[\s\S]*?^\s*[-*]\s+[^\n]*(?:\bdigest)' +flags: mi +target: + source: file + path: .work/quiet-hours/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/behavior-is-a-claim.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/behavior-is-a-claim.md new file mode 100644 index 00000000000..4bdd36e14a2 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/behavior-is-a-claim.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: '^\s*[-*]\s*B\d+:\s*(?:Edit|Update|Modify|Change|Add)\b' +flags: mi +match: not_contains +target: + source: file + path: .work/quiet-hours/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/blast-radius-medium.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/blast-radius-medium.md new file mode 100644 index 00000000000..f228aa6e914 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/blast-radius-medium.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bMEDIUM\b' +flags: i +target: + source: file + path: .work/quiet-hours/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/keeps-outbox.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/keeps-outbox.md new file mode 100644 index 00000000000..5fb2408605d --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/keeps-outbox.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: 'smtplib|send_mail|\bSMTP\b' +flags: i +match: not_contains +weight: 0.001 +target: + source: file + path: .work/quiet-hours/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/prompt.md new file mode 100644 index 00000000000..018fd06e215 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/prompt.md @@ -0,0 +1,10 @@ +--- +description: "MEDIUM tier, no trigger: a five-file draft holds non-critical alerts during quiet hours. Seeded: a Behavior item names a file and a step (agents/plan-reviewer.md third check), and AC3 (held alerts reach the morning digest) is claimed through B3 while no work item touches alerts/digest.py. hard: the script passes both, and the held alerts are silently dropped by the plan as written. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] +runs: 3 +max_turns: 200 +timeout_seconds: 3600 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md has a work item that routes held alerts into the morning digest, rewrites the file-naming Behavior item as a claim, and keeps paging through the outbox rather than sending mail directly; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/quiet-hours/PLAN.md`. Its Brief, its phases and the design early exit (`.work/quiet-hours/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/ac3-built.json b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/ac3-built.json new file mode 100644 index 00000000000..601742b9e01 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/ac3-built.json @@ -0,0 +1,30 @@ +{ + "pass": [ + { + "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `alerts/evaluate.py`, pass each held alert to `digest.add` so the 07:30 digest lists it.\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", + "why": "the plan with the seeded defect fixed" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Call `digest.add(alert)` for every alert held in quiet hours.\n", + "why": "call paraphrase" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Queue each held warning for the 07:30 morning digest.\n", + "why": "prose paraphrase" + } + ], + "fail": [ + { + "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", + "why": "the draft as seeded" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Describe quiet hours in README.md.\n", + "why": "no digest step" + }, + { + "answer": "- B3: Held alerts reach the morning digest (AC3).\n### Phase 2: Next step [TODO]\n- Describe quiet hours in README.md.\n", + "why": "only the Behavior item mentions it" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/behavior-is-a-claim.json b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/behavior-is-a-claim.json new file mode 100644 index 00000000000..6e0dd364f6a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/behavior-is-a-claim.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Warnings and info alerts raised in quiet hours do not page (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/blast-radius-medium.json b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/blast-radius-medium.json new file mode 100644 index 00000000000..6c3f8378e4e --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/blast-radius-medium.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: MEDIUM\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** MEDIUM", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: HIGH\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/keeps-outbox.json b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/keeps-outbox.json new file mode 100644 index 00000000000..62fa1e3b341 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/keeps-outbox.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, send the page with `smtplib` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..9b655266bac --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/quiet-hours/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/quiet-hours/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/scaffold.sh new file mode 100755 index 00000000000..90cad6c726c --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/scaffold.sh @@ -0,0 +1,137 @@ +#!/usr/bin/env bash +# An alerts package that pages for every alert at any hour, and a draft plan for quiet hours. +set -euo pipefail + +mkdir -p .work/quiet-hours/design alerts ops +cat >README.md <<'EOF' +# alerts + +Raises alerts, pages on-call through the outbox, and sends a morning digest. +EOF +: >alerts/__init__.py +cat >alerts/settings.py <<'EOF' +from datetime import time + +QUIET_START = time(22, 0) +QUIET_END = time(7, 0) +DIGEST_AT = time(7, 30) +EOF +cat >alerts/rules.py <<'EOF' +from dataclasses import dataclass + + +@dataclass +class Alert: + name: str + severity: str # "critical", "warning" or "info" +EOF +cat >alerts/outbox.py <<'EOF' +"""Pages are queued here; ops/schedule.txt runs flush() every minute, which sends them.""" +QUEUE = [] + + +def enqueue(alert): + QUEUE.append(alert) + + +def flush(send): + while QUEUE: + send(QUEUE.pop(0)) +EOF +cat >alerts/digest.py <<'EOF' +"""Morning digest, built at settings.DIGEST_AT from the alerts in DIGEST_QUEUE.""" +DIGEST_QUEUE = [] + + +def add(alert): + DIGEST_QUEUE.append(alert) + + +def build_digest(): + lines = ["%s (%s)" % (alert.name, alert.severity) for alert in DIGEST_QUEUE] + DIGEST_QUEUE.clear() + return "\n".join(lines) +EOF +cat >alerts/evaluate.py <<'EOF' +from alerts import outbox + + +def dispatch(alert, now): + outbox.enqueue(alert) +EOF +cat >ops/schedule.txt <<'EOF' +* * * * * python -m alerts.flush +30 7 * * * python -m alerts.send_digest +EOF +cat >.work/quiet-hours/PLAN.md <<'EOF' +# Quiet hours for alerts + +## Brief + +### Problem +On-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning. + +### Goal +Non-critical alerts raised in quiet hours wait for the morning digest instead of paging. + +### Acceptance criteria +- AC1: A critical alert in quiet hours is enqueued as today. +- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued. +- AC3: Each held alert appears in the next morning digest. +- AC4: Outside quiet hours every alert is enqueued as today. + +### Out-of-scope +- Per-person quiet hours. +- Snoozing an alert. + +## Design + +Early exit (tier C): Changes existing functions and reads existing settings; no new types. + +## Plan + +### Behavior +- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1. +- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1. +- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue | + +### Phase 1: Hold non-critical alerts [STATUS] +**Satisfies:** AC1, AC2, AC4 +**Delivers:** B1, B2 + +- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight. +- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`. +- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert. + +**Sanity Check:** `python -m unittest tests.test_rules` exits 0. + +### Phase 2: Morning hand-off [STATUS] +**Satisfies:** AC3 +**Delivers:** B3 + +- In `README.md`, describe quiet hours and which severities still page. +- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `alerts/rules.py` | modify | +| `alerts/evaluate.py` | modify | +| `README.md` | modify | +| `tests/test_rules.py` | create | +| `tests/test_evaluate.py` | create | +EOF +cat >.work/quiet-hours/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: Changes existing functions and reads existing settings; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/quiet-hours diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/case.yaml b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/case.yaml new file mode 100644 index 00000000000..6232944453d --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-medium-reorder-points +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/blast-radius-medium.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/blast-radius-medium.md new file mode 100644 index 00000000000..3964f89e9d6 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/blast-radius-medium.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bMEDIUM\b' +flags: i +target: + source: file + path: .work/reorder-points/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/no-new-dependency.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/no-new-dependency.md new file mode 100644 index 00000000000..29f3b751aa7 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/no-new-dependency.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: '^\|\s*`?(?:requirements[\w.-]*\.txt|pyproject\.toml|setup\.cfg|Pipfile)`?\s*\||tabulate' +flags: mi +match: not_contains +weight: 0.001 +target: + source: file + path: .work/reorder-points/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/out-of-scope-cut.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/out-of-scope-cut.md new file mode 100644 index 00000000000..2680458bf84 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/out-of-scope-cut.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: '^\s*[-*]\s+(?![^\n]*\b(?:out[ -]of[ -]scope|not|no|never|without|leave|leaves|unchanged|drop\w*|cut|remov\w*|exclud\w*)\b)[^\n]*(?:weekly_report|weekly report)' +flags: mi +match: not_contains +target: + source: file + path: .work/reorder-points/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reuses-available.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reuses-available.md new file mode 100644 index 00000000000..9dd9b4010f9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reuses-available.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: 'available\(|`available`|stock\.available' +flags: i +target: + source: file + path: .work/reorder-points/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/prompt.md new file mode 100644 index 00000000000..541630d8b19 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/prompt.md @@ -0,0 +1,10 @@ +--- +description: "MEDIUM tier, no trigger: a five-file draft adds a reorder command and computes on-hand minus reserved itself although inventory/stock.py already exposes that as `available` (context/plan-reviewer.md: build what existing code already does). Also seeded: a work item changes the weekly report, which the Brief's Out-of-scope excludes. hard for the duplicate; routine for the out-of-scope item, which Step 4.7's reading check also covers. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] +runs: 3 +max_turns: 200 +timeout_seconds: 3600 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md uses stock.available, drops the weekly report change, and adds no table-formatting dependency; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/reorder-points/PLAN.md`. Its Brief, its phases and the design early exit (`.work/reorder-points/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/blast-radius-medium.json b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/blast-radius-medium.json new file mode 100644 index 00000000000..4b1d196a11a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/blast-radius-medium.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: MEDIUM\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** MEDIUM", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU.\n- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `inventory/report.py` | modify |\n| `tests/test_reorder.py` | create |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: HIGH\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/no-new-dependency.json b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/no-new-dependency.json new file mode 100644 index 00000000000..5f0f5ce0f8c --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/no-new-dependency.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU.\n- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `inventory/report.py` | modify |\n| `tests/test_reorder.py` | create |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU through `tabulate`.\n- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `inventory/report.py` | modify |\n| `tests/test_reorder.py` | create |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/out-of-scope-cut.json b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/out-of-scope-cut.json new file mode 100644 index 00000000000..59b62b404ea --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/out-of-scope-cut.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `tests/test_reorder.py` | create |\n", + "why": "the plan with the seeded defect fixed" + }, + { + "answer": "- The weekly report is out of scope and stays unchanged.\n", + "why": "names it while leaving it alone" + } + ], + "fail": [ + { + "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU.\n- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `inventory/report.py` | modify |\n| `tests/test_reorder.py` | create |\n", + "why": "the draft as seeded" + }, + { + "answer": "- Extend `weekly_report()` with the reorder column.\n", + "why": "paraphrased scope creep" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reuses-available.json b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reuses-available.json new file mode 100644 index 00000000000..eec93e85b86 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reuses-available.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, take `stock.available(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU.\n- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `inventory/report.py` | modify |\n| `tests/test_reorder.py` | create |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU.\n- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `inventory/report.py` | modify |\n| `tests/test_reorder.py` | create |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..343740a60d9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/reorder-points/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/reorder-points/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/scaffold.sh new file mode 100755 index 00000000000..2418f72611a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/scaffold.sh @@ -0,0 +1,125 @@ +#!/usr/bin/env bash +# An inventory package with an available-stock helper and a weekly report, and a draft plan for reorder points. +set -euo pipefail + +mkdir -p .work/reorder-points/design inventory +cat >README.md <<'EOF' +# inventory + +Stock levels, the weekly report and the buyers' CLI (`python -m inventory.cli`). +EOF +: >inventory/__init__.py +cat >inventory/stock.py <<'EOF' +"""Stock levels. STOCK and RESERVED are loaded from the warehouse export at start-up.""" +STOCK = {} +RESERVED = {} + + +def on_hand(sku): + return STOCK.get(sku, 0) + + +def reserved(sku): + return RESERVED.get(sku, 0) + + +def available(sku): + """Stock free to sell: on hand minus reserved, never below zero.""" + return max(on_hand(sku) - reserved(sku), 0) +EOF +cat >inventory/report.py <<'EOF' +from inventory import stock + + +def weekly_report(): + rows = [(sku, stock.on_hand(sku), stock.reserved(sku)) for sku in sorted(stock.STOCK)] + return "\n".join("%-10s %6d %6d" % row for row in rows) +EOF +cat >inventory/cli.py <<'EOF' +import argparse + +from inventory import report + + +def main(argv=None): + parser = argparse.ArgumentParser(prog="inventory") + sub = parser.add_subparsers(dest="command", required=True) + sub.add_parser("weekly") + args = parser.parse_args(argv) + if args.command == "weekly": + print(report.weekly_report()) + + +if __name__ == "__main__": + main() +EOF +cat >.work/reorder-points/PLAN.md <<'EOF' +# Reorder points + +## Brief + +### Problem +Buyers notice a product is short only when an order cannot be filled. + +### Goal +A `reorder` command lists the products whose free stock has fallen below their reorder point. + +### Acceptance criteria +- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point. +- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed. +- AC3: The list is sorted by SKU. + +### Out-of-scope +- Changing the weekly stock report. +- Placing orders with suppliers. + +## Design + +Early exit (tier C): A new command over existing stock functions; no new types. + +## Plan + +### Behavior +- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2. +- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue | + +### Phase 1: Reorder check [STATUS] +**Satisfies:** AC1, AC2 +**Delivers:** B1, B2 + +- Create `inventory/data/reorder_points.json` with the buyers' points. +- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point. +- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point. + +**Sanity Check:** `python -m unittest tests.test_reorder` exits 0. + +### Phase 2: Command and report [STATUS] +**Satisfies:** AC1, AC3 +**Delivers:** B1 + +- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU. +- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `inventory/data/reorder_points.json` | create | +| `inventory/reorder.py` | create | +| `inventory/cli.py` | modify | +| `inventory/report.py` | modify | +| `tests/test_reorder.py` | create | +EOF +cat >.work/reorder-points/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: A new command over existing stock functions; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/reorder-points diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/case.yaml b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/case.yaml new file mode 100644 index 00000000000..f55b6c1dd1e --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-medium-shipping-zones +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/blast-radius-medium.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/blast-radius-medium.md new file mode 100644 index 00000000000..a17afa8a022 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/blast-radius-medium.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bMEDIUM\b' +flags: i +target: + source: file + path: .work/shipping-zones/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/callers-untouched.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/callers-untouched.md new file mode 100644 index 00000000000..9a798239d0b --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/callers-untouched.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: '^\|\s*`?shop/views\.py`?' +flags: mi +match: not_contains +weight: 0.001 +target: + source: file + path: .work/shipping-zones/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reuses-country-region.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reuses-country-region.md new file mode 100644 index 00000000000..8f27c426e1f --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reuses-country-region.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: 'country_region|shop\.geo|shop/geo\.py|\bgeo\.' +flags: i +target: + source: file + path: .work/shipping-zones/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/sanity-check-command.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/sanity-check-command.md new file mode 100644 index 00000000000..f9f0f1140cf --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/sanity-check-command.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'Sanity Check[^\n]*\b(?:looks? (?:right|correct|good|fine)|as expected|works (?:correctly|properly))' +flags: i +match: not_contains +target: + source: file + path: .work/shipping-zones/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/prompt.md new file mode 100644 index 00000000000..8c8d7033203 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/prompt.md @@ -0,0 +1,10 @@ +--- +description: "MEDIUM tier, no trigger: a five-file draft prices shipping by destination zone and builds a new country-to-zone table although shop/geo.py already maps countries to regions (context/plan-reviewer.md: build what existing code already does). Also seeded: a vague Sanity Check. hard: the duplicate needs a read of a module the draft never mentions. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] +runs: 3 +max_turns: 200 +timeout_seconds: 3600 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md reuses shop.geo.country_region, gives every phase a command Sanity Check, and does not touch shop/views.py since checkout_summary already takes the country; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/shipping-zones/PLAN.md`. Its Brief, its phases and the design early exit (`.work/shipping-zones/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/blast-radius-medium.json b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/blast-radius-medium.json new file mode 100644 index 00000000000..8d8cb9cbe84 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/blast-radius-medium.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: MEDIUM\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** MEDIUM", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`).\n- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** checkout totals look correct for each zone.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `tests/test_shipping.py` | modify |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: HIGH\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/callers-untouched.json b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/callers-untouched.json new file mode 100644 index 00000000000..b4ab38f501b --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/callers-untouched.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`).\n- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** checkout totals look correct for each zone.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `tests/test_shipping.py` | modify |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`).\n- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** checkout totals look correct for each zone.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `shop/views.py` | modify |\n| `tests/test_shipping.py` | modify |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reuses-country-region.json b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reuses-country-region.json new file mode 100644 index 00000000000..7e7acbaf49f --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reuses-country-region.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Use `shop.geo.country_region`, which already maps codes to `home`, `eu` and `world`.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** checkout totals look correct for each zone.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `tests/test_shipping.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`).\n- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** checkout totals look correct for each zone.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `tests/test_shipping.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..ac8117a2c1d --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/shipping-zones/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/shipping-zones/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/sanity-check-command.json b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/sanity-check-command.json new file mode 100644 index 00000000000..112b3a85f4f --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/sanity-check-command.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`).\n- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** `python -m unittest tests.test_shipping` exits 0.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `tests/test_shipping.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`).\n- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** checkout totals look correct for each zone.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `tests/test_shipping.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/scaffold.sh new file mode 100755 index 00000000000..499d1642717 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/scaffold.sh @@ -0,0 +1,146 @@ +#!/usr/bin/env bash +# A shop package with a country-to-region map and flat shipping, and a draft plan for zone rates that builds its own map. +set -euo pipefail + +mkdir -p .work/shipping-zones/design shop tests +cat >README.md <<'EOF' +# shop + +Cart, checkout and shipping for the web shop. Country data lives in shop/geo.py. +EOF +: >shop/__init__.py +cat >shop/geo.py <<'EOF' +"""Country facts shared by checkout, tax and shipping.""" +REGIONS = {"GB": "home", "IE": "eu", "FR": "eu", "DE": "eu", "ES": "eu", "US": "world", "CA": "world", "AU": "world"} + + +def country_region(code): + """'home', 'eu' or 'world'; any code not listed is 'world'.""" + return REGIONS.get(code.upper(), "world") +EOF +cat >shop/settings.py <<'EOF' +from decimal import Decimal + +FREE_SHIPPING_MIN = Decimal("50.00") +FLAT_RATE = Decimal("4.95") +EOF +cat >shop/shipping.py <<'EOF' +from decimal import Decimal + +from shop import settings + + +def shipping_cost(cart_total): + if cart_total >= settings.FREE_SHIPPING_MIN: + return Decimal("0.00") + return settings.FLAT_RATE +EOF +cat >shop/checkout.py <<'EOF' +from shop import geo, shipping + + +def checkout_summary(cart, country): + total = sum(line["price"] * line["qty"] for line in cart) + return {"region": geo.country_region(country), "items": total, "shipping": shipping.shipping_cost(total)} +EOF +cat >shop/views.py <<'EOF' +from shop.checkout import checkout_summary + + +def checkout_page(request): + return checkout_summary(request["cart"], request["country"]) +EOF +cat >tests/test_shipping.py <<'EOF' +import unittest +from decimal import Decimal + +from shop.shipping import shipping_cost + + +class ShippingTest(unittest.TestCase): + def test_flat_rate(self): + self.assertEqual(shipping_cost(Decimal("10")), Decimal("4.95")) + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/shipping-zones/PLAN.md <<'EOF' +# Shipping by zone + +## Brief + +### Problem +Every order pays the domestic flat rate, so orders shipped abroad lose money on postage. + +### Goal +Shipping cost depends on the destination's region. + +### Acceptance criteria +- AC1: Region `home` keeps 4.95, free from a 50.00 cart. +- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only. +- AC3: The checkout summary shows the rate for the destination country. +- AC4: A country code the shop does not know pays the `world` rate. + +### Out-of-scope +- Carrier rate quotes. +- Weight-based pricing. + +## Design + +Early exit (tier C): Changes existing functions; no new types. + +## Plan + +### Behavior +- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2. +- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2. +- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue | + +### Phase 1: Zone table [STATUS] +**Satisfies:** AC2, AC4 +**Delivers:** B1 + +- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`). +- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks. + +**Sanity Check:** `python -c "from shop.zones import zone_for; assert zone_for('ZZ') == 'world'"` exits 0. + +### Phase 2: Rate by zone [STATUS] +**Satisfies:** AC1, AC2 +**Delivers:** B1, B2 + +- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95. +- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold. + +**Sanity Check:** checkout totals look correct for each zone. + +### Phase 3: Show the rate at checkout [STATUS] +**Satisfies:** AC3 +**Delivers:** B3 + +- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `shop/data/zones.json` | create | +| `shop/zones.py` | create | +| `shop/shipping.py` | modify | +| `shop/checkout.py` | modify | +| `tests/test_shipping.py` | modify | +EOF +cat >.work/shipping-zones/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: Changes existing functions; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/shipping-zones From b6843805a972682908bf841c99615d2f0e4ff505 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Wed, 11 Nov 2026 17:19:41 -0500 Subject: [PATCH 4/7] test(planning): clear typos and comment-marker findings in the eval cases Spell "unparsable" in the notes-since case and its samples, spell out the "exclud" regex stem in two out-of-scope graders (same matches), and drop the status-marker word from init-repo.sh's comment. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VQb9Q4F46UumaUgXeUdpYZ --- .../planning/evals/fixtures/plan-reviewer-cut/init-repo.sh | 3 ++- .../samples/appetite-fit-line.json | 4 ++-- .../samples/assumption-row.json | 4 ++-- .../samples/blast-radius-low.json | 2 +- .../samples/no-dateutil.json | 4 ++-- .../evals/plan-reviewer-cut-low-notes-since/scaffold.sh | 2 +- .../graders/out-of-scope-cut.md | 2 +- .../graders/out-of-scope-cut.md | 2 +- 8 files changed, 12 insertions(+), 11 deletions(-) diff --git a/plugins/planning/evals/fixtures/plan-reviewer-cut/init-repo.sh b/plugins/planning/evals/fixtures/plan-reviewer-cut/init-repo.sh index 82f1752a667..da87d9d2f68 100755 --- a/plugins/planning/evals/fixtures/plan-reviewer-cut/init-repo.sh +++ b/plugins/planning/evals/fixtures/plan-reviewer-cut/init-repo.sh @@ -4,7 +4,8 @@ # # init-repo.sh # -# Each draft's phase headings carry the placeholder [STATUS]; this script sets it to [TODO]. +# Each draft's phase headings carry the placeholder [STATUS]; this script sets it to the +# not-started status marker the plan skill expects. set -euo pipefail branch="$1" diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/appetite-fit-line.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/appetite-fit-line.json index c69ee174d26..52d3dd18d9f 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/appetite-fit-line.json +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/appetite-fit-line.json @@ -1,13 +1,13 @@ { "pass": [ { - "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparseable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\nAppetite fit: one flag and one filter on one subcommand fits half a day.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\nAppetite fit: one flag and one filter on one subcommand fits half a day.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", "why": "the plan with the seeded defect fixed" } ], "fail": [ { - "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparseable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", "why": "the draft as seeded" } ] diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/assumption-row.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/assumption-row.json index 5141e3d6088..deabaa3afc6 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/assumption-row.json +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/assumption-row.json @@ -1,13 +1,13 @@ { "pass": [ { - "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparseable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Note `created` values are ISO 8601 dates | Brief | Phase 1 Sanity Check | Major: stop and replan |\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Note `created` values are ISO 8601 dates | Brief | Phase 1 Sanity Check | Major: stop and replan |\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", "why": "the plan with the seeded defect fixed" } ], "fail": [ { - "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparseable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", "why": "the draft as seeded" } ] diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/blast-radius-low.json index 236ee18df6a..7f4c110048f 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/blast-radius-low.json +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/blast-radius-low.json @@ -11,7 +11,7 @@ ], "fail": [ { - "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparseable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", "why": "the draft as seeded has no assessment" }, { diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/no-dateutil.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/no-dateutil.json index 4f6494b64c5..200422d9f9d 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/no-dateutil.json +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/no-dateutil.json @@ -1,13 +1,13 @@ { "pass": [ { - "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparseable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", "why": "the draft as seeded keeps the sound part" } ], "fail": [ { - "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparseable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=dateutil.parser.parse`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=dateutil.parser.parse`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", "why": "the plan made the over-flagged change" } ] diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/scaffold.sh index e50fa5c1cd5..7740f7fa6a7 100755 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/scaffold.sh +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/scaffold.sh @@ -76,7 +76,7 @@ cat >.work/notes-since/PLAN.md <<'EOF' ### Acceptance criteria - AC1: `notes list` without `--since` prints what it prints today. - AC2: `notes list --since 2026-03-01` prints only notes created on or after that date. -- AC3: An unparseable `--since` value exits with status 2 and an error naming the flag. +- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag. ### Out-of-scope - Relative dates such as "last week". diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md index 168467bfd74..df01edc1a71 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '^\s*[-*]\s+(?![^\n]*\b(?:out[ -]of[ -]scope|not|no|never|without|leave|leaves|unchanged|drop\w*|cut|remov\w*|exclud\w*)\b)[^\n]*(?:fahrenheit|unit\s*=|°F)' +pattern: '^\s*[-*]\s+(?![^\n]*\b(?:out[ -]of[ -]scope|not|no|never|without|leave|leaves|unchanged|drop\w*|cut|remov\w*|exclude|excluded|excludes|excluding\w*)\b)[^\n]*(?:fahrenheit|unit\s*=|°F)' flags: mi match: not_contains target: diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/out-of-scope-cut.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/out-of-scope-cut.md index 2680458bf84..abd35743bfd 100644 --- a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/out-of-scope-cut.md +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/out-of-scope-cut.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '^\s*[-*]\s+(?![^\n]*\b(?:out[ -]of[ -]scope|not|no|never|without|leave|leaves|unchanged|drop\w*|cut|remov\w*|exclud\w*)\b)[^\n]*(?:weekly_report|weekly report)' +pattern: '^\s*[-*]\s+(?![^\n]*\b(?:out[ -]of[ -]scope|not|no|never|without|leave|leaves|unchanged|drop\w*|cut|remov\w*|exclude|excluded|excludes|excluding\w*)\b)[^\n]*(?:weekly_report|weekly report)' flags: mi match: not_contains target: From e4b97be947462c7c9d0fcc3a917b87bdff5ceed9 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Wed, 11 Nov 2026 17:59:02 -0500 Subject: [PATCH 5/7] test(planning): drop the unused reviewer control and a retired term The owner chose a before/after compare without the degraded-reviewer arm, so its two fixture texts and prepare-arm.sh's --degrade-reviewer option go (orphaned-fixtures). Case descriptions say "before behavior", not the retired "contract" sense near "Brief" (vocabulary). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VQb9Q4F46UumaUgXeUdpYZ --- .../plan-reviewer-degraded-body.md.txt | 1 - .../plan-reviewer-degraded-prompt.md.txt | 17 -------------- .../fixtures/plan-reviewer-cut/prepare-arm.sh | 22 ++++--------------- .../prompt.md | 2 +- .../prompt.md | 2 +- .../prompt.md | 2 +- .../prompt.md | 2 +- .../prompt.md | 2 +- .../prompt.md | 2 +- .../prompt.md | 2 +- .../prompt.md | 2 +- .../prompt.md | 2 +- .../prompt.md | 2 +- .../prompt.md | 2 +- .../prompt.md | 2 +- 15 files changed, 16 insertions(+), 48 deletions(-) delete mode 100644 plugins/planning/evals/fixtures/plan-reviewer-cut/plan-reviewer-degraded-body.md.txt delete mode 100644 plugins/planning/evals/fixtures/plan-reviewer-cut/plan-reviewer-degraded-prompt.md.txt diff --git a/plugins/planning/evals/fixtures/plan-reviewer-cut/plan-reviewer-degraded-body.md.txt b/plugins/planning/evals/fixtures/plan-reviewer-cut/plan-reviewer-degraded-body.md.txt deleted file mode 100644 index 83426eda278..00000000000 --- a/plugins/planning/evals/fixtures/plan-reviewer-cut/plan-reviewer-degraded-body.md.txt +++ /dev/null @@ -1 +0,0 @@ -You are a plan reviewer. Read the plan you are given and say what you think of it. diff --git a/plugins/planning/evals/fixtures/plan-reviewer-cut/plan-reviewer-degraded-prompt.md.txt b/plugins/planning/evals/fixtures/plan-reviewer-cut/plan-reviewer-degraded-prompt.md.txt deleted file mode 100644 index 028fc7f4ff0..00000000000 --- a/plugins/planning/evals/fixtures/plan-reviewer-cut/plan-reviewer-degraded-prompt.md.txt +++ /dev/null @@ -1,17 +0,0 @@ -# Plan Reviewer: Sub-Agent Dispatch - -Fresh-context plan review for `/planning:plan` Step 3. - -## Orchestrator inputs - -Attach the plan draft and the Brief. - -## Sub-agent prompt template - -```text -Here is a plan. Tell me what you think of it. -``` - -## After dispatch - -The main thread reads the reply, then proceeds to Step 3b (blast radius). diff --git a/plugins/planning/evals/fixtures/plan-reviewer-cut/prepare-arm.sh b/plugins/planning/evals/fixtures/plan-reviewer-cut/prepare-arm.sh index 68d181d9488..872e9c29fce 100755 --- a/plugins/planning/evals/fixtures/plan-reviewer-cut/prepare-arm.sh +++ b/plugins/planning/evals/fixtures/plan-reviewer-cut/prepare-arm.sh @@ -3,27 +3,21 @@ # repository, holding this checkout's plan-reviewer-cut cases. `claude plugin eval` then runs it # with --ablation none. # -# prepare-arm.sh [--degrade-reviewer] +# prepare-arm.sh # -# is a commit of this repository (the before arm pins the PR's merge base). With -# --degrade-reviewer, the copy's plan-reviewer agent keeps its frontmatter and takes its body from -# plan-reviewer-degraded-body.md.txt, and the dispatch prompt (skills/plan/context/plan-reviewer.md) -# is replaced by plan-reviewer-degraded-prompt.md.txt: the suite's negative control. Both are -# replaced because the review axes and the report format live in the dispatch prompt. +# is a commit of this repository (the before arm pins the PR's merge base). # # Exit: 0 written; 2 a usage error, or a dest that already exists. set -euo pipefail usage() { - echo "usage: prepare-arm.sh [--degrade-reviewer]" >&2 + echo "usage: prepare-arm.sh " >&2 exit 2 } -[[ $# -ge 2 && $# -le 3 ]] || usage +[[ $# -eq 2 ]] || usage ref="$1" dest="$2" -degrade="${3:-}" -[[ -z "$degrade" || "$degrade" == "--degrade-reviewer" ]] || usage if [[ -e "$dest" ]]; then echo "prepare-arm.sh: $dest already exists" >&2 exit 2 @@ -48,12 +42,4 @@ rm -rf "${dest:?}/evals/fixtures/plan-reviewer-cut" "$dest"/evals/plan-reviewer- cp -R "$here" "$dest/evals/fixtures/plan-reviewer-cut" cp -R "$plugin"/evals/plan-reviewer-cut-* "$dest/evals/" -if [[ -n "$degrade" ]]; then - agent="$dest/agents/plan-reviewer.md" - frontmatter="$(awk 'NR == 1 || !done { print } NR > 1 && /^---$/ { done = 1 }' "$agent")" - printf '%s\n\n' "$frontmatter" >"$agent" - cat "$here/plan-reviewer-degraded-body.md.txt" >>"$agent" - cp "$here/plan-reviewer-degraded-prompt.md.txt" "$dest/skills/plan/context/plan-reviewer.md" -fi - echo "$dest" diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/prompt.md index 398ae68fc21..51177f031a9 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/prompt.md +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/prompt.md @@ -1,5 +1,5 @@ --- -description: "LOW tier, no trigger: a draft adds post permalinks and writes its own slug helper although blog/text.py already has `slugify`, which AC2's accent folding needs (context/plan-reviewer.md: build what existing code already does). Also seeded: a Behavior item that names a file and a step instead of an observable claim (agents/plan-reviewer.md third check). hard: the duplicate is visible only by reading code the plan never cites. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +description: "LOW tier, no trigger: a draft adds post permalinks and writes its own slug helper although blog/text.py already has `slugify`, which AC2's accent folding needs (context/plan-reviewer.md: build what existing code already does). Also seeded: a Behavior item that names a file and a step instead of an observable claim (agents/plan-reviewer.md third check). hard: the duplicate is visible only by reading code the plan never cites. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] runs: 3 max_turns: 150 diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/prompt.md index 4d81fed2834..3113ec31ad4 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/prompt.md +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/prompt.md @@ -1,5 +1,5 @@ --- -description: "LOW tier, no trigger: a two-file draft adds a delimiter option to a CSV export. Seeded: a Brief captured assumption with no Load-bearing row (agents/plan-reviewer.md second check), and AC3 claimed by Phase 1 that no work item builds (the Delivers check). hard: both seeds pass check-plan-outcome.sh, so only a reader who compares the Brief with the phases catches them. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +description: "LOW tier, no trigger: a two-file draft adds a delimiter option to a CSV export. Seeded: a Brief captured assumption with no Load-bearing row (agents/plan-reviewer.md second check), and AC3 claimed by Phase 1 that no work item builds (the Delivers check). hard: both seeds pass check-plan-outcome.sh, so only a reader who compares the Brief with the phases catches them. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] runs: 3 max_turns: 150 diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/prompt.md index 9843dbf0488..cf629502818 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/prompt.md +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/prompt.md @@ -1,5 +1,5 @@ --- -description: "LOW tier, no trigger: a draft adds a --since filter to a notes CLI. Seeded: the Brief records an appetite but the plan has no Appetite fit line (agents/plan-reviewer.md appetite check), and a Brief captured assumption about the date format has no Load-bearing row. hard: neither is a script criterion, and a run that resumes at the review step skips Step 2, where the main thread would write the line. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +description: "LOW tier, no trigger: a draft adds a --since filter to a notes CLI. Seeded: the Brief records an appetite but the plan has no Appetite fit line (agents/plan-reviewer.md appetite check), and a Brief captured assumption about the date format has no Load-bearing row. hard: neither is a script criterion, and a run that resumes at the review step skips Step 2, where the main thread would write the line. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] runs: 3 max_turns: 150 diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/prompt.md index fd54b683ee5..a3014faa15e 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/prompt.md +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/prompt.md @@ -1,5 +1,5 @@ --- -description: "LOW tier, no trigger: a bug-fix draft caps retry backoff in one module. Seeded: it names a config constant `MAX_DELAY` that fetcher/config.py does not define (the ceiling is `MAX_BACKOFF_SECONDS`), and the bug fix names no regression test (context/plan-reviewer.md Design alignment). hard: the author believes its own constant name, and the mechanical gate checks neither. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +description: "LOW tier, no trigger: a bug-fix draft caps retry backoff in one module. Seeded: it names a config constant `MAX_DELAY` that fetcher/config.py does not define (the ceiling is `MAX_BACKOFF_SECONDS`), and the bug fix names no regression test (context/plan-reviewer.md Design alignment). hard: the author believes its own constant name, and the mechanical gate checks neither. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] runs: 3 max_turns: 150 diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/prompt.md index f8295b5dd5b..8b244f0d782 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/prompt.md +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/prompt.md @@ -1,5 +1,5 @@ --- -description: "LOW tier, no trigger: a draft adds a decimals option to a temperature formatter. Seeded: a work item adds a unit switch the Brief's Out-of-scope excludes (context/plan-reviewer.md Plan mechanics), and the phase's Sanity Check is a vague `looks right` the script counts as present. routine for the out-of-scope seed, which Step 4.7's reading check also covers; hard for the vague Sanity Check. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +description: "LOW tier, no trigger: a draft adds a decimals option to a temperature formatter. Seeded: a work item adds a unit switch the Brief's Out-of-scope excludes (context/plan-reviewer.md Plan mechanics), and the phase's Sanity Check is a vague `looks right` the script counts as present. routine for the out-of-scope seed, which Step 4.7's reading check also covers; hard for the vague Sanity Check. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] runs: 3 max_turns: 150 diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/prompt.md index 4edabc3a3e6..639dfb0ced0 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/prompt.md +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/prompt.md @@ -1,5 +1,5 @@ --- -description: "LOW tier, no trigger: a draft adds price_for and claims load_prices already returns an empty dict for a missing file; catalog/prices.py opens the file unguarded, so it raises FileNotFoundError (context/plan-reviewer.md: findings backed by a code path). Also seeded: a Behavior item that is a step of work, not a claim. hard: the false claim reads as settled fact and only a read of the code disproves it. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +description: "LOW tier, no trigger: a draft adds price_for and claims load_prices already returns an empty dict for a missing file; catalog/prices.py opens the file unguarded, so it raises FileNotFoundError (context/plan-reviewer.md: findings backed by a code path). Also seeded: a Behavior item that is a step of work, not a claim. hard: the false claim reads as settled fact and only a read of the code disproves it. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] runs: 3 max_turns: 150 diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/prompt.md index 5307e940181..640fa26b7dd 100644 --- a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/prompt.md +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/prompt.md @@ -1,5 +1,5 @@ --- -description: "MEDIUM tier, no trigger: a five-file draft adds author search. Seeded: the Brief records an appetite and the plan has no Appetite fit line, and the plan says search_title already sorts by title, so AC3 needs no work, while library/search.py returns matches in catalogue order. hard: the sorting claim is false only on reading the code, and no script criterion covers either seed. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +description: "MEDIUM tier, no trigger: a five-file draft adds author search. Seeded: the Brief records an appetite and the plan has no Appetite fit line, and the plan says search_title already sorts by title, so AC3 needs no work, while library/search.py returns matches in catalogue order. hard: the sorting claim is false only on reading the code, and no script criterion covers either seed. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] runs: 3 max_turns: 200 diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/prompt.md index 6417de96f46..d37b3bc1b0b 100644 --- a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/prompt.md +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/prompt.md @@ -1,5 +1,5 @@ --- -description: "MEDIUM tier, no trigger: a three-phase-file draft applies a stored line discount to invoice totals. Seeded: Phase 2 claims AC4 and B3 (the invoice text shows the percentage) but no work item touches billing/render.py (agents/plan-reviewer.md Delivers check), and a Brief captured assumption has no Load-bearing row. hard: the script sees AC4 on a Satisfies line and passes it. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +description: "MEDIUM tier, no trigger: a three-phase-file draft applies a stored line discount to invoice totals. Seeded: Phase 2 claims AC4 and B3 (the invoice text shows the percentage) but no work item touches billing/render.py (agents/plan-reviewer.md Delivers check), and a Brief captured assumption has no Load-bearing row. hard: the script sees AC4 on a Satisfies line and passes it. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] runs: 3 max_turns: 200 diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/prompt.md index ea7550923de..40afc9ec722 100644 --- a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/prompt.md +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/prompt.md @@ -1,5 +1,5 @@ --- -description: "MEDIUM tier, no trigger: a three-file bug-fix draft makes overnight shifts count forward to the next day. Seeded: the bug fix names no regression test (context/plan-reviewer.md Design alignment), and a Brief captured assumption has no Load-bearing row. hard: the Sanity Checks are real commands, so the plan looks verified while the bug has no test that would catch its return. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +description: "MEDIUM tier, no trigger: a three-file bug-fix draft makes overnight shifts count forward to the next day. Seeded: the bug fix names no regression test (context/plan-reviewer.md Design alignment), and a Brief captured assumption has no Load-bearing row. hard: the Sanity Checks are real commands, so the plan looks verified while the bug has no test that would catch its return. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] runs: 3 max_turns: 200 diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/prompt.md index 018fd06e215..ff3f4d52b11 100644 --- a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/prompt.md +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/prompt.md @@ -1,5 +1,5 @@ --- -description: "MEDIUM tier, no trigger: a five-file draft holds non-critical alerts during quiet hours. Seeded: a Behavior item names a file and a step (agents/plan-reviewer.md third check), and AC3 (held alerts reach the morning digest) is claimed through B3 while no work item touches alerts/digest.py. hard: the script passes both, and the held alerts are silently dropped by the plan as written. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +description: "MEDIUM tier, no trigger: a five-file draft holds non-critical alerts during quiet hours. Seeded: a Behavior item names a file and a step (agents/plan-reviewer.md third check), and AC3 (held alerts reach the morning digest) is claimed through B3 while no work item touches alerts/digest.py. hard: the script passes both, and the held alerts are silently dropped by the plan as written. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] runs: 3 max_turns: 200 diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/prompt.md index 541630d8b19..cb30e10f38f 100644 --- a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/prompt.md +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/prompt.md @@ -1,5 +1,5 @@ --- -description: "MEDIUM tier, no trigger: a five-file draft adds a reorder command and computes on-hand minus reserved itself although inventory/stock.py already exposes that as `available` (context/plan-reviewer.md: build what existing code already does). Also seeded: a work item changes the weekly report, which the Brief's Out-of-scope excludes. hard for the duplicate; routine for the out-of-scope item, which Step 4.7's reading check also covers. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +description: "MEDIUM tier, no trigger: a five-file draft adds a reorder command and computes on-hand minus reserved itself although inventory/stock.py already exposes that as `available` (context/plan-reviewer.md: build what existing code already does). Also seeded: a work item changes the weekly report, which the Brief's Out-of-scope excludes. hard for the duplicate; routine for the out-of-scope item, which Step 4.7's reading check also covers. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] runs: 3 max_turns: 200 diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/prompt.md index 8c8d7033203..ab742efab59 100644 --- a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/prompt.md +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/prompt.md @@ -1,5 +1,5 @@ --- -description: "MEDIUM tier, no trigger: a five-file draft prices shipping by destination zone and builds a new country-to-zone table although shop/geo.py already maps countries to regions (context/plan-reviewer.md: build what existing code already does). Also seeded: a vague Sanity Check. hard: the duplicate needs a read of a module the draft never mentions. Source: written for this suite from the before contract (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +description: "MEDIUM tier, no trigger: a five-file draft prices shipping by destination zone and builds a new country-to-zone table although shop/geo.py already maps countries to regions (context/plan-reviewer.md: build what existing code already does). Also seeded: a vague Sanity Check. hard: the duplicate needs a read of a module the draft never mentions. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] runs: 3 max_turns: 200 From a5ce584795d382c8e51e4b05d1a4785709f77213 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Wed, 11 Nov 2026 18:22:57 -0500 Subject: [PATCH 6/7] feat(planning): keep the plan reviewer on MEDIUM plans The before/after eval measured only LOW plans (change -0.03, interval -0.08 to +0.03, no case drop); MEDIUM went unmeasured. So auto now skips the plan-reviewer on a LOW plan with no trigger only, and runs it at MEDIUM or higher, as the owner decided (ship only measured parts). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VQb9Q4F46UumaUgXeUdpYZ --- .changes/planning/feat-plan-reviewer-risk-gated-140d7eb6.md | 2 +- docs/contracts/verification-level/README.md | 2 +- plugins/planning/README.md | 2 +- plugins/planning/reference/config.md | 6 +++--- plugins/planning/schemas/planning.schema.json | 2 +- plugins/planning/skills/plan/SKILL.md | 2 +- .../planning/skills/plan/context/stress-test-triggers.md | 2 +- plugins/planning/skills/plan/templates/checklist.md | 2 +- 8 files changed, 10 insertions(+), 10 deletions(-) diff --git a/.changes/planning/feat-plan-reviewer-risk-gated-140d7eb6.md b/.changes/planning/feat-plan-reviewer-risk-gated-140d7eb6.md index 992442afb1a..a39c54621bc 100644 --- a/.changes/planning/feat-plan-reviewer-risk-gated-140d7eb6.md +++ b/.changes/planning/feat-plan-reviewer-risk-gated-140d7eb6.md @@ -4,5 +4,5 @@ bump: minor ### Changed -- `/planning:plan` no longer dispatches the fresh-context `plan-reviewer` on every plan. Under the default `plan_stress_test: auto` it runs only when the blast radius is HIGH or CRITICAL, a stress-test trigger matches, or the user asks for a review; the blast radius is now assessed before that decision, and the Step 5 presentation states a skip and its reason (`Reviewer: skipped (blast radius LOW, no trigger)`). `plan_stress_test` `reviewer` and `both` and verification `strict` still always run it; `light` and `off` still skip it. Basis: Anthropic's [Prompting Claude Opus 5, "Task scope and over-verification"](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5#task-scope-and-over-verification): explicit verification steps and "use a subagent to verify" cause over-verification, and removing them reduces wasted tokens with no loss in quality. +- `/planning:plan` no longer dispatches the fresh-context `plan-reviewer` on LOW plans. Under the default `plan_stress_test: auto` it runs when the blast radius is MEDIUM or higher, a stress-test trigger matches, or the user asks for a review; a before/after eval on LOW plans measured a change of -0.03 (95% interval -0.08 to +0.03) with no case drop, and MEDIUM keeps the reviewer until it is measured; the blast radius is now assessed before that decision, and the Step 5 presentation states a skip and its reason (`Reviewer: skipped (blast radius LOW, no trigger)`). `plan_stress_test` `reviewer` and `both` and verification `strict` still always run it; `light` and `off` still skip it. Basis: Anthropic's [Prompting Claude Opus 5, "Task scope and over-verification"](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5#task-scope-and-over-verification): explicit verification steps and "use a subagent to verify" cause over-verification, and removing them reduces wasted tokens with no loss in quality. - New eval case `plan-low-risk-skips-reviewer`: a LOW blast-radius draft with default settings dispatches no plan-reviewer and reports the skip line. diff --git a/docs/contracts/verification-level/README.md b/docs/contracts/verification-level/README.md index 5075cd72044..6792b3c9d03 100644 --- a/docs/contracts/verification-level/README.md +++ b/docs/contracts/verification-level/README.md @@ -81,7 +81,7 @@ A skill that is itself the independent check when invoked does not read the leve | `/discovery:research`, `/discovery:explore`, `/discovery:trace-intent` | the sibling verifier (parent contract, "The sibling verifier, stated once"); `strict` widens research's grading to every row and claim | | `/discipline:do-your-research`, `/discipline:recheck-against-upstream`, `/discipline:sweep-all` | the fresh-context confirmation of a clean audit on load-bearing work (shared method, step 2), the fresh-context re-derivation (step 3) and the fan-out tiers' dispatch; `strict` takes do-your-research's fan-out tier at `full` depth | | `/education:quiz-me` | the fresh-context answer-key pass | -| `/planning:plan` | the plan-reviewer dispatch in Step 3; `strict` dispatches it for every plan, which `plan_stress_test`'s `auto` otherwise gives only to high-risk plans | +| `/planning:plan` | the plan-reviewer dispatch in Step 3; `strict` dispatches it for every plan, which `plan_stress_test`'s `auto` otherwise skips on LOW plans | | `/session-flow:orchestrate`, `/session-flow:workflow`, `/session-flow:retro` | imperative 3's verifier in a primed session; the fresh-context reviewer in pre-PR step 2, without which Stage 10 is not done; retro's single-pass analysis and scoring under `retro_lenses: 1` | | `/harness-config:audit-automation-gaps` | the fresh-context refusal gate (2.3), whose verdicts `light` lists as `unverified`, and the Phase 4 step 6 reviewer | | `/harness-config:audit-instructions` | the Phase C verifiers (per lane, B2 and I33) | diff --git a/plugins/planning/README.md b/plugins/planning/README.md index fd120761051..f29b20ab3d9 100644 --- a/plugins/planning/README.md +++ b/plugins/planning/README.md @@ -19,7 +19,7 @@ where artifacts land in the consuming repo. | `/planning:design` | Design space | Explores types, contracts, module boundaries, and package topology through collaborative discussion rounds, producing capability-matrix / type-inventory / design-threads / topology artifacts; its `handoff` action delegates to `/planning:design-handoff`. Its `data`, `integration`, and `system` artifacts are typed: each carries a `scope` + `dialect` label and a fenced block in the dialect the `diagram_dialect` convention key resolves to (mermaid `erDiagram` or DBML for data; mermaid `sequenceDiagram` plus an OpenAPI 3.1 sketch for integration; a LikeC4 or C4-PlantUML container view for system, and no C4 view at all when that key is unset). `library` and `module` emit no typed artifact. | | `/planning:design-handoff` | Design→plan gate | Gates a finished design for `/planning:plan`. The gate is a binary check that every `design-threads.md` thread is RESOLVED, directional, or TAGGED-DEFERRED. Then it packages the plan-ready summary and resume prompt, or FAILs and routes back to `/planning:design`. On PASS and FAIL alike it also emits an advisory six-dimension coverage table (what, how, where, who, when, why) that never contributes to the verdict; uncovered dimensions ride along in the handoff summary and the resume prompt. | | `/planning:devils-advocate` | Adversarial review | Stress-tests plans via assumption extraction, evidence checks, failure scenarios, and operational-gotcha sweeps. Every finding evidence-backed, never generic warnings. An `incumbent` mode turns the same lens on the status quo: an Alternatives Sweep that stress-tests keeping an incumbent tool/approach against alternatives (native > official > vetted ladder, coupling priced, KEEP / MIGRATE / RESEARCH verdict), exploring the incumbent first-hand in a fresh sub-agent. | -| `/planning:plan` | Implementation plan | Produces a structured plan (goal, approach, test strategy, blast radius, parallelism analysis, tagged unilateral decisions) with a fresh-context `plan-reviewer` stress-test, run by default on high-risk plans and on request and set per mode by `plan_stress_test`, and a user approval gate, persisted to PLAN.md. | +| `/planning:plan` | Implementation plan | Produces a structured plan (goal, approach, test strategy, blast radius, parallelism analysis, tagged unilateral decisions) with a fresh-context `plan-reviewer` stress-test, run by default on MEDIUM-or-higher plans and on request and set per mode by `plan_stress_test`, and a user approval gate, persisted to PLAN.md. | | `/planning:setup` | Configuration | `check` inspects the standards index read-only; `apply` interviews the consumer and bootstraps the standards index (idempotent; re-run to reconfigure). | The pipeline composes end-to-end. `wayfind` charts the fog upstream when an effort diff --git a/plugins/planning/reference/config.md b/plugins/planning/reference/config.md index b15852d2b99..51ca90316c7 100644 --- a/plugins/planning/reference/config.md +++ b/plugins/planning/reference/config.md @@ -166,9 +166,9 @@ layer. Its values: -- `auto`: the `plan-reviewer` agent when Step 3b's blast radius is HIGH or CRITICAL, a trigger - matches, or the user asks for a review; `/planning:devils-advocate` in plan-review mode when - Step 3b's blast radius is MEDIUM or higher or a trigger matches. +- `auto`: the `plan-reviewer` agent and `/planning:devils-advocate` in plan-review mode when Step + 3b's blast radius is MEDIUM or higher or a trigger matches; the `plan-reviewer` agent also when the + user asks for a review. A LOW plan with no trigger runs neither. - `reviewer`: the `plan-reviewer` agent only, whatever the blast radius. - `devils-advocate`: `/planning:devils-advocate` only, whatever the blast radius. - `both`: both, whatever the blast radius. diff --git a/plugins/planning/schemas/planning.schema.json b/plugins/planning/schemas/planning.schema.json index 2a98bc58d2e..4d23b764085 100644 --- a/plugins/planning/schemas/planning.schema.json +++ b/plugins/planning/schemas/planning.schema.json @@ -65,7 +65,7 @@ "default": 27 }, "plan_stress_test": { - "description": "Per mode of the draft, which stress-tests /planning:plan runs. auto: plan-reviewer when the blast radius is HIGH or CRITICAL, a stress-test trigger matches, or the user asks for a review; /planning:devils-advocate when the blast-radius assessment calls for it. reviewer: plan-reviewer only. devils-advocate: /planning:devils-advocate only. both: both, whatever the blast radius. Only /planning:plan reads this key: a direct /planning:devils-advocate request ignores it and runs the mode its own arguments select.", + "description": "Per mode of the draft, which stress-tests /planning:plan runs. auto: plan-reviewer and /planning:devils-advocate when the blast radius is MEDIUM or higher or a stress-test trigger matches, plan-reviewer also when the user asks for a review; a LOW plan with no trigger runs neither. reviewer: plan-reviewer only. devils-advocate: /planning:devils-advocate only. both: both, whatever the blast radius. Only /planning:plan reads this key: a direct /planning:devils-advocate request ignores it and runs the mode its own arguments select.", "type": "object", "additionalProperties": false, "properties": { diff --git a/plugins/planning/skills/plan/SKILL.md b/plugins/planning/skills/plan/SKILL.md index c6cc81ea43c..ffa488d6c4e 100644 --- a/plugins/planning/skills/plan/SKILL.md +++ b/plugins/planning/skills/plan/SKILL.md @@ -63,7 +63,7 @@ These hold after a compaction re-attach. Later steps say how to carry them out. **Verification.** These gates sit near the top so they stay inside the part of this skill that is re-attached after auto-compaction; `tests/reattach-slice.test.sh` stands in for that part with the first 20,000 bytes. How much is re-attached: [Skill content lifecycle](https://code.claude.com/docs/en/skills#skill-content-lifecycle), as of 2026-11-10; recheck when that section changes how much of a skill's invocation is re-attached, or stops describing a per-skill re-attach. -- **Reviewer.** Before presenting, dispatch a fresh-context plan-reviewer sub-agent only when the first matching rule says so: the `verification` level (Step 3) `light` or `off` skips it; `plan_stress_test` (Step 3) `devils-advocate` for the draft's mode skips it; `reviewer` or `both`, or the `verification` level `strict`, runs it; under `auto` it runs when the Step 3b blast radius is HIGH or CRITICAL, any stress-test trigger matches, or the user asks for a plan review or stress-test (a `/planning:plan review` re-plan included), and is skipped otherwise. A skip is stated, never filled by the producing thread self-attacking the plan inline. A change skipped under the one-sentence rule (Purpose) has no plan to review. +- **Reviewer.** Before presenting, dispatch a fresh-context plan-reviewer sub-agent only when the first matching rule says so: the `verification` level (Step 3) `light` or `off` skips it; `plan_stress_test` (Step 3) `devils-advocate` for the draft's mode skips it; `reviewer` or `both`, or the `verification` level `strict`, runs it; under `auto` it runs when the Step 3b blast radius is MEDIUM or higher, any stress-test trigger matches, or the user asks for a plan review or stress-test (a `/planning:plan review` re-plan included), and is skipped otherwise. A skip is stated, never filled by the producing thread self-attacking the plan inline. A change skipped under the one-sentence rule (Purpose) has no plan to review. - **Hard-to-reverse decisions escalate EARLY** regardless of confidence. Below-bar judgment calls go to an interview round before the plan locks. - **Agent teams.** Route a phase to an agent team only when the parallel-safe workers must message each other and agent teams are enabled; otherwise use sub-agent workers or sequential. Disjoint file ownership between teammates is mandatory; for why, see [agent teams: avoid file conflicts](https://code.claude.com/docs/en/agent-teams#avoid-file-conflicts) (as of 2026-11-01; recheck when that section changes how teammates share files). diff --git a/plugins/planning/skills/plan/context/stress-test-triggers.md b/plugins/planning/skills/plan/context/stress-test-triggers.md index 5442a69295d..b985c74d783 100644 --- a/plugins/planning/skills/plan/context/stress-test-triggers.md +++ b/plugins/planning/skills/plan/context/stress-test-triggers.md @@ -1,6 +1,6 @@ # Stress-Test Trigger Criteria -When to invoke `/planning:devils-advocate` on a plan under `plan_stress_test`'s `auto`, where a matching trigger, or a HIGH or CRITICAL blast radius, also runs the Step 3 `plan-reviewer`; the other values decide without these criteria (SKILL.md Step 3b). The goal is to catch plans that carry enough risk to warrant systematic adversarial review, without stress-testing every trivial change. +When to invoke `/planning:devils-advocate` on a plan under `plan_stress_test`'s `auto`, where a matching trigger, or a MEDIUM or higher blast radius, also runs the Step 3 `plan-reviewer`; the other values decide without these criteria (SKILL.md Step 3b). The goal is to catch plans that carry enough risk to warrant systematic adversarial review, without stress-testing every trivial change. ## Always stress-test when ANY of these match diff --git a/plugins/planning/skills/plan/templates/checklist.md b/plugins/planning/skills/plan/templates/checklist.md index 96c8d694a23..f6fb7e5864a 100644 --- a/plugins/planning/skills/plan/templates/checklist.md +++ b/plugins/planning/skills/plan/templates/checklist.md @@ -16,7 +16,7 @@ Copy into `//plan-checklist.md` (default `.work/`; the t ## Skip criteria -- Step 3's reviewer SKIPPED under `plan_stress_test`'s `devils-advocate` (Step 4 runs instead), under verification `light` or `off`, and under `auto` when Step 3b's blast radius is LOW or MEDIUM, no trigger matches, verification is not `strict` and the user did not ask for a review; tick Step 3 with the skip and its reason +- Step 3's reviewer SKIPPED under `plan_stress_test`'s `devils-advocate` (Step 4 runs instead), under verification `light` or `off`, and under `auto` when Step 3b's blast radius is LOW, no trigger matches, verification is not `strict` and the user did not ask for a review; tick Step 3 with the skip and its reason - Step 4 SKIPPED under `plan_stress_test`'s `reviewer`, and under `auto` when Step 3b verdict = LOW blast radius and no trigger matches - Step 4.5 SKIPPED when the plan is single-phase (no parallelism axis, all-main-session default) - Step 4.6 SKIPPED when no unilateral decisions were made (user approved every choice during Q&A) From cc4acc02ca985107674de979c3ef851453c5cfce Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Wed, 11 Nov 2026 18:31:20 -0500 Subject: [PATCH 7/7] test(planning): move the plan-reviewer-cut eval cases to their own PR 158 case files pushed this PR past one review's read budget, so both review lanes read it only in part. The cases land in a separate PR on branch test/plan-reviewer-cut-evals; this PR keeps the skill change and the plan-low-risk-skips-reviewer case. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VQb9Q4F46UumaUgXeUdpYZ --- .../fixtures/plan-reviewer-cut/init-repo.sh | 24 --- .../fixtures/plan-reviewer-cut/prepare-arm.sh | 45 ----- .../case.yaml | 4 - .../graders/behavior-is-a-claim.md | 9 - .../graders/blast-radius-low.md | 9 - .../graders/model-untouched.md | 10 -- .../graders/reuses-slugify.md | 8 - .../graders/reviewer-not-dispatched.md | 9 - .../prompt.md | 10 -- .../samples/behavior-is-a-claim.json | 14 -- .../samples/blast-radius-low.json | 22 --- .../samples/model-untouched.json | 14 -- .../samples/reuses-slugify.json | 14 -- .../samples/reviewer-not-dispatched.json | 54 ------ .../scaffold.sh | 116 ------------ .../case.yaml | 4 - .../graders/ac3-built.md | 8 - .../graders/assumption-row.md | 8 - .../graders/blast-radius-low.md | 9 - .../graders/no-new-dependency.md | 10 -- .../graders/reviewer-not-dispatched.md | 9 - .../prompt.md | 10 -- .../samples/ac3-built.json | 30 ---- .../samples/assumption-row.json | 14 -- .../samples/blast-radius-low.json | 22 --- .../samples/no-new-dependency.json | 14 -- .../samples/reviewer-not-dispatched.json | 54 ------ .../scaffold.sh | 105 ----------- .../case.yaml | 4 - .../graders/appetite-fit-line.md | 8 - .../graders/assumption-row.md | 8 - .../graders/blast-radius-low.md | 9 - .../graders/no-dateutil.md | 10 -- .../graders/reviewer-not-dispatched.md | 9 - .../prompt.md | 10 -- .../samples/appetite-fit-line.json | 14 -- .../samples/assumption-row.json | 14 -- .../samples/blast-radius-low.json | 22 --- .../samples/no-dateutil.json | 14 -- .../samples/reviewer-not-dispatched.json | 54 ------ .../scaffold.sh | 129 ------------- .../case.yaml | 4 - .../graders/blast-radius-low.md | 9 - .../graders/no-jitter.md | 10 -- .../graders/real-ceiling-constant.md | 8 - .../graders/regression-test-named.md | 8 - .../graders/reviewer-not-dispatched.md | 9 - .../prompt.md | 10 -- .../samples/blast-radius-low.json | 22 --- .../samples/no-jitter.json | 14 -- .../samples/real-ceiling-constant.json | 14 -- .../samples/regression-test-named.json | 22 --- .../samples/reviewer-not-dispatched.json | 54 ------ .../scaffold.sh | 97 ---------- .../case.yaml | 4 - .../graders/blast-radius-low.md | 9 - .../graders/no-decimal-type.md | 9 - .../graders/out-of-scope-cut.md | 9 - .../graders/reviewer-not-dispatched.md | 9 - .../graders/sanity-check-command.md | 9 - .../prompt.md | 10 -- .../samples/blast-radius-low.json | 22 --- .../samples/no-decimal-type.json | 14 -- .../samples/out-of-scope-cut.json | 22 --- .../samples/reviewer-not-dispatched.json | 54 ------ .../samples/sanity-check-command.json | 14 -- .../scaffold.sh | 88 --------- .../case.yaml | 4 - .../graders/behavior-is-a-claim.md | 9 - .../graders/blast-radius-low.md | 9 - .../graders/missing-file-handled.md | 8 - .../graders/no-cache.md | 10 -- .../graders/reviewer-not-dispatched.md | 9 - .../prompt.md | 10 -- .../samples/behavior-is-a-claim.json | 14 -- .../samples/blast-radius-low.json | 22 --- .../samples/missing-file-handled.json | 26 --- .../samples/no-cache.json | 14 -- .../samples/reviewer-not-dispatched.json | 54 ------ .../scaffold.sh | 105 ----------- .../case.yaml | 4 - .../graders/ac3-sort-built.md | 8 - .../graders/appetite-fit-line.md | 8 - .../graders/blast-radius-medium.md | 9 - .../graders/no-new-dependency.md | 10 -- .../graders/reviewer-not-dispatched.md | 9 - .../prompt.md | 10 -- .../samples/ac3-sort-built.json | 26 --- .../samples/appetite-fit-line.json | 14 -- .../samples/blast-radius-medium.json | 22 --- .../samples/no-new-dependency.json | 14 -- .../samples/reviewer-not-dispatched.json | 54 ------ .../scaffold.sh | 170 ------------------ .../case.yaml | 4 - .../graders/ac4-built.md | 8 - .../graders/assumption-row.md | 8 - .../graders/blast-radius-medium.md | 9 - .../graders/keeps-decimal-rounding.md | 10 -- .../graders/reviewer-not-dispatched.md | 9 - .../prompt.md | 10 -- .../samples/ac4-built.json | 30 ---- .../samples/assumption-row.json | 14 -- .../samples/blast-radius-medium.json | 22 --- .../samples/keeps-decimal-rounding.json | 14 -- .../samples/reviewer-not-dispatched.json | 54 ------ .../scaffold.sh | 158 ---------------- .../case.yaml | 4 - .../graders/assumption-row.md | 8 - .../graders/blast-radius-medium.md | 9 - .../graders/export-untouched.md | 10 -- .../graders/regression-test-named.md | 8 - .../graders/reviewer-not-dispatched.md | 9 - .../prompt.md | 10 -- .../samples/assumption-row.json | 14 -- .../samples/blast-radius-medium.json | 22 --- .../samples/export-untouched.json | 14 -- .../samples/regression-test-named.json | 14 -- .../samples/reviewer-not-dispatched.json | 54 ------ .../scaffold.sh | 146 --------------- .../case.yaml | 4 - .../graders/ac3-built.md | 8 - .../graders/behavior-is-a-claim.md | 9 - .../graders/blast-radius-medium.md | 9 - .../graders/keeps-outbox.md | 10 -- .../graders/reviewer-not-dispatched.md | 9 - .../prompt.md | 10 -- .../samples/ac3-built.json | 30 ---- .../samples/behavior-is-a-claim.json | 14 -- .../samples/blast-radius-medium.json | 22 --- .../samples/keeps-outbox.json | 14 -- .../samples/reviewer-not-dispatched.json | 54 ------ .../scaffold.sh | 137 -------------- .../case.yaml | 4 - .../graders/blast-radius-medium.md | 9 - .../graders/no-new-dependency.md | 10 -- .../graders/out-of-scope-cut.md | 9 - .../graders/reuses-available.md | 8 - .../graders/reviewer-not-dispatched.md | 9 - .../prompt.md | 10 -- .../samples/blast-radius-medium.json | 22 --- .../samples/no-new-dependency.json | 14 -- .../samples/out-of-scope-cut.json | 22 --- .../samples/reuses-available.json | 14 -- .../samples/reviewer-not-dispatched.json | 54 ------ .../scaffold.sh | 125 ------------- .../case.yaml | 4 - .../graders/blast-radius-medium.md | 9 - .../graders/callers-untouched.md | 10 -- .../graders/reuses-country-region.md | 8 - .../graders/reviewer-not-dispatched.md | 9 - .../graders/sanity-check-command.md | 9 - .../prompt.md | 10 -- .../samples/blast-radius-medium.json | 22 --- .../samples/callers-untouched.json | 14 -- .../samples/reuses-country-region.json | 14 -- .../samples/reviewer-not-dispatched.json | 54 ------ .../samples/sanity-check-command.json | 14 -- .../scaffold.sh | 146 --------------- 158 files changed, 3805 deletions(-) delete mode 100755 plugins/planning/evals/fixtures/plan-reviewer-cut/init-repo.sh delete mode 100755 plugins/planning/evals/fixtures/plan-reviewer-cut/prepare-arm.sh delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/case.yaml delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/behavior-is-a-claim.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/blast-radius-low.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/model-untouched.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reuses-slugify.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reviewer-not-dispatched.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/prompt.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/behavior-is-a-claim.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/blast-radius-low.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/model-untouched.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reuses-slugify.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reviewer-not-dispatched.json delete mode 100755 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/scaffold.sh delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/case.yaml delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/assumption-row.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/blast-radius-low.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/no-new-dependency.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/reviewer-not-dispatched.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/prompt.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/ac3-built.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/assumption-row.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/blast-radius-low.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/no-new-dependency.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/reviewer-not-dispatched.json delete mode 100755 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/scaffold.sh delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/case.yaml delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/appetite-fit-line.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/assumption-row.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/blast-radius-low.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/no-dateutil.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/reviewer-not-dispatched.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/prompt.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/appetite-fit-line.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/assumption-row.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/blast-radius-low.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/no-dateutil.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/reviewer-not-dispatched.json delete mode 100755 plugins/planning/evals/plan-reviewer-cut-low-notes-since/scaffold.sh delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/case.yaml delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/blast-radius-low.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/no-jitter.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/real-ceiling-constant.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/reviewer-not-dispatched.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/prompt.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/blast-radius-low.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/no-jitter.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/real-ceiling-constant.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/reviewer-not-dispatched.json delete mode 100755 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/scaffold.sh delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/case.yaml delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/blast-radius-low.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/no-decimal-type.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/reviewer-not-dispatched.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/sanity-check-command.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/prompt.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/blast-radius-low.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/no-decimal-type.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/out-of-scope-cut.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/reviewer-not-dispatched.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/sanity-check-command.json delete mode 100755 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/scaffold.sh delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/case.yaml delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/behavior-is-a-claim.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/blast-radius-low.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/missing-file-handled.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/no-cache.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/reviewer-not-dispatched.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/prompt.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/behavior-is-a-claim.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/blast-radius-low.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/missing-file-handled.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/no-cache.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/reviewer-not-dispatched.json delete mode 100755 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/scaffold.sh delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/case.yaml delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/ac3-sort-built.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/appetite-fit-line.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/blast-radius-medium.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/no-new-dependency.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/reviewer-not-dispatched.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/prompt.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/ac3-sort-built.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/appetite-fit-line.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/blast-radius-medium.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/no-new-dependency.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/reviewer-not-dispatched.json delete mode 100755 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/scaffold.sh delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/case.yaml delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/ac4-built.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/assumption-row.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/blast-radius-medium.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/keeps-decimal-rounding.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/reviewer-not-dispatched.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/prompt.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/ac4-built.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/assumption-row.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/blast-radius-medium.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/keeps-decimal-rounding.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/reviewer-not-dispatched.json delete mode 100755 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/scaffold.sh delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/case.yaml delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/assumption-row.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/blast-radius-medium.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/export-untouched.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/reviewer-not-dispatched.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/prompt.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/assumption-row.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/blast-radius-medium.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/export-untouched.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/reviewer-not-dispatched.json delete mode 100755 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/scaffold.sh delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/case.yaml delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/ac3-built.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/behavior-is-a-claim.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/blast-radius-medium.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/keeps-outbox.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/reviewer-not-dispatched.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/prompt.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/ac3-built.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/behavior-is-a-claim.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/blast-radius-medium.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/keeps-outbox.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/reviewer-not-dispatched.json delete mode 100755 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/scaffold.sh delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/case.yaml delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/blast-radius-medium.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/no-new-dependency.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/out-of-scope-cut.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reuses-available.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reviewer-not-dispatched.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/prompt.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/blast-radius-medium.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/no-new-dependency.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/out-of-scope-cut.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reuses-available.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reviewer-not-dispatched.json delete mode 100755 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/scaffold.sh delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/case.yaml delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/blast-radius-medium.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/callers-untouched.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reuses-country-region.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reviewer-not-dispatched.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/sanity-check-command.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/prompt.md delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/blast-radius-medium.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/callers-untouched.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reuses-country-region.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reviewer-not-dispatched.json delete mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/sanity-check-command.json delete mode 100755 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/scaffold.sh diff --git a/plugins/planning/evals/fixtures/plan-reviewer-cut/init-repo.sh b/plugins/planning/evals/fixtures/plan-reviewer-cut/init-repo.sh deleted file mode 100755 index da87d9d2f68..00000000000 --- a/plugins/planning/evals/fixtures/plan-reviewer-cut/init-repo.sh +++ /dev/null @@ -1,24 +0,0 @@ -#!/usr/bin/env bash -# Run by each plan-reviewer-cut-* scaffold after it writes the project files and the draft plan: -# commits the project on main, pushes it to a local bare origin, and moves to a feature branch. -# -# init-repo.sh -# -# Each draft's phase headings carry the placeholder [STATUS]; this script sets it to the -# not-started status marker the plan skill expects. -set -euo pipefail - -branch="$1" -printf '.work/\n__pycache__/\n' >.gitignore -for plan in .work/*/PLAN.md; do - text="$(cat "$plan")" - printf '%s\n' "${text//'[STATUS]'/[TODO]}" >"$plan" -done -git init -q -b main -git add . -git -c user.name=dev -c user.email=dev@example.invalid commit -q -m "chore: import project" -mkdir -p .work/origin.git -git init -q --bare -b main .work/origin.git -git remote add origin "$PWD/.work/origin.git" -git push -q origin main -git checkout -q -b "$branch" diff --git a/plugins/planning/evals/fixtures/plan-reviewer-cut/prepare-arm.sh b/plugins/planning/evals/fixtures/plan-reviewer-cut/prepare-arm.sh deleted file mode 100755 index 872e9c29fce..00000000000 --- a/plugins/planning/evals/fixtures/plan-reviewer-cut/prepare-arm.sh +++ /dev/null @@ -1,45 +0,0 @@ -#!/usr/bin/env bash -# Writes one arm of the plan-reviewer-cut suite: a copy of the planning plugin at , outside any -# repository, holding this checkout's plan-reviewer-cut cases. `claude plugin eval` then runs it -# with --ablation none. -# -# prepare-arm.sh -# -# is a commit of this repository (the before arm pins the PR's merge base). -# -# Exit: 0 written; 2 a usage error, or a dest that already exists. -set -euo pipefail - -usage() { - echo "usage: prepare-arm.sh " >&2 - exit 2 -} - -[[ $# -eq 2 ]] || usage -ref="$1" -dest="$2" -if [[ -e "$dest" ]]; then - echo "prepare-arm.sh: $dest already exists" >&2 - exit 2 -fi - -parent="$(dirname "$dest")" -if [[ ! -d "$parent" ]] || git -C "$parent" rev-parse --git-dir >/dev/null 2>&1; then - echo "prepare-arm.sh: $dest must sit in an existing directory outside any git repository" >&2 - exit 2 -fi - -here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -plugin="$(cd "$here/../../.." && pwd)" -repo="$(git -C "$plugin" rev-parse --show-toplevel)" -git -C "$repo" cat-file -e "$ref^{commit}" || usage - -mkdir -p "$dest" -git -C "$repo" archive "$ref" plugins/planning | tar -x -C "$dest" --strip-components=2 -rm -rf "${dest:?}/evals/results" -mkdir -p "$dest/evals/fixtures" -rm -rf "${dest:?}/evals/fixtures/plan-reviewer-cut" "$dest"/evals/plan-reviewer-cut-* -cp -R "$here" "$dest/evals/fixtures/plan-reviewer-cut" -cp -R "$plugin"/evals/plan-reviewer-cut-* "$dest/evals/" - -echo "$dest" diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/case.yaml b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/case.yaml deleted file mode 100644 index 61e68ab1a79..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/case.yaml +++ /dev/null @@ -1,4 +0,0 @@ -schema_version: "1.1" -name: plan-reviewer-cut-low-blog-permalink -context: - scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/behavior-is-a-claim.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/behavior-is-a-claim.md deleted file mode 100644 index a3312120a58..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/behavior-is-a-claim.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: '^\s*[-*]\s*B\d+:\s*(?:Edit|Update|Modify|Change|Add)\b' -flags: mi -match: not_contains -target: - source: file - path: .work/post-permalink/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/blast-radius-low.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/blast-radius-low.md deleted file mode 100644 index 4e4b39320a2..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/blast-radius-low.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b' -flags: i -target: - source: file - path: .work/post-permalink/PLAN.md -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/model-untouched.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/model-untouched.md deleted file mode 100644 index 2fe65e17da1..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/model-untouched.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -type: regex -pattern: '^\|\s*`?blog/models\.py`?\s*\|\s*(?:modify|create|edit)' -flags: mi -match: not_contains -weight: 0.001 -target: - source: file - path: .work/post-permalink/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reuses-slugify.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reuses-slugify.md deleted file mode 100644 index 21de46dc3d5..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reuses-slugify.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '\bslugify\b' -flags: i -target: - source: file - path: .work/post-permalink/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reviewer-not-dispatched.md deleted file mode 100644 index 1f7d67fa7b9..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reviewer-not-dispatched.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: tool_used -tool: Agent -input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' -min: 0 -max: 0 -arm: both -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/prompt.md deleted file mode 100644 index 51177f031a9..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/prompt.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -description: "LOW tier, no trigger: a draft adds post permalinks and writes its own slug helper although blog/text.py already has `slugify`, which AC2's accent folding needs (context/plan-reviewer.md: build what existing code already does). Also seeded: a Behavior item that names a file and a step instead of an observable claim (agents/plan-reviewer.md third check). hard: the duplicate is visible only by reading code the plan never cites. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." -tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] -runs: 3 -max_turns: 150 -timeout_seconds: 2700 -allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] -expected_outcome: "The final PLAN.md reuses blog.text.slugify, rewrites the file-naming Behavior item as a claim, and does not change blog/models.py; the arm's reviewer dispatch matches its contract." ---- -/planning:plan Pick up the draft plan at `.work/post-permalink/PLAN.md`. Its Brief, its phases and the design early exit (`.work/post-permalink/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/behavior-is-a-claim.json b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/behavior-is-a-claim.json deleted file mode 100644 index e1e411ba17d..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/behavior-is-a-claim.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: The slug is lowercase ASCII words with accents folded (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n", - "why": "the plan with the seeded defect fixed" - } - ], - "fail": [ - { - "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n", - "why": "the draft as seeded" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/blast-radius-low.json deleted file mode 100644 index f1eb1c1bdd9..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/blast-radius-low.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "Blast radius: LOW\nReason: two files.", - "why": "plain line" - }, - { - "answer": "**Blast radius:** LOW", - "why": "bold label" - } - ], - "fail": [ - { - "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n", - "why": "the draft as seeded has no assessment" - }, - { - "answer": "Blast radius: MEDIUM\nReason: shared code.", - "why": "another tier" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/model-untouched.json b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/model-untouched.json deleted file mode 100644 index 6f77d4bfdde..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/model-untouched.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n", - "why": "the draft as seeded keeps the sound part" - } - ], - "fail": [ - { - "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n| `blog/models.py` | modify |\n", - "why": "the plan made the over-flagged change" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reuses-slugify.json b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reuses-slugify.json deleted file mode 100644 index ae7329ce5ef..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reuses-slugify.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: The slug is lowercase ASCII words with accents folded (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, import `slugify` from `blog.text`, which already folds accents.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{slugify(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n", - "why": "the plan with the seeded defect fixed" - } - ], - "fail": [ - { - "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n", - "why": "the draft as seeded" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reviewer-not-dispatched.json deleted file mode 100644 index 86031c3df49..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reviewer-not-dispatched.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "pass": [ - { - "answer": [ - { - "tool": "Read", - "input": { - "file_path": ".work/post-permalink/PLAN.md" - } - } - ], - "why": "no agent dispatched" - }, - { - "answer": [ - { - "tool": "Skill", - "input": { - "skill": "planning:devils-advocate" - } - } - ], - "why": "devils-advocate only" - }, - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "general-purpose", - "description": "Stress-test", - "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." - } - } - ], - "why": "another agent whose prompt names the reviewer" - } - ], - "fail": [ - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "planning:plan-reviewer", - "description": "Review plan", - "prompt": "Review .work/post-permalink/PLAN.md" - } - } - ], - "why": "dispatched the plan-reviewer" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/scaffold.sh deleted file mode 100755 index e5ab7f6a532..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/scaffold.sh +++ /dev/null @@ -1,116 +0,0 @@ -#!/usr/bin/env bash -# A blog package with a shared slugify helper, and a draft plan for permalinks that writes its own. -set -euo pipefail - -mkdir -p .work/post-permalink/design blog tests -cat >README.md <<'EOF' -# blog - -Posts, listings and shared text helpers (blog/text.py). -EOF -: >blog/__init__.py -cat >blog/text.py <<'EOF' -"""Text helpers shared by the blog modules.""" -import re -import unicodedata - - -def slugify(title): - """Lowercase ASCII words joined by hyphens: 'Café Notes!' -> 'cafe-notes'.""" - ascii_title = unicodedata.normalize("NFKD", title).encode("ascii", "ignore").decode() - return "-".join(re.findall(r"[a-z0-9]+", ascii_title.lower())) -EOF -cat >blog/models.py <<'EOF' -from dataclasses import dataclass -from datetime import date - - -@dataclass -class Post: - title: str - body: str - published_on: date -EOF -cat >blog/posts.py <<'EOF' -"""Post listing helpers.""" - - -def recent(posts, limit=10): - return sorted(posts, key=lambda post: post.published_on, reverse=True)[:limit] -EOF -cat >tests/test_posts.py <<'EOF' -import unittest -from datetime import date - -from blog.models import Post -from blog.posts import recent - - -class RecentTest(unittest.TestCase): - def test_newest_first(self): - old = Post("a", "", date(2026, 1, 1)) - new = Post("b", "", date(2026, 2, 1)) - self.assertEqual(recent([old, new]), [new, old]) - - -if __name__ == "__main__": - unittest.main() -EOF -cat >.work/post-permalink/PLAN.md <<'EOF' -# Post permalinks - -## Brief - -### Problem -Posts have no stable address: the site links to them by list position, so links break when a post is added. - -### Goal -Each post has a permalink built from its publication date and title. - -### Acceptance criteria -- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded. -- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`. -- AC3: Two posts with the same title in different months get different permalinks. - -### Out-of-scope -- Redirects from the old list-position links. -- Custom slugs chosen per post. - -## Design - -Early exit (tier C): One new function over an existing dataclass; no new types. - -## Plan - -### Behavior -- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1. -- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1. - -### Load-bearing assumptions - -| Assumption | Source | Falsifying check | Response if false | -|---|---|---|---| -| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note | - -### Phase 1: Add permalink [STATUS] -**Satisfies:** AC1, AC2, AC3 -**Delivers:** B1, B2 - -- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`. -- Add `permalink(post)` returning `f"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/"`. -- In `tests/test_posts.py`, add one case per criterion. - -**Sanity Check:** `python -m unittest discover -s tests` exits 0. - -### Files affected -| File | Action | -|---|---| -| `blog/posts.py` | modify | -| `tests/test_posts.py` | modify | -EOF -cat >.work/post-permalink/design/design-resolution.md <<'EOF' -outcome: early-exit -tier: C -reason: One new function over an existing dataclass; no new types. -EOF -bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/post-permalink diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/case.yaml b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/case.yaml deleted file mode 100644 index c786263e163..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/case.yaml +++ /dev/null @@ -1,4 +0,0 @@ -schema_version: "1.1" -name: plan-reviewer-cut-low-ledger-delimiter -context: - scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md deleted file mode 100644 index 1bd8b345899..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '^###\s*Phase[\s\S]*?^\s*[-*]\s+[^\n]*(?:ValueError|unsupported delimiter|\brefus|\breject|\braise)' -flags: mi -target: - source: file - path: .work/export-delimiter/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/assumption-row.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/assumption-row.md deleted file mode 100644 index 9ae87fcafc6..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/assumption-row.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '###\s*Load-bearing assumptions[^\n]*\n(?:(?!#)[^\n]*\n)*?\|[^\n]*(?:semicolon|separator|spreadsheet|import)' -flags: i -target: - source: file - path: .work/export-delimiter/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/blast-radius-low.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/blast-radius-low.md deleted file mode 100644 index 05787b008a6..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/blast-radius-low.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b' -flags: i -target: - source: file - path: .work/export-delimiter/PLAN.md -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/no-new-dependency.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/no-new-dependency.md deleted file mode 100644 index 065d6e7ee20..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/no-new-dependency.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -type: regex -pattern: '^\|\s*`?(?:requirements[\w.-]*\.txt|pyproject\.toml|setup\.cfg|Pipfile)`?\s*\|' -flags: mi -match: not_contains -weight: 0.001 -target: - source: file - path: .work/export-delimiter/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/reviewer-not-dispatched.md deleted file mode 100644 index 1f7d67fa7b9..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/reviewer-not-dispatched.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: tool_used -tool: Agent -input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' -min: 0 -max: 0 -arm: both -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/prompt.md deleted file mode 100644 index 3113ec31ad4..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/prompt.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -description: "LOW tier, no trigger: a two-file draft adds a delimiter option to a CSV export. Seeded: a Brief captured assumption with no Load-bearing row (agents/plan-reviewer.md second check), and AC3 claimed by Phase 1 that no work item builds (the Delivers check). hard: both seeds pass check-plan-outcome.sh, so only a reader who compares the Brief with the phases catches them. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." -tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] -runs: 3 -max_turns: 150 -timeout_seconds: 2700 -allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] -expected_outcome: "The final PLAN.md carries a Load-bearing row for the semicolon-import assumption and a Phase 1 work item that refuses an unsupported delimiter with ValueError, and adds no dependency; the arm's reviewer dispatch matches its contract." ---- -/planning:plan Pick up the draft plan at `.work/export-delimiter/PLAN.md`. Its Brief, its phases and the design early exit (`.work/export-delimiter/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/ac3-built.json b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/ac3-built.json deleted file mode 100644 index 141c0d9a301..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/ac3-built.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- Raise `ValueError` when `delimiter` is not `,` or `;`, before `open`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n", - "why": "the plan with the seeded defect fixed" - }, - { - "answer": "### Phase 2: Next step [TODO]\n- Reject any delimiter outside `,` and `;` before opening the file.\n", - "why": "reject paraphrase" - }, - { - "answer": "### Phase 2: Next step [TODO]\n- Raise when `delimiter not in (\",\", \";\")`.\n", - "why": "raise paraphrase" - } - ], - "fail": [ - { - "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n", - "why": "the draft as seeded" - }, - { - "answer": "### Phase 2: Next step [TODO]\n- Pass `delimiter` to `csv.writer`.\n", - "why": "no validation step" - }, - { - "answer": "- B3: An unsupported delimiter is refused (AC3).\n### Phase 2: Next step [TODO]\n- Pass `delimiter` to `csv.writer`.\n", - "why": "only the Behavior item mentions it" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/assumption-row.json b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/assumption-row.json deleted file mode 100644 index 8689092a92d..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/assumption-row.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Finance's spreadsheet imports semicolon-separated files with no hint line | Brief | Phase 1 Sanity Check | Moderate: flag and continue |\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n", - "why": "the plan with the seeded defect fixed" - } - ], - "fail": [ - { - "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n", - "why": "the draft as seeded" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/blast-radius-low.json deleted file mode 100644 index d15f46ec0d4..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/blast-radius-low.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "Blast radius: LOW\nReason: two files.", - "why": "plain line" - }, - { - "answer": "**Blast radius:** LOW", - "why": "bold label" - } - ], - "fail": [ - { - "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n", - "why": "the draft as seeded has no assessment" - }, - { - "answer": "Blast radius: MEDIUM\nReason: shared code.", - "why": "another tier" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/no-new-dependency.json b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/no-new-dependency.json deleted file mode 100644 index abb4884c5d1..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/no-new-dependency.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n", - "why": "the draft as seeded keeps the sound part" - } - ], - "fail": [ - { - "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n| `requirements.txt` | modify |\n", - "why": "the plan made the over-flagged change" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/reviewer-not-dispatched.json deleted file mode 100644 index 492c1e86a57..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/reviewer-not-dispatched.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "pass": [ - { - "answer": [ - { - "tool": "Read", - "input": { - "file_path": ".work/export-delimiter/PLAN.md" - } - } - ], - "why": "no agent dispatched" - }, - { - "answer": [ - { - "tool": "Skill", - "input": { - "skill": "planning:devils-advocate" - } - } - ], - "why": "devils-advocate only" - }, - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "general-purpose", - "description": "Stress-test", - "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." - } - } - ], - "why": "another agent whose prompt names the reviewer" - } - ], - "fail": [ - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "planning:plan-reviewer", - "description": "Review plan", - "prompt": "Review .work/export-delimiter/PLAN.md" - } - } - ], - "why": "dispatched the plan-reviewer" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/scaffold.sh deleted file mode 100755 index 2a2875cdcc5..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/scaffold.sh +++ /dev/null @@ -1,105 +0,0 @@ -#!/usr/bin/env bash -# A small ledger package whose CSV export takes no delimiter, and a draft plan to add one. -set -euo pipefail - -mkdir -p .work/export-delimiter/design ledger tests -cat >README.md <<'EOF' -# ledger - -Ledger rows and the CSV export finance opens in a spreadsheet. -EOF -: >ledger/__init__.py -cat >ledger/export.py <<'EOF' -"""Write ledger rows to CSV for the finance spreadsheet.""" -import csv - -HEADER = ["date", "account", "amount"] - - -def export_csv(rows, path): - with open(path, "w", newline="", encoding="utf-8") as handle: - writer = csv.writer(handle) - writer.writerow(HEADER) - for row in rows: - writer.writerow([row["date"], row["account"], row["amount"]]) -EOF -cat >tests/test_export.py <<'EOF' -import os -import tempfile -import unittest - -from ledger.export import export_csv - - -class ExportTest(unittest.TestCase): - def test_default_is_comma(self): - path = os.path.join(tempfile.mkdtemp(), "out.csv") - export_csv([{"date": "2026-03-01", "account": "4000", "amount": "12.50"}], path) - with open(path, encoding="utf-8") as handle: - self.assertEqual(handle.read().splitlines(), ["date,account,amount", "2026-03-01,4000,12.50"]) - - -if __name__ == "__main__": - unittest.main() -EOF -cat >.work/export-delimiter/PLAN.md <<'EOF' -# Export delimiter - -## Brief - -### Problem -Finance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column. - -### Goal -`export_csv` takes an optional delimiter, so those staff get one value per column. - -### Acceptance criteria -- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today. -- AC2: `export_csv(rows, path, delimiter=";")` writes every row, header included, with semicolons between fields. -- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened. - -### Out-of-scope -- Changing the column set or order. -- Spreadsheet (.xlsx) output. - -### Captured assumptions -- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column. - -## Design - -Early exit (tier C): One optional parameter on one existing function; no new types. - -## Plan - -### Behavior -- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1. -- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1. -- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1. - -### Load-bearing assumptions - -| Assumption | Source | Falsifying check | Response if false | -|---|---|---|---| -| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note | - -### Phase 1: Add a delimiter option [STATUS] -**Satisfies:** AC1, AC2, AC3 -**Delivers:** B1, B2, B3 - -- In `ledger/export.py`, add `delimiter=","` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`. -- In `tests/test_export.py`, add a case that writes two rows with `delimiter=";"` and reads the file back line by line. - -**Sanity Check:** `python -m unittest discover -s tests` exits 0. - -### Files affected -| File | Action | -|---|---| -| `ledger/export.py` | modify | -| `tests/test_export.py` | modify | -EOF -cat >.work/export-delimiter/design/design-resolution.md <<'EOF' -outcome: early-exit -tier: C -reason: One optional parameter on one existing function; no new types. -EOF -bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/export-delimiter diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/case.yaml b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/case.yaml deleted file mode 100644 index 018be92f519..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/case.yaml +++ /dev/null @@ -1,4 +0,0 @@ -schema_version: "1.1" -name: plan-reviewer-cut-low-notes-since -context: - scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/appetite-fit-line.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/appetite-fit-line.md deleted file mode 100644 index a778eb9cf9c..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/appetite-fit-line.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: 'appetite fit' -flags: i -target: - source: file - path: .work/notes-since/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/assumption-row.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/assumption-row.md deleted file mode 100644 index df70dbaece4..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/assumption-row.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '###\s*Load-bearing assumptions[^\n]*\n(?:(?!#)[^\n]*\n)*?\|[^\n]*(?:ISO|8601|YYYY|created|mobile|date format)' -flags: i -target: - source: file - path: .work/notes-since/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/blast-radius-low.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/blast-radius-low.md deleted file mode 100644 index 320a897819e..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/blast-radius-low.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b' -flags: i -target: - source: file - path: .work/notes-since/PLAN.md -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/no-dateutil.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/no-dateutil.md deleted file mode 100644 index 1711b43e365..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/no-dateutil.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -type: regex -pattern: 'dateutil' -flags: i -match: not_contains -weight: 0.001 -target: - source: file - path: .work/notes-since/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/reviewer-not-dispatched.md deleted file mode 100644 index 1f7d67fa7b9..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/reviewer-not-dispatched.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: tool_used -tool: Agent -input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' -min: 0 -max: 0 -arm: both -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/prompt.md deleted file mode 100644 index cf629502818..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/prompt.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -description: "LOW tier, no trigger: a draft adds a --since filter to a notes CLI. Seeded: the Brief records an appetite but the plan has no Appetite fit line (agents/plan-reviewer.md appetite check), and a Brief captured assumption about the date format has no Load-bearing row. hard: neither is a script criterion, and a run that resumes at the review step skips Step 2, where the main thread would write the line. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." -tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] -runs: 3 -max_turns: 150 -timeout_seconds: 2700 -allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] -expected_outcome: "The final PLAN.md has an Appetite fit line and a Load-bearing row for the ISO date assumption, and adds no dateutil dependency; the arm's reviewer dispatch matches its contract." ---- -/planning:plan Pick up the draft plan at `.work/notes-since/PLAN.md`. Its Brief, its phases and the design early exit (`.work/notes-since/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/appetite-fit-line.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/appetite-fit-line.json deleted file mode 100644 index 52d3dd18d9f..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/appetite-fit-line.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\nAppetite fit: one flag and one filter on one subcommand fits half a day.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", - "why": "the plan with the seeded defect fixed" - } - ], - "fail": [ - { - "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", - "why": "the draft as seeded" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/assumption-row.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/assumption-row.json deleted file mode 100644 index deabaa3afc6..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/assumption-row.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Note `created` values are ISO 8601 dates | Brief | Phase 1 Sanity Check | Major: stop and replan |\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", - "why": "the plan with the seeded defect fixed" - } - ], - "fail": [ - { - "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", - "why": "the draft as seeded" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/blast-radius-low.json deleted file mode 100644 index 7f4c110048f..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/blast-radius-low.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "Blast radius: LOW\nReason: two files.", - "why": "plain line" - }, - { - "answer": "**Blast radius:** LOW", - "why": "bold label" - } - ], - "fail": [ - { - "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", - "why": "the draft as seeded has no assessment" - }, - { - "answer": "Blast radius: MEDIUM\nReason: shared code.", - "why": "another tier" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/no-dateutil.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/no-dateutil.json deleted file mode 100644 index 200422d9f9d..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/no-dateutil.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", - "why": "the draft as seeded keeps the sound part" - } - ], - "fail": [ - { - "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=dateutil.parser.parse`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", - "why": "the plan made the over-flagged change" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/reviewer-not-dispatched.json deleted file mode 100644 index 0760295f5a8..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/reviewer-not-dispatched.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "pass": [ - { - "answer": [ - { - "tool": "Read", - "input": { - "file_path": ".work/notes-since/PLAN.md" - } - } - ], - "why": "no agent dispatched" - }, - { - "answer": [ - { - "tool": "Skill", - "input": { - "skill": "planning:devils-advocate" - } - } - ], - "why": "devils-advocate only" - }, - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "general-purpose", - "description": "Stress-test", - "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." - } - } - ], - "why": "another agent whose prompt names the reviewer" - } - ], - "fail": [ - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "planning:plan-reviewer", - "description": "Review plan", - "prompt": "Review .work/notes-since/PLAN.md" - } - } - ], - "why": "dispatched the plan-reviewer" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/scaffold.sh deleted file mode 100755 index 7740f7fa6a7..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/scaffold.sh +++ /dev/null @@ -1,129 +0,0 @@ -#!/usr/bin/env bash -# A notes CLI whose list command has no date filter, and a draft plan to add --since. -set -euo pipefail - -mkdir -p .work/notes-since/design notes tests -cat >README.md <<'EOF' -# notes - -A small notes CLI: `python -m notes.cli list`. -EOF -: >notes/__init__.py -cat >notes/store.py <<'EOF' -"""Notes kept in notes.json as a list of {"created": "YYYY-MM-DD", "text": ...}.""" -import json -from pathlib import Path - -STORE = Path("notes.json") - - -def list_notes(path=STORE): - if not path.exists(): - return [] - return json.loads(path.read_text(encoding="utf-8")) -EOF -cat >notes/cli.py <<'EOF' -import argparse - -from notes import store - - -def main(argv=None): - parser = argparse.ArgumentParser(prog="notes") - sub = parser.add_subparsers(dest="command", required=True) - sub.add_parser("list") - args = parser.parse_args(argv) - if args.command == "list": - for note in store.list_notes(): - print(note["created"], note["text"]) - - -if __name__ == "__main__": - main() -EOF -cat >tests/test_cli.py <<'EOF' -import io -import unittest -from contextlib import redirect_stdout -from unittest import mock - -from notes import cli - - -class ListTest(unittest.TestCase): - def test_lists_all(self): - notes = [{"created": "2026-03-01", "text": "a"}] - out = io.StringIO() - with mock.patch("notes.store.list_notes", return_value=notes), redirect_stdout(out): - cli.main(["list"]) - self.assertEqual(out.getvalue(), "2026-03-01 a\n") - - -if __name__ == "__main__": - unittest.main() -EOF -cat >.work/notes-since/PLAN.md <<'EOF' -# notes list --since - -## Brief - -### Problem -`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's. - -### Goal -`notes list --since DATE` shows only notes created on or after DATE. - -### Acceptance criteria -- AC1: `notes list` without `--since` prints what it prints today. -- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date. -- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag. - -### Out-of-scope -- Relative dates such as "last week". -- An `--until` flag. - -### Work size -Appetite: half a day. - -### Captured assumptions -- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format. - -## Design - -Early exit (tier C): One flag on an existing subcommand; no new types. - -## Plan - -### Behavior -- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1. -- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1. -- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1. - -### Load-bearing assumptions - -| Assumption | Source | Falsifying check | Response if false | -|---|---|---|---| -| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note | - -### Phase 1: Add --since [STATUS] -**Satisfies:** AC1, AC2, AC3 -**Delivers:** B1, B2, B3 - -- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`. -- Keep only notes whose `date.fromisoformat(note["created"])` is on or after the value. -- In `tests/test_cli.py`, add one case per criterion. - -**Sanity Check:** `python -m unittest discover -s tests` exits 0. - -### Files affected -| File | Action | -|---|---| -| `notes/cli.py` | modify | -| `tests/test_cli.py` | modify | -EOF -cat >.work/notes-since/design/design-resolution.md <<'EOF' -outcome: early-exit -tier: C -reason: One flag on an existing subcommand; no new types. -EOF -bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/notes-since diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/case.yaml b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/case.yaml deleted file mode 100644 index c149fddff97..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/case.yaml +++ /dev/null @@ -1,4 +0,0 @@ -schema_version: "1.1" -name: plan-reviewer-cut-low-retry-backoff-cap -context: - scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/blast-radius-low.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/blast-radius-low.md deleted file mode 100644 index 03849010240..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/blast-radius-low.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b' -flags: i -target: - source: file - path: .work/backoff-cap/PLAN.md -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/no-jitter.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/no-jitter.md deleted file mode 100644 index ae714377cab..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/no-jitter.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -type: regex -pattern: 'import random|random\.(?:uniform|random|randint)' -flags: i -match: not_contains -weight: 0.001 -target: - source: file - path: .work/backoff-cap/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/real-ceiling-constant.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/real-ceiling-constant.md deleted file mode 100644 index 7c157b61ad0..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/real-ceiling-constant.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: 'MAX_BACKOFF_SECONDS' -flags: i -target: - source: file - path: .work/backoff-cap/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md deleted file mode 100644 index eeecbed8589..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '^## Plan[\s\S]*?(?:regression|\btest_\w+|\btests?\b)' -flags: mi -target: - source: file - path: .work/backoff-cap/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/reviewer-not-dispatched.md deleted file mode 100644 index 1f7d67fa7b9..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/reviewer-not-dispatched.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: tool_used -tool: Agent -input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' -min: 0 -max: 0 -arm: both -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/prompt.md deleted file mode 100644 index a3014faa15e..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/prompt.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -description: "LOW tier, no trigger: a bug-fix draft caps retry backoff in one module. Seeded: it names a config constant `MAX_DELAY` that fetcher/config.py does not define (the ceiling is `MAX_BACKOFF_SECONDS`), and the bug fix names no regression test (context/plan-reviewer.md Design alignment). hard: the author believes its own constant name, and the mechanical gate checks neither. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." -tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] -runs: 3 -max_turns: 150 -timeout_seconds: 2700 -allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] -expected_outcome: "The final PLAN.md uses MAX_BACKOFF_SECONDS for the ceiling and names a regression test for the uncapped delay, and adds no jitter; the arm's reviewer dispatch matches its contract." ---- -/planning:plan Pick up the draft plan at `.work/backoff-cap/PLAN.md`. Its Brief, its phases and the design early exit (`.work/backoff-cap/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/blast-radius-low.json deleted file mode 100644 index b111684f896..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/blast-radius-low.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "Blast radius: LOW\nReason: two files.", - "why": "plain line" - }, - { - "answer": "**Blast radius:** LOW", - "why": "bold label" - } - ], - "fail": [ - { - "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", - "why": "the draft as seeded has no assessment" - }, - { - "answer": "Blast radius: MEDIUM\nReason: shared code.", - "why": "another tier" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/no-jitter.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/no-jitter.json deleted file mode 100644 index ed62bec4877..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/no-jitter.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", - "why": "the draft as seeded keeps the sound part" - } - ], - "fail": [ - { - "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Add `random.uniform(0, 1)` seconds to each delay.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", - "why": "the plan made the over-flagged change" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/real-ceiling-constant.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/real-ceiling-constant.json deleted file mode 100644 index 9fa76210a60..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/real-ceiling-constant.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_BACKOFF_SECONDS)`; `MAX_BACKOFF_SECONDS` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", - "why": "the plan with the seeded defect fixed" - } - ], - "fail": [ - { - "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", - "why": "the draft as seeded" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json deleted file mode 100644 index 34790649a43..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- In `tests/test_retry.py`, add a regression case: `backoff(8)` returns 30.0.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", - "why": "the plan with the seeded defect fixed" - }, - { - "answer": "## Plan\n- Add a unit test asserting `backoff(8) == 30.0`.\n", - "why": "test without the word regression" - } - ], - "fail": [ - { - "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", - "why": "the draft as seeded" - }, - { - "answer": "## Plan\n- Bound the delay with `min`.\n", - "why": "no test" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/reviewer-not-dispatched.json deleted file mode 100644 index c2529d5c8b5..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/reviewer-not-dispatched.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "pass": [ - { - "answer": [ - { - "tool": "Read", - "input": { - "file_path": ".work/backoff-cap/PLAN.md" - } - } - ], - "why": "no agent dispatched" - }, - { - "answer": [ - { - "tool": "Skill", - "input": { - "skill": "planning:devils-advocate" - } - } - ], - "why": "devils-advocate only" - }, - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "general-purpose", - "description": "Stress-test", - "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." - } - } - ], - "why": "another agent whose prompt names the reviewer" - } - ], - "fail": [ - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "planning:plan-reviewer", - "description": "Review plan", - "prompt": "Review .work/backoff-cap/PLAN.md" - } - } - ], - "why": "dispatched the plan-reviewer" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/scaffold.sh deleted file mode 100755 index cebd79632a0..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/scaffold.sh +++ /dev/null @@ -1,97 +0,0 @@ -#!/usr/bin/env bash -# A fetcher package whose retry backoff grows past the configured ceiling, and a draft bug-fix plan. -set -euo pipefail - -mkdir -p .work/backoff-cap/design fetcher tests -cat >README.md <<'EOF' -# fetcher - -HTTP fetcher with retries. Tunables live in fetcher/config.py. -EOF -: >fetcher/__init__.py -cat >fetcher/config.py <<'EOF' -"""Tunables for the HTTP fetcher.""" -BASE_DELAY_SECONDS = 0.5 -MAX_BACKOFF_SECONDS = 30.0 -MAX_ATTEMPTS = 8 -EOF -cat >fetcher/retry.py <<'EOF' -"""Retry delays for failed fetches.""" -from fetcher import config - - -def backoff(attempt): - """Seconds to wait before retry number `attempt` (1-based).""" - return config.BASE_DELAY_SECONDS * 2 ** (attempt - 1) -EOF -cat >tests/test_retry.py <<'EOF' -import unittest - -from fetcher.retry import backoff - - -class BackoffTest(unittest.TestCase): - def test_doubles(self): - self.assertEqual([backoff(n) for n in (1, 2, 3)], [0.5, 1.0, 2.0]) - - -if __name__ == "__main__": - unittest.main() -EOF -cat >.work/backoff-cap/PLAN.md <<'EOF' -# Cap retry backoff - -## Brief - -### Problem -With eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes. - -### Goal -No retry waits longer than the configured ceiling. - -### Acceptance criteria -- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling. -- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more. -- AC3: `backoff(0)` and negative attempts raise `ValueError`. - -### Out-of-scope -- Adding jitter. -- Changing the number of attempts. - -## Design - -Early exit (tier C): A bound inside one existing function; no new types. - -## Plan - -### Behavior -- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1. -- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1. -- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1. - -### Load-bearing assumptions - -| Assumption | Source | Falsifying check | Response if false | -|---|---|---|---| -| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note | - -### Phase 1: Bound the delay [STATUS] -**Satisfies:** AC1, AC2, AC3 -**Delivers:** B1, B2, B3 - -- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling. -- Raise `ValueError` at the top of `backoff` when `attempt` is below 1. - -**Sanity Check:** `python -c "from fetcher.retry import backoff; assert backoff(8) <= 30"` exits 0. - -### Files affected -| File | Action | -|---|---| -| `fetcher/retry.py` | modify | -EOF -cat >.work/backoff-cap/design/design-resolution.md <<'EOF' -outcome: early-exit -tier: C -reason: A bound inside one existing function; no new types. -EOF -bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" fix/backoff-cap diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/case.yaml b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/case.yaml deleted file mode 100644 index 1143281a83a..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/case.yaml +++ /dev/null @@ -1,4 +0,0 @@ -schema_version: "1.1" -name: plan-reviewer-cut-low-temperature-decimals -context: - scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/blast-radius-low.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/blast-radius-low.md deleted file mode 100644 index de151c94dad..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/blast-radius-low.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b' -flags: i -target: - source: file - path: .work/temp-decimals/PLAN.md -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/no-decimal-type.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/no-decimal-type.md deleted file mode 100644 index b726461ee3a..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/no-decimal-type.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: '\bDecimal\b' -match: not_contains -weight: 0.001 -target: - source: file - path: .work/temp-decimals/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md deleted file mode 100644 index df01edc1a71..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: '^\s*[-*]\s+(?![^\n]*\b(?:out[ -]of[ -]scope|not|no|never|without|leave|leaves|unchanged|drop\w*|cut|remov\w*|exclude|excluded|excludes|excluding\w*)\b)[^\n]*(?:fahrenheit|unit\s*=|°F)' -flags: mi -match: not_contains -target: - source: file - path: .work/temp-decimals/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/reviewer-not-dispatched.md deleted file mode 100644 index 1f7d67fa7b9..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/reviewer-not-dispatched.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: tool_used -tool: Agent -input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' -min: 0 -max: 0 -arm: both -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/sanity-check-command.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/sanity-check-command.md deleted file mode 100644 index 752cff52031..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/sanity-check-command.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: 'Sanity Check[^\n]*\b(?:looks? (?:right|correct|good|fine)|as expected|works (?:correctly|properly))' -flags: i -match: not_contains -target: - source: file - path: .work/temp-decimals/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/prompt.md deleted file mode 100644 index 8b244f0d782..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/prompt.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -description: "LOW tier, no trigger: a draft adds a decimals option to a temperature formatter. Seeded: a work item adds a unit switch the Brief's Out-of-scope excludes (context/plan-reviewer.md Plan mechanics), and the phase's Sanity Check is a vague `looks right` the script counts as present. routine for the out-of-scope seed, which Step 4.7's reading check also covers; hard for the vague Sanity Check. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." -tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] -runs: 3 -max_turns: 150 -timeout_seconds: 2700 -allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] -expected_outcome: "The final PLAN.md drops the unit switch, replaces the vague Sanity Check with a command, and does not switch to Decimal; the arm's reviewer dispatch matches its contract." ---- -/planning:plan Pick up the draft plan at `.work/temp-decimals/PLAN.md`. Its Brief, its phases and the design early exit (`.work/temp-decimals/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/blast-radius-low.json deleted file mode 100644 index 6e74cd6ff9a..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/blast-radius-low.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "Blast radius: LOW\nReason: two files.", - "why": "plain line" - }, - { - "answer": "**Blast radius:** LOW", - "why": "bold label" - } - ], - "fail": [ - { - "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, return `f\"{celsius:.{digits}f}°C\"`, and raise `ValueError` when it is below 0.\n- Add `unit=\"C\"` and convert to Fahrenheit when `unit=\"F\"`, so the panel can switch later.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** the formatted output looks right on the dashboard.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", - "why": "the draft as seeded has no assessment" - }, - { - "answer": "Blast radius: MEDIUM\nReason: shared code.", - "why": "another tier" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/no-decimal-type.json b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/no-decimal-type.json deleted file mode 100644 index 1f05a1b730a..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/no-decimal-type.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, return `f\"{celsius:.{digits}f}°C\"`, and raise `ValueError` when it is below 0.\n- Add `unit=\"C\"` and convert to Fahrenheit when `unit=\"F\"`, so the panel can switch later.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** the formatted output looks right on the dashboard.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", - "why": "the draft as seeded keeps the sound part" - } - ], - "fail": [ - { - "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, quantize a `Decimal` of the value, and raise `ValueError` when it is below 0.\n- Add `unit=\"C\"` and convert to Fahrenheit when `unit=\"F\"`, so the panel can switch later.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** the formatted output looks right on the dashboard.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", - "why": "the plan made the over-flagged change" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/out-of-scope-cut.json b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/out-of-scope-cut.json deleted file mode 100644 index a80fbe72be8..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/out-of-scope-cut.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, return `f\"{celsius:.{digits}f}°C\"`, and raise `ValueError` when it is below 0.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** the formatted output looks right on the dashboard.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", - "why": "the plan with the seeded defect fixed" - }, - { - "answer": "- Fahrenheit output dropped: out of scope.\n", - "why": "names it while cutting it" - } - ], - "fail": [ - { - "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, return `f\"{celsius:.{digits}f}°C\"`, and raise `ValueError` when it is below 0.\n- Add `unit=\"C\"` and convert to Fahrenheit when `unit=\"F\"`, so the panel can switch later.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** the formatted output looks right on the dashboard.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", - "why": "the draft as seeded" - }, - { - "answer": "- Support Fahrenheit through a `unit=` argument.\n", - "why": "paraphrased scope creep" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/reviewer-not-dispatched.json deleted file mode 100644 index 614f6243110..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/reviewer-not-dispatched.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "pass": [ - { - "answer": [ - { - "tool": "Read", - "input": { - "file_path": ".work/temp-decimals/PLAN.md" - } - } - ], - "why": "no agent dispatched" - }, - { - "answer": [ - { - "tool": "Skill", - "input": { - "skill": "planning:devils-advocate" - } - } - ], - "why": "devils-advocate only" - }, - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "general-purpose", - "description": "Stress-test", - "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." - } - } - ], - "why": "another agent whose prompt names the reviewer" - } - ], - "fail": [ - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "planning:plan-reviewer", - "description": "Review plan", - "prompt": "Review .work/temp-decimals/PLAN.md" - } - } - ], - "why": "dispatched the plan-reviewer" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/sanity-check-command.json b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/sanity-check-command.json deleted file mode 100644 index 2b927d5093a..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/sanity-check-command.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, return `f\"{celsius:.{digits}f}°C\"`, and raise `ValueError` when it is below 0.\n- Add `unit=\"C\"` and convert to Fahrenheit when `unit=\"F\"`, so the panel can switch later.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", - "why": "the plan with the seeded defect fixed" - } - ], - "fail": [ - { - "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, return `f\"{celsius:.{digits}f}°C\"`, and raise `ValueError` when it is below 0.\n- Add `unit=\"C\"` and convert to Fahrenheit when `unit=\"F\"`, so the panel can switch later.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** the formatted output looks right on the dashboard.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", - "why": "the draft as seeded" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/scaffold.sh deleted file mode 100755 index a279cb20c6a..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/scaffold.sh +++ /dev/null @@ -1,88 +0,0 @@ -#!/usr/bin/env bash -# A weather formatter that rounds to whole degrees, and a draft plan to show decimals. -set -euo pipefail - -mkdir -p .work/temp-decimals/design tests weather -cat >README.md <<'EOF' -# weather - -Formatting helpers for the greenhouse dashboard. -EOF -: >weather/__init__.py -cat >weather/format.py <<'EOF' -def format_temp(celsius): - return f"{round(celsius)}°C" -EOF -cat >tests/test_format.py <<'EOF' -import unittest - -from weather.format import format_temp - - -class FormatTest(unittest.TestCase): - def test_whole_degrees(self): - self.assertEqual(format_temp(21.6), "22°C") - - -if __name__ == "__main__": - unittest.main() -EOF -cat >.work/temp-decimals/PLAN.md <<'EOF' -# Temperature decimals - -## Brief - -### Problem -The greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift. - -### Goal -`format_temp` can show a fixed number of decimals. - -### Acceptance criteria -- AC1: `format_temp(c)` with no `digits` returns what it returns today. -- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`. -- AC3: A negative `digits` raises `ValueError`. - -### Out-of-scope -- Units other than Celsius. -- Localized decimal separators. - -## Design - -Early exit (tier C): One optional parameter on one existing function; no new types. - -## Plan - -### Behavior -- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1. -- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1. -- B3: A negative `digits` is refused (AC3). Delivered by Phase 1. - -### Load-bearing assumptions - -| Assumption | Source | Falsifying check | Response if false | -|---|---|---|---| -| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note | - -### Phase 1: Add digits [STATUS] -**Satisfies:** AC1, AC2, AC3 -**Delivers:** B1, B2, B3 - -- In `weather/format.py`, add `digits=None`; when set, return `f"{celsius:.{digits}f}°C"`, and raise `ValueError` when it is below 0. -- Add `unit="C"` and convert to Fahrenheit when `unit="F"`, so the panel can switch later. -- In `tests/test_format.py`, add one case per criterion. - -**Sanity Check:** the formatted output looks right on the dashboard. - -### Files affected -| File | Action | -|---|---| -| `weather/format.py` | modify | -| `tests/test_format.py` | modify | -EOF -cat >.work/temp-decimals/design/design-resolution.md <<'EOF' -outcome: early-exit -tier: C -reason: One optional parameter on one existing function; no new types. -EOF -bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/temp-decimals diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/case.yaml b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/case.yaml deleted file mode 100644 index 6fb6a7a9488..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/case.yaml +++ /dev/null @@ -1,4 +0,0 @@ -schema_version: "1.1" -name: plan-reviewer-cut-low-till-price-lookup -context: - scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/behavior-is-a-claim.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/behavior-is-a-claim.md deleted file mode 100644 index da60f2a2ed1..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/behavior-is-a-claim.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: '^\s*[-*]\s*B\d+:\s*(?:Edit|Update|Modify|Change|Add)\b' -flags: mi -match: not_contains -target: - source: file - path: .work/price-lookup/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/blast-radius-low.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/blast-radius-low.md deleted file mode 100644 index c26a8c96bdf..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/blast-radius-low.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b' -flags: i -target: - source: file - path: .work/price-lookup/PLAN.md -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/missing-file-handled.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/missing-file-handled.md deleted file mode 100644 index 65c4f088af4..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/missing-file-handled.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: 'FileNotFoundError|OSError|\.exists\(\)|\.is_file\(\)' -flags: i -target: - source: file - path: .work/price-lookup/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/no-cache.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/no-cache.md deleted file mode 100644 index e2b0f063e2c..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/no-cache.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -type: regex -pattern: 'lru_cache|functools\.cache|@cache\b' -flags: i -match: not_contains -weight: 0.001 -target: - source: file - path: .work/price-lookup/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/reviewer-not-dispatched.md deleted file mode 100644 index 1f7d67fa7b9..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/reviewer-not-dispatched.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: tool_used -tool: Agent -input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' -min: 0 -max: 0 -arm: both -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/prompt.md deleted file mode 100644 index 639dfb0ced0..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/prompt.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -description: "LOW tier, no trigger: a draft adds price_for and claims load_prices already returns an empty dict for a missing file; catalog/prices.py opens the file unguarded, so it raises FileNotFoundError (context/plan-reviewer.md: findings backed by a code path). Also seeded: a Behavior item that is a step of work, not a claim. hard: the false claim reads as settled fact and only a read of the code disproves it. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." -tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] -runs: 3 -max_turns: 150 -timeout_seconds: 2700 -allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] -expected_outcome: "The final PLAN.md handles a missing prices file (FileNotFoundError or an existence check), rewrites the step-of-work Behavior item as a claim, and adds no cache; the arm's reviewer dispatch matches its contract." ---- -/planning:plan Pick up the draft plan at `.work/price-lookup/PLAN.md`. Its Brief, its phases and the design early exit (`.work/price-lookup/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/behavior-is-a-claim.json b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/behavior-is-a-claim.json deleted file mode 100644 index 9a349712051..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/behavior-is-a-claim.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: An edit to the price file shows in the next lookup (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own.\n- Read the file on every call, so a staff edit shows at once.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", - "why": "the plan with the seeded defect fixed" - } - ], - "fail": [ - { - "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own.\n- Read the file on every call, so a staff edit shows at once.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", - "why": "the draft as seeded" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/blast-radius-low.json deleted file mode 100644 index 00e7f5d22b2..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/blast-radius-low.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "Blast radius: LOW\nReason: two files.", - "why": "plain line" - }, - { - "answer": "**Blast radius:** LOW", - "why": "bold label" - } - ], - "fail": [ - { - "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own.\n- Read the file on every call, so a staff edit shows at once.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", - "why": "the draft as seeded has no assessment" - }, - { - "answer": "Blast radius: MEDIUM\nReason: shared code.", - "why": "another tier" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/missing-file-handled.json b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/missing-file-handled.json deleted file mode 100644 index 5c37a9f6c2f..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/missing-file-handled.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "pass": [ - { - "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; catch `FileNotFoundError` from `load_prices` and return `None` (AC3).\n- Read the file on every call, so a staff edit shows at once.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", - "why": "the plan with the seeded defect fixed" - }, - { - "answer": "- Wrap `load_prices` in `try`/`except OSError` and return `None`.\n", - "why": "OSError paraphrase" - }, - { - "answer": "- Return `None` when `path.exists()` is false.\n", - "why": "existence check" - } - ], - "fail": [ - { - "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own.\n- Read the file on every call, so a staff edit shows at once.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", - "why": "the draft as seeded" - }, - { - "answer": "- `load_prices` already returns {} for a missing file.\n", - "why": "repeats the false claim" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/no-cache.json b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/no-cache.json deleted file mode 100644 index d379f2f5648..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/no-cache.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own.\n- Read the file on every call, so a staff edit shows at once.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", - "why": "the draft as seeded keeps the sound part" - } - ], - "fail": [ - { - "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own.\n- Wrap `load_prices` in `functools.lru_cache` so repeat lookups skip the disk.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", - "why": "the plan made the over-flagged change" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/reviewer-not-dispatched.json deleted file mode 100644 index 57a464b8eb9..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/reviewer-not-dispatched.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "pass": [ - { - "answer": [ - { - "tool": "Read", - "input": { - "file_path": ".work/price-lookup/PLAN.md" - } - } - ], - "why": "no agent dispatched" - }, - { - "answer": [ - { - "tool": "Skill", - "input": { - "skill": "planning:devils-advocate" - } - } - ], - "why": "devils-advocate only" - }, - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "general-purpose", - "description": "Stress-test", - "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." - } - } - ], - "why": "another agent whose prompt names the reviewer" - } - ], - "fail": [ - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "planning:plan-reviewer", - "description": "Review plan", - "prompt": "Review .work/price-lookup/PLAN.md" - } - } - ], - "why": "dispatched the plan-reviewer" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/scaffold.sh deleted file mode 100755 index 4122aa0bbdd..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/scaffold.sh +++ /dev/null @@ -1,105 +0,0 @@ -#!/usr/bin/env bash -# A catalog package whose price loader raises on a missing file, and a draft plan that says it does not. -set -euo pipefail - -mkdir -p .work/price-lookup/design catalog data tests -cat >README.md <<'EOF' -# catalog - -Price lookups for the till. Staff edit data/prices.json while the shop is open. -EOF -: >catalog/__init__.py -cat >catalog/prices.py <<'EOF' -"""Price lookups from data/prices.json, which staff edit while the shop is open.""" -import json -from pathlib import Path - -PRICES_PATH = Path("data/prices.json") - - -def load_prices(path=PRICES_PATH): - with open(path, encoding="utf-8") as handle: - return json.load(handle) -EOF -cat >data/prices.json <<'EOF' -{"A100": 3.5, "B200": 12.0} -EOF -cat >tests/test_prices.py <<'EOF' -import json -import os -import tempfile -import unittest - -from catalog.prices import load_prices - - -class LoadTest(unittest.TestCase): - def test_reads_json(self): - path = os.path.join(tempfile.mkdtemp(), "prices.json") - with open(path, "w", encoding="utf-8") as handle: - json.dump({"A100": 3.5}, handle) - self.assertEqual(load_prices(path), {"A100": 3.5}) - - -if __name__ == "__main__": - unittest.main() -EOF -cat >.work/price-lookup/PLAN.md <<'EOF' -# Price lookup without a crash - -## Brief - -### Problem -The till crashes with a traceback when a product has no price, and staff restart it mid-sale. - -### Goal -`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price. - -### Acceptance criteria -- AC1: `price_for("A100")` returns the value from `data/prices.json`. -- AC2: An unknown SKU returns `None`. -- AC3: A missing price file returns `None`. -- AC4: An edit to `data/prices.json` shows in the next call without a restart. - -### Out-of-scope -- Editing prices from the till. - -## Design - -Early exit (tier C): One new function beside an existing loader; no new types. - -## Plan - -### Behavior -- B1: A known SKU returns its price (AC1). Delivered by Phase 1. -- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1. -- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1. - -### Load-bearing assumptions - -| Assumption | Source | Falsifying check | Response if false | -|---|---|---|---| -| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note | - -### Phase 1: Add price_for [STATUS] -**Satisfies:** AC1, AC2, AC3, AC4 -**Delivers:** B1, B2, B3 - -- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own. -- Read the file on every call, so a staff edit shows at once. -- In `tests/test_prices.py`, add one case per criterion. - -**Sanity Check:** `python -m unittest discover -s tests` exits 0. - -### Files affected -| File | Action | -|---|---| -| `catalog/prices.py` | modify | -| `tests/test_prices.py` | modify | -EOF -cat >.work/price-lookup/design/design-resolution.md <<'EOF' -outcome: early-exit -tier: C -reason: One new function beside an existing loader; no new types. -EOF -bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" fix/price-lookup diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/case.yaml b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/case.yaml deleted file mode 100644 index a32d574d24d..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/case.yaml +++ /dev/null @@ -1,4 +0,0 @@ -schema_version: "1.1" -name: plan-reviewer-cut-medium-catalogue-author-search -context: - scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/ac3-sort-built.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/ac3-sort-built.md deleted file mode 100644 index a1c4ed18e80..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/ac3-sort-built.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '^###\s*Phase[\s\S]*?^\s*[-*]\s+(?![^\n]*already sorts)[^\n]*\bsort' -flags: mi -target: - source: file - path: .work/author-search/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/appetite-fit-line.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/appetite-fit-line.md deleted file mode 100644 index 4b8bbb0de9b..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/appetite-fit-line.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: 'appetite fit' -flags: i -target: - source: file - path: .work/author-search/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/blast-radius-medium.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/blast-radius-medium.md deleted file mode 100644 index 55e0fe300ed..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/blast-radius-medium.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: 'blast[ -]radius[^\n]{0,12}?\bMEDIUM\b' -flags: i -target: - source: file - path: .work/author-search/PLAN.md -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/no-new-dependency.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/no-new-dependency.md deleted file mode 100644 index 694d3bbc60c..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/no-new-dependency.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -type: regex -pattern: '^\|\s*`?(?:requirements[\w.-]*\.txt|pyproject\.toml|setup\.cfg|Pipfile)`?\s*\||rapidfuzz|whoosh' -flags: mi -match: not_contains -weight: 0.001 -target: - source: file - path: .work/author-search/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/reviewer-not-dispatched.md deleted file mode 100644 index 1f7d67fa7b9..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/reviewer-not-dispatched.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: tool_used -tool: Agent -input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' -min: 0 -max: 0 -arm: both -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/prompt.md deleted file mode 100644 index 640fa26b7dd..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/prompt.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -description: "MEDIUM tier, no trigger: a five-file draft adds author search. Seeded: the Brief records an appetite and the plan has no Appetite fit line, and the plan says search_title already sorts by title, so AC3 needs no work, while library/search.py returns matches in catalogue order. hard: the sorting claim is false only on reading the code, and no script criterion covers either seed. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." -tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] -runs: 3 -max_turns: 200 -timeout_seconds: 3600 -allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] -expected_outcome: "The final PLAN.md has an Appetite fit line and a work item that sorts author results by title, and adds no search dependency; the arm's reviewer dispatch matches its contract." ---- -/planning:plan Pick up the draft plan at `.work/author-search/PLAN.md`. Its Brief, its phases and the design early exit (`.work/author-search/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/ac3-sort-built.json b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/ac3-sort-built.json deleted file mode 100644 index dbcc4894fa7..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/ac3-sort-built.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "pass": [ - { - "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query, sorted by title (AC3); `search_title` returns catalogue order, so it does not sort for us.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n", - "why": "the plan with the seeded defect fixed" - }, - { - "answer": "### Phase 2: Next step [TODO]\n- Return `sorted(matches, key=lambda book: book[\"title\"])` from `search_author`.\n", - "why": "code paraphrase" - }, - { - "answer": "### Phase 2: Next step [TODO]\n- Sort author results by title before printing.\n", - "why": "prose paraphrase" - } - ], - "fail": [ - { - "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n", - "why": "the draft as seeded" - }, - { - "answer": "### Phase 2: Next step [TODO]\n- `search_title` already sorts by title, so nothing to do.\n", - "why": "repeats the false claim" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/appetite-fit-line.json b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/appetite-fit-line.json deleted file mode 100644 index ae7b1fd0d14..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/appetite-fit-line.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\nAppetite fit: one index map, one search function and one flag fit a day.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n", - "why": "the plan with the seeded defect fixed" - } - ], - "fail": [ - { - "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n", - "why": "the draft as seeded" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/blast-radius-medium.json b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/blast-radius-medium.json deleted file mode 100644 index 0cbc3f56192..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/blast-radius-medium.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "Blast radius: MEDIUM\nReason: two files.", - "why": "plain line" - }, - { - "answer": "**Blast radius:** MEDIUM", - "why": "bold label" - } - ], - "fail": [ - { - "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n", - "why": "the draft as seeded has no assessment" - }, - { - "answer": "Blast radius: HIGH\nReason: shared code.", - "why": "another tier" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/no-new-dependency.json b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/no-new-dependency.json deleted file mode 100644 index 6357d9aea67..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/no-new-dependency.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n", - "why": "the draft as seeded keeps the sound part" - } - ], - "fail": [ - { - "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n| `requirements.txt` | modify |\n", - "why": "the plan made the over-flagged change" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/reviewer-not-dispatched.json deleted file mode 100644 index bec7fc2e79a..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/reviewer-not-dispatched.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "pass": [ - { - "answer": [ - { - "tool": "Read", - "input": { - "file_path": ".work/author-search/PLAN.md" - } - } - ], - "why": "no agent dispatched" - }, - { - "answer": [ - { - "tool": "Skill", - "input": { - "skill": "planning:devils-advocate" - } - } - ], - "why": "devils-advocate only" - }, - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "general-purpose", - "description": "Stress-test", - "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." - } - } - ], - "why": "another agent whose prompt names the reviewer" - } - ], - "fail": [ - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "planning:plan-reviewer", - "description": "Review plan", - "prompt": "Review .work/author-search/PLAN.md" - } - } - ], - "why": "dispatched the plan-reviewer" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/scaffold.sh deleted file mode 100755 index 70d99c2ec2f..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/scaffold.sh +++ /dev/null @@ -1,170 +0,0 @@ -#!/usr/bin/env bash -# A library kiosk package that searches titles only, and a draft plan to add author search. -set -euo pipefail - -mkdir -p .work/author-search/design data library tests -cat >README.md <<'EOF' -# library - -The catalogue kiosk: `python -m library.cli search WORD`. -EOF -: >library/__init__.py -cat >data/books.json <<'EOF' -[{"title": "Beloved", "author": "Toni Morrison"}, {"title": "Jazz", "author": "Toni Morrison"}, {"title": "Emma", "author": "Jane Austen"}] -EOF -cat >library/catalogue.py <<'EOF' -import json -from pathlib import Path - - -def load_books(path=Path("data/books.json")): - return json.loads(path.read_text(encoding="utf-8")) -EOF -cat >library/index.py <<'EOF' -"""In-memory index of the catalogue.""" -from library.catalogue import load_books - - -def build(): - books = load_books() - by_word = {} - for book in books: - for word in book["title"].split(): - by_word.setdefault(word.lower(), []).append(book) - return {"by_word": by_word, "books": books} -EOF -cat >library/search.py <<'EOF' -def search_title(index, query): - """Books with a title word equal to the query, in catalogue order.""" - return list(index["by_word"].get(query.lower(), [])) -EOF -cat >library/cli.py <<'EOF' -import argparse - -from library import index, search - - -def main(argv=None): - parser = argparse.ArgumentParser(prog="library") - sub = parser.add_subparsers(dest="command", required=True) - find = sub.add_parser("search") - find.add_argument("query") - args = parser.parse_args(argv) - for book in search.search_title(index.build(), args.query): - print(book["title"], "-", book["author"]) - - -if __name__ == "__main__": - main() -EOF -cat >tests/test_index.py <<'EOF' -import unittest -from unittest import mock - -from library import index - - -class IndexTest(unittest.TestCase): - def test_title_words(self): - with mock.patch("library.index.load_books", return_value=[{"title": "Big Sky", "author": "A"}]): - self.assertIn("sky", index.build()["by_word"]) - - -if __name__ == "__main__": - unittest.main() -EOF -cat >tests/test_cli.py <<'EOF' -import io -import unittest -from contextlib import redirect_stdout -from unittest import mock - -from library import cli - - -class CliTest(unittest.TestCase): - def test_title_search(self): - books = [{"title": "Emma", "author": "Jane Austen"}] - out = io.StringIO() - with mock.patch("library.index.load_books", return_value=books), redirect_stdout(out): - cli.main(["search", "emma"]) - self.assertEqual(out.getvalue(), "Emma - Jane Austen\n") - - -if __name__ == "__main__": - unittest.main() -EOF -cat >.work/author-search/PLAN.md <<'EOF' -# Search by author - -## Brief - -### Problem -Patrons ask at the desk for books by an author because the kiosk searches titles only. - -### Goal -The kiosk finds books by author name. - -### Acceptance criteria -- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case. -- AC2: Title search is unchanged. -- AC3: Author results are listed in title order. - -### Out-of-scope -- Fuzzy matching. -- Search by ISBN. - -### Work size -Appetite: one day. - -## Design - -Early exit (tier C): New functions beside existing ones; no new types. - -## Plan - -### Behavior -- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2. -- B2: Title search answers as before (AC2). Delivered by Phase 2. -- B3: Author results come in title order (AC3). Delivered by Phase 2. - -### Load-bearing assumptions - -| Assumption | Source | Falsifying check | Response if false | -|---|---|---|---| -| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue | - -### Phase 1: Index authors [STATUS] -**Satisfies:** AC1 -**Delivers:** B1 - -- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`. -- In `tests/test_index.py`, add a case for two books by one author. - -**Sanity Check:** `python -m unittest tests.test_index` exits 0. - -### Phase 2: Search and command [STATUS] -**Satisfies:** AC1, AC2, AC3 -**Delivers:** B1, B2, B3 - -- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way. -- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given. -- In `tests/test_cli.py`, add a case for `--author` and keep the title case. - -**Sanity Check:** `python -m unittest discover -s tests` exits 0. - -### Files affected -| File | Action | -|---|---| -| `library/index.py` | modify | -| `library/search.py` | modify | -| `library/cli.py` | modify | -| `tests/test_index.py` | modify | -| `tests/test_cli.py` | modify | -EOF -cat >.work/author-search/design/design-resolution.md <<'EOF' -outcome: early-exit -tier: C -reason: New functions beside existing ones; no new types. -EOF -bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/author-search diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/case.yaml b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/case.yaml deleted file mode 100644 index 1c90e506620..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/case.yaml +++ /dev/null @@ -1,4 +0,0 @@ -schema_version: "1.1" -name: plan-reviewer-cut-medium-invoice-discounts -context: - scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/ac4-built.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/ac4-built.md deleted file mode 100644 index fc00036f9f5..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/ac4-built.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '^###\s*Phase[\s\S]*?^\s*[-*]\s+[^\n]*(?:\brender|-15%|percentage)' -flags: mi -target: - source: file - path: .work/invoice-discounts/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/assumption-row.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/assumption-row.md deleted file mode 100644 index e7295a0e42a..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/assumption-row.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '###\s*Load-bearing assumptions[^\n]*\n(?:(?!#)[^\n]*\n)*?\|[^\n]*(?:discount_pct|\b100\b|credit|between 0)' -flags: i -target: - source: file - path: .work/invoice-discounts/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/blast-radius-medium.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/blast-radius-medium.md deleted file mode 100644 index d35392a090f..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/blast-radius-medium.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: 'blast[ -]radius[^\n]{0,12}?\bMEDIUM\b' -flags: i -target: - source: file - path: .work/invoice-discounts/PLAN.md -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/keeps-decimal-rounding.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/keeps-decimal-rounding.md deleted file mode 100644 index aeb81e8c581..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/keeps-decimal-rounding.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -type: regex -pattern: '\bfloat\(|round\([^)\n]*,\s*2\s*\)' -flags: i -match: not_contains -weight: 0.001 -target: - source: file - path: .work/invoice-discounts/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/reviewer-not-dispatched.md deleted file mode 100644 index 1f7d67fa7b9..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/reviewer-not-dispatched.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: tool_used -tool: Agent -input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' -min: 0 -max: 0 -arm: both -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/prompt.md deleted file mode 100644 index d37b3bc1b0b..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/prompt.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -description: "MEDIUM tier, no trigger: a three-phase-file draft applies a stored line discount to invoice totals. Seeded: Phase 2 claims AC4 and B3 (the invoice text shows the percentage) but no work item touches billing/render.py (agents/plan-reviewer.md Delivers check), and a Brief captured assumption has no Load-bearing row. hard: the script sees AC4 on a Satisfies line and passes it. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." -tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] -runs: 3 -max_turns: 200 -timeout_seconds: 3600 -allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] -expected_outcome: "The final PLAN.md has a work item that changes the rendered invoice to show the percentage and a Load-bearing row for the 0-100 discount assumption, and keeps Decimal rounding through money.round_cents; the arm's reviewer dispatch matches its contract." ---- -/planning:plan Pick up the draft plan at `.work/invoice-discounts/PLAN.md`. Its Brief, its phases and the design early exit (`.work/invoice-discounts/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/ac4-built.json b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/ac4-built.json deleted file mode 100644 index bb5d29700c4..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/ac4-built.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In `billing/render.py`, print `-%` beside each discounted line's total.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", - "why": "the plan with the seeded defect fixed" - }, - { - "answer": "### Phase 2: Next step [TODO]\n- Show `-15%` next to each discounted total in the text invoice.\n", - "why": "output paraphrase" - }, - { - "answer": "### Phase 2: Next step [TODO]\n- Update `render_text` to append the discount percentage.\n", - "why": "function name" - } - ], - "fail": [ - { - "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", - "why": "the draft as seeded" - }, - { - "answer": "### Phase 2: Next step [TODO]\n- Keep summing line totals.\n", - "why": "no render step" - }, - { - "answer": "- B3: The invoice text shows each line's discount percentage (AC4).\n### Phase 2: Next step [TODO]\n- Keep summing line totals.\n", - "why": "only the Behavior item mentions it" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/assumption-row.json b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/assumption-row.json deleted file mode 100644 index abdb285b281..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/assumption-row.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `discount_pct` is between 0 and 100 | Brief | Phase 1 Sanity Check | Major: stop and replan |\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", - "why": "the plan with the seeded defect fixed" - } - ], - "fail": [ - { - "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", - "why": "the draft as seeded" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/blast-radius-medium.json b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/blast-radius-medium.json deleted file mode 100644 index 4d6e19c20a0..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/blast-radius-medium.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "Blast radius: MEDIUM\nReason: two files.", - "why": "plain line" - }, - { - "answer": "**Blast radius:** MEDIUM", - "why": "bold label" - } - ], - "fail": [ - { - "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", - "why": "the draft as seeded has no assessment" - }, - { - "answer": "Blast radius: HIGH\nReason: shared code.", - "why": "another tier" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/keeps-decimal-rounding.json b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/keeps-decimal-rounding.json deleted file mode 100644 index 347c19e3386..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/keeps-decimal-rounding.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", - "why": "the draft as seeded keeps the sound part" - } - ], - "fail": [ - { - "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and round it with `round(float(total), 2)`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", - "why": "the plan made the over-flagged change" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/reviewer-not-dispatched.json deleted file mode 100644 index 958f82f90e5..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/reviewer-not-dispatched.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "pass": [ - { - "answer": [ - { - "tool": "Read", - "input": { - "file_path": ".work/invoice-discounts/PLAN.md" - } - } - ], - "why": "no agent dispatched" - }, - { - "answer": [ - { - "tool": "Skill", - "input": { - "skill": "planning:devils-advocate" - } - } - ], - "why": "devils-advocate only" - }, - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "general-purpose", - "description": "Stress-test", - "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." - } - } - ], - "why": "another agent whose prompt names the reviewer" - } - ], - "fail": [ - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "planning:plan-reviewer", - "description": "Review plan", - "prompt": "Review .work/invoice-discounts/PLAN.md" - } - } - ], - "why": "dispatched the plan-reviewer" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/scaffold.sh deleted file mode 100755 index 3ecaaa1f6bc..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/scaffold.sh +++ /dev/null @@ -1,158 +0,0 @@ -#!/usr/bin/env bash -# A billing package that stores a line discount it never applies, and a draft plan to apply and show it. -set -euo pipefail - -mkdir -p .work/invoice-discounts/design billing tests -cat >README.md <<'EOF' -# billing - -Invoices for the order desk. Money rounding lives in billing/money.py. -EOF -: >billing/__init__.py -cat >billing/models.py <<'EOF' -from dataclasses import dataclass, field -from decimal import Decimal - - -@dataclass -class LineItem: - description: str - quantity: int - unit_price: Decimal - # Recorded by the order form; invoices do not apply it yet. - discount_pct: Decimal = Decimal("0") - - -@dataclass -class Invoice: - number: str - items: list = field(default_factory=list) -EOF -cat >billing/money.py <<'EOF' -from decimal import ROUND_HALF_UP, Decimal - - -def round_cents(amount): - return Decimal(amount).quantize(Decimal("0.01"), rounding=ROUND_HALF_UP) -EOF -cat >billing/pricing.py <<'EOF' -from billing import money - - -def line_total(item): - return money.round_cents(item.quantity * item.unit_price) -EOF -cat >billing/invoice.py <<'EOF' -from decimal import Decimal - -from billing import money, pricing - -TAX_RATE = Decimal("0.20") - - -def totals(invoice): - subtotal = sum((pricing.line_total(item) for item in invoice.items), Decimal("0")) - tax = money.round_cents(subtotal * TAX_RATE) - return {"subtotal": subtotal, "tax": tax, "total": subtotal + tax} -EOF -cat >billing/render.py <<'EOF' -from billing import invoice as invoice_totals -from billing import pricing - - -def render_text(invoice): - lines = ["Invoice " + invoice.number] - for item in invoice.items: - lines.append("%-30s %3d x %8s %10s" % (item.description, item.quantity, item.unit_price, pricing.line_total(item))) - sums = invoice_totals.totals(invoice) - lines.append("Subtotal %s Tax %s Total %s" % (sums["subtotal"], sums["tax"], sums["total"])) - return "\n".join(lines) -EOF -cat >tests/test_pricing.py <<'EOF' -import unittest -from decimal import Decimal - -from billing.models import LineItem -from billing.pricing import line_total - - -class LineTotalTest(unittest.TestCase): - def test_quantity_times_price(self): - self.assertEqual(line_total(LineItem("pens", 3, Decimal("10.00"))), Decimal("30.00")) - - -if __name__ == "__main__": - unittest.main() -EOF -cat >.work/invoice-discounts/PLAN.md <<'EOF' -# Apply line discounts on invoices - -## Brief - -### Problem -Sales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand. - -### Goal -Invoices charge the discounted price and show the discount. - -### Acceptance criteria -- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50. -- AC2: The invoice subtotal and tax use the discounted line totals. -- AC3: An item with `discount_pct` 0 totals as today. -- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`. - -### Out-of-scope -- Invoice-level discounts. -- Changes to the order form. - -### Captured assumptions -- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100. - -## Design - -Early exit (tier C): Uses the existing `discount_pct` field; no new types or fields. - -## Plan - -### Behavior -- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1. -- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2. -- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2. - -### Load-bearing assumptions - -| Assumption | Source | Falsifying check | Response if false | -|---|---|---|---| -| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue | - -### Phase 1: Discount line totals [STATUS] -**Satisfies:** AC1, AC3 -**Delivers:** B1 - -- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`. -- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent. - -**Sanity Check:** `python -m unittest tests.test_pricing` exits 0. - -### Phase 2: Carry discounted totals through the invoice [STATUS] -**Satisfies:** AC2, AC4 -**Delivers:** B2, B3 - -- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount. -- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line. - -**Sanity Check:** `python -m unittest discover -s tests` exits 0. - -### Files affected -| File | Action | -|---|---| -| `billing/pricing.py` | modify | -| `tests/test_pricing.py` | modify | -| `tests/test_invoice.py` | create | -EOF -cat >.work/invoice-discounts/design/design-resolution.md <<'EOF' -outcome: early-exit -tier: C -reason: Uses the existing `discount_pct` field; no new types or fields. -EOF -bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/invoice-discounts diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/case.yaml b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/case.yaml deleted file mode 100644 index c148f12cd52..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/case.yaml +++ /dev/null @@ -1,4 +0,0 @@ -schema_version: "1.1" -name: plan-reviewer-cut-medium-overnight-shifts -context: - scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/assumption-row.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/assumption-row.md deleted file mode 100644 index b2e16befeb4..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/assumption-row.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '###\s*Load-bearing assumptions[^\n]*\n(?:(?!#)[^\n]*\n)*?\|[^\n]*(?:noon|12:00|midday|overnight|night)' -flags: i -target: - source: file - path: .work/overnight-shifts/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/blast-radius-medium.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/blast-radius-medium.md deleted file mode 100644 index ed1d4adc9e5..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/blast-radius-medium.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: 'blast[ -]radius[^\n]{0,12}?\bMEDIUM\b' -flags: i -target: - source: file - path: .work/overnight-shifts/PLAN.md -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/export-untouched.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/export-untouched.md deleted file mode 100644 index e60a9c22946..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/export-untouched.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -type: regex -pattern: '^\|\s*`?timesheet/export\.py`?' -flags: mi -match: not_contains -weight: 0.001 -target: - source: file - path: .work/overnight-shifts/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md deleted file mode 100644 index 2d9159a4ea2..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '^## Plan[\s\S]*?(?:regression|\btest_\w+|\btests?\b)' -flags: mi -target: - source: file - path: .work/overnight-shifts/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/reviewer-not-dispatched.md deleted file mode 100644 index 1f7d67fa7b9..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/reviewer-not-dispatched.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: tool_used -tool: Agent -input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' -min: 0 -max: 0 -arm: both -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/prompt.md deleted file mode 100644 index 40afc9ec722..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/prompt.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -description: "MEDIUM tier, no trigger: a three-file bug-fix draft makes overnight shifts count forward to the next day. Seeded: the bug fix names no regression test (context/plan-reviewer.md Design alignment), and a Brief captured assumption has no Load-bearing row. hard: the Sanity Checks are real commands, so the plan looks verified while the bug has no test that would catch its return. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." -tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] -runs: 3 -max_turns: 200 -timeout_seconds: 3600 -allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] -expected_outcome: "The final PLAN.md names a regression test for the 22:00-06:00 shift and has a Load-bearing row for the before-noon assumption, and does not touch timesheet/export.py; the arm's reviewer dispatch matches its contract." ---- -/planning:plan Pick up the draft plan at `.work/overnight-shifts/PLAN.md`. Its Brief, its phases and the design early exit (`.work/overnight-shifts/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/assumption-row.json b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/assumption-row.json deleted file mode 100644 index c33dea4e82a..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/assumption-row.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Overnight shifts end before noon the next day | Brief | Phase 2 Sanity Check | Major: stop and replan |\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", - "why": "the plan with the seeded defect fixed" - } - ], - "fail": [ - { - "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", - "why": "the draft as seeded" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/blast-radius-medium.json b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/blast-radius-medium.json deleted file mode 100644 index 74a0f7675ac..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/blast-radius-medium.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "Blast radius: MEDIUM\nReason: two files.", - "why": "plain line" - }, - { - "answer": "**Blast radius:** MEDIUM", - "why": "bold label" - } - ], - "fail": [ - { - "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", - "why": "the draft as seeded has no assessment" - }, - { - "answer": "Blast radius: HIGH\nReason: shared code.", - "why": "another tier" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/export-untouched.json b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/export-untouched.json deleted file mode 100644 index b2419860b7d..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/export-untouched.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", - "why": "the draft as seeded keeps the sound part" - } - ], - "fail": [ - { - "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n| `timesheet/export.py` | modify |\n", - "why": "the plan made the over-flagged change" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json deleted file mode 100644 index f417ff75c53..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n- In `tests/test_totals.py`, add a regression case: `22:00-06:00` totals 8.0.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", - "why": "the plan with the seeded defect fixed" - } - ], - "fail": [ - { - "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", - "why": "the draft as seeded" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/reviewer-not-dispatched.json deleted file mode 100644 index bd02cd5b58f..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/reviewer-not-dispatched.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "pass": [ - { - "answer": [ - { - "tool": "Read", - "input": { - "file_path": ".work/overnight-shifts/PLAN.md" - } - } - ], - "why": "no agent dispatched" - }, - { - "answer": [ - { - "tool": "Skill", - "input": { - "skill": "planning:devils-advocate" - } - } - ], - "why": "devils-advocate only" - }, - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "general-purpose", - "description": "Stress-test", - "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." - } - } - ], - "why": "another agent whose prompt names the reviewer" - } - ], - "fail": [ - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "planning:plan-reviewer", - "description": "Review plan", - "prompt": "Review .work/overnight-shifts/PLAN.md" - } - } - ], - "why": "dispatched the plan-reviewer" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/scaffold.sh deleted file mode 100755 index ee58d8924b1..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/scaffold.sh +++ /dev/null @@ -1,146 +0,0 @@ -#!/usr/bin/env bash -# A timesheet package that rejects or miscounts overnight shifts, and a draft bug-fix plan. -set -euo pipefail - -mkdir -p .work/overnight-shifts/design tests timesheet -cat >README.md <<'EOF' -# timesheet - -Parses shift lines and totals hours for payroll (timesheet/export.py writes the payroll file). -EOF -: >timesheet/__init__.py -cat >timesheet/models.py <<'EOF' -from dataclasses import dataclass -from datetime import date, time - - -@dataclass -class Shift: - day: date - start: time - end: time -EOF -cat >timesheet/parse.py <<'EOF' -from datetime import date, time - -from timesheet.models import Shift - - -def parse_line(line): - """'2026-03-02 09:00-17:00' -> Shift.""" - day_text, span = line.split() - start_text, end_text = span.split("-") - start, end = time.fromisoformat(start_text), time.fromisoformat(end_text) - if end <= start: - raise ValueError("end before start: " + line) - return Shift(date.fromisoformat(day_text), start, end) -EOF -cat >timesheet/totals.py <<'EOF' -from collections import defaultdict -from datetime import datetime - - -def hours(shift): - start = datetime.combine(shift.day, shift.start) - end = datetime.combine(shift.day, shift.end) - return (end - start).total_seconds() / 3600 - - -def weekly_hours(shifts): - week = defaultdict(float) - for shift in shifts: - week[shift.day] += hours(shift) - return dict(week) -EOF -cat >timesheet/export.py <<'EOF' -def fmt_hours(value): - return "%.2f" % max(value, 0.0) - - -def payroll_rows(week): - return [(day.isoformat(), fmt_hours(total)) for day, total in sorted(week.items())] -EOF -cat >tests/test_totals.py <<'EOF' -import unittest - -from timesheet.parse import parse_line -from timesheet.totals import hours - - -class HoursTest(unittest.TestCase): - def test_day_shift(self): - self.assertEqual(hours(parse_line("2026-03-02 09:00-17:00")), 8.0) - - -if __name__ == "__main__": - unittest.main() -EOF -cat >.work/overnight-shifts/PLAN.md <<'EOF' -# Count overnight shifts - -## Brief - -### Problem -Night staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes. - -### Goal -A shift that ends at or before its start time counts as ending the next day. - -### Acceptance criteria -- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours. -- AC2: Day shifts total as today. -- AC3: `weekly_hours` counts an overnight shift on the day it starts. - -### Out-of-scope -- Shifts longer than 24 hours. -- Daylight-saving adjustments. - -### Captured assumptions -- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00. - -## Design - -Early exit (tier C): Changes existing functions; no new types. - -## Plan - -### Behavior -- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2. -- B2: Day shifts total as before (AC2). Delivered by Phase 2. - -### Load-bearing assumptions - -| Assumption | Source | Falsifying check | Response if false | -|---|---|---|---| -| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note | - -### Phase 1: Accept overnight lines [STATUS] -**Satisfies:** AC1 -**Delivers:** B1 - -- In `timesheet/parse.py`, drop the `end before start` rejection. -- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`. - -**Sanity Check:** `python -c "from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day"` exits 0. - -### Phase 2: Count the hours [STATUS] -**Satisfies:** AC1, AC2, AC3 -**Delivers:** B1, B2 - -- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`. - -**Sanity Check:** `python -c "from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0"` exits 0. - -### Files affected -| File | Action | -|---|---| -| `timesheet/parse.py` | modify | -| `timesheet/models.py` | modify | -| `timesheet/totals.py` | modify | -EOF -cat >.work/overnight-shifts/design/design-resolution.md <<'EOF' -outcome: early-exit -tier: C -reason: Changes existing functions; no new types. -EOF -bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" fix/overnight-shifts diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/case.yaml b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/case.yaml deleted file mode 100644 index 754b72070ed..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/case.yaml +++ /dev/null @@ -1,4 +0,0 @@ -schema_version: "1.1" -name: plan-reviewer-cut-medium-quiet-hours -context: - scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/ac3-built.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/ac3-built.md deleted file mode 100644 index 1703cbc852a..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/ac3-built.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: '^###\s*Phase[\s\S]*?^\s*[-*]\s+[^\n]*(?:\bdigest)' -flags: mi -target: - source: file - path: .work/quiet-hours/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/behavior-is-a-claim.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/behavior-is-a-claim.md deleted file mode 100644 index 4bdd36e14a2..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/behavior-is-a-claim.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: '^\s*[-*]\s*B\d+:\s*(?:Edit|Update|Modify|Change|Add)\b' -flags: mi -match: not_contains -target: - source: file - path: .work/quiet-hours/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/blast-radius-medium.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/blast-radius-medium.md deleted file mode 100644 index f228aa6e914..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/blast-radius-medium.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: 'blast[ -]radius[^\n]{0,12}?\bMEDIUM\b' -flags: i -target: - source: file - path: .work/quiet-hours/PLAN.md -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/keeps-outbox.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/keeps-outbox.md deleted file mode 100644 index 5fb2408605d..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/keeps-outbox.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -type: regex -pattern: 'smtplib|send_mail|\bSMTP\b' -flags: i -match: not_contains -weight: 0.001 -target: - source: file - path: .work/quiet-hours/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/reviewer-not-dispatched.md deleted file mode 100644 index 1f7d67fa7b9..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/reviewer-not-dispatched.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: tool_used -tool: Agent -input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' -min: 0 -max: 0 -arm: both -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/prompt.md deleted file mode 100644 index ff3f4d52b11..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/prompt.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -description: "MEDIUM tier, no trigger: a five-file draft holds non-critical alerts during quiet hours. Seeded: a Behavior item names a file and a step (agents/plan-reviewer.md third check), and AC3 (held alerts reach the morning digest) is claimed through B3 while no work item touches alerts/digest.py. hard: the script passes both, and the held alerts are silently dropped by the plan as written. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." -tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] -runs: 3 -max_turns: 200 -timeout_seconds: 3600 -allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] -expected_outcome: "The final PLAN.md has a work item that routes held alerts into the morning digest, rewrites the file-naming Behavior item as a claim, and keeps paging through the outbox rather than sending mail directly; the arm's reviewer dispatch matches its contract." ---- -/planning:plan Pick up the draft plan at `.work/quiet-hours/PLAN.md`. Its Brief, its phases and the design early exit (`.work/quiet-hours/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/ac3-built.json b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/ac3-built.json deleted file mode 100644 index 601742b9e01..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/ac3-built.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "pass": [ - { - "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `alerts/evaluate.py`, pass each held alert to `digest.add` so the 07:30 digest lists it.\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", - "why": "the plan with the seeded defect fixed" - }, - { - "answer": "### Phase 2: Next step [TODO]\n- Call `digest.add(alert)` for every alert held in quiet hours.\n", - "why": "call paraphrase" - }, - { - "answer": "### Phase 2: Next step [TODO]\n- Queue each held warning for the 07:30 morning digest.\n", - "why": "prose paraphrase" - } - ], - "fail": [ - { - "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", - "why": "the draft as seeded" - }, - { - "answer": "### Phase 2: Next step [TODO]\n- Describe quiet hours in README.md.\n", - "why": "no digest step" - }, - { - "answer": "- B3: Held alerts reach the morning digest (AC3).\n### Phase 2: Next step [TODO]\n- Describe quiet hours in README.md.\n", - "why": "only the Behavior item mentions it" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/behavior-is-a-claim.json b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/behavior-is-a-claim.json deleted file mode 100644 index 6e0dd364f6a..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/behavior-is-a-claim.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Warnings and info alerts raised in quiet hours do not page (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", - "why": "the plan with the seeded defect fixed" - } - ], - "fail": [ - { - "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", - "why": "the draft as seeded" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/blast-radius-medium.json b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/blast-radius-medium.json deleted file mode 100644 index 6c3f8378e4e..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/blast-radius-medium.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "Blast radius: MEDIUM\nReason: two files.", - "why": "plain line" - }, - { - "answer": "**Blast radius:** MEDIUM", - "why": "bold label" - } - ], - "fail": [ - { - "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", - "why": "the draft as seeded has no assessment" - }, - { - "answer": "Blast radius: HIGH\nReason: shared code.", - "why": "another tier" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/keeps-outbox.json b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/keeps-outbox.json deleted file mode 100644 index 62fa1e3b341..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/keeps-outbox.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", - "why": "the draft as seeded keeps the sound part" - } - ], - "fail": [ - { - "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, send the page with `smtplib` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", - "why": "the plan made the over-flagged change" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/reviewer-not-dispatched.json deleted file mode 100644 index 9b655266bac..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/reviewer-not-dispatched.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "pass": [ - { - "answer": [ - { - "tool": "Read", - "input": { - "file_path": ".work/quiet-hours/PLAN.md" - } - } - ], - "why": "no agent dispatched" - }, - { - "answer": [ - { - "tool": "Skill", - "input": { - "skill": "planning:devils-advocate" - } - } - ], - "why": "devils-advocate only" - }, - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "general-purpose", - "description": "Stress-test", - "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." - } - } - ], - "why": "another agent whose prompt names the reviewer" - } - ], - "fail": [ - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "planning:plan-reviewer", - "description": "Review plan", - "prompt": "Review .work/quiet-hours/PLAN.md" - } - } - ], - "why": "dispatched the plan-reviewer" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/scaffold.sh deleted file mode 100755 index 90cad6c726c..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/scaffold.sh +++ /dev/null @@ -1,137 +0,0 @@ -#!/usr/bin/env bash -# An alerts package that pages for every alert at any hour, and a draft plan for quiet hours. -set -euo pipefail - -mkdir -p .work/quiet-hours/design alerts ops -cat >README.md <<'EOF' -# alerts - -Raises alerts, pages on-call through the outbox, and sends a morning digest. -EOF -: >alerts/__init__.py -cat >alerts/settings.py <<'EOF' -from datetime import time - -QUIET_START = time(22, 0) -QUIET_END = time(7, 0) -DIGEST_AT = time(7, 30) -EOF -cat >alerts/rules.py <<'EOF' -from dataclasses import dataclass - - -@dataclass -class Alert: - name: str - severity: str # "critical", "warning" or "info" -EOF -cat >alerts/outbox.py <<'EOF' -"""Pages are queued here; ops/schedule.txt runs flush() every minute, which sends them.""" -QUEUE = [] - - -def enqueue(alert): - QUEUE.append(alert) - - -def flush(send): - while QUEUE: - send(QUEUE.pop(0)) -EOF -cat >alerts/digest.py <<'EOF' -"""Morning digest, built at settings.DIGEST_AT from the alerts in DIGEST_QUEUE.""" -DIGEST_QUEUE = [] - - -def add(alert): - DIGEST_QUEUE.append(alert) - - -def build_digest(): - lines = ["%s (%s)" % (alert.name, alert.severity) for alert in DIGEST_QUEUE] - DIGEST_QUEUE.clear() - return "\n".join(lines) -EOF -cat >alerts/evaluate.py <<'EOF' -from alerts import outbox - - -def dispatch(alert, now): - outbox.enqueue(alert) -EOF -cat >ops/schedule.txt <<'EOF' -* * * * * python -m alerts.flush -30 7 * * * python -m alerts.send_digest -EOF -cat >.work/quiet-hours/PLAN.md <<'EOF' -# Quiet hours for alerts - -## Brief - -### Problem -On-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning. - -### Goal -Non-critical alerts raised in quiet hours wait for the morning digest instead of paging. - -### Acceptance criteria -- AC1: A critical alert in quiet hours is enqueued as today. -- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued. -- AC3: Each held alert appears in the next morning digest. -- AC4: Outside quiet hours every alert is enqueued as today. - -### Out-of-scope -- Per-person quiet hours. -- Snoozing an alert. - -## Design - -Early exit (tier C): Changes existing functions and reads existing settings; no new types. - -## Plan - -### Behavior -- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1. -- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1. -- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2. - -### Load-bearing assumptions - -| Assumption | Source | Falsifying check | Response if false | -|---|---|---|---| -| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue | - -### Phase 1: Hold non-critical alerts [STATUS] -**Satisfies:** AC1, AC2, AC4 -**Delivers:** B1, B2 - -- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight. -- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`. -- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert. - -**Sanity Check:** `python -m unittest tests.test_rules` exits 0. - -### Phase 2: Morning hand-off [STATUS] -**Satisfies:** AC3 -**Delivers:** B3 - -- In `README.md`, describe quiet hours and which severities still page. -- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`. - -**Sanity Check:** `python -m unittest discover -s tests` exits 0. - -### Files affected -| File | Action | -|---|---| -| `alerts/rules.py` | modify | -| `alerts/evaluate.py` | modify | -| `README.md` | modify | -| `tests/test_rules.py` | create | -| `tests/test_evaluate.py` | create | -EOF -cat >.work/quiet-hours/design/design-resolution.md <<'EOF' -outcome: early-exit -tier: C -reason: Changes existing functions and reads existing settings; no new types. -EOF -bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/quiet-hours diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/case.yaml b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/case.yaml deleted file mode 100644 index 6232944453d..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/case.yaml +++ /dev/null @@ -1,4 +0,0 @@ -schema_version: "1.1" -name: plan-reviewer-cut-medium-reorder-points -context: - scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/blast-radius-medium.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/blast-radius-medium.md deleted file mode 100644 index 3964f89e9d6..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/blast-radius-medium.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: 'blast[ -]radius[^\n]{0,12}?\bMEDIUM\b' -flags: i -target: - source: file - path: .work/reorder-points/PLAN.md -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/no-new-dependency.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/no-new-dependency.md deleted file mode 100644 index 29f3b751aa7..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/no-new-dependency.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -type: regex -pattern: '^\|\s*`?(?:requirements[\w.-]*\.txt|pyproject\.toml|setup\.cfg|Pipfile)`?\s*\||tabulate' -flags: mi -match: not_contains -weight: 0.001 -target: - source: file - path: .work/reorder-points/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/out-of-scope-cut.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/out-of-scope-cut.md deleted file mode 100644 index abd35743bfd..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/out-of-scope-cut.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: '^\s*[-*]\s+(?![^\n]*\b(?:out[ -]of[ -]scope|not|no|never|without|leave|leaves|unchanged|drop\w*|cut|remov\w*|exclude|excluded|excludes|excluding\w*)\b)[^\n]*(?:weekly_report|weekly report)' -flags: mi -match: not_contains -target: - source: file - path: .work/reorder-points/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reuses-available.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reuses-available.md deleted file mode 100644 index 9dd9b4010f9..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reuses-available.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: 'available\(|`available`|stock\.available' -flags: i -target: - source: file - path: .work/reorder-points/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reviewer-not-dispatched.md deleted file mode 100644 index 1f7d67fa7b9..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reviewer-not-dispatched.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: tool_used -tool: Agent -input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' -min: 0 -max: 0 -arm: both -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/prompt.md deleted file mode 100644 index cb30e10f38f..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/prompt.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -description: "MEDIUM tier, no trigger: a five-file draft adds a reorder command and computes on-hand minus reserved itself although inventory/stock.py already exposes that as `available` (context/plan-reviewer.md: build what existing code already does). Also seeded: a work item changes the weekly report, which the Brief's Out-of-scope excludes. hard for the duplicate; routine for the out-of-scope item, which Step 4.7's reading check also covers. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." -tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] -runs: 3 -max_turns: 200 -timeout_seconds: 3600 -allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] -expected_outcome: "The final PLAN.md uses stock.available, drops the weekly report change, and adds no table-formatting dependency; the arm's reviewer dispatch matches its contract." ---- -/planning:plan Pick up the draft plan at `.work/reorder-points/PLAN.md`. Its Brief, its phases and the design early exit (`.work/reorder-points/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/blast-radius-medium.json b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/blast-radius-medium.json deleted file mode 100644 index 4b1d196a11a..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/blast-radius-medium.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "Blast radius: MEDIUM\nReason: two files.", - "why": "plain line" - }, - { - "answer": "**Blast radius:** MEDIUM", - "why": "bold label" - } - ], - "fail": [ - { - "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU.\n- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `inventory/report.py` | modify |\n| `tests/test_reorder.py` | create |\n", - "why": "the draft as seeded has no assessment" - }, - { - "answer": "Blast radius: HIGH\nReason: shared code.", - "why": "another tier" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/no-new-dependency.json b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/no-new-dependency.json deleted file mode 100644 index 5f0f5ce0f8c..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/no-new-dependency.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU.\n- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `inventory/report.py` | modify |\n| `tests/test_reorder.py` | create |\n", - "why": "the draft as seeded keeps the sound part" - } - ], - "fail": [ - { - "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU through `tabulate`.\n- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `inventory/report.py` | modify |\n| `tests/test_reorder.py` | create |\n", - "why": "the plan made the over-flagged change" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/out-of-scope-cut.json b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/out-of-scope-cut.json deleted file mode 100644 index 59b62b404ea..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/out-of-scope-cut.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `tests/test_reorder.py` | create |\n", - "why": "the plan with the seeded defect fixed" - }, - { - "answer": "- The weekly report is out of scope and stays unchanged.\n", - "why": "names it while leaving it alone" - } - ], - "fail": [ - { - "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU.\n- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `inventory/report.py` | modify |\n| `tests/test_reorder.py` | create |\n", - "why": "the draft as seeded" - }, - { - "answer": "- Extend `weekly_report()` with the reorder column.\n", - "why": "paraphrased scope creep" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reuses-available.json b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reuses-available.json deleted file mode 100644 index eec93e85b86..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reuses-available.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, take `stock.available(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU.\n- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `inventory/report.py` | modify |\n| `tests/test_reorder.py` | create |\n", - "why": "the plan with the seeded defect fixed" - } - ], - "fail": [ - { - "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU.\n- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `inventory/report.py` | modify |\n| `tests/test_reorder.py` | create |\n", - "why": "the draft as seeded" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reviewer-not-dispatched.json deleted file mode 100644 index 343740a60d9..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reviewer-not-dispatched.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "pass": [ - { - "answer": [ - { - "tool": "Read", - "input": { - "file_path": ".work/reorder-points/PLAN.md" - } - } - ], - "why": "no agent dispatched" - }, - { - "answer": [ - { - "tool": "Skill", - "input": { - "skill": "planning:devils-advocate" - } - } - ], - "why": "devils-advocate only" - }, - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "general-purpose", - "description": "Stress-test", - "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." - } - } - ], - "why": "another agent whose prompt names the reviewer" - } - ], - "fail": [ - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "planning:plan-reviewer", - "description": "Review plan", - "prompt": "Review .work/reorder-points/PLAN.md" - } - } - ], - "why": "dispatched the plan-reviewer" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/scaffold.sh deleted file mode 100755 index 2418f72611a..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/scaffold.sh +++ /dev/null @@ -1,125 +0,0 @@ -#!/usr/bin/env bash -# An inventory package with an available-stock helper and a weekly report, and a draft plan for reorder points. -set -euo pipefail - -mkdir -p .work/reorder-points/design inventory -cat >README.md <<'EOF' -# inventory - -Stock levels, the weekly report and the buyers' CLI (`python -m inventory.cli`). -EOF -: >inventory/__init__.py -cat >inventory/stock.py <<'EOF' -"""Stock levels. STOCK and RESERVED are loaded from the warehouse export at start-up.""" -STOCK = {} -RESERVED = {} - - -def on_hand(sku): - return STOCK.get(sku, 0) - - -def reserved(sku): - return RESERVED.get(sku, 0) - - -def available(sku): - """Stock free to sell: on hand minus reserved, never below zero.""" - return max(on_hand(sku) - reserved(sku), 0) -EOF -cat >inventory/report.py <<'EOF' -from inventory import stock - - -def weekly_report(): - rows = [(sku, stock.on_hand(sku), stock.reserved(sku)) for sku in sorted(stock.STOCK)] - return "\n".join("%-10s %6d %6d" % row for row in rows) -EOF -cat >inventory/cli.py <<'EOF' -import argparse - -from inventory import report - - -def main(argv=None): - parser = argparse.ArgumentParser(prog="inventory") - sub = parser.add_subparsers(dest="command", required=True) - sub.add_parser("weekly") - args = parser.parse_args(argv) - if args.command == "weekly": - print(report.weekly_report()) - - -if __name__ == "__main__": - main() -EOF -cat >.work/reorder-points/PLAN.md <<'EOF' -# Reorder points - -## Brief - -### Problem -Buyers notice a product is short only when an order cannot be filled. - -### Goal -A `reorder` command lists the products whose free stock has fallen below their reorder point. - -### Acceptance criteria -- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point. -- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed. -- AC3: The list is sorted by SKU. - -### Out-of-scope -- Changing the weekly stock report. -- Placing orders with suppliers. - -## Design - -Early exit (tier C): A new command over existing stock functions; no new types. - -## Plan - -### Behavior -- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2. -- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1. - -### Load-bearing assumptions - -| Assumption | Source | Falsifying check | Response if false | -|---|---|---|---| -| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue | - -### Phase 1: Reorder check [STATUS] -**Satisfies:** AC1, AC2 -**Delivers:** B1, B2 - -- Create `inventory/data/reorder_points.json` with the buyers' points. -- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point. -- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point. - -**Sanity Check:** `python -m unittest tests.test_reorder` exits 0. - -### Phase 2: Command and report [STATUS] -**Satisfies:** AC1, AC3 -**Delivers:** B1 - -- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU. -- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`. - -**Sanity Check:** `python -m unittest discover -s tests` exits 0. - -### Files affected -| File | Action | -|---|---| -| `inventory/data/reorder_points.json` | create | -| `inventory/reorder.py` | create | -| `inventory/cli.py` | modify | -| `inventory/report.py` | modify | -| `tests/test_reorder.py` | create | -EOF -cat >.work/reorder-points/design/design-resolution.md <<'EOF' -outcome: early-exit -tier: C -reason: A new command over existing stock functions; no new types. -EOF -bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/reorder-points diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/case.yaml b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/case.yaml deleted file mode 100644 index f55b6c1dd1e..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/case.yaml +++ /dev/null @@ -1,4 +0,0 @@ -schema_version: "1.1" -name: plan-reviewer-cut-medium-shipping-zones -context: - scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/blast-radius-medium.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/blast-radius-medium.md deleted file mode 100644 index a17afa8a022..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/blast-radius-medium.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: 'blast[ -]radius[^\n]{0,12}?\bMEDIUM\b' -flags: i -target: - source: file - path: .work/shipping-zones/PLAN.md -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/callers-untouched.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/callers-untouched.md deleted file mode 100644 index 9a798239d0b..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/callers-untouched.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -type: regex -pattern: '^\|\s*`?shop/views\.py`?' -flags: mi -match: not_contains -weight: 0.001 -target: - source: file - path: .work/shipping-zones/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reuses-country-region.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reuses-country-region.md deleted file mode 100644 index 8f27c426e1f..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reuses-country-region.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: regex -pattern: 'country_region|shop\.geo|shop/geo\.py|\bgeo\.' -flags: i -target: - source: file - path: .work/shipping-zones/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reviewer-not-dispatched.md deleted file mode 100644 index 1f7d67fa7b9..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reviewer-not-dispatched.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: tool_used -tool: Agent -input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' -min: 0 -max: 0 -arm: both -weight: 0.001 ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/sanity-check-command.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/sanity-check-command.md deleted file mode 100644 index f9f0f1140cf..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/sanity-check-command.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: regex -pattern: 'Sanity Check[^\n]*\b(?:looks? (?:right|correct|good|fine)|as expected|works (?:correctly|properly))' -flags: i -match: not_contains -target: - source: file - path: .work/shipping-zones/PLAN.md ---- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/prompt.md deleted file mode 100644 index ab742efab59..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/prompt.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -description: "MEDIUM tier, no trigger: a five-file draft prices shipping by destination zone and builds a new country-to-zone table although shop/geo.py already maps countries to regions (context/plan-reviewer.md: build what existing code already does). Also seeded: a vague Sanity Check. hard: the duplicate needs a read of a module the draft never mentions. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." -tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] -runs: 3 -max_turns: 200 -timeout_seconds: 3600 -allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] -expected_outcome: "The final PLAN.md reuses shop.geo.country_region, gives every phase a command Sanity Check, and does not touch shop/views.py since checkout_summary already takes the country; the arm's reviewer dispatch matches its contract." ---- -/planning:plan Pick up the draft plan at `.work/shipping-zones/PLAN.md`. Its Brief, its phases and the design early exit (`.work/shipping-zones/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/blast-radius-medium.json b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/blast-radius-medium.json deleted file mode 100644 index 8d8cb9cbe84..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/blast-radius-medium.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "pass": [ - { - "answer": "Blast radius: MEDIUM\nReason: two files.", - "why": "plain line" - }, - { - "answer": "**Blast radius:** MEDIUM", - "why": "bold label" - } - ], - "fail": [ - { - "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`).\n- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** checkout totals look correct for each zone.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `tests/test_shipping.py` | modify |\n", - "why": "the draft as seeded has no assessment" - }, - { - "answer": "Blast radius: HIGH\nReason: shared code.", - "why": "another tier" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/callers-untouched.json b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/callers-untouched.json deleted file mode 100644 index b4ab38f501b..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/callers-untouched.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`).\n- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** checkout totals look correct for each zone.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `tests/test_shipping.py` | modify |\n", - "why": "the draft as seeded keeps the sound part" - } - ], - "fail": [ - { - "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`).\n- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** checkout totals look correct for each zone.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `shop/views.py` | modify |\n| `tests/test_shipping.py` | modify |\n", - "why": "the plan made the over-flagged change" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reuses-country-region.json b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reuses-country-region.json deleted file mode 100644 index 7e7acbaf49f..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reuses-country-region.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Use `shop.geo.country_region`, which already maps codes to `home`, `eu` and `world`.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** checkout totals look correct for each zone.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `tests/test_shipping.py` | modify |\n", - "why": "the plan with the seeded defect fixed" - } - ], - "fail": [ - { - "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`).\n- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** checkout totals look correct for each zone.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `tests/test_shipping.py` | modify |\n", - "why": "the draft as seeded" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reviewer-not-dispatched.json deleted file mode 100644 index ac8117a2c1d..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reviewer-not-dispatched.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "pass": [ - { - "answer": [ - { - "tool": "Read", - "input": { - "file_path": ".work/shipping-zones/PLAN.md" - } - } - ], - "why": "no agent dispatched" - }, - { - "answer": [ - { - "tool": "Skill", - "input": { - "skill": "planning:devils-advocate" - } - } - ], - "why": "devils-advocate only" - }, - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "general-purpose", - "description": "Stress-test", - "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." - } - } - ], - "why": "another agent whose prompt names the reviewer" - } - ], - "fail": [ - { - "answer": [ - { - "tool": "Agent", - "input": { - "subagent_type": "planning:plan-reviewer", - "description": "Review plan", - "prompt": "Review .work/shipping-zones/PLAN.md" - } - } - ], - "why": "dispatched the plan-reviewer" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/sanity-check-command.json b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/sanity-check-command.json deleted file mode 100644 index 112b3a85f4f..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/sanity-check-command.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "pass": [ - { - "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`).\n- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** `python -m unittest tests.test_shipping` exits 0.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `tests/test_shipping.py` | modify |\n", - "why": "the plan with the seeded defect fixed" - } - ], - "fail": [ - { - "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`).\n- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** checkout totals look correct for each zone.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `tests/test_shipping.py` | modify |\n", - "why": "the draft as seeded" - } - ] -} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/scaffold.sh deleted file mode 100755 index 499d1642717..00000000000 --- a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/scaffold.sh +++ /dev/null @@ -1,146 +0,0 @@ -#!/usr/bin/env bash -# A shop package with a country-to-region map and flat shipping, and a draft plan for zone rates that builds its own map. -set -euo pipefail - -mkdir -p .work/shipping-zones/design shop tests -cat >README.md <<'EOF' -# shop - -Cart, checkout and shipping for the web shop. Country data lives in shop/geo.py. -EOF -: >shop/__init__.py -cat >shop/geo.py <<'EOF' -"""Country facts shared by checkout, tax and shipping.""" -REGIONS = {"GB": "home", "IE": "eu", "FR": "eu", "DE": "eu", "ES": "eu", "US": "world", "CA": "world", "AU": "world"} - - -def country_region(code): - """'home', 'eu' or 'world'; any code not listed is 'world'.""" - return REGIONS.get(code.upper(), "world") -EOF -cat >shop/settings.py <<'EOF' -from decimal import Decimal - -FREE_SHIPPING_MIN = Decimal("50.00") -FLAT_RATE = Decimal("4.95") -EOF -cat >shop/shipping.py <<'EOF' -from decimal import Decimal - -from shop import settings - - -def shipping_cost(cart_total): - if cart_total >= settings.FREE_SHIPPING_MIN: - return Decimal("0.00") - return settings.FLAT_RATE -EOF -cat >shop/checkout.py <<'EOF' -from shop import geo, shipping - - -def checkout_summary(cart, country): - total = sum(line["price"] * line["qty"] for line in cart) - return {"region": geo.country_region(country), "items": total, "shipping": shipping.shipping_cost(total)} -EOF -cat >shop/views.py <<'EOF' -from shop.checkout import checkout_summary - - -def checkout_page(request): - return checkout_summary(request["cart"], request["country"]) -EOF -cat >tests/test_shipping.py <<'EOF' -import unittest -from decimal import Decimal - -from shop.shipping import shipping_cost - - -class ShippingTest(unittest.TestCase): - def test_flat_rate(self): - self.assertEqual(shipping_cost(Decimal("10")), Decimal("4.95")) - - -if __name__ == "__main__": - unittest.main() -EOF -cat >.work/shipping-zones/PLAN.md <<'EOF' -# Shipping by zone - -## Brief - -### Problem -Every order pays the domestic flat rate, so orders shipped abroad lose money on postage. - -### Goal -Shipping cost depends on the destination's region. - -### Acceptance criteria -- AC1: Region `home` keeps 4.95, free from a 50.00 cart. -- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only. -- AC3: The checkout summary shows the rate for the destination country. -- AC4: A country code the shop does not know pays the `world` rate. - -### Out-of-scope -- Carrier rate quotes. -- Weight-based pricing. - -## Design - -Early exit (tier C): Changes existing functions; no new types. - -## Plan - -### Behavior -- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2. -- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2. -- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3. - -### Load-bearing assumptions - -| Assumption | Source | Falsifying check | Response if false | -|---|---|---|---| -| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue | - -### Phase 1: Zone table [STATUS] -**Satisfies:** AC2, AC4 -**Delivers:** B1 - -- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`). -- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks. - -**Sanity Check:** `python -c "from shop.zones import zone_for; assert zone_for('ZZ') == 'world'"` exits 0. - -### Phase 2: Rate by zone [STATUS] -**Satisfies:** AC1, AC2 -**Delivers:** B1, B2 - -- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95. -- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold. - -**Sanity Check:** checkout totals look correct for each zone. - -### Phase 3: Show the rate at checkout [STATUS] -**Satisfies:** AC3 -**Delivers:** B3 - -- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`. - -**Sanity Check:** `python -m unittest discover -s tests` exits 0. - -### Files affected -| File | Action | -|---|---| -| `shop/data/zones.json` | create | -| `shop/zones.py` | create | -| `shop/shipping.py` | modify | -| `shop/checkout.py` | modify | -| `tests/test_shipping.py` | modify | -EOF -cat >.work/shipping-zones/design/design-resolution.md <<'EOF' -outcome: early-exit -tier: C -reason: Changes existing functions; no new types. -EOF -bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/shipping-zones