test(planning): add plan-reviewer-cut before/after eval cases

.changes/planning/test-plan-reviewer-cut-evals-151426f4.md

@@ -0,0 +1,5 @@
+---
+bump: none
+---
+
+No release: adds eval cases only (`plan-reviewer-cut-*` and their fixtures), which ship nothing users run.

plugins/planning/evals/fixtures/plan-reviewer-cut/init-repo.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+# Run by each plan-reviewer-cut-* scaffold after it writes the project files and the draft plan:
+# commits the project on main, pushes it to a local bare origin, and moves to a feature branch.
+#
+#   init-repo.sh <branch>
+#
+# Each draft's phase headings carry the placeholder [STATUS]; this script sets it to the
+# not-started status marker the plan skill expects.
+set -euo pipefail
+
+branch="$1"
+printf '.work/\n__pycache__/\n' >.gitignore
+for plan in .work/*/PLAN.md; do
+  text="$(cat "$plan")"
+  printf '%s\n' "${text//'[STATUS]'/[TODO]}" >"$plan"
+done
+git init -q -b main
+git add .
+git -c user.name=dev -c user.email=dev@example.invalid commit -q -m "chore: import project"
+mkdir -p .work/origin.git
+git init -q --bare -b main .work/origin.git
+git remote add origin "$PWD/.work/origin.git"
+git push -q origin main
+git checkout -q -b "$branch"

plugins/planning/evals/fixtures/plan-reviewer-cut/prepare-arm.sh

@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+# Writes one arm of the plan-reviewer-cut suite: a copy of the planning plugin at <ref>, outside any
+# repository, holding this checkout's plan-reviewer-cut cases. `claude plugin eval` then runs it
+# with --ablation none.
+#
+#   prepare-arm.sh <ref> <dest>
+#
+# <ref> is a commit of this repository (the before arm pins the PR's merge base).
+#
+# Exit: 0 written; 2 a usage error, or a dest that already exists.
+set -euo pipefail
+
+usage() {
+  echo "usage: prepare-arm.sh <ref> <dest>" >&2
+  exit 2
+}
+
+[[ $# -eq 2 ]] || usage
+ref="$1"
+dest="$2"
+if [[ -e "$dest" ]]; then
+  echo "prepare-arm.sh: $dest already exists" >&2
+  exit 2
+fi
+
+parent="$(dirname "$dest")"
+if [[ ! -d "$parent" ]] || git -C "$parent" rev-parse --git-dir >/dev/null 2>&1; then
+  echo "prepare-arm.sh: $dest must sit in an existing directory outside any git repository" >&2
+  exit 2
+fi
+
+here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+plugin="$(cd "$here/../../.." && pwd)"
+repo="$(git -C "$plugin" rev-parse --show-toplevel)"
+git -C "$repo" cat-file -e "$ref^{commit}" || usage
+
+mkdir -p "$dest"
+git -C "$repo" archive "$ref" plugins/planning | tar -x -C "$dest" --strip-components=2
+rm -rf "${dest:?}/evals/results"
+mkdir -p "$dest/evals/fixtures"
+rm -rf "${dest:?}/evals/fixtures/plan-reviewer-cut" "$dest"/evals/plan-reviewer-cut-*
+cp -R "$here" "$dest/evals/fixtures/plan-reviewer-cut"
+cp -R "$plugin"/evals/plan-reviewer-cut-* "$dest/evals/"
+
+echo "$dest"

plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: plan-reviewer-cut-low-blog-permalink
+context:
+  scaffold_script: scaffold.sh

plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/behavior-is-a-claim.md

@@ -0,0 +1,9 @@
+---
+type: regex
+pattern: '^\s*[-*]\s*B\d+:\s*(?:Edit|Update|Modify|Change|Add)\b'
+flags: mi
+match: not_contains
+target:
+  source: file
+  path: .work/post-permalink/PLAN.md
+---

plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/blast-radius-low.md

@@ -0,0 +1,9 @@
+---
+type: regex
+pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b'
+flags: i
+target:
+  source: file
+  path: .work/post-permalink/PLAN.md
+weight: 0.001
+---

plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/model-untouched.md

@@ -0,0 +1,10 @@
+---
+type: regex
+pattern: '^\|\s*`?blog/models\.py`?\s*\|\s*(?:modify|create|edit)'
+flags: mi
+match: not_contains
+weight: 0.001
+target:
+  source: file
+  path: .work/post-permalink/PLAN.md
+---

plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reuses-slugify.md

@@ -0,0 +1,8 @@
+---
+type: regex
+pattern: '\bslugify\b'
+flags: i
+target:
+  source: file
+  path: .work/post-permalink/PLAN.md
+---

plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reviewer-not-dispatched.md

@@ -0,0 +1,9 @@
+---
+type: tool_used
+tool: Agent
+input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer'
+min: 0
+max: 0
+arm: both
+weight: 0.001
+---

plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "LOW tier, no trigger: a draft adds post permalinks and writes its own slug helper although blog/text.py already has `slugify`, which AC2's accent folding needs (context/plan-reviewer.md: build what existing code already does). Also seeded: a Behavior item that names a file and a step instead of an observable claim (agents/plan-reviewer.md third check). hard: the duplicate is visible only by reading code the plan never cites. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)."
+tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard]
+runs: 3
+max_turns: 150
+timeout_seconds: 2700
+allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent]
+expected_outcome: "The final PLAN.md reuses blog.text.slugify, rewrites the file-naming Behavior item as a claim, and does not change blog/models.py; the arm's reviewer dispatch matches its contract."
+---
+/planning:plan Pick up the draft plan at `.work/post-permalink/PLAN.md`. Its Brief, its phases and the design early exit (`.work/post-permalink/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code.

plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/behavior-is-a-claim.json

@@ -0,0 +1,14 @@
+{
+  "pass": [
+    {
+      "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM/<slug>/`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: The slug is lowercase ASCII words with accents folded (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n",
+      "why": "the plan with the seeded defect fixed"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM/<slug>/`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n",
+      "why": "the draft as seeded"
+    }
+  ]
+}

plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/blast-radius-low.json

@@ -0,0 +1,22 @@
+{
+  "pass": [
+    {
+      "answer": "Blast radius: LOW\nReason: two files.",
+      "why": "plain line"
+    },
+    {
+      "answer": "**Blast radius:** LOW",
+      "why": "bold label"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM/<slug>/`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n",
+      "why": "the draft as seeded has no assessment"
+    },
+    {
+      "answer": "Blast radius: MEDIUM\nReason: shared code.",
+      "why": "another tier"
+    }
+  ]
+}

plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/model-untouched.json

@@ -0,0 +1,14 @@
+{
+  "pass": [
+    {
+      "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM/<slug>/`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n",
+      "why": "the draft as seeded keeps the sound part"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM/<slug>/`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n| `blog/models.py` | modify |\n",
+      "why": "the plan made the over-flagged change"
+    }
+  ]
+}

plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reuses-slugify.json

@@ -0,0 +1,14 @@
+{
+  "pass": [
+    {
+      "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM/<slug>/`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: The slug is lowercase ASCII words with accents folded (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, import `slugify` from `blog.text`, which already folds accents.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{slugify(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n",
+      "why": "the plan with the seeded defect fixed"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM/<slug>/`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n",
+      "why": "the draft as seeded"
+    }
+  ]
+}

plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reviewer-not-dispatched.json

@@ -0,0 +1,54 @@
+{
+  "pass": [
+    {
+      "answer": [
+        {
+          "tool": "Read",
+          "input": {
+            "file_path": ".work/post-permalink/PLAN.md"
+          }
+        }
+      ],
+      "why": "no agent dispatched"
+    },
+    {
+      "answer": [
+        {
+          "tool": "Skill",
+          "input": {
+            "skill": "planning:devils-advocate"
+          }
+        }
+      ],
+      "why": "devils-advocate only"
+    },
+    {
+      "answer": [
+        {
+          "tool": "Agent",
+          "input": {
+            "subagent_type": "general-purpose",
+            "description": "Stress-test",
+            "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped."
+          }
+        }
+      ],
+      "why": "another agent whose prompt names the reviewer"
+    }
+  ],
+  "fail": [
+    {
+      "answer": [
+        {
+          "tool": "Agent",
+          "input": {
+            "subagent_type": "planning:plan-reviewer",
+            "description": "Review plan",
+            "prompt": "Review .work/post-permalink/PLAN.md"
+          }
+        }
+      ],
+      "why": "dispatched the plan-reviewer"
+    }
+  ]
+}