Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .changes/planning/feat-plan-reviewer-risk-gated-140d7eb6.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
bump: minor
---

### Changed

- `/planning:plan` no longer dispatches the fresh-context `plan-reviewer` on LOW plans. Under the default `plan_stress_test: auto` it runs when the blast radius is MEDIUM or higher, a stress-test trigger matches, or the user asks for a review; a before/after eval on LOW plans measured a change of -0.03 (95% interval -0.08 to +0.03) with no case drop, and MEDIUM keeps the reviewer until it is measured; the blast radius is now assessed before that decision, and the Step 5 presentation states a skip and its reason (`Reviewer: skipped (blast radius LOW, no trigger)`). `plan_stress_test` `reviewer` and `both` and verification `strict` still always run it; `light` and `off` still skip it. Basis: Anthropic's [Prompting Claude Opus 5, "Task scope and over-verification"](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5#task-scope-and-over-verification): explicit verification steps and "use a subagent to verify" cause over-verification, and removing them reduces wasted tokens with no loss in quality.
- New eval case `plan-low-risk-skips-reviewer`: a LOW blast-radius draft with default settings dispatches no plan-reviewer and reports the skip line.
2 changes: 1 addition & 1 deletion docs/contracts/verification-level/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ A skill that is itself the independent check when invoked does not read the leve
| `/discovery:research`, `/discovery:explore`, `/discovery:trace-intent` | the sibling verifier (parent contract, "The sibling verifier, stated once"); `strict` widens research's grading to every row and claim |
| `/discipline:do-your-research`, `/discipline:recheck-against-upstream`, `/discipline:sweep-all` | the fresh-context confirmation of a clean audit on load-bearing work (shared method, step 2), the fresh-context re-derivation (step 3) and the fan-out tiers' dispatch; `strict` takes do-your-research's fan-out tier at `full` depth |
| `/education:quiz-me` | the fresh-context answer-key pass |
| `/planning:plan` | the plan-reviewer dispatch in Step 3 |
| `/planning:plan` | the plan-reviewer dispatch in Step 3; `strict` dispatches it for every plan, which `plan_stress_test`'s `auto` otherwise skips on LOW plans |
| `/session-flow:orchestrate`, `/session-flow:workflow`, `/session-flow:retro` | imperative 3's verifier in a primed session; the fresh-context reviewer in pre-PR step 2, without which Stage 10 is not done; retro's single-pass analysis and scoring under `retro_lenses: 1` |
| `/harness-config:audit-automation-gaps` | the fresh-context refusal gate (2.3), whose verdicts `light` lists as `unverified`, and the Phase 4 step 6 reviewer |
| `/harness-config:audit-instructions` | the Phase C verifiers (per lane, B2 and I33) |
Expand Down
2 changes: 1 addition & 1 deletion plugins/planning/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ where artifacts land in the consuming repo.
| `/planning:design` | Design space | Explores types, contracts, module boundaries, and package topology through collaborative discussion rounds, producing capability-matrix / type-inventory / design-threads / topology artifacts; its `handoff` action delegates to `/planning:design-handoff`. Its `data`, `integration`, and `system` artifacts are typed: each carries a `scope` + `dialect` label and a fenced block in the dialect the `diagram_dialect` convention key resolves to (mermaid `erDiagram` or DBML for data; mermaid `sequenceDiagram` plus an OpenAPI 3.1 sketch for integration; a LikeC4 or C4-PlantUML container view for system, and no C4 view at all when that key is unset). `library` and `module` emit no typed artifact. |
| `/planning:design-handoff` | Design→plan gate | Gates a finished design for `/planning:plan`. The gate is a binary check that every `design-threads.md` thread is RESOLVED, directional, or TAGGED-DEFERRED. Then it packages the plan-ready summary and resume prompt, or FAILs and routes back to `/planning:design`. On PASS and FAIL alike it also emits an advisory six-dimension coverage table (what, how, where, who, when, why) that never contributes to the verdict; uncovered dimensions ride along in the handoff summary and the resume prompt. |
| `/planning:devils-advocate` | Adversarial review | Stress-tests plans via assumption extraction, evidence checks, failure scenarios, and operational-gotcha sweeps. Every finding evidence-backed, never generic warnings. An `incumbent` mode turns the same lens on the status quo: an Alternatives Sweep that stress-tests keeping an incumbent tool/approach against alternatives (native > official > vetted ladder, coupling priced, KEEP / MIGRATE / RESEARCH verdict), exploring the incumbent first-hand in a fresh sub-agent. |
| `/planning:plan` | Implementation plan | Produces a structured plan (goal, approach, test strategy, blast radius, parallelism analysis, tagged unilateral decisions) with a fresh-context `plan-reviewer` stress-test, on by default and set per mode by `plan_stress_test`, and a user approval gate, persisted to PLAN.md. |
| `/planning:plan` | Implementation plan | Produces a structured plan (goal, approach, test strategy, blast radius, parallelism analysis, tagged unilateral decisions) with a fresh-context `plan-reviewer` stress-test, run by default on MEDIUM-or-higher plans and on request and set per mode by `plan_stress_test`, and a user approval gate, persisted to PLAN.md. |
| `/planning:setup` | Configuration | `check` inspects the standards index read-only; `apply` interviews the consumer and bootstraps the standards index (idempotent; re-run to reconfigure). |

The pipeline composes end-to-end. `wayfind` charts the fog upstream when an effort
Expand Down
4 changes: 4 additions & 0 deletions plugins/planning/evals/plan-low-risk-skips-reviewer/case.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
schema_version: "1.1"
name: plan-low-risk-skips-reviewer
context:
scaffold_script: scaffold.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
type: tool_used
tool: Agent
input_match: 'plan-reviewer'
min: 0
max: 0
arm: both
---
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
type: regex
pattern: 'reviewer[*_`]*:[*_`\s]*skipped'
flags: i
---
11 changes: 11 additions & 0 deletions plugins/planning/evals/plan-low-risk-skips-reviewer/prompt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
description: "A LOW blast-radius, one-function draft with no stress-test trigger, in a repository with default settings, dispatches no plan-reviewer in Step 3; the reply that reports Step 3 carries the reviewer skip line"
tags: [plan, plan-reviewer, setting]
runs: 3
max_turns: 60
timeout_seconds: 1200
allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent]
expected_outcome: "Resolves plan_stress_test to auto and verification to standard, assesses blast radius LOW with no trigger matching, dispatches no planning:plan-reviewer, and states Reviewer: skipped (blast radius LOW, no trigger) in the reply that reports Step 3"
---

/planning:plan Continue the draft at `.work/slugify-max-length/PLAN.md` from Step 3. The Brief and the design gate are settled; `.work/slugify-max-length/design/design-resolution.md` records the design early exit. Run Steps 3 and 3b as this skill and this repository's settings say, then report what each step did. Stop there: no Step 4 onward, and edit no code.
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"pass": [
{
"answer": [{"tool": "Read", "input": {"file_path": ".work/slugify-max-length/PLAN.md"}}],
"why": "no agent dispatched"
}
],
"fail": [
{
"answer": [{"tool": "Agent", "input": {"subagent_type": "planning:plan-reviewer", "description": "Review plan", "prompt": "Review .work/slugify-max-length/PLAN.md"}}],
"why": "dispatched the plan-reviewer"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"pass": [
{
"answer": "Blast radius: LOW\nReviewer: skipped (blast radius LOW, no trigger)\nStress-test needed: No",
"why": "the skip line beside the assessment"
},
{
"answer": "- **Reviewer:** skipped (blast radius LOW, no trigger matched)",
"why": "bold label"
}
],
"fail": [
{
"answer": "",
"why": "empty reply"
},
{
"answer": "Reviewer: ran (blast radius LOW). Dispatched planning:plan-reviewer; no plan gaps found.",
"why": "the reviewer ran"
},
{
"answer": "verification: skipped (light)",
"why": "a different skip reason"
}
]
}
102 changes: 102 additions & 0 deletions plugins/planning/evals/plan-low-risk-skips-reviewer/scaffold.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
#!/usr/bin/env bash
# A sandbox repository with no planning settings (verification standard, plan_stress_test auto),
# pushed to a local bare origin, holding a small textkit package with a test, a feature branch that
# changes it, and a settled one-function draft plan with LOW blast radius and no stress-test trigger.
set -euo pipefail

git init -q -b main
mkdir -p textkit tests
printf '# textkit\n\nSmall text helpers. slugify lives in textkit/slug.py.\n' >README.md
: >textkit/__init__.py
cat >textkit/slug.py <<'PY'
def slugify(text):
return "-".join(text.lower().split())
PY
cat >tests/test_slug.py <<'PY'
import unittest

from textkit.slug import slugify


class SlugifyTest(unittest.TestCase):
def test_lowercases_and_hyphenates(self):
self.assertEqual(slugify("Hello World"), "hello-world")


if __name__ == "__main__":
unittest.main()
PY
printf '.work/\n__pycache__/\n' >.gitignore
git add .
git -c user.name=eval -c user.email=eval@example.invalid commit -q -m "scaffold textkit"
mkdir -p .work/origin.git
git init -q --bare -b main .work/origin.git
git remote add origin "$PWD/.work/origin.git"
git push -q origin main
git checkout -q -b feat/strip-punctuation
cat >textkit/slug.py <<'PY'
import re


def slugify(text):
words = re.sub(r"[^\w\s-]", "", text).lower().split()
return "-".join(words)
PY
git -c user.name=eval -c user.email=eval@example.invalid commit -q -am "feat: strip punctuation in slugify"

# A settled draft plan and a design early exit, so the run reaches Step 3 without open questions.
mkdir -p .work/slugify-max-length/design
cat >.work/slugify-max-length/PLAN.md <<'MD'
# slugify max_length

## Brief

### Problem
Slugs from long titles make URLs too long for the share links that embed them.

### Goal
`slugify` takes an optional `max_length` and returns a slug no longer than it, cut on a word boundary.

### Acceptance criteria
- AC1: `slugify(text)` with no `max_length` returns what it returns today.
- AC2: `slugify(text, max_length=n)` drops whole trailing words so the result is at most `n`
characters and never ends in a hyphen; every hyphen in the output counts as a word boundary.
- AC3: When the first word alone is longer than `n`, the result is that word cut to `n` characters.
- AC4: `max_length` below 1 raises `ValueError`.

### Out-of-scope
- Unicode transliteration.

## Plan

### Behavior
- B1: no limit keeps today's output (AC1). Delivered by Phase 1.
- B2: a limit cuts whole words, or the first word, to fit (AC2, AC3). Delivered by Phase 1.
- B3: a limit below 1 is refused (AC4). Delivered by Phase 1.

### Phase 1: Add max_length [STATUS]
**Satisfies:** AC1, AC2, AC3, AC4
**Delivers:** B1, B2, B3

- In `textkit/slug.py`, add `max_length=None`; after joining, when it is set, cut the slug at the
last hyphen at or before `max_length`, or at `max_length` when there is none.
- Raise `ValueError` when `max_length` is below 1.
- Add one test per criterion to `tests/test_slug.py`.

**Sanity Check:** `python -m unittest discover -s tests` exits 0.

### Files affected
| File | Action |
|---|---|
| `textkit/slug.py` | modify |
| `tests/test_slug.py` | modify |
MD
# The phase heading takes its status tag here, outside the heredoc.
plan=.work/slugify-max-length/PLAN.md
text="$(cat "$plan")"
printf '%s\n' "${text//'[STATUS]'/[TODO]}" >"$plan"
cat >.work/slugify-max-length/design/design-resolution.md <<'MD'
outcome: early-exit
tier: C
reason: One optional parameter on one existing function; no new types.
MD
5 changes: 3 additions & 2 deletions plugins/planning/reference/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,8 +166,9 @@ layer.

Its values:

- `auto`: the `plan-reviewer` agent always; `/planning:devils-advocate` in plan-review mode when
Step 3b's blast radius is MEDIUM or higher or a trigger matches.
- `auto`: the `plan-reviewer` agent and `/planning:devils-advocate` in plan-review mode when Step
3b's blast radius is MEDIUM or higher or a trigger matches; the `plan-reviewer` agent also when the
user asks for a review. A LOW plan with no trigger runs neither.
- `reviewer`: the `plan-reviewer` agent only, whatever the blast radius.
- `devils-advocate`: `/planning:devils-advocate` only, whatever the blast radius.
- `both`: both, whatever the blast radius.
Expand Down
2 changes: 1 addition & 1 deletion plugins/planning/schemas/planning.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@
"default": 27
},
"plan_stress_test": {
"description": "Per mode of the draft, which stress-tests /planning:plan runs. auto: plan-reviewer always, /planning:devils-advocate when the blast-radius assessment calls for it. reviewer: plan-reviewer only. devils-advocate: /planning:devils-advocate only. both: both, whatever the blast radius. Only /planning:plan reads this key: a direct /planning:devils-advocate request ignores it and runs the mode its own arguments select.",
"description": "Per mode of the draft, which stress-tests /planning:plan runs. auto: plan-reviewer and /planning:devils-advocate when the blast radius is MEDIUM or higher or a stress-test trigger matches, plan-reviewer also when the user asks for a review; a LOW plan with no trigger runs neither. reviewer: plan-reviewer only. devils-advocate: /planning:devils-advocate only. both: both, whatever the blast radius. Only /planning:plan reads this key: a direct /planning:devils-advocate request ignores it and runs the mode its own arguments select.",
"type": "object",
"additionalProperties": false,
"properties": {
Expand Down
Loading
Loading