From 05ddfa4bec2d76f1add7ed570c27434be96a07ce Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Wed, 11 Nov 2026 18:29:18 -0500 Subject: [PATCH 1/6] test(planning): add plan-reviewer-cut before/after eval cases Twelve seeded plan drafts (six designed LOW, six MEDIUM), each with two defects from what the plan-reviewer checks, a must-not-flag guard, and reviewer-dispatch and blast-radius indicators. They measured #15398's reviewer cut on LOW plans; fixtures/plan-reviewer-cut builds one arm per plugin version (prepare-arm.sh) and sets up each case repo (init-repo.sh). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VQb9Q4F46UumaUgXeUdpYZ --- .../fixtures/plan-reviewer-cut/init-repo.sh | 24 +++ .../fixtures/plan-reviewer-cut/prepare-arm.sh | 45 +++++ .../case.yaml | 4 + .../graders/behavior-is-a-claim.md | 9 + .../graders/blast-radius-low.md | 9 + .../graders/model-untouched.md | 10 ++ .../graders/reuses-slugify.md | 8 + .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/behavior-is-a-claim.json | 14 ++ .../samples/blast-radius-low.json | 22 +++ .../samples/model-untouched.json | 14 ++ .../samples/reuses-slugify.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 116 ++++++++++++ .../case.yaml | 4 + .../graders/ac3-built.md | 8 + .../graders/assumption-row.md | 8 + .../graders/blast-radius-low.md | 9 + .../graders/no-new-dependency.md | 10 ++ .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/ac3-built.json | 30 ++++ .../samples/assumption-row.json | 14 ++ .../samples/blast-radius-low.json | 22 +++ .../samples/no-new-dependency.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 105 +++++++++++ .../case.yaml | 4 + .../graders/appetite-fit-line.md | 8 + .../graders/assumption-row.md | 8 + .../graders/blast-radius-low.md | 9 + .../graders/no-dateutil.md | 10 ++ .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/appetite-fit-line.json | 14 ++ .../samples/assumption-row.json | 14 ++ .../samples/blast-radius-low.json | 22 +++ .../samples/no-dateutil.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 129 +++++++++++++ .../case.yaml | 4 + .../graders/blast-radius-low.md | 9 + .../graders/no-jitter.md | 10 ++ .../graders/real-ceiling-constant.md | 8 + .../graders/regression-test-named.md | 8 + .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/blast-radius-low.json | 22 +++ .../samples/no-jitter.json | 14 ++ .../samples/real-ceiling-constant.json | 14 ++ .../samples/regression-test-named.json | 22 +++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 97 ++++++++++ .../case.yaml | 4 + .../graders/blast-radius-low.md | 9 + .../graders/no-decimal-type.md | 9 + .../graders/out-of-scope-cut.md | 9 + .../graders/reviewer-not-dispatched.md | 9 + .../graders/sanity-check-command.md | 9 + .../prompt.md | 10 ++ .../samples/blast-radius-low.json | 22 +++ .../samples/no-decimal-type.json | 14 ++ .../samples/out-of-scope-cut.json | 22 +++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../samples/sanity-check-command.json | 14 ++ .../scaffold.sh | 88 +++++++++ .../case.yaml | 4 + .../graders/behavior-is-a-claim.md | 9 + .../graders/blast-radius-low.md | 9 + .../graders/missing-file-handled.md | 8 + .../graders/no-cache.md | 10 ++ .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/behavior-is-a-claim.json | 14 ++ .../samples/blast-radius-low.json | 22 +++ .../samples/missing-file-handled.json | 26 +++ .../samples/no-cache.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 105 +++++++++++ .../case.yaml | 4 + .../graders/ac3-sort-built.md | 8 + .../graders/appetite-fit-line.md | 8 + .../graders/blast-radius-medium.md | 9 + .../graders/no-new-dependency.md | 10 ++ .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/ac3-sort-built.json | 26 +++ .../samples/appetite-fit-line.json | 14 ++ .../samples/blast-radius-medium.json | 22 +++ .../samples/no-new-dependency.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 170 ++++++++++++++++++ .../case.yaml | 4 + .../graders/ac4-built.md | 8 + .../graders/assumption-row.md | 8 + .../graders/blast-radius-medium.md | 9 + .../graders/keeps-decimal-rounding.md | 10 ++ .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/ac4-built.json | 30 ++++ .../samples/assumption-row.json | 14 ++ .../samples/blast-radius-medium.json | 22 +++ .../samples/keeps-decimal-rounding.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 158 ++++++++++++++++ .../case.yaml | 4 + .../graders/assumption-row.md | 8 + .../graders/blast-radius-medium.md | 9 + .../graders/export-untouched.md | 10 ++ .../graders/regression-test-named.md | 8 + .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/assumption-row.json | 14 ++ .../samples/blast-radius-medium.json | 22 +++ .../samples/export-untouched.json | 14 ++ .../samples/regression-test-named.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 146 +++++++++++++++ .../case.yaml | 4 + .../graders/ac3-built.md | 8 + .../graders/behavior-is-a-claim.md | 9 + .../graders/blast-radius-medium.md | 9 + .../graders/keeps-outbox.md | 10 ++ .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/ac3-built.json | 30 ++++ .../samples/behavior-is-a-claim.json | 14 ++ .../samples/blast-radius-medium.json | 22 +++ .../samples/keeps-outbox.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 137 ++++++++++++++ .../case.yaml | 4 + .../graders/blast-radius-medium.md | 9 + .../graders/no-new-dependency.md | 10 ++ .../graders/out-of-scope-cut.md | 9 + .../graders/reuses-available.md | 8 + .../graders/reviewer-not-dispatched.md | 9 + .../prompt.md | 10 ++ .../samples/blast-radius-medium.json | 22 +++ .../samples/no-new-dependency.json | 14 ++ .../samples/out-of-scope-cut.json | 22 +++ .../samples/reuses-available.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../scaffold.sh | 125 +++++++++++++ .../case.yaml | 4 + .../graders/blast-radius-medium.md | 9 + .../graders/callers-untouched.md | 10 ++ .../graders/reuses-country-region.md | 8 + .../graders/reviewer-not-dispatched.md | 9 + .../graders/sanity-check-command.md | 9 + .../prompt.md | 10 ++ .../samples/blast-radius-medium.json | 22 +++ .../samples/callers-untouched.json | 14 ++ .../samples/reuses-country-region.json | 14 ++ .../samples/reviewer-not-dispatched.json | 54 ++++++ .../samples/sanity-check-command.json | 14 ++ .../scaffold.sh | 146 +++++++++++++++ 158 files changed, 3805 insertions(+) create mode 100755 plugins/planning/evals/fixtures/plan-reviewer-cut/init-repo.sh create mode 100755 plugins/planning/evals/fixtures/plan-reviewer-cut/prepare-arm.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/behavior-is-a-claim.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/blast-radius-low.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/model-untouched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reuses-slugify.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/behavior-is-a-claim.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/blast-radius-low.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/model-untouched.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reuses-slugify.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/assumption-row.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/blast-radius-low.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/no-new-dependency.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/ac3-built.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/assumption-row.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/blast-radius-low.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/no-new-dependency.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/appetite-fit-line.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/assumption-row.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/blast-radius-low.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/no-dateutil.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/appetite-fit-line.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/assumption-row.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/blast-radius-low.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/no-dateutil.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-low-notes-since/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/blast-radius-low.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/no-jitter.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/real-ceiling-constant.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/blast-radius-low.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/no-jitter.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/real-ceiling-constant.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/blast-radius-low.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/no-decimal-type.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/sanity-check-command.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/blast-radius-low.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/no-decimal-type.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/out-of-scope-cut.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/reviewer-not-dispatched.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/sanity-check-command.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/behavior-is-a-claim.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/blast-radius-low.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/missing-file-handled.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/no-cache.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/behavior-is-a-claim.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/blast-radius-low.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/missing-file-handled.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/no-cache.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/ac3-sort-built.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/appetite-fit-line.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/blast-radius-medium.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/no-new-dependency.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/ac3-sort-built.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/appetite-fit-line.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/blast-radius-medium.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/no-new-dependency.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/ac4-built.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/assumption-row.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/blast-radius-medium.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/keeps-decimal-rounding.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/ac4-built.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/assumption-row.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/blast-radius-medium.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/keeps-decimal-rounding.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/assumption-row.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/blast-radius-medium.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/export-untouched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/assumption-row.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/blast-radius-medium.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/export-untouched.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/ac3-built.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/behavior-is-a-claim.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/blast-radius-medium.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/keeps-outbox.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/ac3-built.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/behavior-is-a-claim.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/blast-radius-medium.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/keeps-outbox.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/blast-radius-medium.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/no-new-dependency.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/out-of-scope-cut.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reuses-available.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/blast-radius-medium.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/no-new-dependency.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/out-of-scope-cut.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reuses-available.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reviewer-not-dispatched.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/scaffold.sh create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/case.yaml create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/blast-radius-medium.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/callers-untouched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reuses-country-region.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reviewer-not-dispatched.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/sanity-check-command.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/prompt.md create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/blast-radius-medium.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/callers-untouched.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reuses-country-region.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reviewer-not-dispatched.json create mode 100644 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/sanity-check-command.json create mode 100755 plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/scaffold.sh diff --git a/plugins/planning/evals/fixtures/plan-reviewer-cut/init-repo.sh b/plugins/planning/evals/fixtures/plan-reviewer-cut/init-repo.sh new file mode 100755 index 00000000000..da87d9d2f68 --- /dev/null +++ b/plugins/planning/evals/fixtures/plan-reviewer-cut/init-repo.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# Run by each plan-reviewer-cut-* scaffold after it writes the project files and the draft plan: +# commits the project on main, pushes it to a local bare origin, and moves to a feature branch. +# +# init-repo.sh +# +# Each draft's phase headings carry the placeholder [STATUS]; this script sets it to the +# not-started status marker the plan skill expects. +set -euo pipefail + +branch="$1" +printf '.work/\n__pycache__/\n' >.gitignore +for plan in .work/*/PLAN.md; do + text="$(cat "$plan")" + printf '%s\n' "${text//'[STATUS]'/[TODO]}" >"$plan" +done +git init -q -b main +git add . +git -c user.name=dev -c user.email=dev@example.invalid commit -q -m "chore: import project" +mkdir -p .work/origin.git +git init -q --bare -b main .work/origin.git +git remote add origin "$PWD/.work/origin.git" +git push -q origin main +git checkout -q -b "$branch" diff --git a/plugins/planning/evals/fixtures/plan-reviewer-cut/prepare-arm.sh b/plugins/planning/evals/fixtures/plan-reviewer-cut/prepare-arm.sh new file mode 100755 index 00000000000..872e9c29fce --- /dev/null +++ b/plugins/planning/evals/fixtures/plan-reviewer-cut/prepare-arm.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash +# Writes one arm of the plan-reviewer-cut suite: a copy of the planning plugin at , outside any +# repository, holding this checkout's plan-reviewer-cut cases. `claude plugin eval` then runs it +# with --ablation none. +# +# prepare-arm.sh +# +# is a commit of this repository (the before arm pins the PR's merge base). +# +# Exit: 0 written; 2 a usage error, or a dest that already exists. +set -euo pipefail + +usage() { + echo "usage: prepare-arm.sh " >&2 + exit 2 +} + +[[ $# -eq 2 ]] || usage +ref="$1" +dest="$2" +if [[ -e "$dest" ]]; then + echo "prepare-arm.sh: $dest already exists" >&2 + exit 2 +fi + +parent="$(dirname "$dest")" +if [[ ! -d "$parent" ]] || git -C "$parent" rev-parse --git-dir >/dev/null 2>&1; then + echo "prepare-arm.sh: $dest must sit in an existing directory outside any git repository" >&2 + exit 2 +fi + +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +plugin="$(cd "$here/../../.." && pwd)" +repo="$(git -C "$plugin" rev-parse --show-toplevel)" +git -C "$repo" cat-file -e "$ref^{commit}" || usage + +mkdir -p "$dest" +git -C "$repo" archive "$ref" plugins/planning | tar -x -C "$dest" --strip-components=2 +rm -rf "${dest:?}/evals/results" +mkdir -p "$dest/evals/fixtures" +rm -rf "${dest:?}/evals/fixtures/plan-reviewer-cut" "$dest"/evals/plan-reviewer-cut-* +cp -R "$here" "$dest/evals/fixtures/plan-reviewer-cut" +cp -R "$plugin"/evals/plan-reviewer-cut-* "$dest/evals/" + +echo "$dest" diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/case.yaml b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/case.yaml new file mode 100644 index 00000000000..61e68ab1a79 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-low-blog-permalink +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/behavior-is-a-claim.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/behavior-is-a-claim.md new file mode 100644 index 00000000000..a3312120a58 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/behavior-is-a-claim.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: '^\s*[-*]\s*B\d+:\s*(?:Edit|Update|Modify|Change|Add)\b' +flags: mi +match: not_contains +target: + source: file + path: .work/post-permalink/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/blast-radius-low.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/blast-radius-low.md new file mode 100644 index 00000000000..4e4b39320a2 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/blast-radius-low.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b' +flags: i +target: + source: file + path: .work/post-permalink/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/model-untouched.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/model-untouched.md new file mode 100644 index 00000000000..2fe65e17da1 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/model-untouched.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: '^\|\s*`?blog/models\.py`?\s*\|\s*(?:modify|create|edit)' +flags: mi +match: not_contains +weight: 0.001 +target: + source: file + path: .work/post-permalink/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reuses-slugify.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reuses-slugify.md new file mode 100644 index 00000000000..21de46dc3d5 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reuses-slugify.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '\bslugify\b' +flags: i +target: + source: file + path: .work/post-permalink/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/prompt.md new file mode 100644 index 00000000000..51177f031a9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/prompt.md @@ -0,0 +1,10 @@ +--- +description: "LOW tier, no trigger: a draft adds post permalinks and writes its own slug helper although blog/text.py already has `slugify`, which AC2's accent folding needs (context/plan-reviewer.md: build what existing code already does). Also seeded: a Behavior item that names a file and a step instead of an observable claim (agents/plan-reviewer.md third check). hard: the duplicate is visible only by reading code the plan never cites. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] +runs: 3 +max_turns: 150 +timeout_seconds: 2700 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md reuses blog.text.slugify, rewrites the file-naming Behavior item as a claim, and does not change blog/models.py; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/post-permalink/PLAN.md`. Its Brief, its phases and the design early exit (`.work/post-permalink/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/behavior-is-a-claim.json b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/behavior-is-a-claim.json new file mode 100644 index 00000000000..e1e411ba17d --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/behavior-is-a-claim.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: The slug is lowercase ASCII words with accents folded (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/blast-radius-low.json new file mode 100644 index 00000000000..f1eb1c1bdd9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/blast-radius-low.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: LOW\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** LOW", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: MEDIUM\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/model-untouched.json b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/model-untouched.json new file mode 100644 index 00000000000..6f77d4bfdde --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/model-untouched.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n| `blog/models.py` | modify |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reuses-slugify.json b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reuses-slugify.json new file mode 100644 index 00000000000..ae7329ce5ef --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reuses-slugify.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: The slug is lowercase ASCII words with accents folded (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, import `slugify` from `blog.text`, which already folds accents.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{slugify(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Post permalinks\n\n## Brief\n\n### Problem\nPosts have no stable address: the site links to them by list position, so links break when a post is added.\n\n### Goal\nEach post has a permalink built from its publication date and title.\n\n### Acceptance criteria\n- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded.\n- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`.\n- AC3: Two posts with the same title in different months get different permalinks.\n\n### Out-of-scope\n- Redirects from the old list-position links.\n- Custom slugs chosen per post.\n\n## Design\n\nEarly exit (tier C): One new function over an existing dataclass; no new types.\n\n## Plan\n\n### Behavior\n- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1.\n- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add permalink [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`.\n- Add `permalink(post)` returning `f\"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/\"`.\n- In `tests/test_posts.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `blog/posts.py` | modify |\n| `tests/test_posts.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..86031c3df49 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/post-permalink/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/post-permalink/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/scaffold.sh new file mode 100755 index 00000000000..e5ab7f6a532 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-blog-permalink/scaffold.sh @@ -0,0 +1,116 @@ +#!/usr/bin/env bash +# A blog package with a shared slugify helper, and a draft plan for permalinks that writes its own. +set -euo pipefail + +mkdir -p .work/post-permalink/design blog tests +cat >README.md <<'EOF' +# blog + +Posts, listings and shared text helpers (blog/text.py). +EOF +: >blog/__init__.py +cat >blog/text.py <<'EOF' +"""Text helpers shared by the blog modules.""" +import re +import unicodedata + + +def slugify(title): + """Lowercase ASCII words joined by hyphens: 'Café Notes!' -> 'cafe-notes'.""" + ascii_title = unicodedata.normalize("NFKD", title).encode("ascii", "ignore").decode() + return "-".join(re.findall(r"[a-z0-9]+", ascii_title.lower())) +EOF +cat >blog/models.py <<'EOF' +from dataclasses import dataclass +from datetime import date + + +@dataclass +class Post: + title: str + body: str + published_on: date +EOF +cat >blog/posts.py <<'EOF' +"""Post listing helpers.""" + + +def recent(posts, limit=10): + return sorted(posts, key=lambda post: post.published_on, reverse=True)[:limit] +EOF +cat >tests/test_posts.py <<'EOF' +import unittest +from datetime import date + +from blog.models import Post +from blog.posts import recent + + +class RecentTest(unittest.TestCase): + def test_newest_first(self): + old = Post("a", "", date(2026, 1, 1)) + new = Post("b", "", date(2026, 2, 1)) + self.assertEqual(recent([old, new]), [new, old]) + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/post-permalink/PLAN.md <<'EOF' +# Post permalinks + +## Brief + +### Problem +Posts have no stable address: the site links to them by list position, so links break when a post is added. + +### Goal +Each post has a permalink built from its publication date and title. + +### Acceptance criteria +- AC1: `permalink(post)` returns `/YYYY/MM//`, with the month zero-padded. +- AC2: The slug is lowercase ASCII words joined by hyphens, accents folded: `Café Notes!` becomes `cafe-notes`. +- AC3: Two posts with the same title in different months get different permalinks. + +### Out-of-scope +- Redirects from the old list-position links. +- Custom slugs chosen per post. + +## Design + +Early exit (tier C): One new function over an existing dataclass; no new types. + +## Plan + +### Behavior +- B1: A post's permalink is its publication year and zero-padded month followed by its slug (AC1, AC3). Delivered by Phase 1. +- B2: Edit `blog/posts.py` to add `_make_slug` (AC2). Delivered by Phase 1. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| Every post has `published_on` set | plan | Phase 1 Sanity Check | Minor: proceed and note | + +### Phase 1: Add permalink [STATUS] +**Satisfies:** AC1, AC2, AC3 +**Delivers:** B1, B2 + +- In `blog/posts.py`, add `_make_slug(title)`: lowercase it, replace spaces with hyphens, and drop every character outside `[a-z0-9-]`. +- Add `permalink(post)` returning `f"/{post.published_on:%Y}/{post.published_on:%m}/{_make_slug(post.title)}/"`. +- In `tests/test_posts.py`, add one case per criterion. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `blog/posts.py` | modify | +| `tests/test_posts.py` | modify | +EOF +cat >.work/post-permalink/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: One new function over an existing dataclass; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/post-permalink diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/case.yaml b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/case.yaml new file mode 100644 index 00000000000..c786263e163 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-low-ledger-delimiter +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md new file mode 100644 index 00000000000..1bd8b345899 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '^###\s*Phase[\s\S]*?^\s*[-*]\s+[^\n]*(?:ValueError|unsupported delimiter|\brefus|\breject|\braise)' +flags: mi +target: + source: file + path: .work/export-delimiter/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/assumption-row.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/assumption-row.md new file mode 100644 index 00000000000..9ae87fcafc6 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/assumption-row.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '###\s*Load-bearing assumptions[^\n]*\n(?:(?!#)[^\n]*\n)*?\|[^\n]*(?:semicolon|separator|spreadsheet|import)' +flags: i +target: + source: file + path: .work/export-delimiter/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/blast-radius-low.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/blast-radius-low.md new file mode 100644 index 00000000000..05787b008a6 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/blast-radius-low.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b' +flags: i +target: + source: file + path: .work/export-delimiter/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/no-new-dependency.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/no-new-dependency.md new file mode 100644 index 00000000000..065d6e7ee20 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/no-new-dependency.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: '^\|\s*`?(?:requirements[\w.-]*\.txt|pyproject\.toml|setup\.cfg|Pipfile)`?\s*\|' +flags: mi +match: not_contains +weight: 0.001 +target: + source: file + path: .work/export-delimiter/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/prompt.md new file mode 100644 index 00000000000..3113ec31ad4 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/prompt.md @@ -0,0 +1,10 @@ +--- +description: "LOW tier, no trigger: a two-file draft adds a delimiter option to a CSV export. Seeded: a Brief captured assumption with no Load-bearing row (agents/plan-reviewer.md second check), and AC3 claimed by Phase 1 that no work item builds (the Delivers check). hard: both seeds pass check-plan-outcome.sh, so only a reader who compares the Brief with the phases catches them. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] +runs: 3 +max_turns: 150 +timeout_seconds: 2700 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md carries a Load-bearing row for the semicolon-import assumption and a Phase 1 work item that refuses an unsupported delimiter with ValueError, and adds no dependency; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/export-delimiter/PLAN.md`. Its Brief, its phases and the design early exit (`.work/export-delimiter/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/ac3-built.json b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/ac3-built.json new file mode 100644 index 00000000000..141c0d9a301 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/ac3-built.json @@ -0,0 +1,30 @@ +{ + "pass": [ + { + "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- Raise `ValueError` when `delimiter` is not `,` or `;`, before `open`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Reject any delimiter outside `,` and `;` before opening the file.\n", + "why": "reject paraphrase" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Raise when `delimiter not in (\",\", \";\")`.\n", + "why": "raise paraphrase" + } + ], + "fail": [ + { + "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n", + "why": "the draft as seeded" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Pass `delimiter` to `csv.writer`.\n", + "why": "no validation step" + }, + { + "answer": "- B3: An unsupported delimiter is refused (AC3).\n### Phase 2: Next step [TODO]\n- Pass `delimiter` to `csv.writer`.\n", + "why": "only the Behavior item mentions it" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/assumption-row.json b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/assumption-row.json new file mode 100644 index 00000000000..8689092a92d --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/assumption-row.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Finance's spreadsheet imports semicolon-separated files with no hint line | Brief | Phase 1 Sanity Check | Moderate: flag and continue |\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/blast-radius-low.json new file mode 100644 index 00000000000..d15f46ec0d4 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/blast-radius-low.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: LOW\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** LOW", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: MEDIUM\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/no-new-dependency.json b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/no-new-dependency.json new file mode 100644 index 00000000000..abb4884c5d1 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/no-new-dependency.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n| `requirements.txt` | modify |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..492c1e86a57 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/export-delimiter/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/export-delimiter/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/scaffold.sh new file mode 100755 index 00000000000..2a2875cdcc5 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/scaffold.sh @@ -0,0 +1,105 @@ +#!/usr/bin/env bash +# A small ledger package whose CSV export takes no delimiter, and a draft plan to add one. +set -euo pipefail + +mkdir -p .work/export-delimiter/design ledger tests +cat >README.md <<'EOF' +# ledger + +Ledger rows and the CSV export finance opens in a spreadsheet. +EOF +: >ledger/__init__.py +cat >ledger/export.py <<'EOF' +"""Write ledger rows to CSV for the finance spreadsheet.""" +import csv + +HEADER = ["date", "account", "amount"] + + +def export_csv(rows, path): + with open(path, "w", newline="", encoding="utf-8") as handle: + writer = csv.writer(handle) + writer.writerow(HEADER) + for row in rows: + writer.writerow([row["date"], row["account"], row["amount"]]) +EOF +cat >tests/test_export.py <<'EOF' +import os +import tempfile +import unittest + +from ledger.export import export_csv + + +class ExportTest(unittest.TestCase): + def test_default_is_comma(self): + path = os.path.join(tempfile.mkdtemp(), "out.csv") + export_csv([{"date": "2026-03-01", "account": "4000", "amount": "12.50"}], path) + with open(path, encoding="utf-8") as handle: + self.assertEqual(handle.read().splitlines(), ["date,account,amount", "2026-03-01,4000,12.50"]) + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/export-delimiter/PLAN.md <<'EOF' +# Export delimiter + +## Brief + +### Problem +Finance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column. + +### Goal +`export_csv` takes an optional delimiter, so those staff get one value per column. + +### Acceptance criteria +- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today. +- AC2: `export_csv(rows, path, delimiter=";")` writes every row, header included, with semicolons between fields. +- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened. + +### Out-of-scope +- Changing the column set or order. +- Spreadsheet (.xlsx) output. + +### Captured assumptions +- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column. + +## Design + +Early exit (tier C): One optional parameter on one existing function; no new types. + +## Plan + +### Behavior +- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1. +- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1. +- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note | + +### Phase 1: Add a delimiter option [STATUS] +**Satisfies:** AC1, AC2, AC3 +**Delivers:** B1, B2, B3 + +- In `ledger/export.py`, add `delimiter=","` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`. +- In `tests/test_export.py`, add a case that writes two rows with `delimiter=";"` and reads the file back line by line. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `ledger/export.py` | modify | +| `tests/test_export.py` | modify | +EOF +cat >.work/export-delimiter/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: One optional parameter on one existing function; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/export-delimiter diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/case.yaml b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/case.yaml new file mode 100644 index 00000000000..018be92f519 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-low-notes-since +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/appetite-fit-line.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/appetite-fit-line.md new file mode 100644 index 00000000000..a778eb9cf9c --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/appetite-fit-line.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: 'appetite fit' +flags: i +target: + source: file + path: .work/notes-since/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/assumption-row.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/assumption-row.md new file mode 100644 index 00000000000..df70dbaece4 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/assumption-row.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '###\s*Load-bearing assumptions[^\n]*\n(?:(?!#)[^\n]*\n)*?\|[^\n]*(?:ISO|8601|YYYY|created|mobile|date format)' +flags: i +target: + source: file + path: .work/notes-since/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/blast-radius-low.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/blast-radius-low.md new file mode 100644 index 00000000000..320a897819e --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/blast-radius-low.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b' +flags: i +target: + source: file + path: .work/notes-since/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/no-dateutil.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/no-dateutil.md new file mode 100644 index 00000000000..1711b43e365 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/no-dateutil.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: 'dateutil' +flags: i +match: not_contains +weight: 0.001 +target: + source: file + path: .work/notes-since/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/prompt.md new file mode 100644 index 00000000000..cf629502818 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/prompt.md @@ -0,0 +1,10 @@ +--- +description: "LOW tier, no trigger: a draft adds a --since filter to a notes CLI. Seeded: the Brief records an appetite but the plan has no Appetite fit line (agents/plan-reviewer.md appetite check), and a Brief captured assumption about the date format has no Load-bearing row. hard: neither is a script criterion, and a run that resumes at the review step skips Step 2, where the main thread would write the line. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] +runs: 3 +max_turns: 150 +timeout_seconds: 2700 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md has an Appetite fit line and a Load-bearing row for the ISO date assumption, and adds no dateutil dependency; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/notes-since/PLAN.md`. Its Brief, its phases and the design early exit (`.work/notes-since/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/appetite-fit-line.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/appetite-fit-line.json new file mode 100644 index 00000000000..52d3dd18d9f --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/appetite-fit-line.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\nAppetite fit: one flag and one filter on one subcommand fits half a day.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/assumption-row.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/assumption-row.json new file mode 100644 index 00000000000..deabaa3afc6 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/assumption-row.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Note `created` values are ISO 8601 dates | Brief | Phase 1 Sanity Check | Major: stop and replan |\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/blast-radius-low.json new file mode 100644 index 00000000000..7f4c110048f --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/blast-radius-low.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: LOW\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** LOW", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: MEDIUM\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/no-dateutil.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/no-dateutil.json new file mode 100644 index 00000000000..200422d9f9d --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/no-dateutil.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# notes list --since\n\n## Brief\n\n### Problem\n`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's.\n\n### Goal\n`notes list --since DATE` shows only notes created on or after DATE.\n\n### Acceptance criteria\n- AC1: `notes list` without `--since` prints what it prints today.\n- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date.\n- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag.\n\n### Out-of-scope\n- Relative dates such as \"last week\".\n- An `--until` flag.\n\n### Work size\nAppetite: half a day.\n\n### Captured assumptions\n- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format.\n\n## Design\n\nEarly exit (tier C): One flag on an existing subcommand; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1.\n- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1.\n- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add --since [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `notes/cli.py`, add `--since` to the `list` subparser with `type=dateutil.parser.parse`, so argparse reports a bad value as a usage error naming `--since`.\n- Keep only notes whose `date.fromisoformat(note[\"created\"])` is on or after the value.\n- In `tests/test_cli.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `notes/cli.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..0760295f5a8 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/notes-since/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/notes-since/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-notes-since/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/scaffold.sh new file mode 100755 index 00000000000..7740f7fa6a7 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-notes-since/scaffold.sh @@ -0,0 +1,129 @@ +#!/usr/bin/env bash +# A notes CLI whose list command has no date filter, and a draft plan to add --since. +set -euo pipefail + +mkdir -p .work/notes-since/design notes tests +cat >README.md <<'EOF' +# notes + +A small notes CLI: `python -m notes.cli list`. +EOF +: >notes/__init__.py +cat >notes/store.py <<'EOF' +"""Notes kept in notes.json as a list of {"created": "YYYY-MM-DD", "text": ...}.""" +import json +from pathlib import Path + +STORE = Path("notes.json") + + +def list_notes(path=STORE): + if not path.exists(): + return [] + return json.loads(path.read_text(encoding="utf-8")) +EOF +cat >notes/cli.py <<'EOF' +import argparse + +from notes import store + + +def main(argv=None): + parser = argparse.ArgumentParser(prog="notes") + sub = parser.add_subparsers(dest="command", required=True) + sub.add_parser("list") + args = parser.parse_args(argv) + if args.command == "list": + for note in store.list_notes(): + print(note["created"], note["text"]) + + +if __name__ == "__main__": + main() +EOF +cat >tests/test_cli.py <<'EOF' +import io +import unittest +from contextlib import redirect_stdout +from unittest import mock + +from notes import cli + + +class ListTest(unittest.TestCase): + def test_lists_all(self): + notes = [{"created": "2026-03-01", "text": "a"}] + out = io.StringIO() + with mock.patch("notes.store.list_notes", return_value=notes), redirect_stdout(out): + cli.main(["list"]) + self.assertEqual(out.getvalue(), "2026-03-01 a\n") + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/notes-since/PLAN.md <<'EOF' +# notes list --since + +## Brief + +### Problem +`notes list` prints every note ever written, so people with hundreds of notes scroll to find this week's. + +### Goal +`notes list --since DATE` shows only notes created on or after DATE. + +### Acceptance criteria +- AC1: `notes list` without `--since` prints what it prints today. +- AC2: `notes list --since 2026-03-01` prints only notes created on or after that date. +- AC3: An unparsable `--since` value exits with status 2 and an error naming the flag. + +### Out-of-scope +- Relative dates such as "last week". +- An `--until` flag. + +### Work size +Appetite: half a day. + +### Captured assumptions +- Every note's `created` value is an ISO 8601 date (YYYY-MM-DD): revisit if a note file from the old mobile app shows another format. + +## Design + +Early exit (tier C): One flag on an existing subcommand; no new types. + +## Plan + +### Behavior +- B1: Without the flag the listing is unchanged (AC1). Delivered by Phase 1. +- B2: The flag hides notes created before the given date (AC2). Delivered by Phase 1. +- B3: A bad date is refused with a usage error naming the flag (AC3). Delivered by Phase 1. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| argparse reports a `type=` conversion error with exit status 2 | plan | Phase 1 Sanity Check | Minor: proceed and note | + +### Phase 1: Add --since [STATUS] +**Satisfies:** AC1, AC2, AC3 +**Delivers:** B1, B2, B3 + +- In `notes/cli.py`, add `--since` to the `list` subparser with `type=date.fromisoformat`, so argparse reports a bad value as a usage error naming `--since`. +- Keep only notes whose `date.fromisoformat(note["created"])` is on or after the value. +- In `tests/test_cli.py`, add one case per criterion. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `notes/cli.py` | modify | +| `tests/test_cli.py` | modify | +EOF +cat >.work/notes-since/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: One flag on an existing subcommand; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/notes-since diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/case.yaml b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/case.yaml new file mode 100644 index 00000000000..c149fddff97 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-low-retry-backoff-cap +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/blast-radius-low.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/blast-radius-low.md new file mode 100644 index 00000000000..03849010240 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/blast-radius-low.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b' +flags: i +target: + source: file + path: .work/backoff-cap/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/no-jitter.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/no-jitter.md new file mode 100644 index 00000000000..ae714377cab --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/no-jitter.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: 'import random|random\.(?:uniform|random|randint)' +flags: i +match: not_contains +weight: 0.001 +target: + source: file + path: .work/backoff-cap/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/real-ceiling-constant.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/real-ceiling-constant.md new file mode 100644 index 00000000000..7c157b61ad0 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/real-ceiling-constant.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: 'MAX_BACKOFF_SECONDS' +flags: i +target: + source: file + path: .work/backoff-cap/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md new file mode 100644 index 00000000000..eeecbed8589 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '^## Plan[\s\S]*?(?:regression|\btest_\w+|\btests?\b)' +flags: mi +target: + source: file + path: .work/backoff-cap/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/prompt.md new file mode 100644 index 00000000000..a3014faa15e --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/prompt.md @@ -0,0 +1,10 @@ +--- +description: "LOW tier, no trigger: a bug-fix draft caps retry backoff in one module. Seeded: it names a config constant `MAX_DELAY` that fetcher/config.py does not define (the ceiling is `MAX_BACKOFF_SECONDS`), and the bug fix names no regression test (context/plan-reviewer.md Design alignment). hard: the author believes its own constant name, and the mechanical gate checks neither. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] +runs: 3 +max_turns: 150 +timeout_seconds: 2700 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md uses MAX_BACKOFF_SECONDS for the ceiling and names a regression test for the uncapped delay, and adds no jitter; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/backoff-cap/PLAN.md`. Its Brief, its phases and the design early exit (`.work/backoff-cap/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/blast-radius-low.json new file mode 100644 index 00000000000..b111684f896 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/blast-radius-low.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: LOW\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** LOW", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: MEDIUM\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/no-jitter.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/no-jitter.json new file mode 100644 index 00000000000..ed62bec4877 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/no-jitter.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Add `random.uniform(0, 1)` seconds to each delay.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/real-ceiling-constant.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/real-ceiling-constant.json new file mode 100644 index 00000000000..9fa76210a60 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/real-ceiling-constant.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_BACKOFF_SECONDS)`; `MAX_BACKOFF_SECONDS` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json new file mode 100644 index 00000000000..34790649a43 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- In `tests/test_retry.py`, add a regression case: `backoff(8)` returns 30.0.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + }, + { + "answer": "## Plan\n- Add a unit test asserting `backoff(8) == 30.0`.\n", + "why": "test without the word regression" + } + ], + "fail": [ + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", + "why": "the draft as seeded" + }, + { + "answer": "## Plan\n- Bound the delay with `min`.\n", + "why": "no test" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..c2529d5c8b5 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/backoff-cap/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/backoff-cap/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/scaffold.sh new file mode 100755 index 00000000000..cebd79632a0 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/scaffold.sh @@ -0,0 +1,97 @@ +#!/usr/bin/env bash +# A fetcher package whose retry backoff grows past the configured ceiling, and a draft bug-fix plan. +set -euo pipefail + +mkdir -p .work/backoff-cap/design fetcher tests +cat >README.md <<'EOF' +# fetcher + +HTTP fetcher with retries. Tunables live in fetcher/config.py. +EOF +: >fetcher/__init__.py +cat >fetcher/config.py <<'EOF' +"""Tunables for the HTTP fetcher.""" +BASE_DELAY_SECONDS = 0.5 +MAX_BACKOFF_SECONDS = 30.0 +MAX_ATTEMPTS = 8 +EOF +cat >fetcher/retry.py <<'EOF' +"""Retry delays for failed fetches.""" +from fetcher import config + + +def backoff(attempt): + """Seconds to wait before retry number `attempt` (1-based).""" + return config.BASE_DELAY_SECONDS * 2 ** (attempt - 1) +EOF +cat >tests/test_retry.py <<'EOF' +import unittest + +from fetcher.retry import backoff + + +class BackoffTest(unittest.TestCase): + def test_doubles(self): + self.assertEqual([backoff(n) for n in (1, 2, 3)], [0.5, 1.0, 2.0]) + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/backoff-cap/PLAN.md <<'EOF' +# Cap retry backoff + +## Brief + +### Problem +With eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes. + +### Goal +No retry waits longer than the configured ceiling. + +### Acceptance criteria +- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling. +- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more. +- AC3: `backoff(0)` and negative attempts raise `ValueError`. + +### Out-of-scope +- Adding jitter. +- Changing the number of attempts. + +## Design + +Early exit (tier C): A bound inside one existing function; no new types. + +## Plan + +### Behavior +- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1. +- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1. +- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note | + +### Phase 1: Bound the delay [STATUS] +**Satisfies:** AC1, AC2, AC3 +**Delivers:** B1, B2, B3 + +- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling. +- Raise `ValueError` at the top of `backoff` when `attempt` is below 1. + +**Sanity Check:** `python -c "from fetcher.retry import backoff; assert backoff(8) <= 30"` exits 0. + +### Files affected +| File | Action | +|---|---| +| `fetcher/retry.py` | modify | +EOF +cat >.work/backoff-cap/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: A bound inside one existing function; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" fix/backoff-cap diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/case.yaml b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/case.yaml new file mode 100644 index 00000000000..1143281a83a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-low-temperature-decimals +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/blast-radius-low.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/blast-radius-low.md new file mode 100644 index 00000000000..de151c94dad --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/blast-radius-low.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b' +flags: i +target: + source: file + path: .work/temp-decimals/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/no-decimal-type.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/no-decimal-type.md new file mode 100644 index 00000000000..b726461ee3a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/no-decimal-type.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: '\bDecimal\b' +match: not_contains +weight: 0.001 +target: + source: file + path: .work/temp-decimals/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md new file mode 100644 index 00000000000..df01edc1a71 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: '^\s*[-*]\s+(?![^\n]*\b(?:out[ -]of[ -]scope|not|no|never|without|leave|leaves|unchanged|drop\w*|cut|remov\w*|exclude|excluded|excludes|excluding\w*)\b)[^\n]*(?:fahrenheit|unit\s*=|°F)' +flags: mi +match: not_contains +target: + source: file + path: .work/temp-decimals/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/sanity-check-command.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/sanity-check-command.md new file mode 100644 index 00000000000..752cff52031 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/sanity-check-command.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'Sanity Check[^\n]*\b(?:looks? (?:right|correct|good|fine)|as expected|works (?:correctly|properly))' +flags: i +match: not_contains +target: + source: file + path: .work/temp-decimals/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/prompt.md new file mode 100644 index 00000000000..8b244f0d782 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/prompt.md @@ -0,0 +1,10 @@ +--- +description: "LOW tier, no trigger: a draft adds a decimals option to a temperature formatter. Seeded: a work item adds a unit switch the Brief's Out-of-scope excludes (context/plan-reviewer.md Plan mechanics), and the phase's Sanity Check is a vague `looks right` the script counts as present. routine for the out-of-scope seed, which Step 4.7's reading check also covers; hard for the vague Sanity Check. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] +runs: 3 +max_turns: 150 +timeout_seconds: 2700 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md drops the unit switch, replaces the vague Sanity Check with a command, and does not switch to Decimal; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/temp-decimals/PLAN.md`. Its Brief, its phases and the design early exit (`.work/temp-decimals/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/blast-radius-low.json new file mode 100644 index 00000000000..6e74cd6ff9a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/blast-radius-low.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: LOW\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** LOW", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, return `f\"{celsius:.{digits}f}°C\"`, and raise `ValueError` when it is below 0.\n- Add `unit=\"C\"` and convert to Fahrenheit when `unit=\"F\"`, so the panel can switch later.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** the formatted output looks right on the dashboard.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: MEDIUM\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/no-decimal-type.json b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/no-decimal-type.json new file mode 100644 index 00000000000..1f05a1b730a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/no-decimal-type.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, return `f\"{celsius:.{digits}f}°C\"`, and raise `ValueError` when it is below 0.\n- Add `unit=\"C\"` and convert to Fahrenheit when `unit=\"F\"`, so the panel can switch later.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** the formatted output looks right on the dashboard.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, quantize a `Decimal` of the value, and raise `ValueError` when it is below 0.\n- Add `unit=\"C\"` and convert to Fahrenheit when `unit=\"F\"`, so the panel can switch later.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** the formatted output looks right on the dashboard.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/out-of-scope-cut.json b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/out-of-scope-cut.json new file mode 100644 index 00000000000..a80fbe72be8 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/out-of-scope-cut.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, return `f\"{celsius:.{digits}f}°C\"`, and raise `ValueError` when it is below 0.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** the formatted output looks right on the dashboard.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + }, + { + "answer": "- Fahrenheit output dropped: out of scope.\n", + "why": "names it while cutting it" + } + ], + "fail": [ + { + "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, return `f\"{celsius:.{digits}f}°C\"`, and raise `ValueError` when it is below 0.\n- Add `unit=\"C\"` and convert to Fahrenheit when `unit=\"F\"`, so the panel can switch later.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** the formatted output looks right on the dashboard.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", + "why": "the draft as seeded" + }, + { + "answer": "- Support Fahrenheit through a `unit=` argument.\n", + "why": "paraphrased scope creep" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..614f6243110 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/temp-decimals/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/temp-decimals/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/sanity-check-command.json b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/sanity-check-command.json new file mode 100644 index 00000000000..2b927d5093a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/samples/sanity-check-command.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, return `f\"{celsius:.{digits}f}°C\"`, and raise `ValueError` when it is below 0.\n- Add `unit=\"C\"` and convert to Fahrenheit when `unit=\"F\"`, so the panel can switch later.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Temperature decimals\n\n## Brief\n\n### Problem\nThe greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift.\n\n### Goal\n`format_temp` can show a fixed number of decimals.\n\n### Acceptance criteria\n- AC1: `format_temp(c)` with no `digits` returns what it returns today.\n- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`.\n- AC3: A negative `digits` raises `ValueError`.\n\n### Out-of-scope\n- Units other than Celsius.\n- Localized decimal separators.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1.\n- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1.\n- B3: A negative `digits` is refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add digits [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `weather/format.py`, add `digits=None`; when set, return `f\"{celsius:.{digits}f}°C\"`, and raise `ValueError` when it is below 0.\n- Add `unit=\"C\"` and convert to Fahrenheit when `unit=\"F\"`, so the panel can switch later.\n- In `tests/test_format.py`, add one case per criterion.\n\n**Sanity Check:** the formatted output looks right on the dashboard.\n\n### Files affected\n| File | Action |\n|---|---|\n| `weather/format.py` | modify |\n| `tests/test_format.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/scaffold.sh new file mode 100755 index 00000000000..a279cb20c6a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/scaffold.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash +# A weather formatter that rounds to whole degrees, and a draft plan to show decimals. +set -euo pipefail + +mkdir -p .work/temp-decimals/design tests weather +cat >README.md <<'EOF' +# weather + +Formatting helpers for the greenhouse dashboard. +EOF +: >weather/__init__.py +cat >weather/format.py <<'EOF' +def format_temp(celsius): + return f"{round(celsius)}°C" +EOF +cat >tests/test_format.py <<'EOF' +import unittest + +from weather.format import format_temp + + +class FormatTest(unittest.TestCase): + def test_whole_degrees(self): + self.assertEqual(format_temp(21.6), "22°C") + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/temp-decimals/PLAN.md <<'EOF' +# Temperature decimals + +## Brief + +### Problem +The greenhouse dashboard rounds 21.6 to 22, and staff need the tenth of a degree to see heater drift. + +### Goal +`format_temp` can show a fixed number of decimals. + +### Acceptance criteria +- AC1: `format_temp(c)` with no `digits` returns what it returns today. +- AC2: `format_temp(21.64, digits=1)` returns `21.6°C`. +- AC3: A negative `digits` raises `ValueError`. + +### Out-of-scope +- Units other than Celsius. +- Localized decimal separators. + +## Design + +Early exit (tier C): One optional parameter on one existing function; no new types. + +## Plan + +### Behavior +- B1: Without `digits` the text is unchanged (AC1). Delivered by Phase 1. +- B2: With `digits` the temperature shows that many decimals (AC2). Delivered by Phase 1. +- B3: A negative `digits` is refused (AC3). Delivered by Phase 1. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| Callers pass floats or ints, never strings | plan | Phase 1 Sanity Check | Minor: proceed and note | + +### Phase 1: Add digits [STATUS] +**Satisfies:** AC1, AC2, AC3 +**Delivers:** B1, B2, B3 + +- In `weather/format.py`, add `digits=None`; when set, return `f"{celsius:.{digits}f}°C"`, and raise `ValueError` when it is below 0. +- Add `unit="C"` and convert to Fahrenheit when `unit="F"`, so the panel can switch later. +- In `tests/test_format.py`, add one case per criterion. + +**Sanity Check:** the formatted output looks right on the dashboard. + +### Files affected +| File | Action | +|---|---| +| `weather/format.py` | modify | +| `tests/test_format.py` | modify | +EOF +cat >.work/temp-decimals/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: One optional parameter on one existing function; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/temp-decimals diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/case.yaml b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/case.yaml new file mode 100644 index 00000000000..6fb6a7a9488 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-low-till-price-lookup +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/behavior-is-a-claim.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/behavior-is-a-claim.md new file mode 100644 index 00000000000..da60f2a2ed1 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/behavior-is-a-claim.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: '^\s*[-*]\s*B\d+:\s*(?:Edit|Update|Modify|Change|Add)\b' +flags: mi +match: not_contains +target: + source: file + path: .work/price-lookup/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/blast-radius-low.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/blast-radius-low.md new file mode 100644 index 00000000000..c26a8c96bdf --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/blast-radius-low.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bLOW\b' +flags: i +target: + source: file + path: .work/price-lookup/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/missing-file-handled.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/missing-file-handled.md new file mode 100644 index 00000000000..65c4f088af4 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/missing-file-handled.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: 'FileNotFoundError|OSError|\.exists\(\)|\.is_file\(\)' +flags: i +target: + source: file + path: .work/price-lookup/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/no-cache.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/no-cache.md new file mode 100644 index 00000000000..e2b0f063e2c --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/no-cache.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: 'lru_cache|functools\.cache|@cache\b' +flags: i +match: not_contains +weight: 0.001 +target: + source: file + path: .work/price-lookup/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/prompt.md b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/prompt.md new file mode 100644 index 00000000000..639dfb0ced0 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/prompt.md @@ -0,0 +1,10 @@ +--- +description: "LOW tier, no trigger: a draft adds price_for and claims load_prices already returns an empty dict for a missing file; catalog/prices.py opens the file unguarded, so it raises FileNotFoundError (context/plan-reviewer.md: findings backed by a code path). Also seeded: a Behavior item that is a step of work, not a claim. hard: the false claim reads as settled fact and only a read of the code disproves it. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-low, hard] +runs: 3 +max_turns: 150 +timeout_seconds: 2700 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md handles a missing prices file (FileNotFoundError or an existence check), rewrites the step-of-work Behavior item as a claim, and adds no cache; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/price-lookup/PLAN.md`. Its Brief, its phases and the design early exit (`.work/price-lookup/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/behavior-is-a-claim.json b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/behavior-is-a-claim.json new file mode 100644 index 00000000000..9a349712051 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/behavior-is-a-claim.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: An edit to the price file shows in the next lookup (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own.\n- Read the file on every call, so a staff edit shows at once.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own.\n- Read the file on every call, so a staff edit shows at once.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/blast-radius-low.json b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/blast-radius-low.json new file mode 100644 index 00000000000..00e7f5d22b2 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/blast-radius-low.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: LOW\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** LOW", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own.\n- Read the file on every call, so a staff edit shows at once.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: MEDIUM\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/missing-file-handled.json b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/missing-file-handled.json new file mode 100644 index 00000000000..5c37a9f6c2f --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/missing-file-handled.json @@ -0,0 +1,26 @@ +{ + "pass": [ + { + "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; catch `FileNotFoundError` from `load_prices` and return `None` (AC3).\n- Read the file on every call, so a staff edit shows at once.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + }, + { + "answer": "- Wrap `load_prices` in `try`/`except OSError` and return `None`.\n", + "why": "OSError paraphrase" + }, + { + "answer": "- Return `None` when `path.exists()` is false.\n", + "why": "existence check" + } + ], + "fail": [ + { + "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own.\n- Read the file on every call, so a staff edit shows at once.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", + "why": "the draft as seeded" + }, + { + "answer": "- `load_prices` already returns {} for a missing file.\n", + "why": "repeats the false claim" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/no-cache.json b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/no-cache.json new file mode 100644 index 00000000000..d379f2f5648 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/no-cache.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own.\n- Read the file on every call, so a staff edit shows at once.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Price lookup without a crash\n\n## Brief\n\n### Problem\nThe till crashes with a traceback when a product has no price, and staff restart it mid-sale.\n\n### Goal\n`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price.\n\n### Acceptance criteria\n- AC1: `price_for(\"A100\")` returns the value from `data/prices.json`.\n- AC2: An unknown SKU returns `None`.\n- AC3: A missing price file returns `None`.\n- AC4: An edit to `data/prices.json` shows in the next call without a restart.\n\n### Out-of-scope\n- Editing prices from the till.\n\n## Design\n\nEarly exit (tier C): One new function beside an existing loader; no new types.\n\n## Plan\n\n### Behavior\n- B1: A known SKU returns its price (AC1). Delivered by Phase 1.\n- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1.\n- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Add price_for [TODO]\n**Satisfies:** AC1, AC2, AC3, AC4\n**Delivers:** B1, B2, B3\n\n- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own.\n- Wrap `load_prices` in `functools.lru_cache` so repeat lookups skip the disk.\n- In `tests/test_prices.py`, add one case per criterion.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `catalog/prices.py` | modify |\n| `tests/test_prices.py` | modify |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..57a464b8eb9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/price-lookup/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/price-lookup/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/scaffold.sh new file mode 100755 index 00000000000..4122aa0bbdd --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-low-till-price-lookup/scaffold.sh @@ -0,0 +1,105 @@ +#!/usr/bin/env bash +# A catalog package whose price loader raises on a missing file, and a draft plan that says it does not. +set -euo pipefail + +mkdir -p .work/price-lookup/design catalog data tests +cat >README.md <<'EOF' +# catalog + +Price lookups for the till. Staff edit data/prices.json while the shop is open. +EOF +: >catalog/__init__.py +cat >catalog/prices.py <<'EOF' +"""Price lookups from data/prices.json, which staff edit while the shop is open.""" +import json +from pathlib import Path + +PRICES_PATH = Path("data/prices.json") + + +def load_prices(path=PRICES_PATH): + with open(path, encoding="utf-8") as handle: + return json.load(handle) +EOF +cat >data/prices.json <<'EOF' +{"A100": 3.5, "B200": 12.0} +EOF +cat >tests/test_prices.py <<'EOF' +import json +import os +import tempfile +import unittest + +from catalog.prices import load_prices + + +class LoadTest(unittest.TestCase): + def test_reads_json(self): + path = os.path.join(tempfile.mkdtemp(), "prices.json") + with open(path, "w", encoding="utf-8") as handle: + json.dump({"A100": 3.5}, handle) + self.assertEqual(load_prices(path), {"A100": 3.5}) + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/price-lookup/PLAN.md <<'EOF' +# Price lookup without a crash + +## Brief + +### Problem +The till crashes with a traceback when a product has no price, and staff restart it mid-sale. + +### Goal +`price_for(sku)` returns the price, or `None` when the product or the price file is missing, so the till can ask staff to enter a price. + +### Acceptance criteria +- AC1: `price_for("A100")` returns the value from `data/prices.json`. +- AC2: An unknown SKU returns `None`. +- AC3: A missing price file returns `None`. +- AC4: An edit to `data/prices.json` shows in the next call without a restart. + +### Out-of-scope +- Editing prices from the till. + +## Design + +Early exit (tier C): One new function beside an existing loader; no new types. + +## Plan + +### Behavior +- B1: A known SKU returns its price (AC1). Delivered by Phase 1. +- B2: An unknown SKU or a missing file gives no price instead of an error (AC2, AC3). Delivered by Phase 1. +- B3: Add cases in `tests/test_prices.py` for an edit between calls (AC4). Delivered by Phase 1. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| `data/prices.json` holds a flat object of SKU to price | plan | Phase 1 Sanity Check | Minor: proceed and note | + +### Phase 1: Add price_for [STATUS] +**Satisfies:** AC1, AC2, AC3, AC4 +**Delivers:** B1, B2, B3 + +- In `catalog/prices.py`, add `price_for(sku, path=PRICES_PATH)` returning `load_prices(path).get(sku)`; `load_prices` already returns an empty dict when the file is missing, so AC3 needs no branch of its own. +- Read the file on every call, so a staff edit shows at once. +- In `tests/test_prices.py`, add one case per criterion. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `catalog/prices.py` | modify | +| `tests/test_prices.py` | modify | +EOF +cat >.work/price-lookup/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: One new function beside an existing loader; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" fix/price-lookup diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/case.yaml b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/case.yaml new file mode 100644 index 00000000000..a32d574d24d --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-medium-catalogue-author-search +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/ac3-sort-built.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/ac3-sort-built.md new file mode 100644 index 00000000000..a1c4ed18e80 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/ac3-sort-built.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '^###\s*Phase[\s\S]*?^\s*[-*]\s+(?![^\n]*already sorts)[^\n]*\bsort' +flags: mi +target: + source: file + path: .work/author-search/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/appetite-fit-line.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/appetite-fit-line.md new file mode 100644 index 00000000000..4b8bbb0de9b --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/appetite-fit-line.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: 'appetite fit' +flags: i +target: + source: file + path: .work/author-search/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/blast-radius-medium.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/blast-radius-medium.md new file mode 100644 index 00000000000..55e0fe300ed --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/blast-radius-medium.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bMEDIUM\b' +flags: i +target: + source: file + path: .work/author-search/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/no-new-dependency.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/no-new-dependency.md new file mode 100644 index 00000000000..694d3bbc60c --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/no-new-dependency.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: '^\|\s*`?(?:requirements[\w.-]*\.txt|pyproject\.toml|setup\.cfg|Pipfile)`?\s*\||rapidfuzz|whoosh' +flags: mi +match: not_contains +weight: 0.001 +target: + source: file + path: .work/author-search/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/prompt.md new file mode 100644 index 00000000000..640fa26b7dd --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/prompt.md @@ -0,0 +1,10 @@ +--- +description: "MEDIUM tier, no trigger: a five-file draft adds author search. Seeded: the Brief records an appetite and the plan has no Appetite fit line, and the plan says search_title already sorts by title, so AC3 needs no work, while library/search.py returns matches in catalogue order. hard: the sorting claim is false only on reading the code, and no script criterion covers either seed. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] +runs: 3 +max_turns: 200 +timeout_seconds: 3600 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md has an Appetite fit line and a work item that sorts author results by title, and adds no search dependency; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/author-search/PLAN.md`. Its Brief, its phases and the design early exit (`.work/author-search/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/ac3-sort-built.json b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/ac3-sort-built.json new file mode 100644 index 00000000000..dbcc4894fa7 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/ac3-sort-built.json @@ -0,0 +1,26 @@ +{ + "pass": [ + { + "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query, sorted by title (AC3); `search_title` returns catalogue order, so it does not sort for us.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Return `sorted(matches, key=lambda book: book[\"title\"])` from `search_author`.\n", + "why": "code paraphrase" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Sort author results by title before printing.\n", + "why": "prose paraphrase" + } + ], + "fail": [ + { + "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the draft as seeded" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- `search_title` already sorts by title, so nothing to do.\n", + "why": "repeats the false claim" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/appetite-fit-line.json b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/appetite-fit-line.json new file mode 100644 index 00000000000..ae7b1fd0d14 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/appetite-fit-line.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\nAppetite fit: one index map, one search function and one flag fit a day.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/blast-radius-medium.json b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/blast-radius-medium.json new file mode 100644 index 00000000000..0cbc3f56192 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/blast-radius-medium.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: MEDIUM\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** MEDIUM", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: HIGH\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/no-new-dependency.json b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/no-new-dependency.json new file mode 100644 index 00000000000..6357d9aea67 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/no-new-dependency.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Search by author\n\n## Brief\n\n### Problem\nPatrons ask at the desk for books by an author because the kiosk searches titles only.\n\n### Goal\nThe kiosk finds books by author name.\n\n### Acceptance criteria\n- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case.\n- AC2: Title search is unchanged.\n- AC3: Author results are listed in title order.\n\n### Out-of-scope\n- Fuzzy matching.\n- Search by ISBN.\n\n### Work size\nAppetite: one day.\n\n## Design\n\nEarly exit (tier C): New functions beside existing ones; no new types.\n\n## Plan\n\n### Behavior\n- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2.\n- B2: Title search answers as before (AC2). Delivered by Phase 2.\n- B3: Author results come in title order (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Index authors [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`.\n- In `tests/test_index.py`, add a case for two books by one author.\n\n**Sanity Check:** `python -m unittest tests.test_index` exits 0.\n\n### Phase 2: Search and command [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way.\n- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given.\n- In `tests/test_cli.py`, add a case for `--author` and keep the title case.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `library/index.py` | modify |\n| `library/search.py` | modify |\n| `library/cli.py` | modify |\n| `tests/test_index.py` | modify |\n| `tests/test_cli.py` | modify |\n| `requirements.txt` | modify |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..bec7fc2e79a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/author-search/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/author-search/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/scaffold.sh new file mode 100755 index 00000000000..70d99c2ec2f --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-catalogue-author-search/scaffold.sh @@ -0,0 +1,170 @@ +#!/usr/bin/env bash +# A library kiosk package that searches titles only, and a draft plan to add author search. +set -euo pipefail + +mkdir -p .work/author-search/design data library tests +cat >README.md <<'EOF' +# library + +The catalogue kiosk: `python -m library.cli search WORD`. +EOF +: >library/__init__.py +cat >data/books.json <<'EOF' +[{"title": "Beloved", "author": "Toni Morrison"}, {"title": "Jazz", "author": "Toni Morrison"}, {"title": "Emma", "author": "Jane Austen"}] +EOF +cat >library/catalogue.py <<'EOF' +import json +from pathlib import Path + + +def load_books(path=Path("data/books.json")): + return json.loads(path.read_text(encoding="utf-8")) +EOF +cat >library/index.py <<'EOF' +"""In-memory index of the catalogue.""" +from library.catalogue import load_books + + +def build(): + books = load_books() + by_word = {} + for book in books: + for word in book["title"].split(): + by_word.setdefault(word.lower(), []).append(book) + return {"by_word": by_word, "books": books} +EOF +cat >library/search.py <<'EOF' +def search_title(index, query): + """Books with a title word equal to the query, in catalogue order.""" + return list(index["by_word"].get(query.lower(), [])) +EOF +cat >library/cli.py <<'EOF' +import argparse + +from library import index, search + + +def main(argv=None): + parser = argparse.ArgumentParser(prog="library") + sub = parser.add_subparsers(dest="command", required=True) + find = sub.add_parser("search") + find.add_argument("query") + args = parser.parse_args(argv) + for book in search.search_title(index.build(), args.query): + print(book["title"], "-", book["author"]) + + +if __name__ == "__main__": + main() +EOF +cat >tests/test_index.py <<'EOF' +import unittest +from unittest import mock + +from library import index + + +class IndexTest(unittest.TestCase): + def test_title_words(self): + with mock.patch("library.index.load_books", return_value=[{"title": "Big Sky", "author": "A"}]): + self.assertIn("sky", index.build()["by_word"]) + + +if __name__ == "__main__": + unittest.main() +EOF +cat >tests/test_cli.py <<'EOF' +import io +import unittest +from contextlib import redirect_stdout +from unittest import mock + +from library import cli + + +class CliTest(unittest.TestCase): + def test_title_search(self): + books = [{"title": "Emma", "author": "Jane Austen"}] + out = io.StringIO() + with mock.patch("library.index.load_books", return_value=books), redirect_stdout(out): + cli.main(["search", "emma"]) + self.assertEqual(out.getvalue(), "Emma - Jane Austen\n") + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/author-search/PLAN.md <<'EOF' +# Search by author + +## Brief + +### Problem +Patrons ask at the desk for books by an author because the kiosk searches titles only. + +### Goal +The kiosk finds books by author name. + +### Acceptance criteria +- AC1: `search --author morrison` lists every book whose author contains `morrison`, in any case. +- AC2: Title search is unchanged. +- AC3: Author results are listed in title order. + +### Out-of-scope +- Fuzzy matching. +- Search by ISBN. + +### Work size +Appetite: one day. + +## Design + +Early exit (tier C): New functions beside existing ones; no new types. + +## Plan + +### Behavior +- B1: An author query lists every book by a matching author, case ignored (AC1). Delivered by Phase 1 and Phase 2. +- B2: Title search answers as before (AC2). Delivered by Phase 2. +- B3: Author results come in title order (AC3). Delivered by Phase 2. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| Each book has exactly one author string | plan | Phase 1 Sanity Check | Moderate: flag and continue | + +### Phase 1: Index authors [STATUS] +**Satisfies:** AC1 +**Delivers:** B1 + +- In `library/index.py`, add a `by_author` map from each lowercased author string to its books, built beside `by_word`. +- In `tests/test_index.py`, add a case for two books by one author. + +**Sanity Check:** `python -m unittest tests.test_index` exits 0. + +### Phase 2: Search and command [STATUS] +**Satisfies:** AC1, AC2, AC3 +**Delivers:** B1, B2, B3 + +- In `library/search.py`, add `search_author(index, query)` returning books whose author contains the lowercased query; AC3 needs no work, since `search_title` already sorts its results by title and `search_author` shapes results the same way. +- In `library/cli.py`, add `--author` to the `search` command and call `search_author` when it is given. +- In `tests/test_cli.py`, add a case for `--author` and keep the title case. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `library/index.py` | modify | +| `library/search.py` | modify | +| `library/cli.py` | modify | +| `tests/test_index.py` | modify | +| `tests/test_cli.py` | modify | +EOF +cat >.work/author-search/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: New functions beside existing ones; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/author-search diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/case.yaml b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/case.yaml new file mode 100644 index 00000000000..1c90e506620 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-medium-invoice-discounts +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/ac4-built.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/ac4-built.md new file mode 100644 index 00000000000..fc00036f9f5 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/ac4-built.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '^###\s*Phase[\s\S]*?^\s*[-*]\s+[^\n]*(?:\brender|-15%|percentage)' +flags: mi +target: + source: file + path: .work/invoice-discounts/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/assumption-row.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/assumption-row.md new file mode 100644 index 00000000000..e7295a0e42a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/assumption-row.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '###\s*Load-bearing assumptions[^\n]*\n(?:(?!#)[^\n]*\n)*?\|[^\n]*(?:discount_pct|\b100\b|credit|between 0)' +flags: i +target: + source: file + path: .work/invoice-discounts/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/blast-radius-medium.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/blast-radius-medium.md new file mode 100644 index 00000000000..d35392a090f --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/blast-radius-medium.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bMEDIUM\b' +flags: i +target: + source: file + path: .work/invoice-discounts/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/keeps-decimal-rounding.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/keeps-decimal-rounding.md new file mode 100644 index 00000000000..aeb81e8c581 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/keeps-decimal-rounding.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: '\bfloat\(|round\([^)\n]*,\s*2\s*\)' +flags: i +match: not_contains +weight: 0.001 +target: + source: file + path: .work/invoice-discounts/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/prompt.md new file mode 100644 index 00000000000..d37b3bc1b0b --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/prompt.md @@ -0,0 +1,10 @@ +--- +description: "MEDIUM tier, no trigger: a three-phase-file draft applies a stored line discount to invoice totals. Seeded: Phase 2 claims AC4 and B3 (the invoice text shows the percentage) but no work item touches billing/render.py (agents/plan-reviewer.md Delivers check), and a Brief captured assumption has no Load-bearing row. hard: the script sees AC4 on a Satisfies line and passes it. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] +runs: 3 +max_turns: 200 +timeout_seconds: 3600 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md has a work item that changes the rendered invoice to show the percentage and a Load-bearing row for the 0-100 discount assumption, and keeps Decimal rounding through money.round_cents; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/invoice-discounts/PLAN.md`. Its Brief, its phases and the design early exit (`.work/invoice-discounts/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/ac4-built.json b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/ac4-built.json new file mode 100644 index 00000000000..bb5d29700c4 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/ac4-built.json @@ -0,0 +1,30 @@ +{ + "pass": [ + { + "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In `billing/render.py`, print `-%` beside each discounted line's total.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", + "why": "the plan with the seeded defect fixed" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Show `-15%` next to each discounted total in the text invoice.\n", + "why": "output paraphrase" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Update `render_text` to append the discount percentage.\n", + "why": "function name" + } + ], + "fail": [ + { + "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", + "why": "the draft as seeded" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Keep summing line totals.\n", + "why": "no render step" + }, + { + "answer": "- B3: The invoice text shows each line's discount percentage (AC4).\n### Phase 2: Next step [TODO]\n- Keep summing line totals.\n", + "why": "only the Behavior item mentions it" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/assumption-row.json b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/assumption-row.json new file mode 100644 index 00000000000..abdb285b281 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/assumption-row.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `discount_pct` is between 0 and 100 | Brief | Phase 1 Sanity Check | Major: stop and replan |\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/blast-radius-medium.json b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/blast-radius-medium.json new file mode 100644 index 00000000000..4d6e19c20a0 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/blast-radius-medium.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: MEDIUM\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** MEDIUM", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: HIGH\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/keeps-decimal-rounding.json b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/keeps-decimal-rounding.json new file mode 100644 index 00000000000..347c19e3386 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/keeps-decimal-rounding.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Apply line discounts on invoices\n\n## Brief\n\n### Problem\nSales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand.\n\n### Goal\nInvoices charge the discounted price and show the discount.\n\n### Acceptance criteria\n- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50.\n- AC2: The invoice subtotal and tax use the discounted line totals.\n- AC3: An item with `discount_pct` 0 totals as today.\n- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`.\n\n### Out-of-scope\n- Invoice-level discounts.\n- Changes to the order form.\n\n### Captured assumptions\n- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100.\n\n## Design\n\nEarly exit (tier C): Uses the existing `discount_pct` field; no new types or fields.\n\n## Plan\n\n### Behavior\n- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1.\n- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2.\n- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Discount line totals [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and round it with `round(float(total), 2)`.\n- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent.\n\n**Sanity Check:** `python -m unittest tests.test_pricing` exits 0.\n\n### Phase 2: Carry discounted totals through the invoice [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B2, B3\n\n- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount.\n- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `billing/pricing.py` | modify |\n| `tests/test_pricing.py` | modify |\n| `tests/test_invoice.py` | create |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..958f82f90e5 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/invoice-discounts/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/invoice-discounts/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/scaffold.sh new file mode 100755 index 00000000000..3ecaaa1f6bc --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-invoice-discounts/scaffold.sh @@ -0,0 +1,158 @@ +#!/usr/bin/env bash +# A billing package that stores a line discount it never applies, and a draft plan to apply and show it. +set -euo pipefail + +mkdir -p .work/invoice-discounts/design billing tests +cat >README.md <<'EOF' +# billing + +Invoices for the order desk. Money rounding lives in billing/money.py. +EOF +: >billing/__init__.py +cat >billing/models.py <<'EOF' +from dataclasses import dataclass, field +from decimal import Decimal + + +@dataclass +class LineItem: + description: str + quantity: int + unit_price: Decimal + # Recorded by the order form; invoices do not apply it yet. + discount_pct: Decimal = Decimal("0") + + +@dataclass +class Invoice: + number: str + items: list = field(default_factory=list) +EOF +cat >billing/money.py <<'EOF' +from decimal import ROUND_HALF_UP, Decimal + + +def round_cents(amount): + return Decimal(amount).quantize(Decimal("0.01"), rounding=ROUND_HALF_UP) +EOF +cat >billing/pricing.py <<'EOF' +from billing import money + + +def line_total(item): + return money.round_cents(item.quantity * item.unit_price) +EOF +cat >billing/invoice.py <<'EOF' +from decimal import Decimal + +from billing import money, pricing + +TAX_RATE = Decimal("0.20") + + +def totals(invoice): + subtotal = sum((pricing.line_total(item) for item in invoice.items), Decimal("0")) + tax = money.round_cents(subtotal * TAX_RATE) + return {"subtotal": subtotal, "tax": tax, "total": subtotal + tax} +EOF +cat >billing/render.py <<'EOF' +from billing import invoice as invoice_totals +from billing import pricing + + +def render_text(invoice): + lines = ["Invoice " + invoice.number] + for item in invoice.items: + lines.append("%-30s %3d x %8s %10s" % (item.description, item.quantity, item.unit_price, pricing.line_total(item))) + sums = invoice_totals.totals(invoice) + lines.append("Subtotal %s Tax %s Total %s" % (sums["subtotal"], sums["tax"], sums["total"])) + return "\n".join(lines) +EOF +cat >tests/test_pricing.py <<'EOF' +import unittest +from decimal import Decimal + +from billing.models import LineItem +from billing.pricing import line_total + + +class LineTotalTest(unittest.TestCase): + def test_quantity_times_price(self): + self.assertEqual(line_total(LineItem("pens", 3, Decimal("10.00"))), Decimal("30.00")) + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/invoice-discounts/PLAN.md <<'EOF' +# Apply line discounts on invoices + +## Brief + +### Problem +Sales enters line discounts on the order form, which stores `discount_pct`, but invoices ignore it: customers are billed full price and sales writes credit notes by hand. + +### Goal +Invoices charge the discounted price and show the discount. + +### Acceptance criteria +- AC1: `line_total` applies `discount_pct`: 3 at 10.00 with 15 percent off totals 25.50. +- AC2: The invoice subtotal and tax use the discounted line totals. +- AC3: An item with `discount_pct` 0 totals as today. +- AC4: The text invoice shows each discounted line's percentage beside its total, as `-15%`. + +### Out-of-scope +- Invoice-level discounts. +- Changes to the order form. + +### Captured assumptions +- `discount_pct` is between 0 and 100: revisit if sales asks to record credit lines as discounts above 100. + +## Design + +Early exit (tier C): Uses the existing `discount_pct` field; no new types or fields. + +## Plan + +### Behavior +- B1: A discounted line totals at the discounted price; an undiscounted line totals as today (AC1, AC3). Delivered by Phase 1. +- B2: Invoice subtotal and tax follow the discounted lines (AC2). Delivered by Phase 2. +- B3: The invoice text shows each line's discount percentage (AC4). Delivered by Phase 2. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| Tax is computed on the subtotal, not per line | plan | Phase 2 Sanity Check | Moderate: flag and continue | + +### Phase 1: Discount line totals [STATUS] +**Satisfies:** AC1, AC3 +**Delivers:** B1 + +- In `billing/pricing.py`, compute `item.quantity * item.unit_price * (100 - item.discount_pct) / 100` and pass it through `money.round_cents`. +- In `tests/test_pricing.py`, add cases for 15 percent and 0 percent. + +**Sanity Check:** `python -m unittest tests.test_pricing` exits 0. + +### Phase 2: Carry discounted totals through the invoice [STATUS] +**Satisfies:** AC2, AC4 +**Delivers:** B2, B3 + +- `billing/invoice.py` keeps summing `pricing.line_total`, which now returns the discounted amount. +- In a new `tests/test_invoice.py`, assert the subtotal and tax of an invoice with one discounted line. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `billing/pricing.py` | modify | +| `tests/test_pricing.py` | modify | +| `tests/test_invoice.py` | create | +EOF +cat >.work/invoice-discounts/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: Uses the existing `discount_pct` field; no new types or fields. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/invoice-discounts diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/case.yaml b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/case.yaml new file mode 100644 index 00000000000..c148f12cd52 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-medium-overnight-shifts +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/assumption-row.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/assumption-row.md new file mode 100644 index 00000000000..b2e16befeb4 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/assumption-row.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '###\s*Load-bearing assumptions[^\n]*\n(?:(?!#)[^\n]*\n)*?\|[^\n]*(?:noon|12:00|midday|overnight|night)' +flags: i +target: + source: file + path: .work/overnight-shifts/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/blast-radius-medium.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/blast-radius-medium.md new file mode 100644 index 00000000000..ed1d4adc9e5 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/blast-radius-medium.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bMEDIUM\b' +flags: i +target: + source: file + path: .work/overnight-shifts/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/export-untouched.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/export-untouched.md new file mode 100644 index 00000000000..e60a9c22946 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/export-untouched.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: '^\|\s*`?timesheet/export\.py`?' +flags: mi +match: not_contains +weight: 0.001 +target: + source: file + path: .work/overnight-shifts/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md new file mode 100644 index 00000000000..2d9159a4ea2 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '^## Plan[\s\S]*?(?:regression|\btest_\w+|\btests?\b)' +flags: mi +target: + source: file + path: .work/overnight-shifts/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/prompt.md new file mode 100644 index 00000000000..40afc9ec722 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/prompt.md @@ -0,0 +1,10 @@ +--- +description: "MEDIUM tier, no trigger: a three-file bug-fix draft makes overnight shifts count forward to the next day. Seeded: the bug fix names no regression test (context/plan-reviewer.md Design alignment), and a Brief captured assumption has no Load-bearing row. hard: the Sanity Checks are real commands, so the plan looks verified while the bug has no test that would catch its return. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] +runs: 3 +max_turns: 200 +timeout_seconds: 3600 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md names a regression test for the 22:00-06:00 shift and has a Load-bearing row for the before-noon assumption, and does not touch timesheet/export.py; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/overnight-shifts/PLAN.md`. Its Brief, its phases and the design early exit (`.work/overnight-shifts/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/assumption-row.json b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/assumption-row.json new file mode 100644 index 00000000000..c33dea4e82a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/assumption-row.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Overnight shifts end before noon the next day | Brief | Phase 2 Sanity Check | Major: stop and replan |\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/blast-radius-medium.json b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/blast-radius-medium.json new file mode 100644 index 00000000000..74a0f7675ac --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/blast-radius-medium.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: MEDIUM\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** MEDIUM", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: HIGH\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/export-untouched.json b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/export-untouched.json new file mode 100644 index 00000000000..b2419860b7d --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/export-untouched.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n| `timesheet/export.py` | modify |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json new file mode 100644 index 00000000000..f417ff75c53 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n- In `tests/test_totals.py`, add a regression case: `22:00-06:00` totals 8.0.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..bd02cd5b58f --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/overnight-shifts/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/overnight-shifts/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/scaffold.sh new file mode 100755 index 00000000000..ee58d8924b1 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/scaffold.sh @@ -0,0 +1,146 @@ +#!/usr/bin/env bash +# A timesheet package that rejects or miscounts overnight shifts, and a draft bug-fix plan. +set -euo pipefail + +mkdir -p .work/overnight-shifts/design tests timesheet +cat >README.md <<'EOF' +# timesheet + +Parses shift lines and totals hours for payroll (timesheet/export.py writes the payroll file). +EOF +: >timesheet/__init__.py +cat >timesheet/models.py <<'EOF' +from dataclasses import dataclass +from datetime import date, time + + +@dataclass +class Shift: + day: date + start: time + end: time +EOF +cat >timesheet/parse.py <<'EOF' +from datetime import date, time + +from timesheet.models import Shift + + +def parse_line(line): + """'2026-03-02 09:00-17:00' -> Shift.""" + day_text, span = line.split() + start_text, end_text = span.split("-") + start, end = time.fromisoformat(start_text), time.fromisoformat(end_text) + if end <= start: + raise ValueError("end before start: " + line) + return Shift(date.fromisoformat(day_text), start, end) +EOF +cat >timesheet/totals.py <<'EOF' +from collections import defaultdict +from datetime import datetime + + +def hours(shift): + start = datetime.combine(shift.day, shift.start) + end = datetime.combine(shift.day, shift.end) + return (end - start).total_seconds() / 3600 + + +def weekly_hours(shifts): + week = defaultdict(float) + for shift in shifts: + week[shift.day] += hours(shift) + return dict(week) +EOF +cat >timesheet/export.py <<'EOF' +def fmt_hours(value): + return "%.2f" % max(value, 0.0) + + +def payroll_rows(week): + return [(day.isoformat(), fmt_hours(total)) for day, total in sorted(week.items())] +EOF +cat >tests/test_totals.py <<'EOF' +import unittest + +from timesheet.parse import parse_line +from timesheet.totals import hours + + +class HoursTest(unittest.TestCase): + def test_day_shift(self): + self.assertEqual(hours(parse_line("2026-03-02 09:00-17:00")), 8.0) + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/overnight-shifts/PLAN.md <<'EOF' +# Count overnight shifts + +## Brief + +### Problem +Night staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes. + +### Goal +A shift that ends at or before its start time counts as ending the next day. + +### Acceptance criteria +- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours. +- AC2: Day shifts total as today. +- AC3: `weekly_hours` counts an overnight shift on the day it starts. + +### Out-of-scope +- Shifts longer than 24 hours. +- Daylight-saving adjustments. + +### Captured assumptions +- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00. + +## Design + +Early exit (tier C): Changes existing functions; no new types. + +## Plan + +### Behavior +- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2. +- B2: Day shifts total as before (AC2). Delivered by Phase 2. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note | + +### Phase 1: Accept overnight lines [STATUS] +**Satisfies:** AC1 +**Delivers:** B1 + +- In `timesheet/parse.py`, drop the `end before start` rejection. +- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`. + +**Sanity Check:** `python -c "from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day"` exits 0. + +### Phase 2: Count the hours [STATUS] +**Satisfies:** AC1, AC2, AC3 +**Delivers:** B1, B2 + +- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`. + +**Sanity Check:** `python -c "from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0"` exits 0. + +### Files affected +| File | Action | +|---|---| +| `timesheet/parse.py` | modify | +| `timesheet/models.py` | modify | +| `timesheet/totals.py` | modify | +EOF +cat >.work/overnight-shifts/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: Changes existing functions; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" fix/overnight-shifts diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/case.yaml b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/case.yaml new file mode 100644 index 00000000000..754b72070ed --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-medium-quiet-hours +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/ac3-built.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/ac3-built.md new file mode 100644 index 00000000000..1703cbc852a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/ac3-built.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: '^###\s*Phase[\s\S]*?^\s*[-*]\s+[^\n]*(?:\bdigest)' +flags: mi +target: + source: file + path: .work/quiet-hours/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/behavior-is-a-claim.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/behavior-is-a-claim.md new file mode 100644 index 00000000000..4bdd36e14a2 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/behavior-is-a-claim.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: '^\s*[-*]\s*B\d+:\s*(?:Edit|Update|Modify|Change|Add)\b' +flags: mi +match: not_contains +target: + source: file + path: .work/quiet-hours/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/blast-radius-medium.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/blast-radius-medium.md new file mode 100644 index 00000000000..f228aa6e914 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/blast-radius-medium.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bMEDIUM\b' +flags: i +target: + source: file + path: .work/quiet-hours/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/keeps-outbox.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/keeps-outbox.md new file mode 100644 index 00000000000..5fb2408605d --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/keeps-outbox.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: 'smtplib|send_mail|\bSMTP\b' +flags: i +match: not_contains +weight: 0.001 +target: + source: file + path: .work/quiet-hours/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/prompt.md new file mode 100644 index 00000000000..ff3f4d52b11 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/prompt.md @@ -0,0 +1,10 @@ +--- +description: "MEDIUM tier, no trigger: a five-file draft holds non-critical alerts during quiet hours. Seeded: a Behavior item names a file and a step (agents/plan-reviewer.md third check), and AC3 (held alerts reach the morning digest) is claimed through B3 while no work item touches alerts/digest.py. hard: the script passes both, and the held alerts are silently dropped by the plan as written. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] +runs: 3 +max_turns: 200 +timeout_seconds: 3600 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md has a work item that routes held alerts into the morning digest, rewrites the file-naming Behavior item as a claim, and keeps paging through the outbox rather than sending mail directly; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/quiet-hours/PLAN.md`. Its Brief, its phases and the design early exit (`.work/quiet-hours/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/ac3-built.json b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/ac3-built.json new file mode 100644 index 00000000000..601742b9e01 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/ac3-built.json @@ -0,0 +1,30 @@ +{ + "pass": [ + { + "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `alerts/evaluate.py`, pass each held alert to `digest.add` so the 07:30 digest lists it.\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", + "why": "the plan with the seeded defect fixed" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Call `digest.add(alert)` for every alert held in quiet hours.\n", + "why": "call paraphrase" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Queue each held warning for the 07:30 morning digest.\n", + "why": "prose paraphrase" + } + ], + "fail": [ + { + "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", + "why": "the draft as seeded" + }, + { + "answer": "### Phase 2: Next step [TODO]\n- Describe quiet hours in README.md.\n", + "why": "no digest step" + }, + { + "answer": "- B3: Held alerts reach the morning digest (AC3).\n### Phase 2: Next step [TODO]\n- Describe quiet hours in README.md.\n", + "why": "only the Behavior item mentions it" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/behavior-is-a-claim.json b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/behavior-is-a-claim.json new file mode 100644 index 00000000000..6e0dd364f6a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/behavior-is-a-claim.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Warnings and info alerts raised in quiet hours do not page (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/blast-radius-medium.json b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/blast-radius-medium.json new file mode 100644 index 00000000000..6c3f8378e4e --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/blast-radius-medium.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: MEDIUM\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** MEDIUM", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: HIGH\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/keeps-outbox.json b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/keeps-outbox.json new file mode 100644 index 00000000000..62fa1e3b341 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/keeps-outbox.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Quiet hours for alerts\n\n## Brief\n\n### Problem\nOn-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning.\n\n### Goal\nNon-critical alerts raised in quiet hours wait for the morning digest instead of paging.\n\n### Acceptance criteria\n- AC1: A critical alert in quiet hours is enqueued as today.\n- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued.\n- AC3: Each held alert appears in the next morning digest.\n- AC4: Outside quiet hours every alert is enqueued as today.\n\n### Out-of-scope\n- Per-person quiet hours.\n- Snoozing an alert.\n\n## Design\n\nEarly exit (tier C): Changes existing functions and reads existing settings; no new types.\n\n## Plan\n\n### Behavior\n- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1.\n- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1.\n- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Hold non-critical alerts [TODO]\n**Satisfies:** AC1, AC2, AC4\n**Delivers:** B1, B2\n\n- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight.\n- In `alerts/evaluate.py`, send the page with `smtplib` only when `rules.is_pageable(alert, now)`.\n- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert.\n\n**Sanity Check:** `python -m unittest tests.test_rules` exits 0.\n\n### Phase 2: Morning hand-off [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `README.md`, describe quiet hours and which severities still page.\n- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `alerts/rules.py` | modify |\n| `alerts/evaluate.py` | modify |\n| `README.md` | modify |\n| `tests/test_rules.py` | create |\n| `tests/test_evaluate.py` | create |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..9b655266bac --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/quiet-hours/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/quiet-hours/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/scaffold.sh new file mode 100755 index 00000000000..90cad6c726c --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-quiet-hours/scaffold.sh @@ -0,0 +1,137 @@ +#!/usr/bin/env bash +# An alerts package that pages for every alert at any hour, and a draft plan for quiet hours. +set -euo pipefail + +mkdir -p .work/quiet-hours/design alerts ops +cat >README.md <<'EOF' +# alerts + +Raises alerts, pages on-call through the outbox, and sends a morning digest. +EOF +: >alerts/__init__.py +cat >alerts/settings.py <<'EOF' +from datetime import time + +QUIET_START = time(22, 0) +QUIET_END = time(7, 0) +DIGEST_AT = time(7, 30) +EOF +cat >alerts/rules.py <<'EOF' +from dataclasses import dataclass + + +@dataclass +class Alert: + name: str + severity: str # "critical", "warning" or "info" +EOF +cat >alerts/outbox.py <<'EOF' +"""Pages are queued here; ops/schedule.txt runs flush() every minute, which sends them.""" +QUEUE = [] + + +def enqueue(alert): + QUEUE.append(alert) + + +def flush(send): + while QUEUE: + send(QUEUE.pop(0)) +EOF +cat >alerts/digest.py <<'EOF' +"""Morning digest, built at settings.DIGEST_AT from the alerts in DIGEST_QUEUE.""" +DIGEST_QUEUE = [] + + +def add(alert): + DIGEST_QUEUE.append(alert) + + +def build_digest(): + lines = ["%s (%s)" % (alert.name, alert.severity) for alert in DIGEST_QUEUE] + DIGEST_QUEUE.clear() + return "\n".join(lines) +EOF +cat >alerts/evaluate.py <<'EOF' +from alerts import outbox + + +def dispatch(alert, now): + outbox.enqueue(alert) +EOF +cat >ops/schedule.txt <<'EOF' +* * * * * python -m alerts.flush +30 7 * * * python -m alerts.send_digest +EOF +cat >.work/quiet-hours/PLAN.md <<'EOF' +# Quiet hours for alerts + +## Brief + +### Problem +On-call staff are paged at 3 a.m. for disk-usage warnings that could wait for the morning. + +### Goal +Non-critical alerts raised in quiet hours wait for the morning digest instead of paging. + +### Acceptance criteria +- AC1: A critical alert in quiet hours is enqueued as today. +- AC2: A warning or info alert raised between `QUIET_START` and `QUIET_END` is not enqueued. +- AC3: Each held alert appears in the next morning digest. +- AC4: Outside quiet hours every alert is enqueued as today. + +### Out-of-scope +- Per-person quiet hours. +- Snoozing an alert. + +## Design + +Early exit (tier C): Changes existing functions and reads existing settings; no new types. + +## Plan + +### Behavior +- B1: Critical alerts page at any hour, and every alert pages outside quiet hours (AC1, AC4). Delivered by Phase 1. +- B2: Update `alerts/evaluate.py` to check quiet hours (AC2). Delivered by Phase 1. +- B3: Held alerts reach the morning digest (AC3). Delivered by Phase 2. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| The quiet window wraps midnight (22:00 to 07:00) | plan | Phase 1 Sanity Check | Moderate: flag and continue | + +### Phase 1: Hold non-critical alerts [STATUS] +**Satisfies:** AC1, AC2, AC4 +**Delivers:** B1, B2 + +- In `alerts/rules.py`, add `is_pageable(alert, now)`: true for a critical alert, or when `now` is outside the window from `settings.QUIET_START` to `settings.QUIET_END`, which wraps midnight. +- In `alerts/evaluate.py`, call `outbox.enqueue(alert)` only when `rules.is_pageable(alert, now)`. +- In `tests/test_rules.py`, cover 03:00 and 12:00 for a warning and a critical alert. + +**Sanity Check:** `python -m unittest tests.test_rules` exits 0. + +### Phase 2: Morning hand-off [STATUS] +**Satisfies:** AC3 +**Delivers:** B3 + +- In `README.md`, describe quiet hours and which severities still page. +- In `tests/test_evaluate.py`, assert that a warning at 03:00 never reaches `outbox.enqueue`. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `alerts/rules.py` | modify | +| `alerts/evaluate.py` | modify | +| `README.md` | modify | +| `tests/test_rules.py` | create | +| `tests/test_evaluate.py` | create | +EOF +cat >.work/quiet-hours/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: Changes existing functions and reads existing settings; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/quiet-hours diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/case.yaml b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/case.yaml new file mode 100644 index 00000000000..6232944453d --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-medium-reorder-points +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/blast-radius-medium.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/blast-radius-medium.md new file mode 100644 index 00000000000..3964f89e9d6 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/blast-radius-medium.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bMEDIUM\b' +flags: i +target: + source: file + path: .work/reorder-points/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/no-new-dependency.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/no-new-dependency.md new file mode 100644 index 00000000000..29f3b751aa7 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/no-new-dependency.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: '^\|\s*`?(?:requirements[\w.-]*\.txt|pyproject\.toml|setup\.cfg|Pipfile)`?\s*\||tabulate' +flags: mi +match: not_contains +weight: 0.001 +target: + source: file + path: .work/reorder-points/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/out-of-scope-cut.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/out-of-scope-cut.md new file mode 100644 index 00000000000..abd35743bfd --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/out-of-scope-cut.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: '^\s*[-*]\s+(?![^\n]*\b(?:out[ -]of[ -]scope|not|no|never|without|leave|leaves|unchanged|drop\w*|cut|remov\w*|exclude|excluded|excludes|excluding\w*)\b)[^\n]*(?:weekly_report|weekly report)' +flags: mi +match: not_contains +target: + source: file + path: .work/reorder-points/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reuses-available.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reuses-available.md new file mode 100644 index 00000000000..9dd9b4010f9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reuses-available.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: 'available\(|`available`|stock\.available' +flags: i +target: + source: file + path: .work/reorder-points/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/prompt.md new file mode 100644 index 00000000000..cb30e10f38f --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/prompt.md @@ -0,0 +1,10 @@ +--- +description: "MEDIUM tier, no trigger: a five-file draft adds a reorder command and computes on-hand minus reserved itself although inventory/stock.py already exposes that as `available` (context/plan-reviewer.md: build what existing code already does). Also seeded: a work item changes the weekly report, which the Brief's Out-of-scope excludes. hard for the duplicate; routine for the out-of-scope item, which Step 4.7's reading check also covers. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] +runs: 3 +max_turns: 200 +timeout_seconds: 3600 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md uses stock.available, drops the weekly report change, and adds no table-formatting dependency; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/reorder-points/PLAN.md`. Its Brief, its phases and the design early exit (`.work/reorder-points/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/blast-radius-medium.json b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/blast-radius-medium.json new file mode 100644 index 00000000000..4b1d196a11a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/blast-radius-medium.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: MEDIUM\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** MEDIUM", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU.\n- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `inventory/report.py` | modify |\n| `tests/test_reorder.py` | create |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: HIGH\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/no-new-dependency.json b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/no-new-dependency.json new file mode 100644 index 00000000000..5f0f5ce0f8c --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/no-new-dependency.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU.\n- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `inventory/report.py` | modify |\n| `tests/test_reorder.py` | create |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU through `tabulate`.\n- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `inventory/report.py` | modify |\n| `tests/test_reorder.py` | create |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/out-of-scope-cut.json b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/out-of-scope-cut.json new file mode 100644 index 00000000000..59b62b404ea --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/out-of-scope-cut.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `tests/test_reorder.py` | create |\n", + "why": "the plan with the seeded defect fixed" + }, + { + "answer": "- The weekly report is out of scope and stays unchanged.\n", + "why": "names it while leaving it alone" + } + ], + "fail": [ + { + "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU.\n- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `inventory/report.py` | modify |\n| `tests/test_reorder.py` | create |\n", + "why": "the draft as seeded" + }, + { + "answer": "- Extend `weekly_report()` with the reorder column.\n", + "why": "paraphrased scope creep" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reuses-available.json b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reuses-available.json new file mode 100644 index 00000000000..eec93e85b86 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reuses-available.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, take `stock.available(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU.\n- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `inventory/report.py` | modify |\n| `tests/test_reorder.py` | create |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Reorder points\n\n## Brief\n\n### Problem\nBuyers notice a product is short only when an order cannot be filled.\n\n### Goal\nA `reorder` command lists the products whose free stock has fallen below their reorder point.\n\n### Acceptance criteria\n- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point.\n- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed.\n- AC3: The list is sorted by SKU.\n\n### Out-of-scope\n- Changing the weekly stock report.\n- Placing orders with suppliers.\n\n## Design\n\nEarly exit (tier C): A new command over existing stock functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Reorder check [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- Create `inventory/data/reorder_points.json` with the buyers' points.\n- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point.\n- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point.\n\n**Sanity Check:** `python -m unittest tests.test_reorder` exits 0.\n\n### Phase 2: Command and report [TODO]\n**Satisfies:** AC1, AC3\n**Delivers:** B1\n\n- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU.\n- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `inventory/data/reorder_points.json` | create |\n| `inventory/reorder.py` | create |\n| `inventory/cli.py` | modify |\n| `inventory/report.py` | modify |\n| `tests/test_reorder.py` | create |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..343740a60d9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/reorder-points/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/reorder-points/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/scaffold.sh new file mode 100755 index 00000000000..2418f72611a --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-reorder-points/scaffold.sh @@ -0,0 +1,125 @@ +#!/usr/bin/env bash +# An inventory package with an available-stock helper and a weekly report, and a draft plan for reorder points. +set -euo pipefail + +mkdir -p .work/reorder-points/design inventory +cat >README.md <<'EOF' +# inventory + +Stock levels, the weekly report and the buyers' CLI (`python -m inventory.cli`). +EOF +: >inventory/__init__.py +cat >inventory/stock.py <<'EOF' +"""Stock levels. STOCK and RESERVED are loaded from the warehouse export at start-up.""" +STOCK = {} +RESERVED = {} + + +def on_hand(sku): + return STOCK.get(sku, 0) + + +def reserved(sku): + return RESERVED.get(sku, 0) + + +def available(sku): + """Stock free to sell: on hand minus reserved, never below zero.""" + return max(on_hand(sku) - reserved(sku), 0) +EOF +cat >inventory/report.py <<'EOF' +from inventory import stock + + +def weekly_report(): + rows = [(sku, stock.on_hand(sku), stock.reserved(sku)) for sku in sorted(stock.STOCK)] + return "\n".join("%-10s %6d %6d" % row for row in rows) +EOF +cat >inventory/cli.py <<'EOF' +import argparse + +from inventory import report + + +def main(argv=None): + parser = argparse.ArgumentParser(prog="inventory") + sub = parser.add_subparsers(dest="command", required=True) + sub.add_parser("weekly") + args = parser.parse_args(argv) + if args.command == "weekly": + print(report.weekly_report()) + + +if __name__ == "__main__": + main() +EOF +cat >.work/reorder-points/PLAN.md <<'EOF' +# Reorder points + +## Brief + +### Problem +Buyers notice a product is short only when an order cannot be filled. + +### Goal +A `reorder` command lists the products whose free stock has fallen below their reorder point. + +### Acceptance criteria +- AC1: `inventory reorder` lists each SKU whose free stock (on hand minus reserved) is below its reorder point. +- AC2: Reorder points are read from `inventory/data/reorder_points.json`; a SKU with no entry is never listed. +- AC3: The list is sorted by SKU. + +### Out-of-scope +- Changing the weekly stock report. +- Placing orders with suppliers. + +## Design + +Early exit (tier C): A new command over existing stock functions; no new types. + +## Plan + +### Behavior +- B1: Products below their reorder point are listed by SKU (AC1, AC3). Delivered by Phase 1 and Phase 2. +- B2: Products with no reorder point are never listed (AC2). Delivered by Phase 1. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| Reserved stock never exceeds stock on hand | plan | Phase 1 Sanity Check | Moderate: flag and continue | + +### Phase 1: Reorder check [STATUS] +**Satisfies:** AC1, AC2 +**Delivers:** B1, B2 + +- Create `inventory/data/reorder_points.json` with the buyers' points. +- Add `inventory/reorder.py` with `below_reorder_point()`: for each SKU in the points file, compute `stock.on_hand(sku) - stock.reserved(sku)` and keep it when below the point. +- In `tests/test_reorder.py`, cover a SKU below, at and above its point, and one with no point. + +**Sanity Check:** `python -m unittest tests.test_reorder` exits 0. + +### Phase 2: Command and report [STATUS] +**Satisfies:** AC1, AC3 +**Delivers:** B1 + +- In `inventory/cli.py`, add `reorder`, printing `below_reorder_point()` sorted by SKU. +- In `inventory/report.py`, add a `Below reorder point` column to `weekly_report()`. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `inventory/data/reorder_points.json` | create | +| `inventory/reorder.py` | create | +| `inventory/cli.py` | modify | +| `inventory/report.py` | modify | +| `tests/test_reorder.py` | create | +EOF +cat >.work/reorder-points/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: A new command over existing stock functions; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/reorder-points diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/case.yaml b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/case.yaml new file mode 100644 index 00000000000..f55b6c1dd1e --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/case.yaml @@ -0,0 +1,4 @@ +schema_version: "1.1" +name: plan-reviewer-cut-medium-shipping-zones +context: + scaffold_script: scaffold.sh diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/blast-radius-medium.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/blast-radius-medium.md new file mode 100644 index 00000000000..a17afa8a022 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/blast-radius-medium.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'blast[ -]radius[^\n]{0,12}?\bMEDIUM\b' +flags: i +target: + source: file + path: .work/shipping-zones/PLAN.md +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/callers-untouched.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/callers-untouched.md new file mode 100644 index 00000000000..9a798239d0b --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/callers-untouched.md @@ -0,0 +1,10 @@ +--- +type: regex +pattern: '^\|\s*`?shop/views\.py`?' +flags: mi +match: not_contains +weight: 0.001 +target: + source: file + path: .work/shipping-zones/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reuses-country-region.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reuses-country-region.md new file mode 100644 index 00000000000..8f27c426e1f --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reuses-country-region.md @@ -0,0 +1,8 @@ +--- +type: regex +pattern: 'country_region|shop\.geo|shop/geo\.py|\bgeo\.' +flags: i +target: + source: file + path: .work/shipping-zones/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reviewer-not-dispatched.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reviewer-not-dispatched.md new file mode 100644 index 00000000000..1f7d67fa7b9 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/reviewer-not-dispatched.md @@ -0,0 +1,9 @@ +--- +type: tool_used +tool: Agent +input_match: 'subagent_type"\s*:\s*"[^"]*plan-reviewer' +min: 0 +max: 0 +arm: both +weight: 0.001 +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/sanity-check-command.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/sanity-check-command.md new file mode 100644 index 00000000000..f9f0f1140cf --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/graders/sanity-check-command.md @@ -0,0 +1,9 @@ +--- +type: regex +pattern: 'Sanity Check[^\n]*\b(?:looks? (?:right|correct|good|fine)|as expected|works (?:correctly|properly))' +flags: i +match: not_contains +target: + source: file + path: .work/shipping-zones/PLAN.md +--- diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/prompt.md b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/prompt.md new file mode 100644 index 00000000000..ab742efab59 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/prompt.md @@ -0,0 +1,10 @@ +--- +description: "MEDIUM tier, no trigger: a five-file draft prices shipping by destination zone and builds a new country-to-zone table although shop/geo.py already maps countries to regions (context/plan-reviewer.md: build what existing code already does). Also seeded: a vague Sanity Check. hard: the duplicate needs a read of a module the draft never mentions. Source: written for this suite from the before behavior (plan SKILL.md Step 3 at 6df2738455, agents/plan-reviewer.md, context/plan-reviewer.md)." +tags: [plan, plan-reviewer-cut, plan-reviewer-cut-medium, hard] +runs: 3 +max_turns: 200 +timeout_seconds: 3600 +allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write, Agent] +expected_outcome: "The final PLAN.md reuses shop.geo.country_region, gives every phase a command Sanity Check, and does not touch shop/views.py since checkout_summary already takes the country; the arm's reviewer dispatch matches its contract." +--- +/planning:plan Pick up the draft plan at `.work/shipping-zones/PLAN.md`. Its Brief, its phases and the design early exit (`.work/shipping-zones/design/design-resolution.md`) are written; take it from there to the approval presentation. Edit no code. diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/blast-radius-medium.json b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/blast-radius-medium.json new file mode 100644 index 00000000000..8d8cb9cbe84 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/blast-radius-medium.json @@ -0,0 +1,22 @@ +{ + "pass": [ + { + "answer": "Blast radius: MEDIUM\nReason: two files.", + "why": "plain line" + }, + { + "answer": "**Blast radius:** MEDIUM", + "why": "bold label" + } + ], + "fail": [ + { + "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`).\n- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** checkout totals look correct for each zone.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `tests/test_shipping.py` | modify |\n", + "why": "the draft as seeded has no assessment" + }, + { + "answer": "Blast radius: HIGH\nReason: shared code.", + "why": "another tier" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/callers-untouched.json b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/callers-untouched.json new file mode 100644 index 00000000000..b4ab38f501b --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/callers-untouched.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`).\n- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** checkout totals look correct for each zone.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `tests/test_shipping.py` | modify |\n", + "why": "the draft as seeded keeps the sound part" + } + ], + "fail": [ + { + "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`).\n- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** checkout totals look correct for each zone.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `shop/views.py` | modify |\n| `tests/test_shipping.py` | modify |\n", + "why": "the plan made the over-flagged change" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reuses-country-region.json b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reuses-country-region.json new file mode 100644 index 00000000000..7e7acbaf49f --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reuses-country-region.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Use `shop.geo.country_region`, which already maps codes to `home`, `eu` and `world`.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** checkout totals look correct for each zone.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `tests/test_shipping.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`).\n- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** checkout totals look correct for each zone.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `tests/test_shipping.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reviewer-not-dispatched.json b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reviewer-not-dispatched.json new file mode 100644 index 00000000000..ac8117a2c1d --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/reviewer-not-dispatched.json @@ -0,0 +1,54 @@ +{ + "pass": [ + { + "answer": [ + { + "tool": "Read", + "input": { + "file_path": ".work/shipping-zones/PLAN.md" + } + } + ], + "why": "no agent dispatched" + }, + { + "answer": [ + { + "tool": "Skill", + "input": { + "skill": "planning:devils-advocate" + } + } + ], + "why": "devils-advocate only" + }, + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "general-purpose", + "description": "Stress-test", + "prompt": "Run /planning:devils-advocate; the plan-reviewer was skipped." + } + } + ], + "why": "another agent whose prompt names the reviewer" + } + ], + "fail": [ + { + "answer": [ + { + "tool": "Agent", + "input": { + "subagent_type": "planning:plan-reviewer", + "description": "Review plan", + "prompt": "Review .work/shipping-zones/PLAN.md" + } + } + ], + "why": "dispatched the plan-reviewer" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/sanity-check-command.json b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/sanity-check-command.json new file mode 100644 index 00000000000..112b3a85f4f --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/samples/sanity-check-command.json @@ -0,0 +1,14 @@ +{ + "pass": [ + { + "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`).\n- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** `python -m unittest tests.test_shipping` exits 0.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `tests/test_shipping.py` | modify |\n", + "why": "the plan with the seeded defect fixed" + } + ], + "fail": [ + { + "answer": "# Shipping by zone\n\n## Brief\n\n### Problem\nEvery order pays the domestic flat rate, so orders shipped abroad lose money on postage.\n\n### Goal\nShipping cost depends on the destination's region.\n\n### Acceptance criteria\n- AC1: Region `home` keeps 4.95, free from a 50.00 cart.\n- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only.\n- AC3: The checkout summary shows the rate for the destination country.\n- AC4: A country code the shop does not know pays the `world` rate.\n\n### Out-of-scope\n- Carrier rate quotes.\n- Weight-based pricing.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2.\n- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2.\n- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue |\n\n### Phase 1: Zone table [TODO]\n**Satisfies:** AC2, AC4\n**Delivers:** B1\n\n- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`).\n- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks.\n\n**Sanity Check:** `python -c \"from shop.zones import zone_for; assert zone_for('ZZ') == 'world'\"` exits 0.\n\n### Phase 2: Rate by zone [TODO]\n**Satisfies:** AC1, AC2\n**Delivers:** B1, B2\n\n- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95.\n- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold.\n\n**Sanity Check:** checkout totals look correct for each zone.\n\n### Phase 3: Show the rate at checkout [TODO]\n**Satisfies:** AC3\n**Delivers:** B3\n\n- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `shop/data/zones.json` | create |\n| `shop/zones.py` | create |\n| `shop/shipping.py` | modify |\n| `shop/checkout.py` | modify |\n| `tests/test_shipping.py` | modify |\n", + "why": "the draft as seeded" + } + ] +} diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/scaffold.sh b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/scaffold.sh new file mode 100755 index 00000000000..499d1642717 --- /dev/null +++ b/plugins/planning/evals/plan-reviewer-cut-medium-shipping-zones/scaffold.sh @@ -0,0 +1,146 @@ +#!/usr/bin/env bash +# A shop package with a country-to-region map and flat shipping, and a draft plan for zone rates that builds its own map. +set -euo pipefail + +mkdir -p .work/shipping-zones/design shop tests +cat >README.md <<'EOF' +# shop + +Cart, checkout and shipping for the web shop. Country data lives in shop/geo.py. +EOF +: >shop/__init__.py +cat >shop/geo.py <<'EOF' +"""Country facts shared by checkout, tax and shipping.""" +REGIONS = {"GB": "home", "IE": "eu", "FR": "eu", "DE": "eu", "ES": "eu", "US": "world", "CA": "world", "AU": "world"} + + +def country_region(code): + """'home', 'eu' or 'world'; any code not listed is 'world'.""" + return REGIONS.get(code.upper(), "world") +EOF +cat >shop/settings.py <<'EOF' +from decimal import Decimal + +FREE_SHIPPING_MIN = Decimal("50.00") +FLAT_RATE = Decimal("4.95") +EOF +cat >shop/shipping.py <<'EOF' +from decimal import Decimal + +from shop import settings + + +def shipping_cost(cart_total): + if cart_total >= settings.FREE_SHIPPING_MIN: + return Decimal("0.00") + return settings.FLAT_RATE +EOF +cat >shop/checkout.py <<'EOF' +from shop import geo, shipping + + +def checkout_summary(cart, country): + total = sum(line["price"] * line["qty"] for line in cart) + return {"region": geo.country_region(country), "items": total, "shipping": shipping.shipping_cost(total)} +EOF +cat >shop/views.py <<'EOF' +from shop.checkout import checkout_summary + + +def checkout_page(request): + return checkout_summary(request["cart"], request["country"]) +EOF +cat >tests/test_shipping.py <<'EOF' +import unittest +from decimal import Decimal + +from shop.shipping import shipping_cost + + +class ShippingTest(unittest.TestCase): + def test_flat_rate(self): + self.assertEqual(shipping_cost(Decimal("10")), Decimal("4.95")) + + +if __name__ == "__main__": + unittest.main() +EOF +cat >.work/shipping-zones/PLAN.md <<'EOF' +# Shipping by zone + +## Brief + +### Problem +Every order pays the domestic flat rate, so orders shipped abroad lose money on postage. + +### Goal +Shipping cost depends on the destination's region. + +### Acceptance criteria +- AC1: Region `home` keeps 4.95, free from a 50.00 cart. +- AC2: Region `eu` pays 9.95 and `world` pays 19.95; free shipping applies to `home` only. +- AC3: The checkout summary shows the rate for the destination country. +- AC4: A country code the shop does not know pays the `world` rate. + +### Out-of-scope +- Carrier rate quotes. +- Weight-based pricing. + +## Design + +Early exit (tier C): Changes existing functions; no new types. + +## Plan + +### Behavior +- B1: Each destination pays its region's rate, unknown countries the world rate (AC2, AC4). Delivered by Phase 1 and Phase 2. +- B2: Domestic orders keep today's rate and free-shipping threshold (AC1). Delivered by Phase 2. +- B3: The checkout summary shows the destination's rate (AC3). Delivered by Phase 3. + +### Load-bearing assumptions + +| Assumption | Source | Falsifying check | Response if false | +|---|---|---|---| +| Country codes reach checkout as ISO 3166 alpha-2 | plan | Phase 3 Sanity Check | Moderate: flag and continue | + +### Phase 1: Zone table [STATUS] +**Satisfies:** AC2, AC4 +**Delivers:** B1 + +- Create `shop/data/zones.json` mapping each country code the shop ships to onto a zone (`home`, `eu`, `world`). +- Add `shop/zones.py` with `zone_for(country)`, which reads that file and returns `world` for a code it lacks. + +**Sanity Check:** `python -c "from shop.zones import zone_for; assert zone_for('ZZ') == 'world'"` exits 0. + +### Phase 2: Rate by zone [STATUS] +**Satisfies:** AC1, AC2 +**Delivers:** B1, B2 + +- In `shop/shipping.py`, add a keyword argument, `shipping_cost(cart_total, country=None)`, where `None` keeps today's domestic rate: `home` 4.95 or free from `settings.FREE_SHIPPING_MIN`, `eu` 9.95, `world` 19.95. +- In `tests/test_shipping.py`, add one case per zone and one at the free-shipping threshold. + +**Sanity Check:** checkout totals look correct for each zone. + +### Phase 3: Show the rate at checkout [STATUS] +**Satisfies:** AC3 +**Delivers:** B3 + +- In `shop/checkout.py`, pass `country`, which `checkout_summary` already receives, to `shipping_cost`. + +**Sanity Check:** `python -m unittest discover -s tests` exits 0. + +### Files affected +| File | Action | +|---|---| +| `shop/data/zones.json` | create | +| `shop/zones.py` | create | +| `shop/shipping.py` | modify | +| `shop/checkout.py` | modify | +| `tests/test_shipping.py` | modify | +EOF +cat >.work/shipping-zones/design/design-resolution.md <<'EOF' +outcome: early-exit +tier: C +reason: Changes existing functions; no new types. +EOF +bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/plan-reviewer-cut/init-repo.sh" feat/shipping-zones From 5e8c1ff9c95891c43bf5430a78b6f327507e0759 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Wed, 11 Nov 2026 18:30:31 -0500 Subject: [PATCH 2/6] chore(planning): add a no-release changelog fragment for the eval cases Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VQb9Q4F46UumaUgXeUdpYZ --- .changes/planning/test-plan-reviewer-cut-evals-151426f4.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 .changes/planning/test-plan-reviewer-cut-evals-151426f4.md diff --git a/.changes/planning/test-plan-reviewer-cut-evals-151426f4.md b/.changes/planning/test-plan-reviewer-cut-evals-151426f4.md new file mode 100644 index 00000000000..38071a2b4d8 --- /dev/null +++ b/.changes/planning/test-plan-reviewer-cut-evals-151426f4.md @@ -0,0 +1,5 @@ +--- +bump: none +--- + +No release: adds eval cases only (`plan-reviewer-cut-*` and their fixtures), which ship nothing users run. From d4e2c5a626f81d3cc7c226c499a9148c4b1886e6 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Wed, 11 Nov 2026 19:02:58 -0500 Subject: [PATCH 3/6] test(planning): tie five plan-reviewer-cut graders to their seeds The code-review lane showed four graders passing a plan that still carried its seeded defect and one failing a correct fix. Each pattern now counts only bullets under a `### Phase` heading and stops at the next heading; the regression-test graders require a test word and the defect on one bullet; the temperature out-of-scope grader excuses only a bullet that starts or ends with a cut verb. Each reviewer counterexample is a new labeled sample. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VQb9Q4F46UumaUgXeUdpYZ --- .../graders/ac3-built.md | 2 +- .../samples/ac3-built.json | 4 ++++ .../graders/regression-test-named.md | 2 +- .../samples/regression-test-named.json | 8 ++++++++ .../graders/out-of-scope-cut.md | 2 +- .../samples/out-of-scope-cut.json | 8 ++++++++ .../graders/regression-test-named.md | 2 +- .../samples/regression-test-named.json | 4 ++++ .../graders/out-of-scope-cut.md | 2 +- .../samples/out-of-scope-cut.json | 4 ++++ 10 files changed, 33 insertions(+), 5 deletions(-) diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md index 1bd8b345899..b5b4e95351e 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '^###\s*Phase[\s\S]*?^\s*[-*]\s+[^\n]*(?:ValueError|unsupported delimiter|\brefus|\breject|\braise)' +pattern: '^###\s*Phase(?:(?!^#{2,3}\s)[\s\S])*?^\s*[-*]\s+[^\n]*(?:ValueError|unsupported delimiter|\brefus|\breject|\braise)' flags: mi target: source: file diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/ac3-built.json b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/ac3-built.json index 141c0d9a301..6f53e13e46a 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/ac3-built.json +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/samples/ac3-built.json @@ -25,6 +25,10 @@ { "answer": "- B3: An unsupported delimiter is refused (AC3).\n### Phase 2: Next step [TODO]\n- Pass `delimiter` to `csv.writer`.\n", "why": "only the Behavior item mentions it" + }, + { + "answer": "# Export delimiter\n\n## Brief\n\n### Problem\nFinance staff whose spreadsheet splits columns on semicolons open the ledger export and get every row in one column.\n\n### Goal\n`export_csv` takes an optional delimiter, so those staff get one value per column.\n\n### Acceptance criteria\n- AC1: `export_csv(rows, path)` with no delimiter writes the same file as today.\n- AC2: `export_csv(rows, path, delimiter=\";\")` writes every row, header included, with semicolons between fields.\n- AC3: A delimiter other than `,` or `;` raises `ValueError` before the file is opened.\n\n### Out-of-scope\n- Changing the column set or order.\n- Spreadsheet (.xlsx) output.\n\n### Captured assumptions\n- The finance spreadsheet imports semicolon-separated files without a separator hint line: revisit if finance reports an import that puts every value in one column.\n\n## Design\n\nEarly exit (tier C): One optional parameter on one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: With no delimiter the export is unchanged (AC1). Delivered by Phase 1.\n- B2: A semicolon delimiter separates every field of every row (AC2). Delivered by Phase 1.\n- B3: An unsupported delimiter is refused before any file is written (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Rows always carry date, account and amount keys | plan | Phase 1 Sanity Check runs the existing export case | Minor: proceed and note |\n\n### Phase 1: Add a delimiter option [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `ledger/export.py`, add `delimiter=\",\"` to `export_csv` and pass it to `csv.writer(handle, delimiter=delimiter)`.\n- In `tests/test_export.py`, add a case that writes two rows with `delimiter=\";\"` and reads the file back line by line.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `ledger/export.py` | modify |\n| `tests/test_export.py` | modify |\n\n## Stress-test summary\n- AC3 should raise ValueError for an unsupported delimiter.\n", + "why": "the seeded draft with the only matching bullet under a later ## section" } ] } diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md index eeecbed8589..2edd33b55d4 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '^## Plan[\s\S]*?(?:regression|\btest_\w+|\btests?\b)' +pattern: '(?:^###\s*Phase|^##\s*Plan\b)(?:(?!^#{2,3}\s)[\s\S])*?^\s*[-*]\s+(?=[^\n]*(?:regression|\btests?\b|\btest_\w+|\bassert))(?=[^\n]*(?:backoff\(\s*(?:[7-9]|\d{2,})\s*\)|\bcap|\bceiling|MAX_DELAY|\b30(?:\.0)?\b))' flags: mi target: source: file diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json index 34790649a43..d44e0d784be 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json @@ -17,6 +17,14 @@ { "answer": "## Plan\n- Bound the delay with `min`.\n", "why": "no test" + }, + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", + "why": "the seeded draft with a tests-exit-0 Sanity Check" + }, + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Test Strategy\n- Bug fixes: no regression test; document an explicit carve-out.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", + "why": "the seeded draft with a Test Strategy section that turns the regression test down" } ] } diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md index df01edc1a71..11fd1309233 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '^\s*[-*]\s+(?![^\n]*\b(?:out[ -]of[ -]scope|not|no|never|without|leave|leaves|unchanged|drop\w*|cut|remov\w*|exclude|excluded|excludes|excluding\w*)\b)[^\n]*(?:fahrenheit|unit\s*=|°F)' +pattern: '(?:^###\s*Phase|^(? Date: Wed, 11 Nov 2026 19:29:17 -0500 Subject: [PATCH 4/6] test(planning): stop plan-reviewer-cut graders failing correct fixes Re-grading the kept re-run plans found graders failing plans that had fixed their seed: - 11 bullet graders counted only - and * items; they now also count numbered work items (1. or 1)). - The retry regression-test grader accepted the wrong MAX_DELAY and not the real MAX_BACKOFF_SECONDS; it accepts both now. - Both regression-test graders now also count items under a Test strategy heading, where the plan template puts tests. - Both sanity-check graders matched a summary quoting the old vague check; they now match only Sanity Check lines. Each false fail is a new labeled pass sample. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VQb9Q4F46UumaUgXeUdpYZ --- .../graders/ac3-built.md | 2 +- .../graders/regression-test-named.md | 2 +- .../samples/regression-test-named.json | 8 ++++++++ .../graders/out-of-scope-cut.md | 2 +- .../graders/sanity-check-command.md | 4 ++-- .../samples/sanity-check-command.json | 4 ++++ .../graders/ac3-sort-built.md | 2 +- .../graders/ac4-built.md | 2 +- .../graders/regression-test-named.md | 2 +- .../graders/ac3-built.md | 2 +- .../graders/out-of-scope-cut.md | 2 +- .../graders/sanity-check-command.md | 4 ++-- .../samples/sanity-check-command.json | 4 ++++ 13 files changed, 28 insertions(+), 12 deletions(-) diff --git a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md index b5b4e95351e..4fdd53a9543 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md +++ b/plugins/planning/evals/plan-reviewer-cut-low-ledger-delimiter/graders/ac3-built.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '^###\s*Phase(?:(?!^#{2,3}\s)[\s\S])*?^\s*[-*]\s+[^\n]*(?:ValueError|unsupported delimiter|\brefus|\breject|\braise)' +pattern: '^###\s*Phase(?:(?!^#{2,3}\s)[\s\S])*?^\s*(?:[-*]|\d+[.)])\s+[^\n]*(?:ValueError|unsupported delimiter|\brefus|\breject|\braise)' flags: mi target: source: file diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md index 2edd33b55d4..d3d6da5cfa6 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '(?:^###\s*Phase|^##\s*Plan\b)(?:(?!^#{2,3}\s)[\s\S])*?^\s*[-*]\s+(?=[^\n]*(?:regression|\btests?\b|\btest_\w+|\bassert))(?=[^\n]*(?:backoff\(\s*(?:[7-9]|\d{2,})\s*\)|\bcap|\bceiling|MAX_DELAY|\b30(?:\.0)?\b))' +pattern: '(?:^###\s*Phase|^###\s*Test\s+strateg|^##\s*Plan\b)(?:(?!^#{2,3}\s)[\s\S])*?^\s*(?:[-*]|\d+[.)])\s+(?=[^\n]*(?:regression|\btests?\b|\btest_\w+|\bassert))(?=[^\n]*(?:backoff\(\s*(?:[7-9]|\d{2,})\s*\)|\bcap|\bceiling|MAX_BACKOFF_SECONDS|MAX_DELAY|\b30(?:\.0)?\b))' flags: mi target: source: file diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json index d44e0d784be..364fd0b1720 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json @@ -7,6 +7,14 @@ { "answer": "## Plan\n- Add a unit test asserting `backoff(8) == 30.0`.\n", "why": "test without the word regression" + }, + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_BACKOFF_SECONDS)`; `MAX_BACKOFF_SECONDS` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- In `tests/test_retry.py`, add a regression case: `backoff(8)` returns 30.0.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", + "why": "the fixed plan naming the real constant, config.MAX_BACKOFF_SECONDS, in its regression-test bullet" + }, + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n\n### Test strategy\n\n- `test_caps_at_ceiling` is the regression test for the bug: `backoff(8) == 30.0`.\n", + "why": "the seed plus a Test strategy item naming the ceiling regression test" } ], "fail": [ diff --git a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md index 11fd1309233..880093af866 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md +++ b/plugins/planning/evals/plan-reviewer-cut-low-temperature-decimals/graders/out-of-scope-cut.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '(?:^###\s*Phase|^(? Date: Wed, 11 Nov 2026 19:45:27 -0500 Subject: [PATCH 5/6] test(planning): reject declined regression tests under Test strategy Counting Test strategy items let a line that names the bug but declines the regression test pass both regression-test graders. A bullet carrying no, none, not, without, skip or carve-out no longer counts. The review's three counterexamples are fail samples, and overnight-shifts gains a Test strategy pass sample. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VQb9Q4F46UumaUgXeUdpYZ --- .../graders/regression-test-named.md | 2 +- .../samples/regression-test-named.json | 4 ++++ .../graders/regression-test-named.md | 2 +- .../samples/regression-test-named.json | 12 ++++++++++++ 4 files changed, 18 insertions(+), 2 deletions(-) diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md index d3d6da5cfa6..f93e66ea4f3 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '(?:^###\s*Phase|^###\s*Test\s+strateg|^##\s*Plan\b)(?:(?!^#{2,3}\s)[\s\S])*?^\s*(?:[-*]|\d+[.)])\s+(?=[^\n]*(?:regression|\btests?\b|\btest_\w+|\bassert))(?=[^\n]*(?:backoff\(\s*(?:[7-9]|\d{2,})\s*\)|\bcap|\bceiling|MAX_BACKOFF_SECONDS|MAX_DELAY|\b30(?:\.0)?\b))' +pattern: '(?:^###\s*Phase|^###\s*Test\s+strateg|^##\s*Plan\b)(?:(?!^#{2,3}\s)[\s\S])*?^\s*(?:[-*]|\d+[.)])\s+(?![^\n]*\b(?:no|none|not|without|skip|skips|skipped|skipping|carve[- ]?out|carved out)\b)(?=[^\n]*(?:regression|\btests?\b|\btest_\w+|\bassert))(?=[^\n]*(?:backoff\(\s*(?:[7-9]|\d{2,})\s*\)|\bcap|\bceiling|MAX_BACKOFF_SECONDS|MAX_DELAY|\b30(?:\.0)?\b))' flags: mi target: source: file diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json index 364fd0b1720..66965aea567 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json @@ -33,6 +33,10 @@ { "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Test Strategy\n- Bug fixes: no regression test; document an explicit carve-out.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n", "why": "the seeded draft with a Test Strategy section that turns the regression test down" + }, + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n\n### Test Strategy\n\n- Bug fixes: no new regression test; the Phase 1 Sanity Check already asserts `backoff(8) <= 30`.\n", + "why": "the seed plus a Test Strategy carve-out that names the bug but declines the regression test" } ] } diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md index fc538b6fc27..41a642667c2 100644 --- a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '(?:^###\s*Phase|^###\s*Test\s+strateg|^##\s*Plan\b)(?:(?!^#{2,3}\s)[\s\S])*?^\s*(?:[-*]|\d+[.)])\s+(?=[^\n]*(?:regression|\btests?\b|\btest_\w+|\bassert))(?=[^\n]*(?:22:00|overnight|next[ -_]?day|midnight))' +pattern: '(?:^###\s*Phase|^###\s*Test\s+strateg|^##\s*Plan\b)(?:(?!^#{2,3}\s)[\s\S])*?^\s*(?:[-*]|\d+[.)])\s+(?![^\n]*\b(?:no|none|not|without|skip|skips|skipped|skipping|carve[- ]?out|carved out)\b)(?=[^\n]*(?:regression|\btests?\b|\btest_\w+|\bassert))(?=[^\n]*(?:22:00|overnight|next[ -_]?day|midnight))' flags: mi target: source: file diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json index 63a99991de1..44fb27c5578 100644 --- a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json @@ -3,6 +3,10 @@ { "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n- In `tests/test_totals.py`, add a regression case: `22:00-06:00` totals 8.0.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", "why": "the plan with the seeded defect fixed" + }, + { + "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n\n### Test strategy\n\n- `test_overnight_shift_counts_next_day` is the regression test for the 22:00-06:00 case.\n", + "why": "the seed plus a Test strategy item naming the overnight regression test" } ], "fail": [ @@ -13,6 +17,14 @@ { "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -m unittest discover -s tests` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n", "why": "the seeded draft with a tests-exit-0 Phase 2 Sanity Check" + }, + { + "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n\n### Test Strategy\n\n- Existing tests that need updating: none; `tests/test_totals.py` has no overnight case yet.\n", + "why": "the seed plus a Test Strategy line that mentions the overnight case but adds no test" + }, + { + "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n\n### Test Strategy\n\n- Bug fixes: no regression test for the overnight case; the Phase 2 Sanity Check covers it.\n", + "why": "the seed plus a Test Strategy carve-out for the overnight regression test" } ] } From 5aa784addd603ecdc5fc68c8c7d29984bd404d2b Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Wed, 11 Nov 2026 20:03:48 -0500 Subject: [PATCH 6/6] test(planning): exclude only declined regression tests, not negatives The any-negation exclusion failed correct tests phrased as negatives ("does not exceed", "is not rejected"). Both regression-test graders now reject only an actual decline: "no (new) regression/test", "none", "without a test", skip, or carve-out. The review's negative phrasings are pass samples; every earlier fail sample still fails. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VQb9Q4F46UumaUgXeUdpYZ --- .../graders/regression-test-named.md | 2 +- .../samples/regression-test-named.json | 4 ++++ .../graders/regression-test-named.md | 2 +- .../samples/regression-test-named.json | 4 ++++ 4 files changed, 10 insertions(+), 2 deletions(-) diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md index f93e66ea4f3..a4f7f95d4da 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/graders/regression-test-named.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '(?:^###\s*Phase|^###\s*Test\s+strateg|^##\s*Plan\b)(?:(?!^#{2,3}\s)[\s\S])*?^\s*(?:[-*]|\d+[.)])\s+(?![^\n]*\b(?:no|none|not|without|skip|skips|skipped|skipping|carve[- ]?out|carved out)\b)(?=[^\n]*(?:regression|\btests?\b|\btest_\w+|\bassert))(?=[^\n]*(?:backoff\(\s*(?:[7-9]|\d{2,})\s*\)|\bcap|\bceiling|MAX_BACKOFF_SECONDS|MAX_DELAY|\b30(?:\.0)?\b))' +pattern: '(?:^###\s*Phase|^###\s*Test\s+strateg|^##\s*Plan\b)(?:(?!^#{2,3}\s)[\s\S])*?^\s*(?:[-*]|\d+[.)])\s+(?![^\n]*(?:\bno\s+(?:new\s+)?(?:regression|tests?)\b|\bnone\b|\bwithout\s+(?:a\s+)?(?:new\s+)?(?:regression|tests?)\b|\b(?:skip|skips|skipped|skipping)\b|carve[- ]?out|carved out))(?=[^\n]*(?:regression|\btests?\b|\btest_\w+|\bassert))(?=[^\n]*(?:backoff\(\s*(?:[7-9]|\d{2,})\s*\)|\bcap|\bceiling|MAX_BACKOFF_SECONDS|MAX_DELAY|\b30(?:\.0)?\b))' flags: mi target: source: file diff --git a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json index 66965aea567..7cf517cc9bf 100644 --- a/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json +++ b/plugins/planning/evals/plan-reviewer-cut-low-retry-backoff-cap/samples/regression-test-named.json @@ -15,6 +15,10 @@ { "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n\n### Test strategy\n\n- `test_caps_at_ceiling` is the regression test for the bug: `backoff(8) == 30.0`.\n", "why": "the seed plus a Test strategy item naming the ceiling regression test" + }, + { + "answer": "# Cap retry backoff\n\n## Brief\n\n### Problem\nWith eight attempts, the last retries wait 32 and 64 seconds, past the 30-second ceiling the operations runbook promises, so a stuck fetch holds a worker for minutes.\n\n### Goal\nNo retry waits longer than the configured ceiling.\n\n### Acceptance criteria\n- AC1: `backoff(n)` returns what it returns today while the doubled delay is at or under the ceiling.\n- AC2: `backoff(n)` never returns more than the ceiling for any `n` of 1 or more.\n- AC3: `backoff(0)` and negative attempts raise `ValueError`.\n\n### Out-of-scope\n- Adding jitter.\n- Changing the number of attempts.\n\n## Design\n\nEarly exit (tier C): A bound inside one existing function; no new types.\n\n## Plan\n\n### Behavior\n- B1: Short delays keep doubling as today (AC1). Delivered by Phase 1.\n- B2: No delay exceeds the ceiling (AC2). Delivered by Phase 1.\n- B3: Attempt numbers below 1 are refused (AC3). Delivered by Phase 1.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| `BASE_DELAY_SECONDS` stays positive | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Bound the delay [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2, B3\n\n- In `fetcher/retry.py`, return `min(config.BASE_DELAY_SECONDS * 2 ** (attempt - 1), config.MAX_DELAY)`; `MAX_DELAY` is already defined in `fetcher/config.py` as the 30-second ceiling.\n- Raise `ValueError` at the top of `backoff` when `attempt` is below 1.\n\n**Sanity Check:** `python -c \"from fetcher.retry import backoff; assert backoff(8) <= 30\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `fetcher/retry.py` | modify |\n\n### Phase 1: Bound the delay\n\n- In `tests/test_retry.py`, add a regression test that `backoff(n)` does not exceed `config.MAX_BACKOFF_SECONDS` for any `n` from 1 to 20.\n", + "why": "the seed plus a Phase item whose regression test is phrased as a negative" } ], "fail": [ diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md index 41a642667c2..dbab413acd3 100644 --- a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/graders/regression-test-named.md @@ -1,6 +1,6 @@ --- type: regex -pattern: '(?:^###\s*Phase|^###\s*Test\s+strateg|^##\s*Plan\b)(?:(?!^#{2,3}\s)[\s\S])*?^\s*(?:[-*]|\d+[.)])\s+(?![^\n]*\b(?:no|none|not|without|skip|skips|skipped|skipping|carve[- ]?out|carved out)\b)(?=[^\n]*(?:regression|\btests?\b|\btest_\w+|\bassert))(?=[^\n]*(?:22:00|overnight|next[ -_]?day|midnight))' +pattern: '(?:^###\s*Phase|^###\s*Test\s+strateg|^##\s*Plan\b)(?:(?!^#{2,3}\s)[\s\S])*?^\s*(?:[-*]|\d+[.)])\s+(?![^\n]*(?:\bno\s+(?:new\s+)?(?:regression|tests?)\b|\bnone\b|\bwithout\s+(?:a\s+)?(?:new\s+)?(?:regression|tests?)\b|\b(?:skip|skips|skipped|skipping)\b|carve[- ]?out|carved out))(?=[^\n]*(?:regression|\btests?\b|\btest_\w+|\bassert))(?=[^\n]*(?:22:00|overnight|next[ -_]?day|midnight))' flags: mi target: source: file diff --git a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json index 44fb27c5578..2796feccdda 100644 --- a/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json +++ b/plugins/planning/evals/plan-reviewer-cut-medium-overnight-shifts/samples/regression-test-named.json @@ -7,6 +7,10 @@ { "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n\n### Test strategy\n\n- `test_overnight_shift_counts_next_day` is the regression test for the 22:00-06:00 case.\n", "why": "the seed plus a Test strategy item naming the overnight regression test" + }, + { + "answer": "# Count overnight shifts\n\n## Brief\n\n### Problem\nNight staff lose pay: a 22:00-06:00 line is rejected as ending before it starts, and a hand-entered one totals -16 hours, which payroll zeroes.\n\n### Goal\nA shift that ends at or before its start time counts as ending the next day.\n\n### Acceptance criteria\n- AC1: `2026-03-02 22:00-06:00` parses and totals 8.0 hours.\n- AC2: Day shifts total as today.\n- AC3: `weekly_hours` counts an overnight shift on the day it starts.\n\n### Out-of-scope\n- Shifts longer than 24 hours.\n- Daylight-saving adjustments.\n\n### Captured assumptions\n- Overnight shifts end before noon the next day: revisit if a site reports a shift ending after 12:00.\n\n## Design\n\nEarly exit (tier C): Changes existing functions; no new types.\n\n## Plan\n\n### Behavior\n- B1: An overnight line is accepted and counts the hours to its end the next morning (AC1, AC3). Delivered by Phase 1 and Phase 2.\n- B2: Day shifts total as before (AC2). Delivered by Phase 2.\n\n### Load-bearing assumptions\n\n| Assumption | Source | Falsifying check | Response if false |\n|---|---|---|---|\n| Shift lines use 24-hour HH:MM times | plan | Phase 1 Sanity Check | Minor: proceed and note |\n\n### Phase 1: Accept overnight lines [TODO]\n**Satisfies:** AC1\n**Delivers:** B1\n\n- In `timesheet/parse.py`, drop the `end before start` rejection.\n- In `timesheet/models.py`, add a `Shift.ends_next_day` property that is true when `end <= start`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; assert parse_line('2026-03-02 22:00-06:00').ends_next_day\"` exits 0.\n\n### Phase 2: Count the hours [TODO]\n**Satisfies:** AC1, AC2, AC3\n**Delivers:** B1, B2\n\n- In `timesheet/totals.py`, build the end as `datetime.combine(shift.day, shift.end)` plus one day when `shift.ends_next_day`, and keep `weekly_hours` keyed on `shift.day`.\n\n**Sanity Check:** `python -c \"from timesheet.parse import parse_line; from timesheet.totals import hours; assert hours(parse_line('2026-03-02 22:00-06:00')) == 8.0\"` exits 0.\n\n### Files affected\n| File | Action |\n|---|---|\n| `timesheet/parse.py` | modify |\n| `timesheet/models.py` | modify |\n| `timesheet/totals.py` | modify |\n\n### Phase 2: Count overnight shifts\n\n- In `tests/test_totals.py`, add a regression test that `22:00-06:00` is not rejected and totals 8.0.\n", + "why": "the seed plus a Phase item whose overnight regression test is phrased as a negative" } ], "fail": [