Skip to content

Bump the Claude workflows to Opus 5 and enforce source-discipline rule 3 - #20518

Draft
CamSoper wants to merge 1 commit into
masterfrom
CamSoper/opus5-model-bump
Draft

Bump the Claude workflows to Opus 5 and enforce source-discipline rule 3#20518
CamSoper wants to merge 1 commit into
masterfrom
CamSoper/opus5-model-bump

Conversation

@CamSoper

Copy link
Copy Markdown
Contributor

Moves five Claude workflows to --effort low, bumps the Opus lanes to Opus 5, and turns source-discipline rule 3 from prompt guidance into a deterministic gate.

Why --effort low is the load-bearing part

Opus 5 defaults to high effort when claude_args doesn't set it, and every one of these workflows left the flag unset. A bare model swap on the pre-merge composer measured 2.20× the Opus 4.8 incumbent. At --effort low the same swap is 1.05× cost and 0.81× wall-clock — faster than what we run today.

So the flag isn't a tuning nicety; it's what makes the bump cost-neutral. Setting it explicitly everywhere also stops the next model change from silently inheriting a default.

The five workflows

Workflow Model Evidence
claude-code-review.yml opus-4-8opus-5 Measured — 1.05× cost, 0.81× wall-clock, validator/trail/lane checks clean
claude-update.yml sonnet-5 (unchanged) Measured — 0.59× cost, 0.64× wall-clock, 0.56× turns
check-links.yml opus-4-7opus-5 Reasoned
content-review-article.yml opus-4-8opus-5 Reasoned
blog-review-index.yml opus-4-8opus-5 Reasoned

Measured arms ran on matched fork fixtures with same-day controls — 3 fixtures per composer arm, and interleaved control/arm rounds for the update lane (ordering favored the control; low still won, and with much lower variance: $2.147 vs $2.160 across rounds against the control's $3.18–$4.12 spread).

check-links.yml was also a model behind the rest at 4.7. That drift is fixed here rather than tracked separately.

The three unmeasured lanes are bets, and I want that called out in review. Each sits behind a deterministic gate that doesn't care which model produced the work — make lint + make build plus human PR review for check-links, publish-gate.py's line-range enforcement for content-review-article, and the flag-only contract plus findings-schema validation for blog-review-index. Their realistic failure mode is "costs more than expected" or "quality dips and we notice," not silent corruption. If you'd rather split them out, they're one line each.

Source-discipline rule 3 becomes a gate

#20349 put four hard rules in the verifier prompt after a ledger re-adjudication found 17 of 22 contradicted verdicts were false. Rule 3 — a generated-from-data page transcribes product metadata, so disagreement with the external framework it cites is upstream feedback, not a doc contradiction — turned out to be advisory in practice.

A model sweep over 50 real ledger claims found every Opus configuration flagged 5–6 of 12 policy-pack claims contradicted, against Sonnet's 1 of 12 — and the rate did not move with reasoning effort:

config pack claims flagged
Sonnet 5 (shipping) 1/12
Opus 5, thinking off 6/12
Opus 5, effort low 6/12
Opus 5, effort medium 5/12

Flat across the dial. A rule a stronger model overrides is not a rule, so the downgrade is now deterministic.

verify-claims.py downgrades contradictedunverifiable on any section carrying a _content.gotmpl — the marker that makes a section generated from data/ rather than authored — stamps source_discipline_gate: "generated-from-data" on the verdict, and prefixes evidence with the reason. Discovering roots from the marker keeps the gate in step with the content tree instead of pinning a path list that goes stale.

The gate reclassifies; it doesn't silence. The model's reasoning is preserved verbatim so the upstream concern still reaches the reviewer, and mismatch is untouched — a transcription that disagrees with its own data file is still reported as the doc bug it is. The prompt now tells the verifier the gate exists, so the model and the harness agree rather than fighting.

Scope caveat worth flagging

The rule-3 gate protects the verify lane and everything downstream of .verified-claims.json. It does not constrain blog-review-index.yml's own Opus review layer, which adjudicates blog posts — and blog posts are never generated-from-data, so rule 3 doesn't apply there. That lane's assertiveness risk is real but unmeasured, and this PR doesn't address it. Noting it explicitly so the gate isn't read as broader cover than it is.

Testing

  • New test_verify_claims_source_discipline.py — 7 cases: marker discovery, the downgrade, non-contradicted pass-through, authored pages keeping contradicted, inertness without a repo root, and sibling-prefix false-match protection.
  • Full .claude/commands/docs-review/scripts/ suite: 96 passed.
  • make lint clean.

Follow-ups not in this PR

  • routed-metadata-reconciles validator rule — routed metadata drifts from the trail in 5 of 6 reviews, both models, nothing catches it.
  • candidate-claims-coverage is unenforced on the re-entrant update path — claude-update.yml runs only validate-pinned.py count-buckets, never the validate → splice → validator-fix chain.
  • Batch API for the two daily Opus flows (blog-review-index, content-review-article), which are not latency-sensitive.

🤖 Generated with Claude Code

https://claude.ai/code/session_019WR5eKWHo5Hb7ZFHFdLLuG

Five workflows move to `--effort low`, and the generated-from-data rule
becomes a deterministic gate instead of prompt guidance.

**Why `--effort low` is not optional.** Opus 5 defaults to `high` effort when
`claude_args` doesn't set it, and every one of these workflows left the flag
unset. A bare model swap on the pre-merge composer measured 2.20x the Opus 4.8
incumbent; at `--effort low` the same swap is 1.05x cost and 0.81x wall-clock.
Setting the flag explicitly is what makes the bump cost-neutral, and it stops
the next model change from silently inheriting a default again.

Measured on matched fork fixtures (3 fixtures/arm, same-day controls):

| workflow | model | measured |
|---|---|---|
| claude-code-review.yml | opus-4-8 -> opus-5 | 1.05x cost, 0.81x wall-clock |
| claude-update.yml | sonnet-5 (unchanged) | 0.59x cost, 0.64x wall-clock |
| check-links.yml | opus-4-7 -> opus-5 | not measured |
| content-review-article.yml | opus-4-8 -> opus-5 | not measured |
| blog-review-index.yml | opus-4-8 -> opus-5 | not measured |

check-links.yml was also a model behind the others at 4.7; that drift is
fixed here rather than tracked separately. The three unmeasured lanes are
reasoned bets on job shape, each behind a deterministic gate that does not
care which model produced the work: `make lint` + `make build` plus human PR
review for check-links, publish-gate.py's line-range enforcement for
content-review-article, and the flag-only contract plus findings-schema
validation for blog-review-index.

**Source-discipline rule 3 becomes a gate.** #20349 put four hard rules in the
verifier prompt after a ledger re-adjudication found 17 of 22 `contradicted`
verdicts were false. Rule 3 turned out to be advisory in practice: a model
sweep found every Opus configuration flagged 5-6 of 12 policy-pack claims
`contradicted` against Sonnet's 1 of 12, and the rate did not move with
reasoning effort (6 / 6 / 5 across low / medium / high). A rule a stronger
model overrides is not a rule.

verify-claims.py now downgrades `contradicted` to `unverifiable` on any
section carrying a `_content.gotmpl` -- the marker that makes a section
generated from `data/` rather than authored -- stamps
`source_discipline_gate: "generated-from-data"`, and prefixes `evidence` with
the reason. Discovering the roots from the marker keeps the gate in step with
the content tree instead of pinning a path list that goes stale. The model's
reasoning is preserved verbatim so the upstream concern still reaches the
reviewer; `mismatch` is untouched, so a transcription that disagrees with its
own data file is still reported as the doc bug it is.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019WR5eKWHo5Hb7ZFHFdLLuG
@pulumi-bot

Copy link
Copy Markdown
Collaborator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants