Bump the Claude workflows to Opus 5 and enforce source-discipline rule 3 - #20518
Draft
CamSoper wants to merge 1 commit into
Draft
Bump the Claude workflows to Opus 5 and enforce source-discipline rule 3#20518CamSoper wants to merge 1 commit into
CamSoper wants to merge 1 commit into
Conversation
Five workflows move to `--effort low`, and the generated-from-data rule becomes a deterministic gate instead of prompt guidance. **Why `--effort low` is not optional.** Opus 5 defaults to `high` effort when `claude_args` doesn't set it, and every one of these workflows left the flag unset. A bare model swap on the pre-merge composer measured 2.20x the Opus 4.8 incumbent; at `--effort low` the same swap is 1.05x cost and 0.81x wall-clock. Setting the flag explicitly is what makes the bump cost-neutral, and it stops the next model change from silently inheriting a default again. Measured on matched fork fixtures (3 fixtures/arm, same-day controls): | workflow | model | measured | |---|---|---| | claude-code-review.yml | opus-4-8 -> opus-5 | 1.05x cost, 0.81x wall-clock | | claude-update.yml | sonnet-5 (unchanged) | 0.59x cost, 0.64x wall-clock | | check-links.yml | opus-4-7 -> opus-5 | not measured | | content-review-article.yml | opus-4-8 -> opus-5 | not measured | | blog-review-index.yml | opus-4-8 -> opus-5 | not measured | check-links.yml was also a model behind the others at 4.7; that drift is fixed here rather than tracked separately. The three unmeasured lanes are reasoned bets on job shape, each behind a deterministic gate that does not care which model produced the work: `make lint` + `make build` plus human PR review for check-links, publish-gate.py's line-range enforcement for content-review-article, and the flag-only contract plus findings-schema validation for blog-review-index. **Source-discipline rule 3 becomes a gate.** #20349 put four hard rules in the verifier prompt after a ledger re-adjudication found 17 of 22 `contradicted` verdicts were false. Rule 3 turned out to be advisory in practice: a model sweep found every Opus configuration flagged 5-6 of 12 policy-pack claims `contradicted` against Sonnet's 1 of 12, and the rate did not move with reasoning effort (6 / 6 / 5 across low / medium / high). A rule a stronger model overrides is not a rule. verify-claims.py now downgrades `contradicted` to `unverifiable` on any section carrying a `_content.gotmpl` -- the marker that makes a section generated from `data/` rather than authored -- stamps `source_discipline_gate: "generated-from-data"`, and prefixes `evidence` with the reason. Discovering the roots from the marker keeps the gate in step with the content tree instead of pinning a path list that goes stale. The model's reasoning is preserved verbatim so the upstream concern still reaches the reviewer; `mismatch` is untouched, so a transcription that disagrees with its own data file is still reported as the doc bug it is. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019WR5eKWHo5Hb7ZFHFdLLuG
Collaborator
|
Your site preview for commit 44ccdbb is ready! 🎉 http://www-testing-pulumi-docs-origin-pr-20518-44ccdbb0.s3-website.us-west-2.amazonaws.com |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves five Claude workflows to
--effort low, bumps the Opus lanes to Opus 5, and turns source-discipline rule 3 from prompt guidance into a deterministic gate.Why
--effort lowis the load-bearing partOpus 5 defaults to
higheffort whenclaude_argsdoesn't set it, and every one of these workflows left the flag unset. A bare model swap on the pre-merge composer measured 2.20× the Opus 4.8 incumbent. At--effort lowthe same swap is 1.05× cost and 0.81× wall-clock — faster than what we run today.So the flag isn't a tuning nicety; it's what makes the bump cost-neutral. Setting it explicitly everywhere also stops the next model change from silently inheriting a default.
The five workflows
claude-code-review.ymlopus-4-8→opus-5claude-update.ymlsonnet-5(unchanged)check-links.ymlopus-4-7→opus-5content-review-article.ymlopus-4-8→opus-5blog-review-index.ymlopus-4-8→opus-5Measured arms ran on matched fork fixtures with same-day controls — 3 fixtures per composer arm, and interleaved control/arm rounds for the update lane (ordering favored the control; low still won, and with much lower variance: $2.147 vs $2.160 across rounds against the control's $3.18–$4.12 spread).
check-links.ymlwas also a model behind the rest at 4.7. That drift is fixed here rather than tracked separately.The three unmeasured lanes are bets, and I want that called out in review. Each sits behind a deterministic gate that doesn't care which model produced the work —
make lint+make buildplus human PR review for check-links,publish-gate.py's line-range enforcement for content-review-article, and the flag-only contract plus findings-schema validation for blog-review-index. Their realistic failure mode is "costs more than expected" or "quality dips and we notice," not silent corruption. If you'd rather split them out, they're one line each.Source-discipline rule 3 becomes a gate
#20349 put four hard rules in the verifier prompt after a ledger re-adjudication found 17 of 22
contradictedverdicts were false. Rule 3 — a generated-from-data page transcribes product metadata, so disagreement with the external framework it cites is upstream feedback, not a doc contradiction — turned out to be advisory in practice.A model sweep over 50 real ledger claims found every Opus configuration flagged 5–6 of 12 policy-pack claims
contradicted, against Sonnet's 1 of 12 — and the rate did not move with reasoning effort:Flat across the dial. A rule a stronger model overrides is not a rule, so the downgrade is now deterministic.
verify-claims.pydowngradescontradicted→unverifiableon any section carrying a_content.gotmpl— the marker that makes a section generated fromdata/rather than authored — stampssource_discipline_gate: "generated-from-data"on the verdict, and prefixesevidencewith the reason. Discovering roots from the marker keeps the gate in step with the content tree instead of pinning a path list that goes stale.The gate reclassifies; it doesn't silence. The model's reasoning is preserved verbatim so the upstream concern still reaches the reviewer, and
mismatchis untouched — a transcription that disagrees with its own data file is still reported as the doc bug it is. The prompt now tells the verifier the gate exists, so the model and the harness agree rather than fighting.Scope caveat worth flagging
The rule-3 gate protects the verify lane and everything downstream of
.verified-claims.json. It does not constrainblog-review-index.yml's own Opus review layer, which adjudicates blog posts — and blog posts are never generated-from-data, so rule 3 doesn't apply there. That lane's assertiveness risk is real but unmeasured, and this PR doesn't address it. Noting it explicitly so the gate isn't read as broader cover than it is.Testing
test_verify_claims_source_discipline.py— 7 cases: marker discovery, the downgrade, non-contradictedpass-through, authored pages keepingcontradicted, inertness without a repo root, and sibling-prefix false-match protection..claude/commands/docs-review/scripts/suite: 96 passed.make lintclean.Follow-ups not in this PR
routed-metadata-reconcilesvalidator rule — routed metadata drifts from the trail in 5 of 6 reviews, both models, nothing catches it.candidate-claims-coverageis unenforced on the re-entrant update path —claude-update.ymlruns onlyvalidate-pinned.py count-buckets, never the validate → splice → validator-fix chain.blog-review-index,content-review-article), which are not latency-sensitive.🤖 Generated with Claude Code
https://claude.ai/code/session_019WR5eKWHo5Hb7ZFHFdLLuG