Skip to content

ci: raise Claude review turn budget and bump to Opus 4.8 - #1130

Merged
shane-moore merged 1 commit into
sigp:stablefrom
shane-moore:ci/claude-review-turn-budget
Jul 11, 2026
Merged

ci: raise Claude review turn budget and bump to Opus 4.8#1130
shane-moore merged 1 commit into
sigp:stablefrom
shane-moore:ci/claude-review-turn-budget

Conversation

@shane-moore

Copy link
Copy Markdown
Member

Problem, Evidence, and Context

Both Claude review workflows failed on PR #1128 with Reached maximum number of turns (50), losing all review work roughly 13 minutes in and posting nothing:

  • claude-pr-review: run 29113701689 - ~15 of 50 turns were wasted fighting the diff: gh pr diff output (~84KB) overflowed the tool-output limit, every attempt to save it to a file was blocked by the tool allowlist (no Write), and the agent fell back to re-paging the diff one sed slice per turn.
  • claude-mentions: run 29115026820 - only 5 permission denials, so minimal friction, and it still exhausted 50 turns mid-review. The budget itself is too small for a ~1,400-line PR at Opus pace (~15s/turn, one tool call per turn).

Each failed attempt costs ~$3.30 in Bedrock spend with zero output.

Change Overview

  • Raise --max-turns 50 -> 80 and timeout-minutes 15 -> 25 in both workflows. These are the same limit at Opus pace (50 turns ~= 13.5 min), so they must move together.
  • claude-pr-review only: pre-stage the PR diff into .claude-pr-diff.txt in a workflow step and point the prompt at it, so the agent reads it with the paginated Read tool instead of burning turns on gh pr diff. The file lives only on the ephemeral runner; nothing commits or uploads it.
  • review.md: instruct the agent to post each inline comment as soon as it is confirmed, so a capped run still delivers partial findings instead of losing everything.
  • Bump the model to us.anthropic.claude-opus-4-8 in both workflows: same Bedrock pricing ($5/$25 per MTok), no API changes from 4.7, stronger at code review.

Intentionally unchanged: the tool allowlists and the silent-watchdog prompt semantics.

Risks, Trade-offs, and Mitigations

  • Worst-case cost/latency per review rises (80 turns / 25 min ceiling). Typical runs finish well under the old cap, so expected cost is nearly flat; only pathological runs use the headroom.
  • Opus 4.8 follows conservative-reporting instructions more literally than 4.7, so borderline findings may drop slightly. That matches the silent-watchdog intent.
  • Requires Opus 4.8 model access in the AWS account (us-west-2). If missing, the run fails fast with an access error rather than burning the turn budget.

Validation

  • Both workflow files parse as valid YAML.
  • Failure analysis is from the run logs of the two failed jobs (turn counts, permission denials, and per-turn pacing).
  • The end-to-end check requires this to land on stable first (pull_request_target/issue_comment run workflow files from the default branch); after merge, applying the claude-recheck label to PR feat(validator_store): use decided votes for aggregation #1128 re-runs the review and doubles as verification of Bedrock Opus 4.8 access.

Rollback

Revert the commit. No config, data, or operational impact beyond the workflows themselves.

Additional Info / Next Steps

After merge: apply claude-recheck to PR #1128 to confirm the review completes within budget.

Both Claude review workflows hit the 50-turn cap on PR sigp#1128 (runs
29113701689 and 29115026820), losing all review work ~13 min in. The
turn cap and the 15-minute step timeout are the same limit at Opus
pace (~15s/turn), so raise both together to 80 turns / 25 minutes.

Pre-stage the PR diff into .claude-pr-diff.txt: large diffs overflow
the agent tool-output limit and the allowlist blocks saving output to
a file, so the agent burned ~15 turns re-paging gh pr diff. Point the
prompt at the staged file instead.

Post inline comments incrementally so a capped run still delivers
partial findings, and bump the model to Opus 4.8 (same Bedrock
pricing, stronger code review).

@jnhsigmap jnhsigmap left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good - I wonder if it needs to be told explicitly not to duplicate commentary and to ensure that issues are logically segregated into mutually exclusive components. This would prevent it from wasting tokens examining something that it has already addressed previously - for example if an issue is confirmed at a surface level and a comment is posted, then further exploration results in a deeper root cause or wider blast radius being identified. An attempt to totally re-resolve may leave a previous comment defunct? I guess we can only find out by observing here.

@jnhsigmap

Copy link
Copy Markdown
Contributor

The test-suite checks should be fixed on unstable. Can ignore them I believe

@shane-moore
shane-moore merged commit 38d2892 into sigp:stable Jul 11, 2026
14 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants