fix(triage): remove 500-char cap on verdict summary - #60
Conversation
A real triage may need to cite multiple claims, reproductions, and evidence to honestly convey the verdict. Capping summary at 500 chars forced the agent to truncate analysis or, as observed in production on issue #16, overflowed the cap and discarded a $1.15 / 21-turn run. Detail is unaffected: TRIAGE.md remains uncapped, reproduction.details keeps its 2000-char allowance, and evidence[] is structured. Prompt template and a stale comment are aligned with the loosened schema. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 54 minutes and 13 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📝 WalkthroughWalkthroughThe Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (2)
src/workflows/handlers/triage.ts (2)
74-74: Schema relaxation looks fine; consider a sanity ceiling for downstream comment safety.Removing
.max(500)is well-motivated by the cited incident. One thing to keep in mind:verdict.summaryis interpolated intoresult.reasonat line 188 (triage rejected as invalid: ${verdict.summary}), and persrc/workflows/orchestrator.ts:74-91thatreasonis embedded verbatim into the parent run'shumanMessage, which becomes a GitHub comment. GitHub enforces a 65,536-character limit on issue/PR comment bodies, so a pathologically long agent-produced summary could now cause the parent "ship halted…" comment to be rejected by the API.In practice this is unlikely with current agent behavior, but since the only guardrail has been removed, you may want either:
- a generous defensive cap (e.g.
.max(20_000)) purely to protect downstream surfaces, or- a truncation step when composing
reason(and orchestratorhumanMessage) rather than at the schema layer.Per the PR description, a separate
headlinefield is the preferred long-term answer; this is just a near-term safety net.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/workflows/handlers/triage.ts` at line 74, Add a defensive ceiling to the verdict summary to prevent extremely large GitHub comments: in src/workflows/handlers/triage.ts constrain verdict.summary (the z.string() schema) with a generous .max(20000) so that when it is interpolated into result.reason (used by orchestrator.ts to build the parent run humanMessage) it cannot exceed GitHub's comment size and risk rejection; alternatively, if you prefer not to change the schema, ensure truncation of verdict.summary before composing result.reason in the same triage handler (truncate to ~20k chars and append an ellipsis) so orchestrator.ts receives a bounded humanMessage.
320-320: Prompt guidance is now open-ended — consider a soft length hint.The new instruction
"<as long as needed to faithfully convey the verdict>"removes any size guidance for the agent. Sincesummaryis meant to be a verdict headline that gets embedded into the parent ship comment (line 188 → orchestrator parenthumanMessage), it may be worth nudging the agent toward, e.g., "1–3 sentences; place longer reasoning inTRIAGE.md" so the field stays useful as a one-glance reason while remaining uncapped at the schema level. The TRIAGE.md template at lines 297–298 already calls for "one-paragraph summary", so the JSON instruction is now slightly out of step with the markdown instruction.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/workflows/handlers/triage.ts` at line 320, Update the open-ended JSON prompt guidance for the "summary" field to include a soft length hint (e.g., "1–3 sentences; place longer reasoning in TRIAGE.md") so that the generated verdict remains a concise one-glance headline; modify the string used for the "summary" instruction in triage.ts (the `"summary": "<as long as needed to faithfully convey the verdict>"` entry) to include this hint and ensure it references TRIAGE.md's one-paragraph guidance and the orchestrator's parent humanMessage so the instructions stay consistent.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@src/workflows/handlers/triage.ts`:
- Line 74: Add a defensive ceiling to the verdict summary to prevent extremely
large GitHub comments: in src/workflows/handlers/triage.ts constrain
verdict.summary (the z.string() schema) with a generous .max(20000) so that when
it is interpolated into result.reason (used by orchestrator.ts to build the
parent run humanMessage) it cannot exceed GitHub's comment size and risk
rejection; alternatively, if you prefer not to change the schema, ensure
truncation of verdict.summary before composing result.reason in the same triage
handler (truncate to ~20k chars and append an ellipsis) so orchestrator.ts
receives a bounded humanMessage.
- Line 320: Update the open-ended JSON prompt guidance for the "summary" field
to include a soft length hint (e.g., "1–3 sentences; place longer reasoning in
TRIAGE.md") so that the generated verdict remains a concise one-glance headline;
modify the string used for the "summary" instruction in triage.ts (the
`"summary": "<as long as needed to faithfully convey the verdict>"` entry) to
include this hint and ensure it references TRIAGE.md's one-paragraph guidance
and the orchestrator's parent humanMessage so the instructions stay consistent.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 0307942b-d574-4876-ab6a-7a3e2a69a73b
📒 Files selected for processing (1)
src/workflows/handlers/triage.ts
…hema Removes the "one-paragraph" framing — summary is now uncapped and may span multiple paragraphs when the verdict honestly requires it. Notes where the field is interpolated downstream so future readers know why length matters. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
## [1.3.2](v1.3.1...v1.3.2) (2026-04-26) ### Bug Fixes * **triage:** remove 500-char cap on verdict summary ([#60](#60)) ([ed3c657](ed3c657))
|
🎉 This PR is included in version 1.3.2 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
1 participant
Description
Drops the artificial
.max(500)constraint onverdict.summaryin the triage workflow. A real triage may need to cite multiple claims, reproductions, and evidence to honestly convey the verdict — capping the summary to 500 chars forced the agent to either truncate analysis or, as observed in production, overflow the cap and have a $1.15 / 21-turn run thrown away by post-validation.The schema already supports detail elsewhere (
TRIAGE.mdis the full uncapped report,reproduction.detailsallows 2000 chars, structuredevidence[]is unbounded). The 500-char cap onsummarywas the only field standing between a successful agent run and afailedoutcome on cosmetic overflow.Prompt template and stale comment updated to match the loosened schema.
Before
After
Related Issues
chrisleekr/github-app-playground#16(run34e361c2-9e9a-433d-b419-f7568221923d) where the agent produced a validVALIDverdict (confidence 0.9) but the post-run validator rejectedTRIAGE_VERDICT.jsonwithsummary too_big — expected ≤500 characters.Testing
bun run typecheck)bun run lint— 0 errors, only pre-existing warnings unrelated to this change)min(1), structured fields,recommendedNextenum, etc.)Notes for reviewer
summaryis interpolated into the tracking-commentreasonline astriage rejected as invalid: ${verdict.summary}. With the cap removed, that line may now span multiple paragraphs. It still renders as valid markdown. If a one-liner headline becomes valuable for log/UI purposes later, the right fix is a separateheadlinefield — not re-imposing a cap onsummary.🤖 Generated with Claude Code
Summary by CodeRabbit