Skip to content

review-kit 0.5.0: cost estimate, measured budget, shorter listing entry - #20

Merged
aoreshkov merged 1 commit into
mainfrom
feat/review-kit-0.5.0
Oct 28, 2026
Merged

aoreshkov merged 1 commit into
mainfrom
feat/review-kit-0.5.0

Conversation

@aoreshkov

Copy link
Copy Markdown
Owner

Summary

  • Cost estimate before a capped run. A needs-confirmation stop now carries estimate: the likely and worst-case agents and tokens for the pairs in wouldRun and for all pairs. next tells the caller to show it when asking the user.
    • At the defaults, 12 pairs come to about 32 agents and 1.1M tokens, at most 85 agents and 2.4M.
    • The constants are per-agent averages from a real 62-pair run: 43k tokens per reviewer, 26k per verifier, 1.3 findings per reviewer, 8.5% claimed critical or high. The result says they grow with the repository's size.
  • Measured token budget. Sizing under a +500k-style budget uses measured output per agent (4k per reviewer, 2k per verifier, 25k fixed) and now follows votes, maxVerifyPerJob and verify. A job costs 16k at the defaults instead of a flat 25k, so +500k runs 29 jobs instead of 19.
  • Shorter listing entry. whenToUse drops each argument's shape (the bad-args messages already give it) and points one-branch or one-PR reviews at /code-review.
    • The entry is 932 chars, down from 1,366 in 0.4.0. A test keeps it under 993, half of 0.2.2's 1,986.
    • On 2.1.312, claude plugin details reports ~152 tokens because it counts only the agents. /context puts the agents at 169 and an installed 0.3.1 entry (1,303 chars) at ~440–460 tokens. The 0.5.0 entry works out to ~320 tokens at the same ratio (derived, not measured).
  • README. New "When to use which" section, the estimate, the re-derived token budget, measured run costs, and reviewer-model advice: model: { reviewer: "opus" } for a release gate, the defaults for routine sweeps.

Eval (2.1.312, one real run)

The shorter whenToUse could stop the workflow being picked, so the seeded-defects case ran once as a real session against this branch: 11/12 graders, workflow-used passed (one Workflow call with matching input), 5/5 defects and 1/4 decoys confirmed, 119 s, $1.84. The decoy count is within 0.4.0's range on the default models.

Test plan

  • node --test (206 pass)
  • claude plugin validate --strict for ., ./review-kit, ./token-audit, ./token-meter
  • claude plugin test ./token-meter (40 pass)
  • Seeded-defects real run as above

After merge: tag review-kit-v0.5.0 with notes from the changelog, then confirm the listing figure with /context on an installed 0.5.0.

🤖 Generated with Claude Code

- A needs-confirmation stop carries an estimate of agents and tokens
  (likely and worst case) for the pairs in wouldRun and for all pairs,
  and next tells the caller to show it. The constants are per-agent
  averages from a real 62-pair run.
- Token-budget sizing uses measured output per agent and follows
  votes, maxVerifyPerJob and verify: 16k per job at the defaults
  instead of a flat 25k, so "+500k" runs 29 jobs instead of 19.
- whenToUse is shorter and points one-branch or one-PR reviews at
  /code-review. The listing entry is 932 chars, down from 1,366; a
  test keeps it under 993, half of 0.2.2's 1,986.
- README: "When to use which", the estimate, the re-derived token
  budget, measured run costs and reviewer-model advice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aoreshkov
aoreshkov merged commit 798fdc7 into main Oct 28, 2026
5 checks passed
@aoreshkov
aoreshkov deleted the feat/review-kit-0.5.0 branch October 28, 2026 06:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant