Claude PR review as a CI check, triggered by a `/review` comment

[walkerhughes]

Problem

CI here gates on lint, typecheck, coverage, task verifiers, the eval suites, and the plugin version bump. All of that is mechanical. Nothing in CI reads the diff and says whether the change is any good — whether a tool schema still steers the model toward valid calls, whether a skill's instructions actually match what the code does, whether an eval task was edited to fit a failing run rather than to measure something.

That review currently only happens by hand, and only when someone remembers to do it.

Proposal

A workflow that runs a Claude review over the PR diff and leaves its findings as review comments on the PR. Driven by the Claude Agent SDK, with the poneytail plugin loaded so the review uses its review skill rather than an ad-hoc prompt in a YAML file.

• Trigger: a /review comment on the PR. On demand, not on every push — a review that runs on every commit becomes noise and spends tokens on work-in-progress.
• Auth: CLAUDE_CODE_OAUTH_TOKEN, the same subscription-auth path the eval gates already use. Not ANTHROPIC_API_KEY.
• Output: review comments on the PR, anchored to lines where the finding is line-specific.

How it hangs together

Sketch, not a spec:

• on: issue_comment (types [created]), filtered to github.event.issue.pull_request != null and a body matching /review. Note that issue_comment workflows always run the copy of the file on the default branch, so iterating on this cannot be done from a PR branch the way the other workflows can.
• Resolve the PR head from the API and check it out; issue_comment gives no pull_request payload, so the ref has to be looked up.
• React 👀 on the triggering comment immediately, so a slow review does not look like a dead one.
• concurrency keyed on the PR number, cancel-in-progress: true — a second /review should supersede the first, not queue behind it.
• The SDK run gets the diff and the repo, loads poneytail, and returns findings; a step turns those into a pending review with line comments and submits it.

Auth and permissions

The eval gates get their subscription auth for free because pull_request runs on forks carry no secrets. issue_comment is different: it runs in the base repo's context with secrets available, on a comment any user can leave, against code from the PR head. So the guard has to be explicit:

• Gate on author_association (OWNER / MEMBER / COLLABORATOR) so a drive-by commenter cannot spend a rate-limit window.
• Do not build, install, or execute anything from the checked-out head with the token in the environment. The review reads the diff; it should not need to run the code.
• Same token-expiry probe as tastytrade.yml — a review job that silently no-ops on a revoked token is worse than one that fails.
• pull_request: write for the review, issues: write for the reaction.

Open pieces

• poneytail. Nothing in this repo or its marketplace references it, so this needs @walkerhughes to say where it comes from — a plugin to be written here, one from another marketplace, or an existing internal one. If it lands in this repo it also needs a plugins/ entry, a marketplace entry, and the version-bump gate applies to it.
• Is it a check or a comment? The title says CI check, but a comment-triggered job cannot be a required status check — a PR nobody types /review on would sit pending forever. Either it stays advisory, or something also runs it automatically on PR open so there is always a result to gate on.
• Model and cost. The eval gates are already serialized against a shared subscription rate limit. A review job draws on the same account, so it may want to join that concurrency group rather than compete with it.
• Scope of the diff. Whole PR every time, or only what changed since the last review comment? Re-reviewing an unchanged file repeats findings that were already addressed.
• Duplicate findings. Repeated /review calls should not re-post comments that are already on the PR, and probably should not reopen threads that were resolved.
• Forks. With the author-association gate, a fork PR from a non-collaborator gets no review at all. Decide whether that is the intended behavior or whether a maintainer typing /review on a fork PR should run it.