RubyRT: Ruby (but mostly Rails) Review Thing

RubyRT is an opinionated but helpful and flexible AI code review tool for Ruby and Rails projects. It is heavily inspired by Gito and brings the same LLM-driven review workflow to Ruby codebases, with extra context pulled from the Ruby LSP.

Features

• AI-powered code review via any OpenAI-compatible LLM (powered by ruby_llm).
• Ruby and Rails-aware review prompts out of the box.
• Reads project skills and rules from configurable directories (defaults to .agents, .claude, and .cursor).
• Supports auxiliary files (aux_files) for injecting individual files as extra context into review prompts.
• Configurable request timeouts, retries, and logging to fail fast on unreachable providers.
• Pulls extra context from language servers (LSP) during review — ruby-lsp first, any LSP via config. (Run static analyzers like RuboCop as a separate step.)
• Cuts false positives with a critic pass ([verify]): every surviving finding is re-checked by a fresh, skeptical LLM call before it's reported.
• Runs as a GitHub Action composite action, posting feedback as precise PR comments and (with a suitable token) resolving stale review threads automatically.

Quickstart (after release)

Add to your repository:

# .github/workflows/rubyrt-review.yml
name: "RubyRT Review"
on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  review:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: Bonusly/rubyrt/.github/actions/rubyrt@v1
        with:
          api_key: ${{ secrets.LLM_API_KEY }}
          provider: openrouter
          model: moonshotai/kimi-k2.6

Ruby version and run command

• ruby_version — Ruby the action installs to run RubyRT. Defaults to 3.4.
• rubyrt_command — how the CLI is invoked. Defaults to rubyrt. Set it to bundle exec rubyrt, direnv exec . rubyrt, etc. when your environment needs it.
• rubyrt_version — gem version to install (local builds from the checked-out repo). Set to skip to install nothing when rubyrt_command already provides RubyRT (e.g. it's in your Gemfile).

      - uses: Bonusly/rubyrt/.github/actions/rubyrt@v1
        with:
          api_key: ${{ secrets.LLM_API_KEY }}
          ruby_version: "4.0"
          rubyrt_command: bundle exec rubyrt
          rubyrt_version: skip   # rubyrt comes from the project's Gemfile

Full working example (the workflow this repo runs)

RubyRT reviews its own pull requests. Below is the actual .github/workflows/rubyrt-review.yml from this repo — copy it as a known-good starting point. It mints a GitHub App token when one is configured (so comments post under the app's name) and otherwise falls back to the default token.

# .github/workflows/rubyrt-review.yml
name: "RubyRT Review"

on:
  pull_request:
    types: [opened, synchronize, reopened]
  workflow_dispatch:
    inputs:
      pr_number:
        description: "Pull Request number"
        required: true

jobs:
  review:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v7
        with:
          ref: ${{ github.event.pull_request.head.sha }}
      # Mint an app installation token so review comments post under the app's
      # name/avatar. Skipped (falls back to GITHUB_TOKEN) when the app isn't set up.
      - uses: actions/create-github-app-token@v3
        id: app_token
        if: ${{ vars.RUBYRT_APP_ID != '' }}
        with:
          client-id: ${{ vars.RUBYRT_APP_ID }}
          private-key: ${{ secrets.RUBYRT_APP_PRIVATE_KEY }}
      - name: Run RubyRT review
        uses: Bonusly/rubyrt/.github/actions/rubyrt@v1
        with:
          api_key: ${{ secrets.LLM_API_KEY }}
          model: ${{ vars.LLM_MODEL }}
          provider: ${{ vars.LLM_PROVIDER }}
          # Post as the app when configured, otherwise as github-actions.
          github_token: ${{ steps.app_token.outputs.token || github.token }}
          # PAT (user token) with pull-requests write so stale review threads can
          # be auto-resolved — GITHUB_TOKEN and App tokens cannot. Skipped if unset.
          resolve_token: ${{ secrets.RUBYRT_RESOLVE_TOKEN }}

Model and provider come from repo variables (vars.LLM_MODEL, vars.LLM_PROVIDER) so you can change them without editing the workflow. This repo runs the review on OpenRouter and re-checks findings with a stronger model in the critic pass — see The critic pass.

The action referenced above is the composite action at .github/actions/rubyrt/action.yml; it installs Ruby, installs RubyRT, computes the base ref (unshallowing as needed so merge-base is correct), runs the review, posts the comment, and uploads the JSON and Markdown reports as artifacts.

Required secrets and permissions

• secrets.LLM_API_KEY: Your LLM provider API key.
• secrets.GITHUB_TOKEN is provided automatically by GitHub Actions. It must have at least pull-requests: write permission (set in the workflow permissions block) so RubyRT can post review comments.
• Want comments to post under a custom name and avatar instead of "github-actions"? See Posting as a GitHub App below.
• Resolving stale review threads needs an extra token — see Resolving stale review threads below. The default GITHUB_TOKEN cannot do it.

Posting as a GitHub App (custom name and avatar)

By default RubyRT posts as github-actions[bot] (the workflow GITHUB_TOKEN). To post under your own bot name and avatar, authenticate with a GitHub App installation token and pass it as the action's github_token input — review comments then appear as YourApp[bot].

1. Create the app: Settings → Developer settings → GitHub Apps → New GitHub App (under your org or account). Give it the name and avatar you want to see on comments.
   • Repository permissions → Pull requests: Read and write. (No webhook needed — uncheck Active.)
2. Generate a private key and note the App ID.
3. Install the app on the repositories that run RubyRT.
4. Store the App ID as a variable (e.g. vars.RUBYRT_APP_ID) and the private key as a secret (e.g. secrets.RUBYRT_APP_PRIVATE_KEY).
5. Mint an installation token with actions/create-github-app-token and pass it as github_token:

jobs:
  review:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/create-github-app-token@v2
        id: app_token
        with:
          app-id: ${{ vars.RUBYRT_APP_ID }}
          private-key: ${{ secrets.RUBYRT_APP_PRIVATE_KEY }}
      - uses: Bonusly/rubyrt/.github/actions/rubyrt@v1
        with:
          api_key: ${{ secrets.LLM_API_KEY }}
          github_token: ${{ steps.app_token.outputs.token }}   # post as the app
          resolve_token: ${{ secrets.RUBYRT_RESOLVE_TOKEN }}    # resolve threads (PAT — see below)

The app installation token posts comments fine, but cannot resolve review threads (resolveReviewThread rejects server-to-server tokens). If you also want stale threads auto-resolved, pair it with a user PAT in resolve_token as shown — see the next section.

Resolving stale review threads

When RubyRT re-reviews a PR, it can mark its earlier threads as resolved once the issue is fixed or the line is outdated. This uses the GraphQL resolveReviewThread mutation, which requires a user-to-server token (a personal access token).

The default GITHUB_TOKEN cannot do this, and neither can a GitHub App. Both are server-to-server tokens, and resolveReviewThread returns Resource not accessible by integration for them even with pull-requests: write — including the installation token from actions/create-github-app-token. Only a user PAT works.

To enable auto-resolving, create a PAT and pass it via the resolve_token action input (or the RUBYRT_RESOLVE_TOKEN env var / --resolve-token flag for the github-comment CLI). If you don't set one, RubyRT skips resolving and still posts the new review.

1. Create a token as a user with write access to the repo:
   • Classic PAT with the repo scope (recommended — reliably works for resolveReviewThread). Settings → Developer settings → Personal access tokens → Tokens (classic). If your org enforces SSO, click Configure SSO on the token and authorize it for the org.
   • Fine-grained PATs are not recommended here. Even with Pull requests: Read and write they often fail resolveReviewThread with Resource not accessible by personal access token — and for an org repo the write permission stays read-only until an org owner approves it, so the fetch succeeds but resolving fails. If you must use one, get it org-approved and expect to troubleshoot.
2. Store it as a repository (or org) secret, e.g. secrets.RUBYRT_RESOLVE_TOKEN.
3. Pass it to the action:

jobs:
  review:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: Bonusly/rubyrt/.github/actions/rubyrt@v1
        with:
          api_key: ${{ secrets.LLM_API_KEY }}
          provider: openrouter
          model: moonshotai/kimi-k2.6
          resolve_token: ${{ secrets.RUBYRT_RESOLVE_TOKEN }}

Threads are resolved as whichever user owns the PAT. (A machine/bot user account is a good fit if you don't want resolutions attributed to a person.)

Auto-approving PRs

RubyRT can submit an Approve review when a PR is clean enough. It's off by default — enable it under [approve] in .rubyrt/config.toml:

[approve]
enabled = true
max_changes = 500          # additions + deletions ceiling; skipped if GitHub doesn't report the size
max_severity = 3           # issues at/below this severity number (1=Critical) block approval
skip_label = "rubyrt-skip-approve"
protected_paths = ["app/billing/**", "config/secrets.yml"]  # globs that block approval when changed
dry_run = false            # evaluate and log the decision without approving/dismissing

A PR is approved only when all of these hold:

• It isn't a draft and doesn't carry the skip_label.
• It doesn't modify .rubyrt/config.toml — a PR can't weaken the approval rules and wave itself through.
• No changed file matches a protected_paths glob — e.g. billing code or sensitive config you always want a human to review.
• Total changes are within max_changes (this check is skipped when the size is unknown).
• This run produced no findings at or above max_severity.
• No RubyRT findings at or above max_severity are still unresolved.
• Every resolved RubyRT finding at or above max_severity was resolved by someone who is neither the PR author nor a contributor to the PR — an author can't clear their own findings to earn an approval.

Re-runs are idempotent: RubyRT won't stack a second approval on a head commit it already approved, and it dismisses its earlier approval if a later push stops meeting the rules. With dry_run = true it logs the decision and reasons but takes no action — useful for rollout.

The approval needs the workflow's pull-requests: write permission (already required for comments). RubyRT tries the approval with the main github_token first, then falls back to the resolve_token PAT if that attempt fails.

Heads up on which token approves. An approval submitted by GITHUB_TOKEN (or a GitHub App installation token) does not count toward branch-protection "required approvals", and the fallback only triggers when the first attempt errors — a non-counting approval from GITHUB_TOKEN succeeds and won't fall through to the PAT. If you need the approval to satisfy required reviewers, run RubyRT with the PAT as its main github_token (or as a bot user) so the counting identity is the one that approves.

Limitation. "Contributor" is determined from commit author/committer GitHub logins, so a Co-authored-by: trailer doesn't count as having committed. If a thread's resolver can't be determined (e.g. a deleted account), RubyRT fails safe and does not approve.

Configuration options

RubyRT reads configuration from layers (later layers override earlier ones):

1. lib/rubyrt/config/default.toml bundled with the gem.
2. .rubyrt/config.toml in your project root.
3. ~/.rubyrt/.env (loaded into your environment).
4. Environment variables.
5. CLI flags.

Key settings:

Setting	Default	Config key	Environment variable	CLI flag
LLM provider	openai	provider	LLM_PROVIDER	-p, --provider
LLM model	gpt-4o	model	LLM_MODEL	-m, --model
API key	—	llm_api_key	LLM_API_KEY	—
API base URL	provider default	llm_api_base	LLM_API_BASE	—
Request timeout (seconds)	120	request_timeout	LLM_REQUEST_TIMEOUT	—
Retries	3	retries	LLM_RETRIES	—
Log file	stdout	log_file	RUBYRT_LOG_FILE	—
Log level	info	log_level	RUBYRT_LOG_LEVEL	—
Max concurrent file reviews	10	max_concurrent_tasks	MAX_CONCURRENT_TASKS	—
Confidence threshold (keep if ≤)	1	post_process.max_confidence	—	—
Severity threshold (keep if ≤)	3	post_process.max_severity	—	—
Critic pass enabled	true	verify.enabled	—	—
Critic pass model	review model	verify.model	—	—
Auto-approve PRs	false	approve.enabled	—	—
Auto-approve change limit	500	approve.max_changes	—	—
Auto-approve severity gate	3	approve.max_severity	—	—
Auto-approve skip label	rubyrt-skip-approve	approve.skip_label	—	—
Auto-approve protected paths	[]	approve.protected_paths	—	—
Auto-approve dry run	false	approve.dry_run	—	—
Skill directories	.agents, .claude, .cursor	skill_directories	—	—
Auxiliary files	[]	aux_files	—	—
Language servers	none	lsp.<name>	—	—

Supported providers match whatever RubyLLM supports, including openai, anthropic, gemini, ollama, deepseek, openrouter, mistral, perplexity, xai, azure, bedrock, vertexai, and gpustack.

Example .rubyrt/config.toml:

provider = "openrouter"
model = "moonshotai/kimi-k2.6"
request_timeout = 60
log_file = "log/rubyrt.log"
log_level = "debug"

# Add extra files as context to every review prompt
aux_files = ["docs/conventions.md", ".rubyrt/style-guide.md"]

# Customize which directories are scanned for skill fragments
skill_directories = [".agents", ".github"]

# Only keep the most-confident findings (1) up to medium severity (3).
[post_process]
max_confidence = 1
max_severity = 3

# Re-check every surviving finding with a stronger model before reporting it.
[verify]
enabled = true
model = "anthropic/claude-opus-4.8"

# Give the LLM access to a language server for extra code context.
[lsp.ruby]
command = ["ruby-lsp"]
extensions = [".rb", ".rake"]

Skills and auxiliary files

RubyRT automatically discovers markdown files (.md) in skill directories and injects them as additional rules in every review prompt. By default it scans .agents/, .claude/, and .cursor/ in the project root. Override this with the skill_directories config key.

For individual files (rather than whole directories), use aux_files to list paths relative to the project root. Their contents are included as extra context in every review prompt:

aux_files = ["docs/coding-standards.md", "docs/security-rules.md"]

The critic pass (reducing false positives)

The biggest source of noise in AI review is confident-but-wrong findings — claims that simply aren't true about the code. Tightening the confidence and severity thresholds only filters by the model's self-reported confidence, which it routinely over-states.

The [verify] critic pass attacks this directly. After the normal review and threshold filtering, each surviving finding gets a second, fresh LLM call framed adversarially: "try to refute this; uphold it only if you can point to the lines that make it true; when uncertain, reject." The critic has the diff, the full file, and (if configured) the LSP symbol tool, so it can confirm API and method claims instead of trusting them. Findings it can't uphold are dropped.

[verify]
enabled = true                      # on by default; set false to skip the pass
model = "anthropic/claude-opus-4.8" # optional: re-check on a stronger model

• verify.enabled — turn the critic pass on or off. Default true.
• verify.model — run the critic on a different (usually stronger) model than the review, using the same provider and API key. Leave it unset to reuse the review model. With OpenRouter the model slug is provider-prefixed (e.g. anthropic/claude-opus-4.8), so one key covers both the cheap review model and the strong critic.

Because the critic runs only on findings that survived filtering — not on every file — its cost scales with the number of findings, not the size of the diff. It also fails open: if a critic call errors or returns an unparseable verdict, the finding is kept and a processing warning is recorded, so a broken critic never silently swallows a real bug.

Filtering findings (post_process)

The model scores every finding on a 1–4 severity scale (1 = Critical) and a 1–4 confidence scale (1 = highest). post_process drops anything above your thresholds before the critic pass runs:

[post_process]
max_confidence = 1   # keep only the model's highest-confidence findings
max_severity = 3     # keep Critical/High/Medium, drop Low

Lower numbers are stricter. An omitted threshold means "no limit". This is a cheap first filter; the critic pass is the precision filter on top of it.

Language servers (LSP)

When you configure a language server, RubyRT exposes a symbol-lookup tool to the LLM during both review and verification. The model uses it to confirm that a class, method, or constant actually exists (and how a third-party API is shaped) before reporting an "undefined" or "misused API" issue — a major source of hallucinated findings.

# Each [lsp.<name>] entry is launched in the reviewed project's root, so use
# that project's own LSP binary. Add more languages with more entries.
[lsp.ruby]
command = ["ruby-lsp"]
extensions = [".rb", ".rake"]

A server is only started when the changeset contains a file whose extension matches one of its extensions, so unrelated reviews don't pay the startup cost. LSP is disabled entirely when no [lsp.*] entry is configured.

Logging

Set log_file to a file path to persist RubyLLM request logs for debugging. Set log_level to debug to see full request and response bodies. Valid levels are debug, info, warn, error, and fatal.

Structured output

RubyRT uses structured output to ensure the LLM returns valid JSON. Not all models support this feature; see the RubyLLM available models page for a list of compatible models. For models that don't support structured output, RubyRT falls back to parsing the response as JSON from the prompt instructions.

Development

bundle install
bundle exec rake

License

MIT. See LICENSE.txt.