needlefish

Strict, local PR review that acts like a senior engineer — it flags only real defects and stays silent on everything else.

Needlefish reviews your diff before merge and reports only real defects — bugs, regressions, security, data loss, migration/upgrade risk, missing validation, duplicate behavior — never style.

Why it's different:

Prefer-zero findings. A strict senior reviewer's bar: if it isn't worth blocking merge, it's dropped. No style nits, no noise.
Deterministic verdicts. The pass / needs_human / changes_requested verdict is derived from the surviving findings by fixed rules, never freehanded by the model.
Isolated review targets. Reviews run against a throwaway clean clone and are checked for tampering after every model call.
Guarded evals. Every prompt/pipeline change is measured against an 84-scenario harness with active anti-cheat guards before it ships (see Benchmarks).

Small PRs use a review pass plus an adversarial critic; large PRs use map/deep passes before the same critic. Codex is the default runner; Claude Code, opencode, OpenAI-compatible HTTP, Grok, pi, and ACP agents are also supported.

Install

From inside any git repo you want reviewed:

npx needlefish

Requires Node 20+ and at least one authed runner CLI on PATH. Needlefish auto-detects codex, then claude, then opencode. Pass --runner or set NEEDLEFISH_RUNNER when you want a specific runner.

GitHub Action quick start

Add .github/workflows/needlefish.yml to your repo:

name: needlefish
on:
  pull_request:
    types: [opened, synchronize, reopened]
permissions:
  contents: read
  pull-requests: write
  checks: write
jobs:
  review:
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: frankekn/needlefish@v0
        env:
          CODEX_AUTH_JSON: ${{ secrets.CODEX_AUTH_JSON }}

Set one secret — CODEX_AUTH_JSON (the contents of a logged-in codex CLI's ~/.codex/auth.json) or CODEX_API_KEY — and open a PR. Findings arrive as inline review comments anchored to the diff; pushes update the same review in place (fresh / still-open / resolved) instead of stacking new ones.

Cost: 2 model calls per review on small PRs (~56s at the workflow default, gpt-5.6-terra at high effort), 1 map + N deep calls + 1 critic on large ones. Docs-only PRs and unchanged heads skip the model entirely. Maintainers can comment @needlefish recheck or @needlefish explain <finding> on the PR.

Benchmarks

Needlefish ships with a guarded evaluation harness (eval/) and is measured against it before any prompt or pipeline change ships. The fixture set is 84 review scenarios — synthetic planted-bug/negative/honeypot cases plus fixtures mined from real PRs — each run 3 times. Anti-cheat guards are active on every recorded run: per-draw ephemeral HOME, a planted bait answer key with a per-run canary, and a full-transcript scan (cheatDetectedCount: 0 on all numbers below; any structured bait use voids a report).

Production review lane (codex runner, gpt-5.6-terra, high reasoning effort):

Test date	Lane	Anchored recall	False-positive rate	Verdict match
2026-07-19	terra high (current baseline)	0.874	0.056	0.944
2026-07-18	terra high	0.885	0.014	0.972
2026-07-18	sol medium (previous default)	0.879	0.111	0.944

Methodology notes, learned the hard way and enforced by the harness:

Recall is anchored: a finding only counts if it matches the expected pattern AND the expected file. Positives carry difficulty tiers; tier-1 misses disqualify a lane outright.
Provider-side behavior drifts within a day on an identical prompt and config (observed false-positive envelope: 1–5 draws per 72 negatives), so prompt A/B comparisons are only trusted as same-window paired runs.
Reports are comparable only when prompt hash, fixture-set hash, scorer hash, and anti-cheat generation all match; the harness refuses anything else, including its own pre-guard baselines.

Development install

Requires:

Node 20+
Corepack (recommended) or the pinned pnpm from packageManager
One supported model CLI authed locally: Codex, Claude Code, or opencode
GitHub CLI (gh) for --pr, pr, and GitHub Action mode

git clone https://github.com/frankekn/needlefish
cd needlefish
PNPM_VERSION=$(node -p "require('./package.json').packageManager")
corepack enable
corepack prepare "$PNPM_VERSION" --activate
pnpm install --frozen-lockfile

If Corepack is unavailable, install the package manager pinned in package.json:

PNPM_VERSION=$(node -p "require('./package.json').packageManager")
npm exec --yes --package "$PNPM_VERSION" -- pnpm install --frozen-lockfile

Make the development shim resolve on PATH (optional)

The repo keeps a bin/needlefish development shim. After clone, symlink it onto a directory that's on your PATH so you can invoke needlefish from any cwd/shell:

ln -sf "$PWD/bin/needlefish" ~/.local/bin/needlefish   # or any PATH dir
needlefish --version

The shim resolves symlinks and runs the repo-local tsx against src/cli.ts, so it survives the repo being linked from elsewhere and works in non-interactive shells (unlike a shell alias). Without this step, invoke via the full path below.

Local use (read-only, no GitHub writes)

Run from inside any repo you want reviewed, on a branch with changes:

# One-line package install/run:
cd /path/to/some-repo
npx needlefish

# If the development shim is linked (above), from inside the target repo:
needlefish

# Otherwise, full path (cwd is the target):
/path/to/needlefish/node_modules/.bin/tsx /path/to/needlefish/src/cli.ts

# Uncommitted code (no branch/PR needed): if the working tree is dirty —
# or the repo has no commits yet — `needlefish` reviews your uncommitted
# changes, untracked files included. Not a git repo yet? Run `git init` first.
needlefish --repo /path/to/some-repo --uncommitted  # force working-tree review
needlefish --repo /path/to/some-repo --branch       # force merge-base..HEAD review

# Local diff review of committed work. Point at the target with --repo from anywhere:
needlefish --repo /path/to/some-repo --focus security
needlefish --repo /path/to/some-repo --deep
needlefish --repo /path/to/some-repo --pr 123  # attaches PR metadata to the local diff
needlefish --repo /path/to/some-repo --base develop

# PR ref review from any branch:
needlefish pr 123 --repo /path/to/some-repo

# Runner selection:
needlefish --repo /path/to/some-repo --runner claude
needlefish --repo /path/to/some-repo --runner opencode --model zai-coding-plan/glm-5.2
NEEDLEFISH_ACP_BIN=/path/to/acp-agent needlefish --repo /path/to/some-repo --runner acp

Output: Markdown to stdout, JSON saved to ~/.cache/needlefish/<repo>/last-review.json. Pass --json to print the same ReviewResult JSON to stdout instead:

needlefish --repo . --json | jq .verdict

Machine interface

needlefish --repo <path> --json and needlefish pr <number> --json print a versioned ReviewResult JSON object to stdout. The local cache stores the same serialized object at ~/.cache/needlefish/<repo>/last-review.json.

Within a schemaVersion, fields are only added, never changed or removed. Breaking shape changes require a new schemaVersion and changelog entry.

Field	Shape
`schemaVersion`	Literal `1`.
`verdict`	`pass`, `needs_human`, or `changes_requested`.
`reviewTarget`	Optional review target string.
`findings[]`	Finding objects with `severity`, `title`, `category`, `file`, `lineStart`, `lineEnd`, `confidence`, `whyItBreaks`, `suggestedFix`, and `validation`.
`findings[].consumerFile`	Optional downstream file affected by the finding.
`findings[].consumerLine`	Optional downstream line affected by the finding.
`residualRisks[]`	Residual-risk objects with `text` and `blocks`.
`checked[]`	Evidence strings describing what the review examined.
`stats`	Optional per-runner-call timing and attempt stats.
`totalDurationMs`	Optional total review duration in milliseconds.

Base detection

--base → origin/HEAD → main. Pass --base <ref> to override.

GitHub Action mode (self-hosted runner)

needlefish --github --pr N collects the PR via gh api, runs the same core, and posts a non-sticky COMMENT review with the full rendered review body plus the authoritative Needlefish check-run. Verdict → surface mapping:

verdict	review event	check
pass	COMMENT	success
changes_requested	COMMENT	failure
needs_human	COMMENT	neutral
run failed	(none)	failure

All verdict reviews are COMMENT, not approval or blocking-review events. The GITHUB_TOKEN bot is not permitted to formally approve PRs, and sticky blocking reviews can outlive a fixed head. The check-run is the merge gate: a failed review never passes a PR because the check goes failure.

When a finding includes a validated exact replacement, its inline comment adds a native GitHub suggestion block; failed validation falls back to the normal comment without a suggestion.

The reusable workflow skips closed or forked pull_request events before the self-hosted job starts. Manual and reusable dispatch resolve PR metadata first, then skip closed or forked PRs before checkout or model invocation. Before posting any result, the CLI re-reads the PR and skips output if the PR closed or the head SHA moved.

Runner setup (one-time)

Target repos consume needlefish by calling the reusable workflow in this repo. Add a thin caller in the target repo (e.g. .github/workflows/needlefish.yml):

name: needlefish
on:
  pull_request:
    types: [opened, synchronize, reopened]
  workflow_dispatch:
    inputs:
      pr_number: { description: PR number to review (manual trigger), required: true }
permissions:
  contents: read
  pull-requests: write
  checks: write
jobs:
  review:
    uses: frankekn/needlefish/.github/workflows/review.yml@main
    with:
      pr_number: ${{ github.event.inputs.pr_number || github.event.pull_request.number }}
      # Optional:
      # runner: codex
      # model: gpt-5.6-terra
      # codex_reasoning_effort: high
      # timeout_ms: "600000"
    secrets: inherit

To use Grok 4.5, replace the runner and model overrides with runner: grok and model: grok-4.5. The self-hosted workflow requires the authenticated grok CLI on the runner's PATH; it does not install or log in to that CLI for you.

For a one-off Grok review without editing a caller workflow:

PR_NUMBER=123 # replace with the PR number
gh workflow run review.yml -R frankekn/needlefish --ref main \
  -f pr_number="$PR_NUMBER" -f runner=grok -f model=grok-4.5

All production model runners execute without their own process-level permission restrictions. Use them only on a self-hosted runner you control.

Because the caller pins @main, fixes to needlefish's review.yml propagate to every target repo automatically. The runner must have needlefish deployed at ~/.local/bin/needlefish; the workflow does not reinstall the tool on every PR. Hardened installed releases should also publish ~/.local/share/needlefish/current/release.json with the installed Needlefish SHA so review jobs can fail before spending model tokens when a runner is stale.

Register a self-hosted runner on the target repo (free, unlimited minutes). Keep it on a machine you control (EC2/pod/Mac).
Deploy needlefish once on that runner. Future pushes to main run needlefish-deploy and update the runner automatically:
```
ssh termtek@ubuntu 'sh -s' < scripts/deploy-ubuntu.sh
```
The current production fleet uses one shared x64 installation plus two separate ARM installations. Deploy the same release SHA to all three installations and verify their installed metadata before trusting the fleet.
Ensure the runner has gh and the selected model CLI on PATH.
On that runner, auth the selected CLI once. For Codex:
```
printf '%s' "$CODEX_API_KEY" | codex login --with-api-key -c 'service_tier="fast"'
```
For Grok, complete the provider's CLI login or key setup as appropriate and verify that grok runs as the runner service account.
If needlefish is private, the caller repo must be allowed to call this reusable workflow; otherwise (public) the default GITHUB_TOKEN is enough.
Runner global-instructions caveat: model CLIs may auto-load global instructions from the runner's home directory. needlefish instructs the model to ignore anything outside the target repo's AGENTS.md as policy, but if you want zero leakage, keep the runner home free of unrelated instruction files.

Self-hosted runners execute PR code on your machine. Fine for solo use on your own repos; if you ever open PRs to outside contributors, isolate the runner (ephemeral container) so contributor code can't touch your persistent host.

GitHub Action (hosted, any repo)

No self-hosted runner required: this repo doubles as a composite action that runs on GitHub-hosted ubuntu-latest. Add a workflow to the target repo. The hosted action installs the runners listed in action.yml; use the self-hosted workflow above for Grok 4.5 because the hosted action does not install the Grok CLI.

name: needlefish
on:
  pull_request:
    types: [opened, synchronize, reopened]
permissions:
  contents: read
  pull-requests: write
  checks: write
jobs:
  review:
    # Fork PRs don't receive secrets; skip them instead of failing at model auth.
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0 # full history: needlefish needs the merge base
      - uses: frankekn/needlefish@v0
        env:
          CODEX_AUTH_JSON: ${{ secrets.CODEX_AUTH_JSON }}

Runner authentication (repo secrets, passed via env on the action step):

runner	secret(s)
codex	`CODEX_AUTH_JSON` (contents of a logged-in `~/.codex/auth.json`) or `CODEX_API_KEY`
claude	`ANTHROPIC_API_KEY`
opencode	provider key for the chosen model (e.g. `OPENAI_API_KEY`)
openai	`OPENAI_API_KEY`
grok	Grok CLI auth or provider-specific key (self-hosted lane)
pi	`PI_AUTH_JSON` (contents of a logged-in `~/.pi/agent/auth.json`)
acp	agent-specific auth plus `NEEDLEFISH_ACP_BIN` on the runner

The claude auth vars (ANTHROPIC_API_KEY, CLAUDE_CODE_OAUTH_TOKEN) and opencode's OPENAI_API_KEY are allowlisted through to the runner subprocess; other providers' keys need NEEDLEFISH_RUNNER_ENV_PASSTHROUGH=VAR (see "Runner subprocess environment").

Inputs (all optional): pr_number (defaults to the event PR), runner (default codex), model, timeout_ms, codex_reasoning_effort, runner_version (npm version of the runner CLI), repo_path (defaults to the workspace checkout), github_token (defaults to the workflow token).

Cost and behavior notes:

Small PRs use 2 model calls per PR (review + critic), about 56s at the workflow default, gpt-5.6-terra at high effort. Large PRs use 1 map call + N deep calls (concurrency 3 by default) + 1 critic. Docs-only PRs use 0 model calls. Same-head re-runs use 0 model calls unless forced with --recheck.
Fork PRs don't receive secrets by default. The if: gate above skips them. pull_request_target would hand secrets to workflows triggered by fork code — avoid it unless you fully understand the exposure.
The hosted path cold-starts on every run (pnpm install + runner CLI install, roughly a minute). The self-hosted path above stays the low-latency option.

Model runner invocation

src/shared/codex.ts invokes the selected runner. Use --runner, --model, and --timeout-ms, or the matching env vars:

option	env	default
runner	`NEEDLEFISH_RUNNER`	auto-detects `codex`, then `claude`, then `opencode`
model	`NEEDLEFISH_MODEL`	runner default
Codex reasoning effort	`CODEX_REASONING_EFFORT`	`medium` (reusable workflow: `high` for `gpt-5.6-terra`)
timeout	`NEEDLEFISH_TIMEOUT_MS`	`600000`

Runner-specific binary env vars are CODEX_BIN, CLAUDE_BIN, OPENCODE_BIN, GROK_BIN, PI_BIN, and NEEDLEFISH_ACP_BIN. NEEDLEFISH_ACP_BIN is required for the acp runner. Existing CODEX_MODEL, CODEX_TIMEOUT_MS, and CODEX_RETRY_MS still work for Codex compatibility.

When neither --runner nor NEEDLEFISH_RUNNER is set and none of codex, claude, or opencode can be found, Needlefish exits with install commands for those three CLIs instead of a stack trace.

Codex runs with --ignore-user-config --ignore-rules --dangerously-bypass-approvals-and-sandbox so its inspection commands are not blocked by execpolicy rules, approval prompts, or the host sandbox. Needlefish still runs it inside a throwaway clean clone, strips GitHub tokens, fixes the expected HEAD, and rejects any worktree mutation. medium is the default; set CODEX_REASONING_EFFORT=high to restore the old default, or xhigh for the highest-effort mode. Claude Code runs with --dangerously-skip-permissions, --safe-mode, and no session persistence. Grok runs with --always-approve --permission-mode bypassPermissions --no-plan --sandbox off. opencode runs with --auto in headless mode and an inline permission: "allow" override for its global and build-agent permissions. pi runs with --no-session --mode text --provider openai-codex --thinking <level> and its default full toolset. ACP runs a JSON-RPC 2.0 Agent Client Protocol process over stdio from NEEDLEFISH_ACP_BIN; Needlefish sends session/cancel on timeout, then applies the same process-group kill path as the CLI runners. Closed PRs are skipped before diffing or model invocation. All CLI runners execute inside a throwaway clean clone at the review head commit; needlefish checks that clone with git status --porcelain --untracked-files=all --ignored=matching and verifies HEAD did not move after each successful model call.

Runner subprocess environment

Runner CLIs (codex, claude, opencode, grok, pi, acp) are spawned with an allowlisted environment, not the full parent process.env — only locale/proxy/path basics plus each runner's own _BIN/_MODEL-style variables are passed through. To pass an additional variable to the runner subprocess, set NEEDLEFISH_RUNNER_ENV_PASSTHROUGH=VAR1,VAR2 (comma-separated names).

ACP env authentication additionally requires an explicit credential declaration: set NEEDLEFISH_ACP_AUTH_ENV_VARS to the credential names and include those same names in NEEDLEFISH_RUNNER_ENV_PASSTHROUGH, for example NEEDLEFISH_ACP_AUTH_ENV_VARS=MY_AGENT_TOKEN with NEEDLEFISH_RUNNER_ENV_PASSTHROUGH=MY_AGENT_TOKEN. Arbitrary passthrough configuration does not prove authentication. Alternatively, set NEEDLEFISH_ACP_AUTH_FILES to comma-separated HOME-relative credential files; Needlefish copies only those files into the disposable runner HOME.

Verdict derivation (deterministic)

any P0 / P1 / P2 finding → changes_requested
otherwise a blocking residual risk → needs_human
otherwise → pass

P3-only findings are reported but do not block (check stays green).

Status

v0.3.4. Read-only. Shipped: inline review comments, sticky re-review (fresh/open/resolved across pushes), @needlefish recheck / @needlefish explain maintainer commands, docs-only fast path (no model calls), same-head dedupe, hosted-runner repo inspection (best-effort AppArmor sysctl). --fix stays unimplemented by design.

Name		Name	Last commit message	Last commit date
Latest commit History 215 Commits
.conductor		.conductor
.github/workflows		.github/workflows
assets		assets
bin		bin
docs		docs
eval		eval
plans		plans
prompts		prompts
scripts		scripts
src		src
.gitignore		.gitignore
AGENTS.md		AGENTS.md
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
FUTURE_TODO.md		FUTURE_TODO.md
LICENSE		LICENSE
README.md		README.md
README.zh-TW.md		README.zh-TW.md
action.yml		action.yml
eslint.config.js		eslint.config.js
package.json		package.json
paseo.json		paseo.json
pnpm-lock.yaml		pnpm-lock.yaml
pnpm-workspace.yaml		pnpm-workspace.yaml
tsconfig.build.json		tsconfig.build.json
tsconfig.json		tsconfig.json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

needlefish

Contents

Install

GitHub Action quick start

Benchmarks

Development install

Make the development shim resolve on PATH (optional)

Local use (read-only, no GitHub writes)

Machine interface

Base detection

GitHub Action mode (self-hosted runner)

Runner setup (one-time)

GitHub Action (hosted, any repo)

Model runner invocation

Runner subprocess environment

Verdict derivation (deterministic)

Status

About

Releases

Packages

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

needlefish

Contents

Install

GitHub Action quick start

Benchmarks

Development install

Make the development shim resolve on PATH (optional)

Local use (read-only, no GitHub writes)

Machine interface

Base detection

GitHub Action mode (self-hosted runner)

Runner setup (one-time)

GitHub Action (hosted, any repo)

Model runner invocation

Runner subprocess environment

Verdict derivation (deterministic)

Status

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages