AI first-pass review on outside fork PRs — earned, not assumed

[simonoppowa]

Destination

Outside fork PRs stop sitting unreviewed — but only if a review is worth having.

Originally drawn as "Jules posts advisory reviews on every PR I did not author". Research killed that: Jules has no PR-typed input anywhere in its API and cannot review anyone's pull request. The need survives the tool — both currently-open outside PRs have zero reviews — so the map was redrawn rather than closed.

The route is now earn it, then automate it:

1. Prove the value by hand on a real fork PR with the local /code-review skill. Nothing installed, nothing granted, no third party given contents: write.
2. Only if that verdict is "worth automating", choose a tool and ship it.

A verdict of worth doing by hand or not worth doing closes this map, and both are real outcomes rather than failures.

Status, 2026-09-04. Step 1 answered — #1035 proved AI first-pass review worth automating. Step 2 is half done: the repository is prepared and the tool chosen, but nothing is installed and no review has run. All children are closed; the remaining work is a short checklist on the maintainer's own accounts, recorded on #1031. Reopen or cut a fresh effort when that is done and there is a verdict to record.

Notes

Domain. Jules is Google's asynchronous coding agent (jules.google.com). Its documented loop is authoring: clone into a cloud VM, plan, edit, open a PR. Whether it reviews PRs as a first-class mode is the open question this map opens on. Account: Google AI Pro subscription.

The gap this closes. 63 outside PRs from 19 contributors have landed here (jordan-lee-code alone at 31), plus app/copilot-swe-agent at 2. Both currently-open outside PRs — #1010 and #995 — have zero reviews, zero comments, no assignee. Dependabot adds 11 open bumps, the oldest (#770, #773) months stale. Outside work arrives faster than it gets looked at.

What makes this repo hard to review by hand. AGENTS.md is 442 lines / 30 KB and explicitly canonical (CLAUDE.md and .github/copilot-instructions.md are thin pointers to it). It carries conventions no generic reviewer would know: Semantics(identifier:) on every new interactive widget with the container: true flex gotcha, nine ARBs held at equal key count, no Co-authored-by: trailer naming any AI, 120-char width, lib/generated/ never hand-edited, codegen rerun on DBO/DTO/.env change, and the Row overflow rules (Expanded + maxLines + ellipsis, AutoSizeText for prominent titles). Most are convention, not lint — nothing enforces them today except a reviewer noticing.

Why the GitHub App and not a workflow. Every outside PR comes from a fork. A pull_request run on a fork gets a read-only token and no secrets, so it cannot post; pull_request_target can, and is the best-known footgun in GitHub Actions. This repo's Actions secrets hold ANDROID_KEYSTORE_BASE, ANDROID_KEY_PASSWORD, APP_STORE_CONNECT_API_KEY_*, GOOGLE_PLAY_SERVICE_ACCOUNT_JSON and MATCH_* — release signing material for both stores. A GitHub App cannot read Actions secrets; that separation is the reason this wiring was chosen. No workflow in this repo uses pull_request_target today and this map does not introduce one.

Standing concern, stated once. General code review + public + advisory + first-time contributors is the noisiest available combination. A stranger's first PR meeting a wall of generic LLM commentary is a contributor-retention cost. Tuning that is a ticket in this map, not a thing to pre-solve.

Skills every session should consult: /grilling and /domain-modeling for decisions; /research for anything about Jules' current capabilities — the product moves fast and model knowledge of it is stale.

Repo conventions (AGENTS.md): conventional commits; never add an AI assistant as co-author or name it in a commit subject or body. Do not run repo-wide dart format — the pinned SDK rewrites ~270 committed files.

Execution override. This map is plan-only except its final ticket, which carries execution: merge the configuration, run it against a real PR, and record the verdict.

Decisions so far

Settled while charting (2026-09-02), before any ticket existed:

• Jules is a reviewer here, not a feature author. The async-backlog-worker use case was considered and set aside.
• Scope is author != simonoppowa — outside contributors and Dependabot.
• The review covers conventions and general code review, not conventions alone.
• Output is a public GitHub review, advisory, never blocking.
• Wiring is the native Jules GitHub App — ruling out pull_request_target and any workflow holding a JULES_API_KEY.

Settled by the four research tickets, all closed 2026-09-02:

• Does Jules review pull requests natively, and what fires it? — No, and this is now settled against a live surface rather than the stale changelog. The API discovery doc (revision 20260901) contains "review" zero times; GitHubRepoContext accepts only a branch name — no PR number, no ref, no PR-typed input anywhere. Behaviourally: 1,317 sampled google-labs-jules[bot] review records, all COMMENTED with body length 0, none from a fork — threaded replies on Jules' own branches, not reviews. The destination as drawn is unreachable.
• What the Jules GitHub App asks for — the premise behind the wiring decision does not hold. The App requests contents: write + workflows: write + actions: write. GitHub's own guidance: write access is secret access, because the actor can author and run a workflow that echoes them. checks: read separately confirms it could never post a check run. Also: Google may train on sessions linked to public repositories, with human review of a sample; opt-out is off by default.
• What Google AI Pro actually buys — 100 tasks/rolling 24h, 15 concurrent; this repo needs ~38–57/month. Not a budget constraint; a burst one — over 15 in flight returns 429 rather than queueing.
• How a Jules review is steered — Jules does read AGENTS.md, documented explicitly, root only. But its VM ships no Dart, no Flutter, no Android SDK and no Xcode, and injects env vars as process variables rather than writing .env.

Settled 2026-09-02, after the research landed:

• The destination is redrawn around the need, not the tool. Jules cannot review; the reviewer will be a different tool or no tool.
• Nothing gets installed until a hand-run review proves it is worth automating. The trial is #1035, and it gates the tool choice in #1034. Chosen over installing the leading candidate directly because the trial costs nothing, grants nobody contents: write, and answers a question no amount of tool research can: whether an AI first pass on this repo's PRs is actually useful.
• Jules for authoring — a weekly docs-drift patrol and validated dependency sweeps — was considered and not taken up now. It is a defensible use of Jules and the repo's merge history argues for it, but it belongs to a different destination. Not ruled out; simply not this effort.

Settled by the hand-run trial, #1035, closed 2026-09-02:

• AI first-pass review on outside fork PRs is worth automating. /code-review on fork PR feat: add scan and create-custom actions to no-results search state (#578) #995 caught a documented-standard violation CI structurally cannot see — three of four new strings left verbatim English across all eight non-English locales, against CONTRIBUTING.md:54, while just check_l10n only fails on missing keys. Green pipeline, real violation.
• It passed the two conventions the contributor got right, which matters more than the catches: ARB parity verified at 946 real keys per locale (correctly discounting the template-only @key metadata that would fool a naive check), and both Semantics(identifier:) wrappers correct with container: true rightly absent under a Wrap. No manufactured findings on a first-timer's PR.
• The Spec axis earns its separate existence — it found two behavioural regressions invisible to a standards pass (the Recent tab asserting "no foods matched" when no search ran; an all-provider outage reported as zero results), and 1-of-5 spec'd test states covered.
• The bar for Choose the review tool, now that Jules cannot be it #1034 is now concrete: a candidate must read AGENTS.md/CONTRIBUTING.md well enough to reason about the @key exception and the container: true gotcha, not merely grep for Semantics.

Settled by Can an OpenAI / Google / Anthropic subscription drive PR review directly?, closed 2026-09-02:

• Only a vendor-side integration can serve fork PRs, and four independent parties agree: Google's own Gemini action gates forks out (head.repo.fork == false), Anthropic labels the bypass "(RISKY)", all three third-party Jules wrappers ship skip_forks: true, and Antigravity's reviewer does the same. Ruling out pull_request_target was the industry's settled position, not caution.
• OpenAI is the only consumer subscription that buys it — Codex review on ChatGPT Plus, $20/mo, vendor-side via chatgpt-codex-connector, fork reviews proven live, reads AGENTS.md, advisory by permission (checks: read/statuses: read — it cannot block). But: no author filter at all (contradicting this map's author != simonoppowa scope), a 32 KiB instruction cap this repo sits at 90.6% of with silent truncation, and contents/workflows/actions: write — making #1032 a hard prerequisite. It is also a new subscription; the one held is Google AI Pro.
• Anthropic and Google are closed. Managed Claude Code Review is Team/Enterprise-only and never auto-reviews forks; Google AI Pro buys no PR review at all, and Gemini Code Assist for individuals is gone.
• A separate, live defect: Claude Code reads CLAUDE.md, not AGENTS.md, and this repo's stub reference is a backticked markdown link — the documented way to prevent an import. The manual /code-review fallback therefore sees none of the 442 lines unless a bare @AGENTS.md is added.

Settled by Choose the review tool, closed 2026-09-02:

• The reviewer is Codex, on a ChatGPT Plus subscription. Chosen over the free CodeRabbit because it is advisory by permission rather than by configuration — checks: read and statuses: read mean no setting and no future default can turn it into a merge gate. Vendor-side, fork reviews proven live, reads AGENTS.md on the review path.
• REVERSAL — the author != simonoppowa scope decision no longer holds. Codex has no author filter; the only author-scoped setting is the inverse. Own PRs will be reviewed too, overlapping the local /code-review. Recorded as a reversal rather than dropped quietly; the consequences move to #1028.
• #1032 is promoted to hard prerequisite. The Codex App requests contents/workflows/actions: write — the same shape that made the Jules grant unacceptable. Nothing gets installed until it lands.
• CodeRabbit is the recorded fallback, needing no further research — only a .coderabbit.yaml with base_branches covering develop, feature/.*, release/.*, hotfix/.*.

Settled by Protect the Actions secret store, closed 2026-09-03:

• Two environments, split where work leaves for a store. release-build (main only, no reviewer) holds the four ANDROID_* signing secrets for android-package. release-ship (main only, required reviewer) holds APP_STORE_CONNECT_*, MATCH_* and GOOGLE_PLAY_SERVICE_ACCOUNT_JSON for ios-package, ios-deploy and android-deploy. Net property: signed Android artifacts can be produced unattended; nothing reaches a store, and no iOS artifact is signed, without an approval a GitHub App cannot give. That is the mechanism that makes this map's original premise true rather than aspirational.
• ios-package is gated despite only building — its fastlane ios build lane calls match(readonly: true, api_key: ...), so it genuinely needs App Store Connect credentials. Provisioning a second role-scoped key was considered and declined; iOS therefore cannot build unattended.
• Scoping without a reviewer was rejected as ineffective: an actor with workflows: write can add environment: to a job it writes, so the branch policy alone closes nothing.
• SENTRY_DNS and SUPABASE_* stay repo-level deliberately — they ship inside the app binary and the anon key is public by design; protecting them would be theatre. Every PR job already uses stub values via .github/actions/write-env-file, so no PR-path job reads a real secret and scoping cannot break PR CI.
• ACCEPTED RESIDUAL RISK — the 11 orphaned secrets stay. AWS_*, CLOUDFLARE_*, R2_*, TF_* and CATALOG_ACCESS_TOKEN are referenced nowhere in .github/ and have no consumer in this repo, yet remain readable by any write-capable actor: DNS for the domain, object storage, and Terraform state. Consciously chosen over closing them, to spend the effort on the deploy path. This is the largest remaining exposure here and this map does not close it.

Settled by Codex reviews every PR including mine, closed 2026-09-03:

• Automatic reviews on for everything, trigger "On PR open" (never On every push), policy left on "Review all PRs". Measured volume over 90 days: 161 mine (80.5%) / 24 Dependabot (12%) / 15 outside (7.5%) — the 80% costs quota and ignorable noise, not effort.
• No narrowing exists that helps. Every author-policy value drops the outside PRs too or first; Review my PRs selects exactly the slice to skip and may also gate the manual mention path. Branch/label/bot filters were closed not_planned (Feature request: Limit Codex auto code review to default branch (or configurable branch allowlist) openai/codex#13597); Dependabot coverage shipped as a requested feature (#5669).
• Mention-only refuted by natural experiment: danny-avila/LibreChat runs Codex manual-only and reviews 1 of 10 outside PRs while covering 87% of the owner's. Human triggers track attention — the exact failure that left docs(readme): name every contact in What leaves your device #989 #1010 and feat: add scan and create-custom actions to no-results search state (#578) #995 unreviewed.
• The no-secret workflow trigger is dead today. Codex refuses github-actions[bot] and posts a visible junk refusal onto the contributor's PR. It worked in May 2026, so it is a vendor change worth a 10-minute retest, not a permanent no.
• Cheapest real win is upstream of Codex: group the four ungrouped Dependabot ecosystems in .github/dependabot.yml (verified: zero groups: keys), cutting the 12% at source and killing the weekly burst that silently starves concurrent reviews.
• Accepted: automatic coverage plateaus near 75% with no error and no annotation, and since the App holds checks: read a missing review is invisible to CI too.

Settled by Ship it, closed 2026-09-04:

• The repo-side work shipped: #1045 (74464cb9) and #1047 (24c25dd4). AGENTS.md at 28,140 bytes with ## Code Review Rules at byte 653, the check_agents_md guard in justfile and linux-checks, Dependabot grouped across all four ecosystems, and the 120-column claim gone repo-wide.
• The account-side work did not, and is deliberately left with the maintainer: #1032's environment checklist (verified unexecuted — main still 0 required reviews, all four environments unprotected), installing the App, the trigger settings, the @codex review backfill on docs(readme): name every contact in What leaves your device #989 #1010 and feat: add scan and create-custom actions to no-results search state (#578) #995, and the keep/tune/drop verdict itself.
• So the destination is not reached. No Codex review has run on an outside PR, and nothing has been judged. The map produced a decision trail and a prepared repository, not a working reviewer.

Corrections to what this map asserted while charting — recorded because the map was wrong, not merely imprecise:

• The Dependabot pile is not stale. All 11 open bumps were created between 2026-08-21 and 2026-08-30 (oldest 12 days). The low PR numbers chore(deps): bump flutter_local_notifications, intl, timezone and meta #770/chore(deps): bump share_plus_platform_interface, intl, file_picker, meta, package_info_plus, health and share_plus #773 read as old and are not. Zero bundler PRs are open — all 11 target develop.
• The genuinely recurring pile is docs drift, not dependencies: 15 docs-correction PRs merged between 2026-08-27 and 2026-09-01, including "correct eight claims the code no longer supports".
• The secret store is far larger than "store signing keys". 26 repo-level secrets including CLOUDFLARE_API_TOKEN, TF_VAR_ZONE_ID_OPENNUTRITRACKER_ORG, AWS_*, R2_*, TF_GITHUB_TOKEN, TF_VAR_STATE_PASSPHRASE. See #1032.
• automationMode has two values, not one — AUTOMATION_MODE_UNSPECIFIED (no branch, no PR) and AUTO_CREATE_PR — and jules-action takes five inputs, not two. Neither rescues the capability; both matter because AUTOMATION_MODE_UNSPECIFIED is the door every third-party wrapper walks through.

Not yet specified

• Whether the destination survives, and in what form. Jules cannot review. The open choice is whether this map keeps Jules repointed at authoring (docs-drift patrol, validated dependency sweeps) alongside a separate review tool, or drops Jules and becomes a review-tooling effort outright. Until that is answered the map's title and destination are provisional.
• What Jules would need to author anything here. Flutter 3.44.6 via a setup script plus Run-and-Snapshot; no macOS runner, so iOS bundler and CocoaPods bumps are unverifiable; commit attribution set to user-as-sole-author to satisfy the no-AI-trailer rule. The .env problem is already solved for free by the stub pattern in .github/actions/write-env-file/action.yml.
• Jules V2. An early-access waitlist opened 2026-04-22; jules.google.com/earlyaccess returns 200 but is sign-in gated and unreadable by any agent. Everything visible about V2 describes authoring. A genuine unknown, not a negative — and only the signed-in maintainer can settle it.
• Whether a User-owned repo can enrol in Gemini Code Assist or Claude's managed Code Review. Both say to pick "the organization that owns the repositories". A hard gate; untested.
• CodeRabbit's real OSS rate band at 2,446 stars — published only as a range (1–10 reviews/developer/hour) with no star-to-band mapping.
• The kill switch. Cannot disable Codex Security Review while keeping Code Review enabled openai/codex#38110 finds the only reliable off switch is removing the repo from the App installation; #26911 reports credits drained with Code review disabled. An "off" state cannot be trusted.
• Whether the code-review quota is separate from the ChatGPT Plus coding budget. Codex Cloud auto code review: silent fail + dashboard shows quota available but review says limit reached openai/codex#15477 reports a distinct display. The highest-value open item — if confirmed, the entire cost case behind the trigger question dissolves. Visible on the settings page to anyone signed in.
• Codex for Open Source. OpenAI runs a programme offering six months of ChatGPT Pro to maintainers of widely-used public projects, plus an API-credit fund aimed explicitly at Codex PR review. A solo-maintained AGPL app with 2,446 stars is a plausible applicant.
• AGENTS.md will breach the 32 KiB cap on its own. At 29,681 of 32,768 bytes with docs-correction PRs landing most weeks, growth alone will cross it — silently. Whether the answer is trimming, splitting root→leaf, reordering so the non-negotiables survive truncation, or a CI guard, is scoped in #1029 but the long-term shape is not settled.

Out of scope

• Jules as a PR reviewer — Connect Jules to the repository, closed. Not a rescoping: there is no PR-typed input in the Jules API at all. Every published route to a Jules-authored review needs JULES_API_KEY in a workflow reacting to fork PRs — the decision this map had already closed. All three third-party wrappers ship skip_forks: true for exactly that reason, and one records the trade-off verbatim.
• Reviewing my own PRs. Covered by /code-review locally.
• pull_request_target or any workflow holding a review tool's API key.
• Blocking / required-check enforcement. Advisory only.
• An agent pushing fix commits to contributor branches.
• The repo's general branch-protection posture. #1032 covers protecting the secret store specifically; the wider enforcement question stays where #1016 left it.