Honest Oracle

A pre-submit reviewer that audits your pull request's claims against the actual evidence — and runs the checks itself instead of just asking for them.

A PR makes two kinds of statements: the diff (what actually changed) and the prose (the description and commit messages, claiming what it does and how it was verified). Reviewers — human and bot — reject PRs when the prose outruns the diff:

"verified end-to-end on macOS" (no logs) · "parity unchanged (21→21, byte-stable)" (the test checks one file) · "3× faster" (no benchmark) · "rename complete, no old refs remain" (one call site left behind)

The claim might even be true — but if it isn't checkable from the artifact, the reviewer can't tell honest evidence from confident assertion, and bounces it. That costs a review round-trip and chips away at your credibility as a contributor.

Honest Oracle is that reviewer, run before you submit. It's a single-folder skill — nothing in it is harness-specific, so it works in any agent that loads skills from a directory and can run shell commands (Claude Code, the Claude Agent SDK, pi, Codex, Letta, or any comparable harness).

What it does

Given your draft PR description (and commit messages) plus the diff, it:

1. Extracts every checkable claim — what was tested, performance, parity, file counts, platform/OS coverage, "verified", "unchanged", "no regressions", specific numbers.
2. Classifies each one — proven by the diff, verifiable now, or unsubstantiated.
3. Verifies the cheap, safe ones itself — it's in your repo, so it runs the typecheck, the targeted test, the grep, the file count, and reports ✓ proven / ✗ refuted with the real command output. A refuted claim means the PR would have shipped a falsehood.
4. Flags the rest with the exact command, log, or reword that would settle each.
5. Offers a rewritten description so every line is provable or correctly scoped.

It reviews claims, not code quality — for bug hunting use a dedicated code reviewer. Staying in that lane is what makes the verdict trustworthy.

Why it's different from CodeRabbit / PR-Agent / Sage

Those run after you open the PR and can only ask you to provide evidence. Honest Oracle runs before, locally, where it can produce the evidence — verifying or refuting claims against the working tree instead of flagging them for someone else to check.

Install

It's a single-folder skill. Drop it wherever your agent harness looks for skills:

git clone https://github.com/klittle32/honest-oracle <your-skills-dir>/honest-oracle

For example, Claude Code and the Claude Agent SDK look in ~/.claude/skills/:

git clone https://github.com/klittle32/honest-oracle ~/.claude/skills/honest-oracle

That's it — SKILL.md is at the repo root, so the skill is immediately available. It triggers on phrases like "review my PR", "check my PR description", "any overclaims?", "is this PR ready?", or just "honest oracle" — and proactively before gh pr create / gh pr edit.

Prefer not to clone the eval suite into your skills dir? Copy just SKILL.md into <your-skills-dir>/honest-oracle/SKILL.md.

Use it

> I'm about to open this PR. Run honest-oracle on it first.

It produces a verdict — SHIP, TIGHTEN (reword to match the evidence), or VERIFY (a claim is refuted or load-bearing-and-unbacked) — with a claim-by-claim table and an offered rewrite.

A real refutation it caught (Python fixture — full output in evals/sample-outputs/):

## Honest Oracle — VERIFY first (do not ship)

Ran: python test_stats.py  →  TypeError: unhashable type: 'list'  (exit 1)
     A/B vs main (temp dir)  →  ok

| Claim                                            | Verdict      | Evidence / fix |
| "All tests pass on Python 3.8–3.12"              | ✗ refuted    | lru_cache can't hash the list arg → existing test throws |
| "Fully backward compatible — callers unaffected" | ✗ refuted    | main → ok; branch → TypeError |
| "3× faster on repeated calls (benchmarked)"      | unsubstantiated | no benchmark in the diff/repo |
| "Added lru_cache memoization to compute_stats"   | proven-by-diff | stats.py:1-3 |

The diff alone looks fine — one line adding a cache. Only running the existing test reveals the break, in a file the diff never touched.

How it works

The full method is in SKILL.md. The core is a verify-vs-flag gate: run a check yourself only if it's read-only/temp-scoped, fast, side-effect-free, and meaningful — otherwise name the command and flag it. It's stack-agnostic; the example commands (gh, git, typecheck) are illustrative — it substitutes npm/pytest/cargo/go test, GitLab/Bitbucket, etc. And it never reports a result for a command it didn't actually run: fabricating evidence is the exact sin it exists to catch.

Validation

This isn't a vibe — it was built and refined against an eval suite spanning three stacks (TypeScript, JavaScript, Python) and both overclaim-recall and false-positive-precision. The cases, assertions, fixtures, and sample outputs are in evals/, reproducible with evals/build_fixtures.sh + any eval harness (it was developed with Anthropic's skill-creator).

Honest caveat, in the spirit of the skill: a capable agent already catches blatant problems on its own. Honest Oracle's lift is the disciplined claim-by-claim pass that catches the subtle overclaims — unmeasured numbers, a narrow test sold as a broad contract, an unscoped platform claim — plus the rewrite that makes the description match the evidence.

Acknowledgements

Inspired by the metafactory — their reviewer bot caught the overclaims in my very first PR's description, and that feedback is the whole reason this skill exists. Thanks for the nudge.

License

MIT. Built with Claude Code. Contributions and new eval cases welcome.