feat(arenabench): a Stop-hook oracle — deterministic probes, judge fallback, bounded

[macanderson]

What

arenabench/hooks/oracle_stop_hook.py — the verify_done successor, as an out-of-process Stop hook, plus 7 tests over its decision core.

Why here, and not as a tool or a pipeline stage

• Not a built-in tool. The built-in surface is pinned at twelve (CLAUDE.md); verify_done was deleted with the rest by feat!: reduce the built-in tool surface to 12 tools (subagents, tasks, state, environment) #3244.
• Not the old pipeline. Its witness stage cost more than it gained, and its warrant waived far more often than it authored — that is the defect this design is built against.
• The shell-hook plane is where it belongs, and Plugin platform: the substrate mostly exists across two planes that do not know about each other — design review and sequencing #3246's O3 makes the case from the code: a verification pass that runs a test suite is exactly the workload the in-process HookBus cannot host (sync closures, cannot await) and the subprocess plane can (10-minute ceiling, own process, any language).

The three rungs

1. Deterministic flip. A configured STELLA_ORACLE_TEST_COMMAND, or probes persisted from an earlier round, are run by the hook. Exit 0 allows; failure denies with the output tail. Unspoofable by prose — verify_done's posture without the shadow-worktree machinery.
2. Judge-derived probes. With no configured oracle, one judge call must return runnable probe commands, not an opinion. The probes are then run exactly like rung 1: the judge creates the oracle; running it decides. The derived probes persist across rounds, so the revision re-runs the same probe — that is the fail→pass flip, observed rather than asserted. Only a probe-less judgement falls back to prose, and an unreachable/unparseable judge allows.
3. Approach diversity. The second consecutive deny appends a step-back-and-try-a-different-approach instruction — sequential best-of-N against the 1-in-3 prior — as advice inside the deny reason rather than machinery, because MAX_STOP_CONSULTS already bounds the loop.

Why there is no warrant

The old pipeline decided up front whether a change deserved proof, and that predicate is where the waivers came from. Here nothing decides eligibility: every completion is probed, and the only "waiver" is a judge that could derive no probe — which is a stated outcome, not a silent skip.

Fail-open, everywhere

A broken oracle must never hold a correct turn hostage: no key, no judge, no parse, no network → allow. An internal error exits non-zero, which the engine surfaces as a diagnostic and never a block (user_hooks.rs's documented Stop posture).

Tests

uv run pytest tests/test_oracle_stop_hook.py -q → 7 passed; ruff check clean on both new files. Covered: both flip directions, cross-round probe persistence with the judge asserted not to be called again, the steering threshold, fail-open on an unreachable judge, probe-less verdicts in both directions, and the engine's decision vocabulary.

Depends on

#3302 (the Stop gate's bounded consultation counter). With the once-per-turn latch this hook can deny but its re-check never happens — the flip is unobservable. Land #3302 first.

Not yet wired

No match TOML sets STELLA_ORACLE_* or installs the hook into a task container yet — that is the next slice, deliberately separate so this can be reviewed on its logic. Filing the wiring issue alongside.

Refs #3234, #3246, #3244

Summary by Sourcery

Introduce a Stop-hook oracle script that verifies task completions via deterministic probes with a judge-based fallback and fail-open behavior, persisting state across rounds to support bounded, repeatable verification.

New Features:

• Add an out-of-process Stop hook oracle that runs configured or judge-derived shell probes to decide whether to allow or deny a turn completion.

Enhancements:

• Persist oracle state across rounds to reuse derived probes and track consecutive denies, adding an approach-diversification hint on repeated failures.
• Limit probe execution time and truncate returned output to keep verification responsive and worker-facing diagnostics concise.

Tests:

• Add a test suite for the oracle’s decision core covering configured commands, probe persistence and flip behavior across rounds, judge fallback semantics, fail-open paths, and adherence to the engine’s Stop-hook decision vocabulary.

[sourcery-ai]

Sorry @macanderson, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
stella-cli-docs	Ignored		Aug 14, 2026 8:34pm

[sourcery-ai]

Reviewer's Guide

Introduce a new out-of-process Stop hook oracle that deterministically runs configured or judge-derived test probes to decide allow/deny, with persistent state and fail-open behavior, plus tests covering the decision logic and engine contract.

File-Level Changes

Change	Details	Files
Add a standalone Stop-hook oracle script that executes configured or judge-derived probes, persists state across rounds, and emits allow/deny decisions with fail-open behavior.	Implement payload reading, cross-round state load/save, and probe execution with timeouts and tailed output for deny reasons Compute filtered workspace diffs and untracked file lists to provide context to the judge model Integrate with an LLM-based judge via OpenRouter, including JSON-response extraction and robust parsing/fail-open semantics Define the main decision algorithm that prioritizes configured commands, falls back to judge-derived probes or prose verdicts, and appends approach-diversity guidance on repeated denies Wire a main entrypoint that runs the decision, prints the JSON action for the engine, and surfaces internal errors via non-zero exit without blocking turns	arenabench/hooks/oracle_stop_hook.py
Add tests that import the hook script by path and exercise the decision core, including configuration, persistence, steering, fail-open, and engine vocabulary behavior.	Import the oracle hook module via importlib from its file path so tests can target the shipped script Test allow/deny behavior for passing and failing configured commands, including presence/absence of diversification note Verify that judge-derived probes persist across rounds and that a flip from failing to passing probe allows without reconsulting the judge Assert fail-open behavior when the judge is unreachable and when judge returns probe-less pass/fail verdicts Validate that emitted decisions match the engine’s expected hook vocabulary and are JSON-serializable	arenabench/tests/test_oracle_stop_hook.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.