Skip to content

feat(self-eval): enforce the gate on config changes + worker-driven run (Phase A wiring) - #66

Merged
QuintinBotes merged 3 commits into
mainfrom
feature-self-eval-wiring
Jul 14, 2026
Merged

feat(self-eval): enforce the gate on config changes + worker-driven run (Phase A wiring)#66
QuintinBotes merged 3 commits into
mainfrom
feature-self-eval-wiring

Conversation

@QuintinBotes

Copy link
Copy Markdown
Owner

Self-Eval Gate enforcement — wiring (A1 + A4)

Builds on the substrate merged in #65 (baseline persistence + live ProductionEvalRunner). This makes the Self-Eval Gate an actually-enforced gate.

A1 — enforce the gate on AO config changes

PUT /ao/role-config/{role} and PUT /ao/settings now consult the Self-Eval Gate before applying a change:

  • self_eval_enforce setting (default OFF) gates the whole path; also a no-op on cold start (no baseline / no private suite).
  • Regression → 409, the block is written to the tamper-evident audit log (ao.config.self_eval_blocked), and the mutation is never applied.
  • ?force=true lets an admin override; audited (ao.config.self_eval_forced), commits atomically with the change.
  • Gate injected via Depends(get_self_eval_gate) — real baseline lookup + an eval runner. The default runner is a no-op (a live private-suite eval is minutes-long, agent-driven, worker-owned, and apps/api can't import forge_worker without a cycle). The gate mechanism is fully wired and tested by injecting a runner.

A4 — worker-driven self-eval run + baseline recording

  • execute_self_eval_run — the run → record loop; drives the live runner and persists the score as the workspace baseline, recording nothing when there's no score (never a bogus baseline).
  • forge.self_eval.run Celery task — wires the real ProductionEvalRunner + SelfEvalService.record_baseline, resolving the published private suite + on-disk case dir + local repo clone. An honest no-op (scored: false + missing-prereq reason) until an operator provisions FORGE_BENCHMARK_DIR + FORGE_SELF_EVAL_REPO_ROOT/<repo_id> + a BYOK model — the foundation ships no repo-clone manager, and nothing is faked.
  • The parked POST /runs docstring is updated: the "no EvalRunner substrate" rationale is gone (the substrate now exists); the public F35 router remains the external submit/verify/rank path for shared community suites.

Verification

  • A1: 6 tests (regression blocked + audited + not-applied, passing allowed, force overrides + audits, flag-off + cold-start no-op, settings gated too). Existing AO router tests still green after the sync→async endpoint change (15 passed).
  • A4: execute_self_eval_run records a baseline (fake + real in-memory SelfEvalService), records nothing on a None score; the task registers under forge.self_eval.run.
  • make typecheck — 587 files, no issues.

Honest status

The gate mechanism, enforcement, audit, force override, run task, and baseline loop are all real and tested. The one remaining gap to a fully-live default deployment is a repo-clone manager (a local checkout of a workspace's connected repo for the sandboxed run) — deliberately not faked; until it exists, live runs no-op and the gate stays off by default.

🤖 Generated with Claude Code

Forge Swarm and others added 3 commits July 14, 2026 08:31
The AO config-change endpoints (PUT /ao/role-config/{role}, PUT /ao/settings)
now consult the Self-Eval Gate before applying a change:

* self_eval_enforce setting (default OFF) gates the whole path — an unconfigured
  instance is never affected, and the gate is additionally a no-op on cold start
  (no baseline / no private suite).
* On a regression the change is refused with 409, the block is written to the
  tamper-evident audit log ("ao.config.self_eval_blocked"), and the mutation is
  never applied.
* ?force=true lets an admin override; the override is audited
  ("ao.config.self_eval_forced") and commits atomically with the change.
* The gate is injected via Depends(get_self_eval_gate) — real baseline lookup
  (SelfEvalService) + an eval runner. The default runner is a no-op because a
  live private-suite eval is minutes-long, agent-driven, and worker-owned
  (apps/api cannot import forge_worker without a cycle); establishing a baseline
  is the worker path (A4). The gate MECHANISM is fully wired and tested by
  injecting a runner.

Tests: regression blocked + audited + not-applied, passing allowed, force
overrides + audits, flag-off and cold-start no-op, settings endpoint gated too.
Existing AO router tests still green after the sync->async endpoint change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Un-parks the internal agent-driven benchmark run for the Self-Eval Gate — as a
worker task, not an inline endpoint, since a private-suite run is minutes-long
and agent-driven:

* execute_self_eval_run — the run -> record loop: drives the (injected) live
  runner and, on a real score, persists it as the workspace baseline via an
  injected record_baseline. Returns None (records nothing) when the runner has
  nothing to score, so a failed/offline run never writes a bogus baseline.
* forge.self_eval.run Celery task — wires the real ProductionEvalRunner (A3) +
  SelfEvalService.record_baseline (A2), resolving the workspace's published
  private suite + its on-disk case dir + a local repo clone. An honest no-op
  (scored: false, with the missing-prereq reason) until an operator provisions
  FORGE_BENCHMARK_DIR + FORGE_SELF_EVAL_REPO_ROOT/<repo_id> + a BYOK model —
  the foundation ships no repo-clone manager, and nothing is faked.
* Updated the benchmarks router docstring: the parked "no EvalRunner substrate"
  rationale is gone (A3 provides it); the public F35 router stays the external
  submit/verify/rank path for shared community suites.

Tests: execute_self_eval_run records a baseline (fake + real in-memory
SelfEvalService), records nothing on a None score; the task registers under
forge.self_eval.run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ed branches

The forge_worker coverage floor (>=90%) tripped because the A4 Celery task file
was 0%-covered. Add a test exercising the honest no-op path (no provisioning env
-> scored:false) and isolate the genuinely infra-gated logic (DB suite resolve,
BYOK client, live run) into # pragma: no cover helpers that need a real repo
clone + provider to run. Worker coverage back to 91%.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@QuintinBotes
QuintinBotes merged commit 24832c7 into main Jul 14, 2026
19 checks passed
@QuintinBotes
QuintinBotes deleted the feature-self-eval-wiring branch July 14, 2026 07:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant