feat(self-eval): enforce the gate on config changes + worker-driven run (Phase A wiring) - #66
Merged
Merged
Conversation
The AO config-change endpoints (PUT /ao/role-config/{role}, PUT /ao/settings)
now consult the Self-Eval Gate before applying a change:
* self_eval_enforce setting (default OFF) gates the whole path — an unconfigured
instance is never affected, and the gate is additionally a no-op on cold start
(no baseline / no private suite).
* On a regression the change is refused with 409, the block is written to the
tamper-evident audit log ("ao.config.self_eval_blocked"), and the mutation is
never applied.
* ?force=true lets an admin override; the override is audited
("ao.config.self_eval_forced") and commits atomically with the change.
* The gate is injected via Depends(get_self_eval_gate) — real baseline lookup
(SelfEvalService) + an eval runner. The default runner is a no-op because a
live private-suite eval is minutes-long, agent-driven, and worker-owned
(apps/api cannot import forge_worker without a cycle); establishing a baseline
is the worker path (A4). The gate MECHANISM is fully wired and tested by
injecting a runner.
Tests: regression blocked + audited + not-applied, passing allowed, force
overrides + audits, flag-off and cold-start no-op, settings endpoint gated too.
Existing AO router tests still green after the sync->async endpoint change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Un-parks the internal agent-driven benchmark run for the Self-Eval Gate — as a worker task, not an inline endpoint, since a private-suite run is minutes-long and agent-driven: * execute_self_eval_run — the run -> record loop: drives the (injected) live runner and, on a real score, persists it as the workspace baseline via an injected record_baseline. Returns None (records nothing) when the runner has nothing to score, so a failed/offline run never writes a bogus baseline. * forge.self_eval.run Celery task — wires the real ProductionEvalRunner (A3) + SelfEvalService.record_baseline (A2), resolving the workspace's published private suite + its on-disk case dir + a local repo clone. An honest no-op (scored: false, with the missing-prereq reason) until an operator provisions FORGE_BENCHMARK_DIR + FORGE_SELF_EVAL_REPO_ROOT/<repo_id> + a BYOK model — the foundation ships no repo-clone manager, and nothing is faked. * Updated the benchmarks router docstring: the parked "no EvalRunner substrate" rationale is gone (A3 provides it); the public F35 router stays the external submit/verify/rank path for shared community suites. Tests: execute_self_eval_run records a baseline (fake + real in-memory SelfEvalService), records nothing on a None score; the task registers under forge.self_eval.run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ed branches The forge_worker coverage floor (>=90%) tripped because the A4 Celery task file was 0%-covered. Add a test exercising the honest no-op path (no provisioning env -> scored:false) and isolate the genuinely infra-gated logic (DB suite resolve, BYOK client, live run) into # pragma: no cover helpers that need a real repo clone + provider to run. Worker coverage back to 91%. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Self-Eval Gate enforcement — wiring (A1 + A4)
Builds on the substrate merged in #65 (baseline persistence + live
ProductionEvalRunner). This makes the Self-Eval Gate an actually-enforced gate.A1 — enforce the gate on AO config changes
PUT /ao/role-config/{role}andPUT /ao/settingsnow consult the Self-Eval Gate before applying a change:self_eval_enforcesetting (default OFF) gates the whole path; also a no-op on cold start (no baseline / no private suite).ao.config.self_eval_blocked), and the mutation is never applied.?force=truelets an admin override; audited (ao.config.self_eval_forced), commits atomically with the change.Depends(get_self_eval_gate)— real baseline lookup + an eval runner. The default runner is a no-op (a live private-suite eval is minutes-long, agent-driven, worker-owned, andapps/apican't importforge_workerwithout a cycle). The gate mechanism is fully wired and tested by injecting a runner.A4 — worker-driven self-eval run + baseline recording
execute_self_eval_run— the run → record loop; drives the live runner and persists the score as the workspace baseline, recording nothing when there's no score (never a bogus baseline).forge.self_eval.runCelery task — wires the realProductionEvalRunner+SelfEvalService.record_baseline, resolving the published private suite + on-disk case dir + local repo clone. An honest no-op (scored: false+ missing-prereq reason) until an operator provisionsFORGE_BENCHMARK_DIR+FORGE_SELF_EVAL_REPO_ROOT/<repo_id>+ a BYOK model — the foundation ships no repo-clone manager, and nothing is faked.POST /runsdocstring is updated: the "no EvalRunner substrate" rationale is gone (the substrate now exists); the public F35 router remains the external submit/verify/rank path for shared community suites.Verification
execute_self_eval_runrecords a baseline (fake + real in-memorySelfEvalService), records nothing on a None score; the task registers underforge.self_eval.run.make typecheck— 587 files, no issues.Honest status
The gate mechanism, enforcement, audit, force override, run task, and baseline loop are all real and tested. The one remaining gap to a fully-live default deployment is a repo-clone manager (a local checkout of a workspace's connected repo for the sandboxed run) — deliberately not faked; until it exists, live runs no-op and the gate stays off by default.
🤖 Generated with Claude Code