feat: Self-Eval Gate — block config changes that regress a workspace's private per-repo suite (trust-layer 4/4) - #64
Merged
Conversation
…lds (migration 0039)
…s/pass-to-pass) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…o_pass_rate metric Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n rate + baseline check) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ivate-suite resolution rate Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…(slice 6/6) A workspace's Self-Eval suite is minted from its own private repo and is published so the workspace can rank against its recorded baseline — but it must never leak to the unauthenticated /public/* API. Enforce that at every public read path (defence in depth): * BenchmarkService.list_suites(public_only=True) — filters to global, non-private suites only; wired into GET /public/benchmarks. * BenchmarkService.leaderboard(public_only=True) — 404s a published-but-private suite as if it did not exist. * BenchmarkService.public_submission — same guard on the submission-detail / bundles routes, closing the back-door leak. * BenchmarkService.is_public_suite(suite) — the single predicate all three share. Tests: parametrized over the `private` flag and `workspace_id` scoping, asserting a published private suite is invisible via listing, leaderboard, and submission detail. Renamed the colliding tests/sweval/test_runner.py -> test_swe_runner.py (basename clash with tests/test_runner.py broke collection). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…void basename clash CI runs 'uv run pytest -q' over the whole repo with the default (prepend) import mode, where two flat test files sharing a basename import as the same top-level module. sweval/test_gate.py collided with deploy-core/tests/test_gate.py, breaking collection (exit 2) for both the main pytest job and the realeval job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pip-audit flags click 8.1.8 for PYSEC-2026-2132 (command injection in click.edit(), CVSS 8.8). Forge imports click nowhere, so the vulnerable path is unreachable, and click<8.2 is transitively pinned by the semgrep dev/CI dependency — click>=8.3.3 cannot be resolved without dropping semgrep. Add a justified, documented --ignore-vuln to both pip-audit invocations (VEX 'vulnerable_code_not_present'); revisit and remove once semgrep lifts the cap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Self-Eval Gate (trust-layer differentiator 4 of 4)
The fourth and final trust-layer differentiator. A workspace mints a private, per-repo benchmark suite from its own merged PRs (hidden fail-to-pass / pass-to-pass tests), records a baseline resolution rate, and then any model/prompt/router config change is refused if it regresses that rate — measured against ground-truth tests the model never sees.
No incumbent gates a config change on the customer's own private task distribution. Combined with Attested Changesets, Time-Travel Runs, and the Red-Team Gate, this closes the trust layer: provenance → replay → adversarial pre-merge check → regression-proof config.
What ships (complete + tested offline)
BenchmarkSuitegains nullableworkspace_id/repo_id+ aprivateflag (migration0039, chained from0038_red_team_gate). Hidden-test fields (fail_to_pass,pass_to_pass,sandbox_image,setup_commands,base_commit) modelled inswe_case.py.forge_eval/mint/pr_miner.py+self_eval_mintworker task) — derives cases from merged PRs via read-only GitHub API helpers (list_pr_files,pr_base_commit,pr_head_commit).forge_eval/sweval/runner.py) — applies a candidate patch in theSandboxSessionexecution seam, runs the hidden tests, scores resolution. Thesolve_fnis handed a redacted case (hidden tests stripped) so the model can never see the answer. Path-traversal-safe patch application. Newagent.fail_to_pass_ratemetric.self_eval.py,gate.py) —SelfEvalScorecard+SelfEvalGate.check_config(...)raisingSelfEvalRegressionErroron regression; no-op on cold start (no baseline / no private suite) and on explicitforce=override, so existing config flows stay green until a suite exists./public/*API at all three read paths (listing, leaderboard, submission-detail), even when published.Honest scope (park-don't-fake)
The eval engine (scoping · mint · runner · aggregation · gate · privacy) is complete and tested offline. The live agent-backed
eval_runner, baseline persistence, and thePOST /benchmarks/{...}/runsendpoint are wired through injected seams and remain parked for the follow-up integration PR — nothing is faked to look done.Verification (local, all green)
make typecheck— 583 files, no issues🤖 Generated with Claude Code