fix(eval): Linux node_modules for sandbox — dynamic eval runs e2e + 3 defects found - #24
Open
explosivebit wants to merge 1 commit into
Open
fix(eval): Linux node_modules for sandbox — dynamic eval runs e2e + 3 defects found#24explosivebit wants to merge 1 commit into
explosivebit wants to merge 1 commit into
Conversation
…2e; 3 defects found Resolves the EVID-031 Linux-portability blocker: build task-pack deps in node:22.18-alpine (musl, matches sandbox) via npm install, point the verify harness at them via FE01_NODE_MODULES (gold/node_modules untouched). vitest now executes inside the sandbox — first successful end-to-end dynamic-eval run. The e2e run surfaced 3 defects (EVID-033, all gating band-separation validity): - CorrectnessEvaluator vacuous-pass: total==0 -> 1.0 scores broken samples as perfect. - CoverageEvaluator ENOENT on read-only /workspace (needs writable coverage dir). - fe_01 calibration samples export only MultiStepForm; gold tests unit-test 4 helpers the samples omit -> automatic band separation structurally impossible as authored. EVID-033 left draft pending owner review (verdict=weakens; Defect 3 is a methodology fork). Tasks #7/#8/#9 track the defects. Refs: EVID-033, NOTE-008 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
explosivebit
added a commit
that referenced
this pull request
Jun 1, 2026
…n sandbox → patch) Design for the candidate-side executor that turns the 2026-06-02 CLI-harness feasibility spikes (7 CLIs proven headless via the LiteLLM proxy) into the production path for non-raw-llm stacks. Two-half sandbox model: Half A = run the harness CLI → patch (this RFC); Half B = run produced code → scores (#24 + evaluators, complementary, no conflict). Per-CLI proxy-recipe map, network=none + proxy-only bridge, cost metered via proxy, non-determinism handled by pass^k (ADR-013). First slice: aider × be_01 → first real "model × harness" number. Draft (not activated). Validate PASS (0 MUST errors). Refs: rfc-stack-executor-run-agent-cli-harnesses-in-the-sandbox-candidate-side-patch, rfc-005, prd-006
explosivebit
added a commit
that referenced
this pull request
Jun 2, 2026
…metrics (#33) * docs(adr): ADR-013 — publish pass@k + pass^k + flakiness reliability metrics (SWE-rebench prior art) Records the methodology decision behind the pass_k.py code (merged #29): report pass@1 + pass@k (ceiling) + pass^k (reliability) + flaky (=pass@k-pass^k) per (model, stack, task-set) over k seeds. No hard pass^k threshold at v0.1 (transparency metric, not a gate — defer). Pairs with best-of-N (PRD-009). Draft (not activated). Refs: adr-publish-reliability-metrics-pass-k-ceiling-pass-k-reliability-flakiness, prd-009, note-swe-rebench-nebius-prior-art-pass-k-borrowed-v2-dataset-ingestible-as * docs(rfc): RFC-006 — stack executor design (run agent-CLI harnesses in sandbox → patch) Design for the candidate-side executor that turns the 2026-06-02 CLI-harness feasibility spikes (7 CLIs proven headless via the LiteLLM proxy) into the production path for non-raw-llm stacks. Two-half sandbox model: Half A = run the harness CLI → patch (this RFC); Half B = run produced code → scores (#24 + evaluators, complementary, no conflict). Per-CLI proxy-recipe map, network=none + proxy-only bridge, cost metered via proxy, non-determinism handled by pass^k (ADR-013). First slice: aider × be_01 → first real "model × harness" number. Draft (not activated). Validate PASS (0 MUST errors). Refs: rfc-stack-executor-run-agent-cli-harnesses-in-the-sandbox-candidate-side-patch, rfc-005, prd-006
explosivebit
added a commit
that referenced
this pull request
Jun 2, 2026
…stale BLOCKER (health unhealthy→needs_attention) (#34) Post-merge artifact-graph cleanup (smith Row #11 tech-debt, autonomous /autorun). The authoritative state lives in LanceDB; this syncs the .md projections. - Activated 13 tier:auto EVIDs that were complete (verdict+CL, linked, R_eff>0) but stuck in draft — dynamic-eval + judge + RFC-004 workstream evidence (EVID-021/022/023/024/034/035/037/038/044/045/046) + EVID-039 (RFC-004 PASS re-gate). - EVID-036 (RFC-004 Slice-1 guardian BLOCKER) retired: activated → deprecated. Its sole precondition (parent ADR-008 draft) is long resolved — ADR-008 active, RFC-004 active + merged (#25/#26). Superseded by EVID-039 (audit trail intact). - RFC-004 phase advanced validate→done (Slice 1+2 shipped). Result: forgeplan_health unhealthy→needs_attention; orphans 1→0, phase-mismatch 1→0, auto-anomalies cleared. Remaining drafts are by-design (frozen v0.2 layer behind EPIC-002 + backlog PRDs) — NOT activated to avoid premature/fabricated proof. NOTE-002 R_eff=0 left as-is (it is the evidence-quality STANDARD, a contract Note legitimately without evidence — not debt). EVID-033 (fe_01 dynamic-eval) intentionally NOT included here — it belongs to open PR #24.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Resolves the EVID-031 Linux-portability blocker and lands the first successful end-to-end dynamic-eval run — then honestly reports the 3 defects that run uncovered.
Follow-up to PR #23. Done autonomously under
/autorun.✅ What landed
node:22.18-alpine(musl, matches the sandbox image) vianpm install(npm cifails on the macOS-generated lockfile's platform optional deps); point the verify harness at them via a newFE01_NODE_MODULESenv override.gold/node_modulesleft untouched.vitestnow executes inside the sandbox —CorrectnessEvaluatoremits real pass-rate scores instead of0.0/skipped.🔴 What the e2e run uncovered (3 defects — band separation RED)
Measured
CorrectnessEvaluatorscores: perfect 0.4348, good 0.4348, mediocre 0.3043, poor 0.0870, broken 1.0000 (inverted).correctness_evaluator.py:180) —pass_rate = 1.0 if total == 0. A non-compiling submission → vitest collects 0 tests → scores 1.0 (best) instead of 0.0. This is whybrokenscored 1.0. → Task [FR-007] Orchestrator writes immutable manifest with hashes of all inputs/outputs #7./workspace—vitest --coveragetries tomkdir /workspace/coverage, but the mount isread_only(frozen security, correct). Coverage output needs a writable tmpfs path. → Task [FR-008] Maintainer can reproduce single eval from manifest (make reproduce) #8.tests.spec.tsximports 5 symbols (MultiStepForm+ 4 helpers); the calibration samples export onlyMultiStepForm, so every helper test fails on every sample, cappingperfectat ~10/23 = 0.4348. Automatic band separation is structurally impossible as authored. Methodology fork (gold tests should test only the public contract? / samples should export the full surface? / calibration is judge-only and the "auto band-sep" premise was wrong?). → Task [FR-009] Orchestrator stores failed evals with error_class (no drop from denominator) #9, needs owner.EVID-033 is intentionally left draft (
verdict: weakens) — Defect 3 is a methodology decision that may invalidate the "perfect ≥ 0.85 / broken ≤ 0.20 via automatic evaluators" premise. Activate after you've ruled on it.Net: no task pack can be honestly promoted to
calibrationvia automatic evaluators until #7/#8/#9 are resolved. This gates PRD-006 Wave 1.Reversibility
One infra script + one draft EVID + a NOTE back-ref. No product code changed. Fully reversible.
Refs
evid-033, note-008
🤖 Generated with Claude Code