diff --git a/.forgeplan/evidence/EVID-049-rfc-006-stackexecutor-live-first-real-harness-model-patch-judged-score-aider-qwen-be-01.md b/.forgeplan/evidence/EVID-049-rfc-006-stackexecutor-live-first-real-harness-model-patch-judged-score-aider-qwen-be-01.md new file mode 100644 index 0000000..ee2c828 --- /dev/null +++ b/.forgeplan/evidence/EVID-049-rfc-006-stackexecutor-live-first-real-harness-model-patch-judged-score-aider-qwen-be-01.md @@ -0,0 +1,62 @@ +--- +depth: standard +id: EVID-049 +kind: evidence +last_modified_at: 2026-06-02T18:31:17.801906+00:00 +last_modified_by: claude-code/2.1.156 +links: +- target: RFC-006 + relation: informs +status: active +title: 'RFC-006 StackExecutor live: first real harness×model patch + judged score (aider×qwen×be_01)' +--- + +## Summary + +RFC-006 StackExecutor (Half A: harness → patch) is implemented and validated +end-to-end on real money. The candidate pipeline produces a real +`harness × model × task` patch with metered cost, and the Half A→B bridge feeds +it to the judge panel for a real score. + +## Structured Fields + +- **verdict**: PASS +- **congruence_level**: CL3 (same context — this is RFC-006's own implementation, verified directly against its acceptance criteria) +- **evidence_type**: live_integration_run + +## What was verified (2026-06-02) + +1. **Network decision A (bastion) — proven, $0.** A container on the Docker + `internal` net `pollmevals-sandbox` reaches the LiteLLM proxy (HTTP 200) but + NOT the internet (DNS resolution fails). No un-metered egress; `cap_drop ALL` + holds (no NET_ADMIN needed). +2. **Half A plumbing — proven, $0.** `DockerHarnessLauncher` runs a no-model + command in the sandbox, writes to the writable `/workspace` bind, and + host-side git captures the diff (`--plumbing` check). Surfaced + fixed a + `DOCKER_HOST` discovery bug before any spend (cheap-signal-before-dear). +3. **First real patch — aider × qwen-3-14b × be_01.** `status=ok`, a 208-line + real Express JWT auth middleware written to `solution.ts`, `cost=$0.000626` + (metered via the proxy), in 1400 / out 2200 tokens, ~55s. +4. **First scored number — same stack, judged.** Half A→B bridge → judge panel + (inversion-free; the be_01 deterministic evaluators invert per EVID-027): + claude-sonnet 5.27 · gemini-3-flash 7.33 · gpt-5-mini 0.00\* · total $0.070. + Panel median 5.27/10; trustworthy 2-judge signal ≈ 6.3. + +## Defect surfaced + +\* **gpt-5-mini judge truncates** its rubric JSON at the 2048 `max_tokens` cap on +the be_01 7-criterion coding rubric → JSON parse-fail → 0.0 fallback, which +drags the panel median + Krippendorff α (α went negative). Fix is +`reasoning_effort` (cap reasoning tokens), NOT raising the cap — EVID-023 bounds +the cap by the OpenRouter HTTP-402 pre-reservation hazard. Tracked as judge +follow-up; does not block the executor. + +## Artifacts + +- Code: `apps/eval-core-py/src/orchestrator/stack_executor.py`, `stack_scoring.py` +- Image: `pollmevals-harness-aider:0.1.0` (aider-chat 0.86.2) +- Smokes: `scripts/stack_exec_live_smoke.py`, `scripts/stack_score_live_smoke.py` +- Runbook: `docs/04-runbook/14-stack-executor.md` +- Branches: `feat/stack-executor-rfc006` (Phases 1-3, PR #39), `feat/stack-scoring-rfc006` (Phase 4a) + + diff --git a/.forgeplan/rfcs/RFC-006-stack-executor-run-agent-cli-harnesses-in-the-sandbox-candidate-side-patch-trace-metered-cost.md b/.forgeplan/rfcs/RFC-006-stack-executor-run-agent-cli-harnesses-in-the-sandbox-candidate-side-patch-trace-metered-cost.md index 51836e4..5591e1a 100644 --- a/.forgeplan/rfcs/RFC-006-stack-executor-run-agent-cli-harnesses-in-the-sandbox-candidate-side-patch-trace-metered-cost.md +++ b/.forgeplan/rfcs/RFC-006-stack-executor-run-agent-cli-harnesses-in-the-sandbox-candidate-side-patch-trace-metered-cost.md @@ -119,3 +119,4 @@ real "model × harness" number**. Then widen to codex/opencode + more tasks. + diff --git a/apps/eval-core-py/scripts/stack_score_live_smoke.py b/apps/eval-core-py/scripts/stack_score_live_smoke.py new file mode 100644 index 0000000..cffc8d2 --- /dev/null +++ b/apps/eval-core-py/scripts/stack_score_live_smoke.py @@ -0,0 +1,164 @@ +#!/usr/bin/env python +"""Live: aider x qwen-3-14b x be_01 -> patch -> JUDGE PANEL -> FIRST scored number. + +RFC-006 Phase 4. Chains Half A (StackExecutor) + the Half A->B bridge +(stack_scoring) + the operational judge panel (#28). Judges are used (not the +be_01 deterministic evaluators, which invert — EVID-027); judged-subjective +scoring is also POLLMEVALS' edge. + +Cost: ~$0.0006 (aider on qwen) + ~$0.05 (3 judges). Gated by --confirm-spend. + +Prereqs: make stack-up && make sandbox-net-up && make harness-image-aider. +Run: + uv run --project apps/eval-core-py python \ + apps/eval-core-py/scripts/stack_score_live_smoke.py --confirm-spend +""" + +from __future__ import annotations + +import argparse +import asyncio +import os +import shutil +import statistics +import sys +import tempfile +from datetime import UTC, datetime +from decimal import Decimal +from pathlib import Path + +import yaml + +REPO = Path(__file__).resolve().parents[3] +sys.path.insert(0, str(REPO / "apps" / "eval-core-py")) + +from src.orchestrator.cost import PricingTuple # noqa: E402 +from src.orchestrator.judge_panel import JudgePanel # noqa: E402 +from src.orchestrator.stack_executor import ( # noqa: E402 + DockerHarnessLauncher, + ExecStatus, + StackAdapter, + StackExecRequest, + StackExecutor, +) +from src.orchestrator.stack_scoring import ( # noqa: E402 + cost_with_judges, + exec_result_to_eval_result, +) + +_CANDIDATE = "openrouter/qwen/qwen-3-14b" +_TASK = "be_01_jwt_auth" +_JUDGES = ["claude-sonnet-4-6-judge", "gpt-5-mini-judge", "gemini-3-flash"] +_QWEN_PRICING = PricingTuple( + model_id=_CANDIDATE, + input_per_mtoken_usd=Decimal("0.07"), + output_per_mtoken_usd=Decimal("0.24"), + snapshot_at=datetime(2026, 6, 2, tzinfo=UTC), +) + + +def _load_env(repo: Path) -> None: + envf = repo / ".env" + if not envf.exists(): + return + for line in envf.read_text(encoding="utf-8").splitlines(): + s = line.strip() + if s and not s.startswith("#") and "=" in s: + k, _, v = s.partition("=") + os.environ.setdefault(k.strip(), v.strip().strip('"').strip("'")) + + +def _seed_candidate_snapshot(dst: Path, pack: Path) -> None: + """be_01 CANDIDATE workspace: pinned deps only — NO gold, NO tests.""" + for f in ("package.json", "tsconfig.json"): + shutil.copy(pack / "gold" / f, dst / f) + (dst / "solution.ts").write_text( + "// Implement the Express JWT auth middleware here (see the task prompt).\n", + encoding="utf-8", + ) + + +async def _main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--confirm-spend", action="store_true", help="real $ (aider + judges)") + args = ap.parse_args() + + # JudgePanel resolves rubric.yaml relative to cwd; anchor at the repo root. + os.chdir(REPO) + _load_env(REPO) + master_key = os.environ.get("LITELLM_MASTER_KEY", "") + if not master_key: + print("ERROR: LITELLM_MASTER_KEY not set (.env)", file=sys.stderr) + return 2 + if not args.confirm_spend: + print("DRY: pass --confirm-spend to run aider + 3 judges for real (~$0.05).") + return 0 + + pack = REPO / "evals" / "task-packs" / _TASK + prompt = str(yaml.safe_load((pack / "task.yaml").read_text())["prompt_template"]) + adapter = StackAdapter.from_yaml_path(REPO / "stacks" / "aider" / "stack.yaml") + + snapshot = Path(tempfile.mkdtemp(prefix="pollmevals-score-")) + _seed_candidate_snapshot(snapshot, pack) + artifacts = snapshot / "artifacts" + + # --- Half A: harness -> patch --- + print(f"[1/3] Half A: aider x qwen x {_TASK} ...") + executor = StackExecutor( + launcher=DockerHarnessLauncher(), + api_key=master_key, + pricing_snapshot={_CANDIDATE: _QWEN_PRICING}, + ) + request = StackExecRequest( + eval_id="score-aider-qwen-be01", + model_id=_CANDIDATE, + model_alias="qwen-3-14b", + stack=adapter, + task_id=_TASK, + task_prompt=prompt, + repo_snapshot_dir=snapshot, + seed=1, + timeout_s=600, + ) + exec_result = await executor.execute(request) + print( + f" status={exec_result.status} cost=${exec_result.cost_usd} " + f"wall={exec_result.wall_ms}ms" + ) + if exec_result.status is not ExecStatus.OK: + print(f" executor did not produce a patch: {exec_result.error_detail}") + return 1 + + # --- Bridge: Half A -> Half B --- + print("[2/3] bridge: StackExecResult -> EvalResult (write submission artifact)") + eval_result = exec_result_to_eval_result(exec_result, log_dir=artifacts) + + # --- Half B: judge panel (inversion-free) --- + print(f"[3/3] Half B: {len(_JUDGES)} judges score the patch via the be_01 rubric ...") + panel = JudgePanel( + judge_models=_JUDGES, + candidate_model_id=_CANDIDATE, + rubric_version="1.0", + ) + judgments = await panel.score(eval_result, _TASK) + agg = panel.aggregate(judgments) + + judge_cost = sum((j.cost_usd for j in judgments), Decimal("0")) + total = cost_with_judges(exec_result.cost_usd, judge_cost) + + print("\n=== FIRST SCORED (model x harness x task) NUMBER ===") + print(f"stack: aider x qwen-3-14b task: {_TASK}") + for j in judgments: + print(f" judge {j.judge_model_id:<24} total={j.total_score:5.2f} cost=${j.cost_usd}") + panel_median = statistics.median([j.total_score for j in judgments]) if judgments else 0.0 + print(f"panel median: {panel_median:.2f} / 10 (per-criterion: {agg.median_per_criterion})") + print( + f"alpha: point={agg.alpha_point} ci_lower={agg.alpha_ci_lower} " + f"status={agg.judge_status}" + ) + print(f"cost: harness=${exec_result.cost_usd} + judges=${judge_cost} = ${total}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(asyncio.run(_main())) diff --git a/apps/eval-core-py/src/orchestrator/stack_scoring.py b/apps/eval-core-py/src/orchestrator/stack_scoring.py new file mode 100644 index 0000000..2f7aff3 --- /dev/null +++ b/apps/eval-core-py/src/orchestrator/stack_scoring.py @@ -0,0 +1,160 @@ +"""Bridge Half A (StackExecResult patch) -> Half B (judge panel / evaluators). + +RFC-006 Phase 4. Maps a ``StackExecResult`` (harness -> patch, Half A) into the +``EvalResult`` shape that ``JudgePanel.score`` (and the evaluators) consume, so a +``(model x harness x task)`` run yields a real score. GridRunner dispatch-by-stack +reuses this so CLI stacks flow through the SAME scoring path as ``raw-llm``. + +The judge panel reads the candidate's output from the ``raw_output`` artifact +URI (a file on disk), so the bridge writes the submission there. + +Submission = the FINAL content of the candidate's changed source files (not the +raw diff), with harness bookkeeping (``.aider*``, ``.gitignore``, +``.pollmevals*``) filtered out — code-quality judging wants the code, not a diff. +""" + +from __future__ import annotations + +import hashlib +import re +from datetime import UTC, datetime +from decimal import Decimal +from pathlib import Path + +from src.contracts import ( + ArtifactRef, + EvalArtifactRefs, + EvalRow, + EvalStats, + EvalStatus, +) +from src.orchestrator.eval_caller import EvalRequest, EvalResult, compute_eval_id +from src.orchestrator.stack_executor import ExecStatus, StackExecResult + +# Harness bookkeeping files that are NOT part of the candidate submission. +_NOISE_RE = re.compile(r"(^|/)(\.aider|\.gitignore|\.pollmevals|node_modules/)") + +# Synthetic run_hash fragment for standalone (non-grid) scoring. GridRunner +# passes the real run_hash; this default keeps single-eval scoring deterministic. +_STANDALONE_RUN_HASH = "sha256:" + "5" * 64 + + +def changed_files(patch: str) -> list[str]: + """Return the candidate's changed source paths from a unified diff. + + Reads ``+++ b/`` headers; drops ``/dev/null`` (pure deletions) and + harness bookkeeping noise. + """ + out: list[str] = [] + for line in patch.splitlines(): + if line.startswith("+++ b/"): + path = line[len("+++ b/") :].strip() + if path and path != "/dev/null" and not _NOISE_RE.search(path): + out.append(path) + return out + + +def extract_submission(snapshot_dir: Path, patch: str) -> str: + """Concatenated final content of the candidate's changed source files. + + Each file is prefixed with a ``// === ===`` banner so a multi-file + submission stays legible to the judge. Files referenced by the patch but + missing on disk (e.g. deletions) are skipped. + """ + parts: list[str] = [] + for rel in changed_files(patch): + f = snapshot_dir / rel + if f.exists(): + body = f.read_text(encoding="utf-8", errors="replace") + parts.append(f"// === {rel} ===\n{body}") + return "\n\n".join(parts) + + +def _artifact(log_dir: Path, eval_id: str, label: str, content: str) -> ArtifactRef: + """Write *content* to a content-addressed file and return its ArtifactRef.""" + sha256 = hashlib.sha256(content.encode()).hexdigest() + mime = "application/json" if label == "evaluator_json" else "text/plain" + ext = ".json" if label == "evaluator_json" else ".txt" + dest = log_dir / eval_id / f"{label}-{sha256}{ext}" + dest.parent.mkdir(parents=True, exist_ok=True) + dest.write_text(content, encoding="utf-8") + return ArtifactRef( + sha256=sha256, + size_bytes=len(content.encode()), + uri=f"file://{dest}", + mime_type=mime, + ) + + +def exec_result_to_eval_result( + exec_result: StackExecResult, + *, + log_dir: Path, + run_hash: str = _STANDALONE_RUN_HASH, +) -> EvalResult: + """Map a StackExecResult into a judge-ready EvalResult. + + The candidate submission (changed source files) is written as the + ``raw_output`` + ``normalized_output`` artifacts so ``JudgePanel.score`` can + read it. ``stack_id`` is the adapter slug; cost/tokens carry over from Half A. + + Raises: + ValueError: the executor did not produce a patch (status != OK) — there + is nothing to score. + """ + req = exec_result.request + if exec_result.status is not ExecStatus.OK or not exec_result.patch: + raise ValueError( + f"cannot score a non-OK execution (status={exec_result.status}, eval_id={req.eval_id})" + ) + + stack_id = req.stack.slug + eval_id = compute_eval_id(run_hash, req.model_id, stack_id, req.task_id, req.seed) + submission = extract_submission(req.repo_snapshot_dir, exec_result.patch) + evaluator_json = ( + f'{{"eval_id":"{eval_id}","harness":"{req.stack.agent_cli}",' + f'"patch_bytes":{len(exec_result.patch.encode())}}}' + ) + + artifact_refs = EvalArtifactRefs( + raw_output=_artifact(log_dir, eval_id, "raw_output", submission), + normalized_output=_artifact(log_dir, eval_id, "normalized_output", submission.strip()), + evaluator_json=_artifact(log_dir, eval_id, "evaluator_json", evaluator_json), + ) + stats = EvalStats( + input_tokens=exec_result.input_tokens, + output_tokens=exec_result.output_tokens, + wall_clock_ms=exec_result.wall_ms, + cost_usd=exec_result.cost_usd, + ) + row = EvalRow( + eval_id=eval_id, + model_id=req.model_id, + stack_id=stack_id, + task_id=req.task_id, + seed=req.seed, + status=EvalStatus.SCORED, + artifact_refs=artifact_refs, + stats=stats, + started_at=exec_result.started_at, + completed_at=exec_result.completed_at, + ) + request = EvalRequest( + eval_id=eval_id, + model_id=req.model_id, + stack_id=stack_id, + task_id=req.task_id, + seed=req.seed, + ) + return EvalResult( + request=request, + eval_row=row, + exception=None, + started_at=exec_result.started_at, + completed_at=datetime.now(UTC), + ) + + +def cost_with_judges(exec_cost: Decimal, judgments_cost: Decimal) -> Decimal: + """Total eval cost = harness spend + judge spend (CONTEXT.md cost rule).""" + return exec_cost + judgments_cost diff --git a/apps/eval-core-py/tests/test_stack_scoring.py b/apps/eval-core-py/tests/test_stack_scoring.py new file mode 100644 index 0000000..39279f4 --- /dev/null +++ b/apps/eval-core-py/tests/test_stack_scoring.py @@ -0,0 +1,132 @@ +"""Unit tests for the Half A -> Half B scoring bridge (RFC-006 Phase 4). + +Pure mapping logic — no Docker, no judges, no network. +""" + +from __future__ import annotations + +from datetime import UTC, datetime +from decimal import Decimal +from pathlib import Path + +import pytest + +from src.contracts import EvalStatus +from src.orchestrator.stack_executor import ( + ExecStatus, + StackAdapter, + StackExecRequest, + StackExecResult, +) +from src.orchestrator.stack_scoring import ( + changed_files, + exec_result_to_eval_result, + extract_submission, +) + +_ADAPTER_YAML = """ +slug: aider +agent_cli: aider +execution: + mode: repository_patch + command: aider +""" + +_PATCH = """diff --git a/.gitignore b/.gitignore +new file mode 100644 +--- /dev/null ++++ b/.gitignore +@@ -0,0 +1 @@ ++.aider* +diff --git a/solution.ts b/solution.ts +--- a/solution.ts ++++ b/solution.ts +@@ -1 +1,2 @@ +-// stub ++export const ok = true; +diff --git a/old.ts b/old.ts +--- a/old.ts ++++ /dev/null +""" + + +def _exec_result( + tmp_path: Path, *, status: ExecStatus = ExecStatus.OK, patch: str = _PATCH +) -> StackExecResult: + adapter = StackAdapter.from_yaml_text(_ADAPTER_YAML) + req = StackExecRequest( + eval_id="raw-id", + model_id="openrouter/qwen/qwen-3-14b", + model_alias="qwen-3-14b", + stack=adapter, + task_id="be_01_jwt_auth", + task_prompt="impl", + repo_snapshot_dir=tmp_path, + seed=1, + ) + return StackExecResult( + request=req, + status=status, + patch=patch, + trace="t", + cost_usd=Decimal("0.000626"), + input_tokens=1400, + output_tokens=2200, + tool_calls=0, + wall_ms=55000, + error_detail=None, + started_at=datetime.now(UTC), + completed_at=datetime.now(UTC), + ) + + +class TestChangedFiles: + def test_parses_and_filters_noise(self) -> None: + files = changed_files(_PATCH) + assert "solution.ts" in files + assert ".gitignore" not in files # harness bookkeeping filtered + assert "old.ts" not in files # deletion (+++ /dev/null) dropped + + def test_empty_patch(self) -> None: + assert changed_files("") == [] + + +class TestExtractSubmission: + def test_reads_changed_file_content(self, tmp_path: Path) -> None: + (tmp_path / "solution.ts").write_text("export const ok = true;\n") + sub = extract_submission(tmp_path, _PATCH) + assert "=== solution.ts ===" in sub + assert "export const ok = true;" in sub + + def test_skips_missing_files(self, tmp_path: Path) -> None: + # solution.ts referenced by the patch but not on disk -> skipped, no raise + assert extract_submission(tmp_path, _PATCH) == "" + + +class TestExecResultToEvalResult: + def test_maps_ok_result(self, tmp_path: Path) -> None: + (tmp_path / "solution.ts").write_text("export const ok = true;\n") + log_dir = tmp_path / "artifacts" + result = _exec_result(tmp_path) + ev = exec_result_to_eval_result(result, log_dir=log_dir) + + assert ev.eval_row is not None + row = ev.eval_row + assert row.stack_id == "aider" # adapter slug, not the raw eval_id stack + assert row.model_id == "openrouter/qwen/qwen-3-14b" + assert row.task_id == "be_01_jwt_auth" + assert row.status is EvalStatus.SCORED + assert len(row.eval_id) == 16 # 16-hex contract + # cost + tokens carried over from Half A + assert row.stats.cost_usd == Decimal("0.000626") + assert row.stats.input_tokens == 1400 + # the raw_output artifact file was written with the submission + uri = row.artifact_refs.raw_output.uri + assert uri.startswith("file://") + written = Path(uri[len("file://") :]).read_text() + assert "export const ok = true;" in written + + def test_rejects_non_ok(self, tmp_path: Path) -> None: + result = _exec_result(tmp_path, status=ExecStatus.NO_PATCH, patch="") + with pytest.raises(ValueError, match="non-OK execution"): + exec_result_to_eval_result(result, log_dir=tmp_path) diff --git a/docs/04-runbook/14-stack-executor.md b/docs/04-runbook/14-stack-executor.md new file mode 100644 index 0000000..dc22b0f --- /dev/null +++ b/docs/04-runbook/14-stack-executor.md @@ -0,0 +1,171 @@ +# 14 — Stack Executor (Half A: harness → patch) + +How to run a **real** `harness × model × task` evaluation: drive an agent-CLI +harness (aider, …) against a task inside an isolated sandbox, capture the +produced **patch + trace + metered cost**, then score it. + +Built per **RFC-006**. Complements the existing pieces: + +- `05-stack-adapter-guide.md` — the `stack.yaml` adapter this reads. +- `09-sandbox-security.md` — the frozen sandbox policy (Half B side). +- `07-judge-panel.md` + `08-scoring-contract.md` — how the patch is scored. + +--- + +## The two-half sandbox + +``` +Half A — candidate (this doc): harness CLI ──► patch + StackExecutor + DockerHarnessLauncher (src/orchestrator/stack_executor.py) + +Half B — evaluator: produced code ──► scores + evaluators/ + evaluators/sandbox/runner.py (+ judge panel) + +Bridge: StackExecResult ──► EvalResult (src/orchestrator/stack_scoring.py) +``` + +`raw-llm` (L0, `execution.mode: direct_completion`) stays on `InspectEvalCaller`. +Every `repository_patch` stack (aider, claude-code, codex, …) runs through the +**StackExecutor**. + +--- + +## Network model — the bastion (RFC-006 decision A) + +The harness must reach the LiteLLM proxy (to pick the model + meter every token) +but must NOT reach anything else (no un-metered egress, no data exfil). + +Solution: a Docker **`internal`** network `pollmevals-sandbox` (no external +route). The proxy is attached to it as the **only reachable host** (a bastion); +it also stays on the default net to reach OpenRouter. The harness joins ONLY +`pollmevals-sandbox`. + +``` +[ harness container ] ──(pollmevals-sandbox, internal)──► [ litellm-proxy ] ──(default)──► OpenRouter + │ + └── no route to anything else (DNS for the open internet fails) +``` + +Consequences: +- No `NET_ADMIN` / iptables needed → `cap_drop ALL` holds. +- Portable Linux / macOS / CI (pure Docker topology, not firewall rules). +- In-sandbox the proxy is addressed by **container name** (`http://pollmevals-litellm-proxy:4000`), + NOT `localhost` — the internal net has no host route. + +--- + +## One-time setup + +```bash +make stack-up # Postgres + NATS + LiteLLM proxy (healthy) +make sandbox-net-up # create pollmevals-sandbox + attach the proxy (idempotent; + # for an already-running stack — no recreate needed) +make harness-image-aider # build pollmevals-harness-aider:0.1.0 (python+aider+git) +``` + +`make stack-up` also creates `pollmevals-sandbox` declaratively (it is in +`infra/docker-compose.litellm.yml`); `sandbox-net-up` is the imperative path for +a stack that is already up. + +Verify the bastion (no spend): + +```bash +# reaches the proxy (expect 200) +docker run --rm --network pollmevals-sandbox pollmevals-harness-aider:0.1.0 \ + python -c "import urllib.request; print(urllib.request.urlopen('http://pollmevals-litellm-proxy:4000/health/liveliness',timeout=8).status)" +# cannot reach the internet (expect DNS failure) +docker run --rm --network pollmevals-sandbox pollmevals-harness-aider:0.1.0 \ + python -c "import urllib.request; urllib.request.urlopen('https://example.com',timeout=6)" +``` + +--- + +## Running it + +All scripts are spend-gated (`--confirm-spend`) and require `LITELLM_MASTER_KEY` +in `.env` (the proxy key — NOT the upstream provider key). + +### Plumbing check (\$0 — no model call) + +```bash +uv run --project apps/eval-core-py python \ + apps/eval-core-py/scripts/stack_exec_live_smoke.py --plumbing +``` + +Runs a no-model command in the sandbox and asserts the launcher captures the +git diff. Use this to debug Docker / mount / network issues before spending. + +### Half A only — harness → patch (≈ \$0.001) + +```bash +uv run --project apps/eval-core-py python \ + apps/eval-core-py/scripts/stack_exec_live_smoke.py --confirm-spend +``` + +### Full chain — harness → patch → judge → scored number (≈ \$0.07) + +```bash +uv run --project apps/eval-core-py python \ + apps/eval-core-py/scripts/stack_score_live_smoke.py --confirm-spend +``` + +Run from the **repo root** (the judge panel resolves `evals/task-packs//rubric.yaml` +via cwd; the score script also `chdir`s to the repo root defensively). + +--- + +## How patch capture works + +The harness command is run as a **clean argv list** (no shell), so the +multi-line task prompt needs no quoting. The patch is captured **host-side** with +git, robust to whether the harness auto-commits: + +1. before the run: `git init` (if needed) + commit the snapshot AS GIVEN → base SHA +2. run the harness container (writable `/workspace` bind on the bastion net) +3. after: `git add -A && git diff --cached ` → the unified patch + (committed + working-tree + new files), with harness bookkeeping + (`.aider*`, `.gitignore`) filtered out by the scoring bridge + +Token/cost: best-effort from the harness self-report for now; the proxy is the +metered source of truth for reconciliation (a Phase-4 follow-up). + +--- + +## Scoring path (be_01 caveat) + +The bridge (`stack_scoring.exec_result_to_eval_result`) writes the candidate's +changed source files as the `raw_output` artifact, then the judge panel scores +it against the task rubric. + +**Use the judge panel for be_01, not the deterministic evaluators** — the be_01 +deterministic evaluators score-INVERT (broken > perfect, EVID-027). Judged +subjective scoring is also POLLMEVALS' edge. `doc_01` is judge-only by design; +`fe_01` has working dynamic evaluators (post-#24). + +--- + +## Known defects / follow-ups + +- **gpt-5-mini judge truncates** its rubric JSON at the 2048 `max_tokens` cap on + a 7-criterion coding rubric → parse-fail → 0.0 fallback, which drags the panel + median + Krippendorff α. Fix is `reasoning_effort` (cap reasoning tokens), NOT + raising the cap (EVID-023: the cap is bounded by the OpenRouter HTTP-402 + pre-reservation hazard). Until then, read the per-judge scores, not just the + median. +- **Proxy-metered cost reconciliation** — replace the best-effort harness + self-report with a proxy `/spend` query (harness-agnostic). +- **GridRunner dispatch-by-stack** — route CLI stacks to StackExecutor inside + GridRunner so a full grid run emits real Board data (RFC-006 Phase 4b). +- **More harnesses** — codex / claude-code / goose / openhands recipes are + registered but pending their per-stack smoke (`_PENDING_RECIPES`, Phase 5). + +--- + +## First real numbers (2026-06-02) + +| stack | task | result | cost | +|---|---|---|---| +| aider × qwen-3-14b | be_01 | patch: 208-line Express JWT middleware, `status=ok` | \$0.00063 | +| aider × qwen-3-14b | be_01 | judged: claude 5.27 · gemini 7.33 · gpt-5-mini 0.00\* | \$0.070 | + +\* gpt-5-mini truncation defect (above); trustworthy 2-judge signal ≈ 6.3.