feat(executor): Phase 4a — Half A→B scoring bridge + first scored number

.forgeplan/evidence/EVID-049-rfc-006-stackexecutor-live-first-real-harness-model-patch-judged-score-aider-qwen-be-01.md

@@ -0,0 +1,62 @@
+---
+depth: standard
+id: EVID-049
+kind: evidence
+last_modified_at: 2026-06-02T18:31:17.801906+00:00
+last_modified_by: claude-code/2.1.156
+links:
+- target: RFC-006
+  relation: informs
+status: active
+title: 'RFC-006 StackExecutor live: first real harness×model patch + judged score (aider×qwen×be_01)'
+---
+
+## Summary
+
+RFC-006 StackExecutor (Half A: harness → patch) is implemented and validated
+end-to-end on real money. The candidate pipeline produces a real
+`harness × model × task` patch with metered cost, and the Half A→B bridge feeds
+it to the judge panel for a real score.
+
+## Structured Fields
+
+- **verdict**: PASS
+- **congruence_level**: CL3 (same context — this is RFC-006's own implementation, verified directly against its acceptance criteria)
+- **evidence_type**: live_integration_run
+
+## What was verified (2026-06-02)
+
+1. **Network decision A (bastion) — proven, $0.** A container on the Docker
+   `internal` net `pollmevals-sandbox` reaches the LiteLLM proxy (HTTP 200) but
+   NOT the internet (DNS resolution fails). No un-metered egress; `cap_drop ALL`
+   holds (no NET_ADMIN needed).
+2. **Half A plumbing — proven, $0.** `DockerHarnessLauncher` runs a no-model
+   command in the sandbox, writes to the writable `/workspace` bind, and
+   host-side git captures the diff (`--plumbing` check). Surfaced + fixed a
+   `DOCKER_HOST` discovery bug before any spend (cheap-signal-before-dear).
+3. **First real patch — aider × qwen-3-14b × be_01.** `status=ok`, a 208-line
+   real Express JWT auth middleware written to `solution.ts`, `cost=$0.000626`
+   (metered via the proxy), in 1400 / out 2200 tokens, ~55s.
+4. **First scored number — same stack, judged.** Half A→B bridge → judge panel
+   (inversion-free; the be_01 deterministic evaluators invert per EVID-027):
+   claude-sonnet 5.27 · gemini-3-flash 7.33 · gpt-5-mini 0.00\* · total $0.070.
+   Panel median 5.27/10; trustworthy 2-judge signal ≈ 6.3.
+
+## Defect surfaced
+
+\* **gpt-5-mini judge truncates** its rubric JSON at the 2048 `max_tokens` cap on
+the be_01 7-criterion coding rubric → JSON parse-fail → 0.0 fallback, which
+drags the panel median + Krippendorff α (α went negative). Fix is
+`reasoning_effort` (cap reasoning tokens), NOT raising the cap — EVID-023 bounds
+the cap by the OpenRouter HTTP-402 pre-reservation hazard. Tracked as judge
+follow-up; does not block the executor.
+
+## Artifacts
+
+- Code: `apps/eval-core-py/src/orchestrator/stack_executor.py`, `stack_scoring.py`
+- Image: `pollmevals-harness-aider:0.1.0` (aider-chat 0.86.2)
+- Smokes: `scripts/stack_exec_live_smoke.py`, `scripts/stack_score_live_smoke.py`
+- Runbook: `docs/04-runbook/14-stack-executor.md`
+- Branches: `feat/stack-executor-rfc006` (Phases 1-3, PR #39), `feat/stack-scoring-rfc006` (Phase 4a)
+
+

.forgeplan/rfcs/RFC-006-stack-executor-run-agent-cli-harnesses-in-the-sandbox-candidate-side-patch-trace-metered-cost.md

@@ -119,3 +119,4 @@ real "model × harness" number**. Then widen to codex/opencode + more tasks.
 
 
 
+

apps/eval-core-py/scripts/stack_score_live_smoke.py

@@ -0,0 +1,164 @@
+#!/usr/bin/env python
+"""Live: aider x qwen-3-14b x be_01 -> patch -> JUDGE PANEL -> FIRST scored number.
+
+RFC-006 Phase 4. Chains Half A (StackExecutor) + the Half A->B bridge
+(stack_scoring) + the operational judge panel (#28). Judges are used (not the
+be_01 deterministic evaluators, which invert — EVID-027); judged-subjective
+scoring is also POLLMEVALS' edge.
+
+Cost: ~$0.0006 (aider on qwen) + ~$0.05 (3 judges). Gated by --confirm-spend.
+
+Prereqs: make stack-up && make sandbox-net-up && make harness-image-aider.
+Run:
+  uv run --project apps/eval-core-py python \
+      apps/eval-core-py/scripts/stack_score_live_smoke.py --confirm-spend
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import os
+import shutil
+import statistics
+import sys
+import tempfile
+from datetime import UTC, datetime
+from decimal import Decimal
+from pathlib import Path
+
+import yaml
+
+REPO = Path(__file__).resolve().parents[3]
+sys.path.insert(0, str(REPO / "apps" / "eval-core-py"))
+
+from src.orchestrator.cost import PricingTuple  # noqa: E402
+from src.orchestrator.judge_panel import JudgePanel  # noqa: E402
+from src.orchestrator.stack_executor import (  # noqa: E402
+    DockerHarnessLauncher,
+    ExecStatus,
+    StackAdapter,
+    StackExecRequest,
+    StackExecutor,
+)
+from src.orchestrator.stack_scoring import (  # noqa: E402
+    cost_with_judges,
+    exec_result_to_eval_result,
+)
+
+_CANDIDATE = "openrouter/qwen/qwen-3-14b"
+_TASK = "be_01_jwt_auth"
+_JUDGES = ["claude-sonnet-4-6-judge", "gpt-5-mini-judge", "gemini-3-flash"]
+_QWEN_PRICING = PricingTuple(
+    model_id=_CANDIDATE,
+    input_per_mtoken_usd=Decimal("0.07"),
+    output_per_mtoken_usd=Decimal("0.24"),
+    snapshot_at=datetime(2026, 6, 2, tzinfo=UTC),
+)
+
+
+def _load_env(repo: Path) -> None:
+    envf = repo / ".env"
+    if not envf.exists():
+        return
+    for line in envf.read_text(encoding="utf-8").splitlines():
+        s = line.strip()
+        if s and not s.startswith("#") and "=" in s:
+            k, _, v = s.partition("=")
+            os.environ.setdefault(k.strip(), v.strip().strip('"').strip("'"))
+
+
+def _seed_candidate_snapshot(dst: Path, pack: Path) -> None:
+    """be_01 CANDIDATE workspace: pinned deps only — NO gold, NO tests."""
+    for f in ("package.json", "tsconfig.json"):
+        shutil.copy(pack / "gold" / f, dst / f)
+    (dst / "solution.ts").write_text(
+        "// Implement the Express JWT auth middleware here (see the task prompt).\n",
+        encoding="utf-8",
+    )
+
+
+async def _main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--confirm-spend", action="store_true", help="real $ (aider + judges)")
+    args = ap.parse_args()
+
+    # JudgePanel resolves rubric.yaml relative to cwd; anchor at the repo root.
+    os.chdir(REPO)
+    _load_env(REPO)
+    master_key = os.environ.get("LITELLM_MASTER_KEY", "")
+    if not master_key:
+        print("ERROR: LITELLM_MASTER_KEY not set (.env)", file=sys.stderr)
+        return 2
+    if not args.confirm_spend:
+        print("DRY: pass --confirm-spend to run aider + 3 judges for real (~$0.05).")
+        return 0
+
+    pack = REPO / "evals" / "task-packs" / _TASK
+    prompt = str(yaml.safe_load((pack / "task.yaml").read_text())["prompt_template"])
+    adapter = StackAdapter.from_yaml_path(REPO / "stacks" / "aider" / "stack.yaml")
+
+    snapshot = Path(tempfile.mkdtemp(prefix="pollmevals-score-"))
+    _seed_candidate_snapshot(snapshot, pack)
+    artifacts = snapshot / "artifacts"
+
+    # --- Half A: harness -> patch ---
+    print(f"[1/3] Half A: aider x qwen x {_TASK} ...")
+    executor = StackExecutor(
+        launcher=DockerHarnessLauncher(),
+        api_key=master_key,
+        pricing_snapshot={_CANDIDATE: _QWEN_PRICING},
+    )
+    request = StackExecRequest(
+        eval_id="score-aider-qwen-be01",
+        model_id=_CANDIDATE,
+        model_alias="qwen-3-14b",
+        stack=adapter,
+        task_id=_TASK,
+        task_prompt=prompt,
+        repo_snapshot_dir=snapshot,
+        seed=1,
+        timeout_s=600,
+    )
+    exec_result = await executor.execute(request)
+    print(
+        f"      status={exec_result.status} cost=${exec_result.cost_usd} "
+        f"wall={exec_result.wall_ms}ms"
+    )
+    if exec_result.status is not ExecStatus.OK:
+        print(f"      executor did not produce a patch: {exec_result.error_detail}")
+        return 1
+
+    # --- Bridge: Half A -> Half B ---
+    print("[2/3] bridge: StackExecResult -> EvalResult (write submission artifact)")
+    eval_result = exec_result_to_eval_result(exec_result, log_dir=artifacts)
+
+    # --- Half B: judge panel (inversion-free) ---
+    print(f"[3/3] Half B: {len(_JUDGES)} judges score the patch via the be_01 rubric ...")
+    panel = JudgePanel(
+        judge_models=_JUDGES,
+        candidate_model_id=_CANDIDATE,
+        rubric_version="1.0",
+    )
+    judgments = await panel.score(eval_result, _TASK)
+    agg = panel.aggregate(judgments)
+
+    judge_cost = sum((j.cost_usd for j in judgments), Decimal("0"))
+    total = cost_with_judges(exec_result.cost_usd, judge_cost)
+
+    print("\n=== FIRST SCORED (model x harness x task) NUMBER ===")
+    print(f"stack:        aider x qwen-3-14b   task: {_TASK}")
+    for j in judgments:
+        print(f"  judge {j.judge_model_id:<24} total={j.total_score:5.2f} cost=${j.cost_usd}")
+    panel_median = statistics.median([j.total_score for j in judgments]) if judgments else 0.0
+    print(f"panel median: {panel_median:.2f} / 10   (per-criterion: {agg.median_per_criterion})")
+    print(
+        f"alpha:        point={agg.alpha_point} ci_lower={agg.alpha_ci_lower} "
+        f"status={agg.judge_status}"
+    )
+    print(f"cost:         harness=${exec_result.cost_usd} + judges=${judge_cost} = ${total}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(asyncio.run(_main()))

apps/eval-core-py/src/orchestrator/stack_scoring.py

@@ -0,0 +1,160 @@
+"""Bridge Half A (StackExecResult patch) -> Half B (judge panel / evaluators).
+
+RFC-006 Phase 4. Maps a ``StackExecResult`` (harness -> patch, Half A) into the
+``EvalResult`` shape that ``JudgePanel.score`` (and the evaluators) consume, so a
+``(model x harness x task)`` run yields a real score. GridRunner dispatch-by-stack
+reuses this so CLI stacks flow through the SAME scoring path as ``raw-llm``.
+
+The judge panel reads the candidate's output from the ``raw_output`` artifact
+URI (a file on disk), so the bridge writes the submission there.
+
+Submission = the FINAL content of the candidate's changed source files (not the
+raw diff), with harness bookkeeping (``.aider*``, ``.gitignore``,
+``.pollmevals*``) filtered out — code-quality judging wants the code, not a diff.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import re
+from datetime import UTC, datetime
+from decimal import Decimal
+from pathlib import Path
+
+from src.contracts import (
+    ArtifactRef,
+    EvalArtifactRefs,
+    EvalRow,
+    EvalStats,
+    EvalStatus,
+)
+from src.orchestrator.eval_caller import EvalRequest, EvalResult, compute_eval_id
+from src.orchestrator.stack_executor import ExecStatus, StackExecResult
+
+# Harness bookkeeping files that are NOT part of the candidate submission.
+_NOISE_RE = re.compile(r"(^|/)(\.aider|\.gitignore|\.pollmevals|node_modules/)")
+
+# Synthetic run_hash fragment for standalone (non-grid) scoring. GridRunner
+# passes the real run_hash; this default keeps single-eval scoring deterministic.
+_STANDALONE_RUN_HASH = "sha256:" + "5" * 64
+
+
+def changed_files(patch: str) -> list[str]:
+    """Return the candidate's changed source paths from a unified diff.
+
+    Reads ``+++ b/<path>`` headers; drops ``/dev/null`` (pure deletions) and
+    harness bookkeeping noise.
+    """
+    out: list[str] = []
+    for line in patch.splitlines():
+        if line.startswith("+++ b/"):
+            path = line[len("+++ b/") :].strip()
+            if path and path != "/dev/null" and not _NOISE_RE.search(path):
+                out.append(path)
+    return out
+
+
+def extract_submission(snapshot_dir: Path, patch: str) -> str:
+    """Concatenated final content of the candidate's changed source files.
+
+    Each file is prefixed with a ``// === <path> ===`` banner so a multi-file
+    submission stays legible to the judge. Files referenced by the patch but
+    missing on disk (e.g. deletions) are skipped.
+    """
+    parts: list[str] = []
+    for rel in changed_files(patch):
+        f = snapshot_dir / rel
+        if f.exists():
+            body = f.read_text(encoding="utf-8", errors="replace")
+            parts.append(f"// === {rel} ===\n{body}")
+    return "\n\n".join(parts)
+
+
+def _artifact(log_dir: Path, eval_id: str, label: str, content: str) -> ArtifactRef:
+    """Write *content* to a content-addressed file and return its ArtifactRef."""
+    sha256 = hashlib.sha256(content.encode()).hexdigest()
+    mime = "application/json" if label == "evaluator_json" else "text/plain"
+    ext = ".json" if label == "evaluator_json" else ".txt"
+    dest = log_dir / eval_id / f"{label}-{sha256}{ext}"
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    dest.write_text(content, encoding="utf-8")
+    return ArtifactRef(
+        sha256=sha256,
+        size_bytes=len(content.encode()),
+        uri=f"file://{dest}",
+        mime_type=mime,
+    )
+
+
+def exec_result_to_eval_result(
+    exec_result: StackExecResult,
+    *,
+    log_dir: Path,
+    run_hash: str = _STANDALONE_RUN_HASH,
+) -> EvalResult:
+    """Map a StackExecResult into a judge-ready EvalResult.
+
+    The candidate submission (changed source files) is written as the
+    ``raw_output`` + ``normalized_output`` artifacts so ``JudgePanel.score`` can
+    read it. ``stack_id`` is the adapter slug; cost/tokens carry over from Half A.
+
+    Raises:
+        ValueError: the executor did not produce a patch (status != OK) — there
+            is nothing to score.
+    """
+    req = exec_result.request
+    if exec_result.status is not ExecStatus.OK or not exec_result.patch:
+        raise ValueError(
+            f"cannot score a non-OK execution (status={exec_result.status}, eval_id={req.eval_id})"
+        )
+
+    stack_id = req.stack.slug
+    eval_id = compute_eval_id(run_hash, req.model_id, stack_id, req.task_id, req.seed)
+    submission = extract_submission(req.repo_snapshot_dir, exec_result.patch)
+    evaluator_json = (
+        f'{{"eval_id":"{eval_id}","harness":"{req.stack.agent_cli}",'
+        f'"patch_bytes":{len(exec_result.patch.encode())}}}'
+    )
+
+    artifact_refs = EvalArtifactRefs(
+        raw_output=_artifact(log_dir, eval_id, "raw_output", submission),
+        normalized_output=_artifact(log_dir, eval_id, "normalized_output", submission.strip()),
+        evaluator_json=_artifact(log_dir, eval_id, "evaluator_json", evaluator_json),
+    )
+    stats = EvalStats(
+        input_tokens=exec_result.input_tokens,
+        output_tokens=exec_result.output_tokens,
+        wall_clock_ms=exec_result.wall_ms,
+        cost_usd=exec_result.cost_usd,
+    )
+    row = EvalRow(
+        eval_id=eval_id,
+        model_id=req.model_id,
+        stack_id=stack_id,
+        task_id=req.task_id,
+        seed=req.seed,
+        status=EvalStatus.SCORED,
+        artifact_refs=artifact_refs,
+        stats=stats,
+        started_at=exec_result.started_at,
+        completed_at=exec_result.completed_at,
+    )
+    request = EvalRequest(
+        eval_id=eval_id,
+        model_id=req.model_id,
+        stack_id=stack_id,
+        task_id=req.task_id,
+        seed=req.seed,
+    )
+    return EvalResult(
+        request=request,
+        eval_row=row,
+        exception=None,
+        started_at=exec_result.started_at,
+        completed_at=datetime.now(UTC),
+    )
+
+
+def cost_with_judges(exec_cost: Decimal, judgments_cost: Decimal) -> Decimal:
+    """Total eval cost = harness spend + judge spend (CONTEXT.md cost rule)."""
+    return exec_cost + judgments_cost