feat(executor): Phase 4b — GridRunner dispatch-by-stack (StackExecutorCaller) - #42
Merged
Conversation
…cher seam (RFC-006) Half A candidate path (run agent-CLI harness -> patch), no Docker / no spend. - StackAdapter: operational parse of stacks/<slug>/stack.yaml (distinct from the manifest StackPin run-pin — separate concern, separate contract) - build_proxy_invocation: per-CLI proxy recipe. aider proven (the RFC-006 first slice); other known harnesses surface HarnessRecipePending (recipe lands at its Phase-5 per-stack smoke); unknown CLI -> UnsupportedHarnessError - HarnessLauncher Protocol seam + deterministic FakeHarnessLauncher; DockerHarnessLauncher is a typed Phase-2 stub (launch() raises) - build_docker_run_kwargs: writable /workspace bind + locked-down flags (cap_drop ALL, no-new-privileges). PROXY_ONLY egress bridge raises NetworkPolicyNotConfigured — the security-sensitive bridge is an open RFC-006 decision wired in Phase 2 - StackExecutor.execute: adapter + recipe -> HarnessRunPlan -> launch -> patch/trace/cost(metered)/status; never drops a result (FR-009 discipline) Gates: ruff + mypy --strict clean; 33 new unit tests; full suite 690 passed. Refs: rfc-006 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sion A) RFC-006 Phase 2 (no spend): the image + network now exist; the real run is gated. - infra/docker/harness-aider/Dockerfile -> pollmevals-harness-aider:0.1.0 (python:3.12-slim + aider-chat 0.86.2 + git 2.47.3, non-root, writable /workspace). Built + smoke-verified (aider --version, git --version). - Network decision A wired declaratively: pollmevals-sandbox `internal: true` net in the litellm compose; the proxy joins it as the bastion (only reachable host, no external route -> harness reaches ONLY the metered proxy, and cap_drop ALL holds because no NET_ADMIN/iptables is needed). Portable Linux/macOS/CI. - stack_executor: PROXY_ONLY now joins the internal bastion net (no longer raises); in-sandbox the harness addresses the proxy by container name (SANDBOX_PROXY_BASE_URL), not localhost (the internal net has no host route); DockerHarnessLauncher.launch moves to Phase 3 (built + validated against the first real run, not blind). - Makefile: harness-image-aider (build) + sandbox-net-up (attach proxy to the bastion net on an already-running stack, idempotent). Gates: ruff + mypy --strict clean; 36 stack-executor tests; full suite 692 passed; compose validates with internal:true preserved. Refs: rfc-006 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ider×qwen×be_01) RFC-006 Phase 3: the candidate pipeline (Half A) runs end-to-end on real money. - DockerHarnessLauncher.launch: docker-py run on the pollmevals-sandbox bastion net (decision A) against a writable /workspace; host-side git base + then `git add -A && git diff --cached <base>` robustly captures the patch (committed + working-tree + new files) without shell-quoting the prompt. Auto-discovers the Docker Desktop socket (~/.docker/run) since docker-py's from_env() defaults to /var/run and misses it. - scripts/stack_exec_live_smoke.py: --plumbing ($0 no-model docker/patch check) + --confirm-spend (real run, gated like smoke_run.py). - Tests: git base/capture + token-parse plumbing (host git, no Docker); the container run itself is validated by the live smoke. FIRST REAL NUMBER — aider × qwen-3-14b × be_01 via the metered bastion proxy: status=ok, 208-line patch (a real Express JWT auth middleware), cost=$0.000626, in=1400/out=2200 tokens, ~55s. Bastion proven end-to-end: a sandbox container reaches the proxy (200) but NOT the internet (DNS fails) — no un-metered egress. Gates: ruff + mypy --strict clean; 39 executor tests; full suite 694 passed. Refs: rfc-006 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RFC-006 Phase 4. Bridges StackExecResult (Half A patch) → EvalResult so the judge panel (and evaluators) score a (model × harness × task) run. - src/orchestrator/stack_scoring.py: changed_files (parse diff, filter harness noise .aider/.gitignore), extract_submission (final content of changed source files — code, not diff), exec_result_to_eval_result (write submission as the raw_output artifact + map to EvalRow; rejects non-OK runs). GridRunner dispatch-by-stack will reuse this same path. - scripts/stack_score_live_smoke.py: executor → bridge → judge panel → scored number. Anchors cwd at repo root (the panel resolves rubric.yaml via cwd). - 6 bridge unit tests (no Docker/judges). FIRST SCORED NUMBER — aider × qwen-3-14b × be_01, judged (inversion-free, since the be_01 deterministic evaluators invert per EVID-027): claude-sonnet 5.27 · gemini-3-flash 7.33 · gpt-5-mini 0.00* · cost $0.070 (*DEFECT: gpt-5-mini truncated its rubric JSON at the 2048 cap → parse-fail → 0.0 fallback, dragging the median + alpha. Trustworthy 2-judge signal ≈ 6.3. Fix is reasoning_effort per EVID-023, NOT raising the cap — separate judge work.) Gates: ruff + mypy --strict clean; 6 bridge tests; full suite 700 passed. Refs: rfc-006 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
How to run a real harness×model×task eval: the bastion network model (decision A), one-time setup (stack-up / sandbox-net-up / harness-image-aider), the $0 plumbing check + the two live smokes (Half A only, full chain to a scored number), how host-side git patch capture works, the be_01 judge-vs-evaluator caveat (EVID-027 inversion), known defects (gpt-5-mini truncation, proxy cost reconciliation, GridRunner dispatch), and the first real numbers. Refs: rfc-006 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rms RFC-006 First-real-run evidence pack: bastion proven ($0), Half A first patch (aider×qwen×be_01, $0.000626), first judged score (median ~6.3 on 2 trustworthy judges), and the gpt-5-mini truncation defect. verdict=PASS, CL3, evidence_type=live_integration_run. Activated; R_eff recomputed. Refs: evid-rfc-006-stackexecutor-live-first-real-harness-model-patch-judged-score Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rCaller) RFC-006 Phase 4b. Makes a CLI stack usable through the EvalCaller Protocol so a real grid run dispatches it identically to raw-llm — the keystone that turns the standalone executor into part of the grid pipeline. - src/orchestrator/stack_caller.py: StackExecutorCaller (EvalCaller adapter) — resolve the stack adapter → seed a candidate snapshot → run StackExecutor → bridge the patch to an EvalResult (stack_scoring); a non-OK run → a graceful FAILED row (FR-009, never dropped). + factory providers (task prompt from task.yaml, be_01 snapshot) and default_model_alias (route → proxy alias). - grid_runner.py: optional caller_for_stack resolver — raw-llm → InspectEvalCaller, CLI stacks → StackExecutorCaller. Backward-compatible (falls back to the single caller when unset). GridRunner's judge hook scores the produced submission with no downstream special-casing. - 6 offline tests: caller OK→SCORED+submission / empty-patch→FAILED, cost/alias carry-over, dispatch routing, single-caller fallback. Gates: ruff + mypy --strict clean; full suite 706 passed. Refs: rfc-006 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #40. The keystone integration: makes a CLI stack usable through the EvalCaller Protocol so GridRunner dispatches it identically to
raw-llm. Turns the standalone executor into part of the grid pipeline.src/orchestrator/stack_caller.py: StackExecutorCaller (EvalCaller adapter) — resolve the stack adapter → seed a candidate snapshot → runStackExecutor→ bridge the patch to anEvalResult; a non-OK run → a graceful FAILED row (FR-009, never dropped). Plus factory providers (make_task_prompt_provider,make_be01_snapshot_provider) anddefault_model_alias.grid_runner.py: optionalcaller_for_stackresolver —raw-llm→InspectEvalCaller, CLI stacks →StackExecutorCaller. Backward-compatible (falls back to the single caller when unset). GridRunner's judge hook scores the produced submission with no downstream special-casing.Gates
ruff + mypy --strict clean · full suite 706 passed.
Pipeline status
With this, the real-data pipeline is built end-to-end: Half A (executor, #39) + bridge (#40) + judge fix (#41) + grid dispatch (this). Remaining: a real grid run → Board emission → point
apps/site(PR #36) at real data (Phase 4c).Refs: rfc-006
🤖 Generated with Claude Code