feat(executor): Phase 4a — Half A→B scoring bridge + first scored number - #40
Closed
explosivebit wants to merge 3 commits into
Closed
feat(executor): Phase 4a — Half A→B scoring bridge + first scored number#40explosivebit wants to merge 3 commits into
explosivebit wants to merge 3 commits into
Conversation
RFC-006 Phase 4. Bridges StackExecResult (Half A patch) → EvalResult so the judge panel (and evaluators) score a (model × harness × task) run. - src/orchestrator/stack_scoring.py: changed_files (parse diff, filter harness noise .aider/.gitignore), extract_submission (final content of changed source files — code, not diff), exec_result_to_eval_result (write submission as the raw_output artifact + map to EvalRow; rejects non-OK runs). GridRunner dispatch-by-stack will reuse this same path. - scripts/stack_score_live_smoke.py: executor → bridge → judge panel → scored number. Anchors cwd at repo root (the panel resolves rubric.yaml via cwd). - 6 bridge unit tests (no Docker/judges). FIRST SCORED NUMBER — aider × qwen-3-14b × be_01, judged (inversion-free, since the be_01 deterministic evaluators invert per EVID-027): claude-sonnet 5.27 · gemini-3-flash 7.33 · gpt-5-mini 0.00* · cost $0.070 (*DEFECT: gpt-5-mini truncated its rubric JSON at the 2048 cap → parse-fail → 0.0 fallback, dragging the median + alpha. Trustworthy 2-judge signal ≈ 6.3. Fix is reasoning_effort per EVID-023, NOT raising the cap — separate judge work.) Gates: ruff + mypy --strict clean; 6 bridge tests; full suite 700 passed. Refs: rfc-006 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
How to run a real harness×model×task eval: the bastion network model (decision A), one-time setup (stack-up / sandbox-net-up / harness-image-aider), the $0 plumbing check + the two live smokes (Half A only, full chain to a scored number), how host-side git patch capture works, the be_01 judge-vs-evaluator caveat (EVID-027 inversion), known defects (gpt-5-mini truncation, proxy cost reconciliation, GridRunner dispatch), and the first real numbers. Refs: rfc-006 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rms RFC-006 First-real-run evidence pack: bastion proven ($0), Half A first patch (aider×qwen×be_01, $0.000626), first judged score (median ~6.3 on 2 trustworthy judges), and the gpt-5-mini truncation defect. verdict=PASS, CL3, evidence_type=live_integration_run. Activated; R_eff recomputed. Refs: evid-rfc-006-stackexecutor-live-first-real-harness-model-patch-judged-score Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #39. Bridges Half A (StackExecResult patch) → Half B (judge panel) so a
model × harness × taskrun yields a real score. GridRunner dispatch-by-stack (Phase 4b) will reuse this same path.src/orchestrator/stack_scoring.py:changed_files(parse diff, filter harness noise) ·extract_submission(final content of changed source files — code, not diff) ·exec_result_to_eval_result(write submission as theraw_outputartifact →EvalRow; rejects non-OK runs).scripts/stack_score_live_smoke.py: executor → bridge → judge panel → scored number (anchors cwd at repo root for rubric resolution).docs/04-runbook/14-stack-executor.md: full runbook (bastion model, setup, smokes, patch capture, scoring caveat, defects).First scored number (live)
aider × qwen-3-14b × be_01, judged (inversion-free — be_01 deterministic evaluators invert, EVID-027):Panel median 5.27/10 · total cost $0.070.
max_tokenscap on the 7-criterion coding rubric → parse-fail → 0.0 fallback, dragging the median + α. Fix isreasoning_effort(EVID-023 bounds the cap by the OpenRouter 402 hazard), and a parse-fail should map to DEGRADED, not 0.0. Trustworthy 2-judge signal ≈ 6.3.Gates
ruff + mypy --strict clean · 6 bridge unit tests · full suite 700 passed.
Note
This branch's EVID-049 auto-link touched
RFC-006.md; reconcile with #37 (RFC-006 phases) at merge.Refs: rfc-006
🤖 Generated with Claude Code