Codex/swe bench pro prod multiagent eval

[areshand]

Summary

• run the actual production multi-agent solver inside SWE Bench Pro task containers
• use one packaged production entrypoint with no scaffold, proxy, custom-solver, or direct-file fallback
• promote reusable contract extraction, final-diff verification, structured findings/todos, and repair gates into multiagent_framework
• enforce machine-checked build/behavior evidence against the exact submitted diff
• add a concise README, architecture/demo docs, benchmark provenance, an MIT license, and scoped upstream contribution drafts
• provide a no-spend local gate demo and a runnable 5-10 task internal pilot harness

Evaluation Boundary

The supported benchmark entrypoint is python3 -m evaluation.swe_bench_pro. It bakes the production repository into each task image, changes to /opt/multiagent, and launches:

python3 -m evaluation.native_solver.solve_swe_prod

That package calls the same launch.sh orchestrator/worker/verifier workflow used outside evaluation. EvalScope receives clean accepted patches normally. A final production gate rejection is scoreable only when the framework publishes the versioned submission_gate_rejection terminal artifact and dedicated exit code 3; the runner then discards the rejected diff and scores a clean no-submission workspace. Legacy rc=2, timeouts, malformed signals, and runner/infrastructure failures remain fail-closed.

The historical first-50 page records a cumulative best-known 36/50 aggregate assembled across focused reruns. It is not a single held-out run from commit 66638d2, and this PR does not claim that it is. docs/benchmark.md records the recovered historical provenance, missing raw artifacts, and exact fresh-run command. A new score is publication-ready only when the provenance capture and validation commands succeed: they bind clean solver/EvalScope/verifier commits, dataset hash, effective configuration, official report, native event log, actual in-container Codex/Node versions, and content-addressed base/derived image identities. The derived image itself carries the full solver-source digest label.

Framework Refactor

The former monolithic native solver is split by responsibility:

• solve_swe_prod.py: 157-line packaged entrypoint and compatibility facade
• swe_prod_lifecycle.py: lifecycle coordination
• contracts, bootstrap, repository, evidence, validation, orchestration, checkpoints, and transitions: explicit modules with strict package-relative imports
• multiagent_framework/coding/contracts.py: framework-neutral public requirement extraction, issue-coverage gating, provenance/history contracts, and contract-ledger rendering
• multiagent_framework/{snapshot,verification,state,gate}.py: exact-diff identity, structured evidence, atomic state, and final submission gate
• multiagent_framework/provenance.py: reusable clean-Git identity and relocatable SHA-256 artifact-bundle primitives; the SWE semantic adapter remains under 300 lines

All repeated try/except ImportError import fallbacks and wildcard imports are removed from evaluation/native_solver. Import-time dependency failures now preserve the original traceback. The legacy entrypoint helper surface remains enumerable for existing named, wildcard, dir(), and module.__dict__ consumers without using wildcard imports internally.

SWE-specific code retains only public metadata sanitization, prompt-envelope handling, adapter paths, and rendering. Framework boundary tests reject SWE Bench, EvalScope, hidden-test, benchmark-row, and EVAL_* dependencies.

System Diagram

flowchart TD
    User["User task"] --> Launch["./launch.sh"]
    Eval["Optional SWE Bench Pro adapter"] --> Bake["Bake production repo into task image"]
    Bake --> Module["python3 -m evaluation.native_solver.solve_swe_prod"]
    Module --> Launch

    subgraph Framework["General multiagent framework"]
        Launch --> Tmux["tmux orchestrator session"]
        Tmux --> Orchestrator["Orchestrator CLI"]
        Orchestrator --> Helper["bin/subagent.sh control plane"]
        Helper --> Scouts["Contract and scope scouts"]
        Helper --> Workers["Path-owned workers"]
        Helper --> Verifiers["Build and behavior verifiers"]
        Scouts --> State[("Findings and contracts")]
        Workers --> Repo[("Target Git repository")]
        Verifiers --> State
        Repo --> Snapshot["Exact binary Git diff + SHA-256"]
        Snapshot --> Verifiers
        State --> Todos["Blocking findings become repair todos"]
        Todos --> Orchestrator
        Verifiers --> Gate{"No blockers and evidence matches final diff?"}
        Gate -- "No: repair and reverify" --> Orchestrator
    end

    Gate -- "Yes" --> Accepted["Accepted final patch"]
    Accepted --> Official["Evaluation only: official scorer"]

Loading

multiagent_framework is a short-lived Python standard-library control plane, not a daemon. The long-lived processes are the existing CLI agents in tmux.

Public Entry Path

• ./scripts/demo.sh runs a deterministic reject -> repair -> accept -> stale-diff reject flow in seconds without a model or API key.
• README.md is reduced from more than 1,100 lines to 134 lines; the detailed operations reference moved to docs/getting-started.md.
• docs/technical-note.md explains the Codex/Claude worker/verifier composition and its limitations.
• examples/internal-pilot/ runs paired baseline/orchestrated cells against 5-10 pinned real tasks and requires independent human review.
• docs/internal-pilot-request.md provides the evidence-bounded one-page internal ask and names the artifacts required before presentation, commitment, or results TODOs can close.
• TODO.md keeps the real internal pilot open; local presentation drafts are not counted as completed external evidence.
• Upstream status is recorded: SWE-agent #1464 was closed because the project is maintenance-only, OpenCode #37388 remains open after template compliance, and OpenHands SDK draft PR #4131 contains the focused tool-span/action-event extraction.

Validation

• bash tests/run.sh passed with Git signing disabled only for temporary fixture commits
• 9 framework contract tests passed on local Python 3.13 and python:3.8-slim
• 6 native package/import tests passed on local Python 3.13 and python:3.8-slim
• 5 typed SWE outcome/aggregation tests passed locally and on Python 3.8 (the Git-dependent lifecycle test runs in CI and skips only in the minimal image without Git)
• 7 generic provenance tests and 7 SWE provenance tests passed locally; Python 3.8 passed all non-Git cases
• 4 no-network internal-pilot tests passed
• ./scripts/demo.sh passed
• shellcheck and bash -n passed for new shell entrypoints
• git diff --check passed

The Python 3.8 container run also found and fixed an existing runtime type-alias incompatibility in multiagent_framework/gate.py; compatibility is now exercised by CI rather than inferred from syntax parsing alone.