[Feature]: evotrace-openeval-adapter — export compiled bundles / validation runs to the EvalPort interchange format

[adhabnr-ux]

Real workflow problem

EvoTrace's compiled bundles (task.json from evotrace.compiler.compile_session(), docs/schema.md §"Compiled task manifest") and its Docker two-state validation runs (evotrace.validation.validate_bundle_in_docker(), §"Saved run") are a genuinely EvoTrace-specific format: task text + a list of verifier shell commands in, a passed/score/checks report out. Anyone who wants to run EvoTrace-compiled assets through a framework-agnostic eval harness, or pull an existing eval suite into EvoTrace's Docker two-state protocol, has to hand-write the mapping today — there's no interchange layer.

EvalPort is a small open JSON spec for exactly this: a Suite (test cases + graders) and a ResultSet (per-test-case results + grader results). Full spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md. There are 36 existing adapters for other eval/observability frameworks under adapters/<name>-openeval-adapter/; an evotrace-openeval-adapter would let a compiled EvoTrace bundle round-trip through anything else that already speaks EvalPort.

Evidence and proposed contract

The mapping is fairly direct on the export side:

• task.json's verifier.commands list (each a shell command run by the generated verifier.py, per compiler.py) maps naturally to a list of EvalPort command-type graders on one Suite test case.
• The run.json produced by validate_bundle_in_docker() — passed, reference.score, reference.checks[] (each with name/passed/output) — maps to one EvalPort ResultSet entry with per-check grader results.

Rough sketch (illustrative only, not tested against the actual EvalPort schema validator):

# evotrace_openeval_adapter/convert.py
from pathlib import Path
from evotrace.util import read_json
from evotrace.catalog import bundle_sha256

def bundle_to_openeval_test_case(bundle: Path) -> dict:
    """task.json (compile_session's manifest) -> one EvalPort Suite test case."""
    manifest = read_json(bundle / "task.json")
    verifier = manifest["verifier"]
    return {
        "id": manifest["id"],
        "input": {"prompt": manifest["task"]},
        "graders": [
            {
                "type": "command",
                "name": f"verifier[{i}]",
                "config": {"command": cmd, "timeout_seconds": verifier["timeout_seconds"]},
            }
            for i, cmd in enumerate(verifier["commands"])
        ],
        "metadata": {
            "source_agent": manifest["source"]["agent"],
            "base_commit": manifest["repository"]["base_commit"],
            "bundle_sha256": bundle_sha256(bundle),
        },
    }

def run_report_to_openeval_result(bundle: Path, report: dict) -> dict:
    """validate_bundle_in_docker()'s run report -> one EvalPort ResultSet entry."""
    return {
        "test_case_id": bundle.name,
        "passed": report["passed"],
        "score": report["reference"]["score"],
        "grader_results": [
            {"name": c["name"], "passed": c["passed"], "output": c.get("output", "")}
            for c in report["reference"]["checks"]
        ],
        "metadata": {
            "protocol": report["protocol"],  # "docker-two-state-v0.1"
            "bundle_sha256": report["bundle_sha256"],
        },
    }

from_openeval() (Suite → EvoTrace bundle) is honestly the harder direction: EvoTrace bundles carry a git base commit, a reference patch, and a sandboxed environment (docs/sandbox-contract.md) that a generic EvalPort Suite has no place to put. A first pass would probably only support Suites whose graders are already command-type (i.e., treat the Suite's grader commands as verifier.commands against a caller-supplied repo + base commit), rather than trying to synthesize an environment from nothing.

If this shape looks right, I'd be glad to open a draft PR under adapters/evotrace-openeval-adapter/ following the layout of the existing 36 adapters. Happy to adjust the mapping if maintainers see a cleaner fit to task.yaml's compact form instead of the full task.json.

Privacy

Only the schema/field names above are referenced (from docs/schema.md, compiler.py, validation.py in this repo); no real trajectory or bundle content is included.