Context
benchmarks/commit0 does not appear to have a corresponding Harbor registry dataset/adapter yet. To fully rely on Harbor for benchmark execution, Commit0 needs a Harbor adapter with parity against the current OpenHands benchmark implementation.
Proposed scope
- Create a Harbor adapter/dataset for Commit0 tasks.
- Preserve task setup, image/build behavior, verifier semantics, and current default dataset/split assumptions.
- Validate Oracle/reference behavior first, then compare an OpenHands/OpenHands SDK agent run against the existing harness.
- Document parity results and known exclusions.
Acceptance criteria
- Commit0 tasks can be run with
harbor run or an equivalent Harbor job config.
- Parity results are recorded against the existing
benchmarks/commit0 workflow.
- The OpenHands wrapper can delegate Commit0 execution to Harbor.
Context
benchmarks/commit0does not appear to have a corresponding Harbor registry dataset/adapter yet. To fully rely on Harbor for benchmark execution, Commit0 needs a Harbor adapter with parity against the current OpenHands benchmark implementation.Proposed scope
Acceptance criteria
harbor runor an equivalent Harbor job config.benchmarks/commit0workflow.