Skip to content

Port Commit0 benchmark to Harbor #721

Description

@neubig

Context

benchmarks/commit0 does not appear to have a corresponding Harbor registry dataset/adapter yet. To fully rely on Harbor for benchmark execution, Commit0 needs a Harbor adapter with parity against the current OpenHands benchmark implementation.

Proposed scope

  • Create a Harbor adapter/dataset for Commit0 tasks.
  • Preserve task setup, image/build behavior, verifier semantics, and current default dataset/split assumptions.
  • Validate Oracle/reference behavior first, then compare an OpenHands/OpenHands SDK agent run against the existing harness.
  • Document parity results and known exclusions.

Acceptance criteria

  • Commit0 tasks can be run with harbor run or an equivalent Harbor job config.
  • Parity results are recorded against the existing benchmarks/commit0 workflow.
  • The OpenHands wrapper can delegate Commit0 execution to Harbor.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions