Context
benchmarks/programbench currently contains substantial benchmark-specific inference and evaluation logic for cleanroom task images and ProgramBench submission/evaluation behavior. A Harbor adapter is needed before this benchmark can be removed from the bespoke OpenHands path.
Proposed scope
- Create a Harbor adapter/dataset for ProgramBench tasks.
- Preserve cleanroom task image usage, workspace submission format, and upstream evaluation semantics.
- Support resource settings appropriate for ProgramBench long rebuild tasks.
- Run parity against the current
programbench-infer/programbench-eval workflow.
Acceptance criteria
- ProgramBench tasks can be run through Harbor.
- Evaluation output matches upstream ProgramBench expectations.
- Parity results and resource caveats are documented.
Context
benchmarks/programbenchcurrently contains substantial benchmark-specific inference and evaluation logic for cleanroom task images and ProgramBench submission/evaluation behavior. A Harbor adapter is needed before this benchmark can be removed from the bespoke OpenHands path.Proposed scope
programbench-infer/programbench-evalworkflow.Acceptance criteria