This repository stores evaluation artifacts for the HWE-bench.
The current layout is:
jobs/
<repo>/
<model>/
pr-<number>/
agent/
verifier/
manifest.json
result.json
trial.log
results/
<repo>/
<model>/
eval/
final_report.json
logs/
run_evaluation.log
gen_report.log
eval_workdir/
<org>/<repo>/evals/pr-<number>/
jobs/ contains the model repair traces and raw patches. The primary patch file
for each case is verifier/model_patch.diff.
results/ contains the evaluator output produced from those patches, including
per-case simulation logs when available.
Included repositories are ibex, cva6, caliptra, rocketchip, and
xiangshan. Included models are gpt5.4, sonnet4.6, opus4.6,
kimi_k2.5, qwen3.6plus, and deepseek_v3.2.