Mapping ReactBench tasks/results onto EvalPort's TestCase/Grader/Result format

[adhabnr-ux]

Hi — really like the two-gate design here (fail-to-pass Playwright/vitest spec and no new React Doctor issues vs. baseline). The clean-room verifier split and the STYLE_RULES/NEW_ISSUE_RULE_BANDS false-positive bands in grader.mjs are the kind of detail that makes a bench trustworthy rather than just impressive-looking.

I maintain EvalPort, an open interchange format (TestCase/Grader/Result/ResultSet/GraderResult) for portable eval data — the idea being a suite or a run's output can move between DeepEval, Inspect AI, Braintrust, etc. without hand-writing a converter each time. ReactBench's shape maps onto it pretty directly, and I think it'd be a genuinely useful adapter for people who want to fold your reward.json output into a mixed-bench leaderboard alongside other evals, so wanted to float the mapping before attempting a PR.

Task → TestCase (from task.toml + instruction.md):

{
  "id": "hello-react",
  "input": "<instruction.md contents>",
  "graders": ["gr_test_suite", "gr_react_doctor"],
  "tags": ["smoke", "react", "hello-world", "clean-room"],
  "metadata": {
    "origin": "<task.toml [metadata].origin, e.g. PR + base commit sha>",
    "difficulty": "easy",
    "category": "smoke",
    "task_format": "harbor"
  }
}

Both gates are opaque scripted checks (a vitest/Playwright run, then a react-doctor diff against a baked baseline) rather than string/regex/schema comparisons, so type: "code" is the honest fit rather than forcing one of the deterministic grader types:

[
  { "id": "gr_test_suite", "type": "code",
    "params": { "entrypoint": "tests/test.sh", "cmd": "<config.json test.cmd>" } },
  { "id": "gr_react_doctor", "type": "code",
    "params": { "mode": "<config.json rd.mode>", "scan_root": "<config.json rd.scan_root>" },
    "description": "no new react-doctor footguns vs. baseline scan" }
]

reward.json → Result — this is the part worth flagging rather than silently papering over: your reward is testOk && (rdOk || telemetryOnly), a gate, not a weighted average, so it doesn't collapse cleanly into EvalPort's default score-weighting. I'd keep both signals as separate GraderResults and let Result.passed carry the actual gated verdict:

{
  "test_case_id": "hello-react",
  "passed": true,
  "grader_results": [
    { "grader_id": "gr_test_suite", "type": "code", "score": 1, "passed": true },
    { "grader_id": "gr_react_doctor", "type": "code", "score": 1, "passed": true }
  ],
  "metadata": { "reward_json": { "reward": 1, "tests": 1, "react_doctor": 1 } }
}

...rolled up into one ResultSet per Harbor job (runner: { name: "harbor" }, results: [...] across jobs/<timestamp>/<task>__*/verifier/reward.json).

For precedent on scope/shape: adapters/autogen-openeval-adapter is the pattern EvalPort's own CONTRIBUTING guide points people to — standalone package, to_openeval()/from_openeval(), tests against the real validator. A reactbench-openeval-adapter would follow the same shape on the JS/TS side, against sdk/typescript's real TestCase/Grader/Result/ResultSet types (sdk/typescript/src/types.ts) — I'd build and maintain it there rather than asking for anything to land in this repo, so this is really just "does the mapping look right to you / is type: code for both gates the right call, or would you rather see tests and react_doctor broken out differently" before I write it.

Open to being told this doesn't matter to you at all, too — no worries either way, just wanted to check before spending the time.

— Sahi, independent contributor (not affiliated with this project)