Feature request: EvalPort import/export for CustomDataset (adapter already built and tested)

[adhabnr-ux]

What

EvalPort is an open interchange format for portable LLM evaluation datasets — a small JSON schema for test cases, graders, suites, and results, designed so an eval dataset built for one framework (DeepEval, Promptfoo, Ragas, Inspect AI, LangSmith, MLflow, and 20+ others) can be exported/imported by another without hand-rewriting it.

I've built and tested a standalone adapter, opencompass-openeval-adapter, that converts between EvalPort and OpenCompass's own CustomDataset (opencompass/datasets/custom.py) — the "bring your own eval data" path documented for users who aren't running one of the 100+ curated benchmarks. It's not a proposal for a specific code change to this repo; it's a working package you (or anyone using both projects) can use today, and I wanted to flag it here in case it's useful or worth pointing users to.

Why CustomDataset specifically

OpenCompass's 100+ curated benchmark loaders (gsm8k.py, mmlu.py, ...) each have their own bespoke schema, so there's no single object to build a generic adapter against. CustomDataset is the one surface OpenCompass itself designed to be generic — a flat list of dict rows (MCQ with single-uppercase-letter option columns, or free-text QA with a reference-answer column) — which maps directly onto EvalPort's own portable TestCase.

What it does

from opencompass.datasets.custom import CustomDataset
from opencompass_openeval_adapter import to_openeval, from_openeval, result_to_openeval

rows = CustomDataset.load(path="my_mcq_dataset.jsonl")
suite = to_openeval(rows, options=["A", "B", "C", "D"], suite_id="my_dataset")
# suite is now a portable EvalPort Suite, importable by any EvalPort-speaking tool

rows2 = from_openeval(suite)  # -> CustomDataset-ready rows, round-trips losslessly

predictions = [my_model(row) for row in rows2]
result_set = result_to_openeval(rows2, predictions, options=["A", "B", "C", "D"], ...)
# scored with your own real, installed OptionSimAccEvaluator/AccEvaluator directly

result_to_openeval() doesn't reimplement your scoring logic — it calls opencompass.datasets.custom.OptionSimAccEvaluator and opencompass.openicl.icl_evaluator.AccEvaluator directly, so the numbers are exactly what a real OpenCompass run would produce.

Grader mapping, if it's useful context

OptionSimAccEvaluator's fuzzy option-matching (exact letter → regex extraction → substring match → Levenshtein fallback) doesn't correspond to EvalPort's literal exact_match grader type, so it maps to custom with the real evaluator as the handler rather than overclaiming precision. AccEvaluator's scoring genuinely is exact string equality after str() coercion (confirmed by reading AccEvaluator._preprocess), so that one honestly does map to exact_match. Full reasoning and a test that cross-checks the per-item breakdown against AccEvaluator's own real aggregate accuracy are in the README.

Testing

28 tests, all passing locally against the real, installed opencompass package (0.5.3, current PyPI release) — real CustomDataset.load() calls against real .jsonl files, real OptionSimAccEvaluator/AccEvaluator scoring, and a full round trip back through OpenCompass's own loader. Not mocked.

Ask

No specific action needed from maintainers — this lives entirely outside your repo as an independent package, so there's nothing to merge. Flagging it here in case it's useful to link from docs, or in case anyone on the team has feedback on whether CustomDataset is the right integration point (vs. something in opencompass/summarizers/ for the aggregate-results side, for example). Happy to adjust the mapping if there's a better fit I'm missing.