Proposal: EvalPort exporter for results/completions (ResultsUploader-based, opt-in)

[adhabnr-ux]

Context

I maintain EvalPort, an open interchange format (TestCase/Grader/Result/ResultSet/GraderResult, JSON Schema + Python/TS SDKs) for moving evaluation data between eval tools without hand-rolled glue for every pair. I looked through eval-framework's actual result model (contract.py, shared/types.py, metrics/base.py, result_processors/) rather than guessing from the README, and the shapes line up closely enough that I think an exporter is a reasonable, low-risk addition — not a redesign ask.

Where the shapes line up

• eval_framework.contract.Sample (id, subject, messages, ground_truth, possible_completions, context) maps directly onto EvalPort's TestCase (id, input, expected_output, context, metadata).
• eval_framework.metrics.base.MetricResult (metric_name, value, higher_is_better, llm_judge_prompt, llm_judge_response, code_execution_trace, error) maps onto EvalPort's GraderResult (grader_id, type, score, passed, reason, metadata) almost field-for-field — llm_judge_response is a natural fit for reason when the metric is an LLM-judge one.
• eval_framework.result_processors.base.Result — the per-(id, subject, metric_name) row written to results.jsonl — is effectively one EvalPort Result + one GraderResult, grouped by (id, subject) across all metrics run for that sample.
• eval_framework.result_processors.base.ResultsUploader (the ABC HFUploader/WandbUploader already implement, with upload(llm_name, config, output_dir) -> bool) is exactly the extension point this would use — an EvalPortExporter(ResultsUploader) that reads output.jsonl (Completions) and results.jsonl (Results) from output_dir and writes an EvalPort ResultSet, no changes to ResultProcessor or the eval loop itself.

Sketch

# eval_framework/result_processors/evalport_exporter.py
from collections import defaultdict
from openeval.types import Result as EPResult, GraderResult, ResultSet, OPENEVAL_VERSION

from eval_framework.result_processors.base import Result, ResultsUploader
from eval_framework.shared.types import Completion

def _grader_type(metric_class_name: str, metric_name: str) -> str:
    if metric_class_name == "Accuracy":
        return "exact_match"
    if "Judge" in metric_class_name:  # ChatbotStyleJudge, InstructionJudge, ...
        return "llm_judge"
    return "custom"  # BLEU/ROUGE/F1/etc — no lossless native equivalent

def to_result_set(
    completions: list[Completion], results: list[Result], *, suite_id: str, run_id: str, started_at: str,
) -> ResultSet:
    by_sample: dict[tuple[int, str], list[Result]] = defaultdict(list)
    for r in results:
        by_sample[(r.id, r.subject)].append(r)

    completion_by_id = {(c.id, c.subject): c for c in completions}
    ep_results = []
    for (sid, subject), rs in by_sample.items():
        completion = completion_by_id.get((sid, subject))
        grader_results = [
            GraderResult(
                grader_id=r.metric_name,
                type=_grader_type(r.metric_class_name, r.metric_name),
                score=r.value,
                passed=bool(r.value) if r.value is not None else False,
                reason=r.llm_judge_response,
                metadata={"metric_class_name": r.metric_class_name, "key": r.key, "subject": subject},
            )
            for r in rs
        ]
        ep_results.append(EPResult(
            test_case_id=str(sid),
            passed=all(gr.passed for gr in grader_results),
            grader_results=grader_results,
            actual_output=completion.completion if completion else None,
            error=({"message": completion.error.message} if completion and completion.error else None),
        ))

    return ResultSet(
        version=OPENEVAL_VERSION, suite_id=suite_id, run_id=run_id,
        started_at=started_at, results=ep_results,
    )


class EvalPortExporter(ResultsUploader):
    def upload(self, llm_name, config, output_dir) -> bool:
        ...  # load_responses()/load_metrics_results() via ResultsFileProcessor, call to_result_set(), write JSON

The one deliberately open question is passed: eval-framework doesn't carry a pass/fail threshold anywhere in Result/MetricResult — only a continuous value + higher_is_better — so passed above is a placeholder (bool(value)), not a real derivation. Happy to either leave that mapping to the caller (pass a threshold_fn per metric name) or omit passed semantics from this exporter's docs as "unknown/not native to the source format" rather than inventing a number. That mirrors how the lm-eval-harness adapter in EvalPort handles metrics with no honest native grader equivalent — it maps them to custom with the real metric name preserved (params.handler = "lm-evaluation-harness:<metric_name>") rather than fabricating one. eval-framework and lm-eval-harness are structurally close (task/benchmark abstraction, per-document samples, HF/W&B result upload), so that adapter is probably the closest existing precedent for how this one would end up shaped.

What I'm asking

Not proposing a spec change on your end — just: would a PR adding eval_framework/result_processors/evalport_exporter.py (as a ResultsUploader, opt-in like the HF/W&B uploaders, evalport-sdk as an optional extra) be something you'd take? If the passed-derivation question above has an opinion from the maintainers (e.g. "don't synthesize passed at all, only score"), I'd rather build to that than guess. Can put up a draft PR with real e2e output (eval_framework --task-name MMLU ... --output-dir ... piped through the exporter, validated against openeval.validate.validate_result_set) if that's useful before a full PR.

— Sahi, independent contributor (not affiliated with this project)