Optional --format openeval output: mapping scenario/expect onto a portable TestCase/Result schema

[adhabnr-ux]

Hi Rubén — I'm the author of EvalPort (https://github.com/adhabnr-ux/evalport), an open, framework-agnostic schema for portable LLM/agent eval data: TestCase (one input + expected behavior + graders), Grader (an evaluation rule), Result/GraderResult (per-test-case, per-grader outcomes), and ResultSet (a full run). The goal is that eval data produced by one harness can be read, re-graded, or diffed by another without hand-translating formats.

langdrift's scenario/expect model maps onto it more cleanly than most agent harnesses I've looked at, because it's already narrowly typed rather than free text:

• A Scenario (id, agent, locales: Record<string, ScenarioLocale>) → an EvalSuite, one TestCase per locale (id: "${scenario.id}_${locale}", input, metadata.locale).
• expect.toolCall{name, arguments} and expect.toolCalls (ordered sequence) → EvalPort already has TestCase.expected_tools: string[] for exactly this (see the AutoGen adapter note below); argument-level matching — including your oneOf scalar matcher — doesn't fit that field alone, so it'd live in a Grader with type: "custom" and params carrying the assertion.
• expect.noToolCall{names} and expect.responseLanguage → also natural custom graders (params.kind: "forbidden_tool" / "response_language").
• LocaleResult{status, failureMode, detail} → a Result with one GraderResult per assertion. Your 8-value FailureMode taxonomy (no_tool_call / wrong_tool / wrong_argument / missing_argument / forbidden_tool / wrong_sequence / wrong_language / target_error) is more granular than anything in the spec today and would sit fine in Result.metadata.failure_mode or GraderResult.reason.
• RunResult{scenarioId, target, iterations, results} → a ResultSet.

Concrete sketch, using the fr locale from your README's example:

// EvalPort TestCase
{
  "id": "refund_request_fr",
  "input": "J'ai été facturé deux fois. Pouvez-vous me rembourser un paiement?",
  "expected_tools": ["create_refund_ticket"],
  "graders": [
    { "id": "tool_call_match", "type": "custom",
      "params": { "name": "create_refund_ticket", "arguments": { "reason": "duplicate_charge" } } },
    { "id": "response_script_fr", "type": "custom",
      "params": { "kind": "response_language", "locale": "fr" } }
  ],
  "metadata": { "scenario_id": "refund_request", "locale": "fr", "agent": "support" }
}

// EvalPort Result, if the actual arg came back as free text
{
  "test_case_id": "refund_request_fr",
  "passed": false,
  "grader_results": [
    { "grader_id": "tool_call_match", "type": "custom", "score": 0, "passed": false,
      "reason": "expected argument reason=duplicate_charge, got \"doble cargo\"" }
  ],
  "metadata": { "failure_mode": "wrong_argument" }
}

Precedent for the shape: adapters/autogen-openeval-adapter does the same expected_tools round-trip for AutoGen's task/result objects — https://github.com/adhabnr-ux/evalport/tree/main/adapters/autogen-openeval-adapter — same idea, different source format.

Given langdrift already supports --format text|json|markdown, this would most naturally land as an additional --format openeval (or a small standalone to_openeval() adapter package, same pattern as the AutoGen one — it wouldn't need to touch langdrift's "CLI, not a library" design choice) rather than any change to the core scenario/assertion model. Happy to sketch a PR if it's useful — otherwise just flagging the mapping in case it fits somewhere down the line. No pressure either way.

— Sahi, independent contributor (not affiliated with this project)