Portable test suites for evaluating Agent orchestrations (GroupChatOrchestration, etc.) — EvalPort interop?

[adhabnr-ux]

Hi SK community,

Looking through python/samples/getting_started_with_agents/multi_agent_orchestration/step3_group_chat.py, the orchestration pattern is clean: build a GroupChatOrchestration from a list of ChatCompletionAgents and a RoundRobinGroupChatManager, runtime.start(), await group_chat_orchestration.invoke(task=..., runtime=runtime), then await orchestration_result.get() for the final value. That's a nice, uniform shape — any orchestration (group chat, sequential, or your other managers) takes a task string in and produces a result out.

That uniformity made me wonder about testing it at scale: has anyone here wanted a portable way to define "given this task, the orchestration's output should satisfy X" test suites — the kind you'd want to run against an SK agent orchestration, and separately also run against a LangChain/AutoGen/raw-API pipeline doing the same job, to compare apples to apples?

I maintain EvalPort (https://github.com/adhabnr-ux/evalport), a small open spec (JSON Schemas under spec/schemas/ for TestSuite/TestCase/ResultSet/Grader, plus Python/TS SDKs) aimed at exactly that — not another eval runner, just a portable format so a test suite isn't locked to one framework's dataset shape.

Rough sketch of what an adapter over GroupChatOrchestration could look like:

async def run_suite(suite: evalport.TestSuite, orchestration, runtime) -> evalport.ResultSet:
    results = []
    for tc in suite.test_cases:
        res = await orchestration.invoke(task=tc.input, runtime=runtime)
        output = await res.get()
        results.append(evalport.Result(test_case_id=tc.id, actual_output=str(output)))
    return evalport.ResultSet(suite_id=suite.id, results=results)

Given your CONTRIBUTING guidance that plugins/connectors are best hosted outside the SK repo, I'd build and host this as its own small package rather than proposing it land in semantic_kernel/. Posting here first (per COMMUNITY.md) to check whether this is actually a gap for anyone, before spending time on it. Spec for reference: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

No pressure either way — genuinely just gauging interest.

[Ama Senevirathne (amasen02)]

This is a legitimate gap in the agent ecosystem. Right now, most teams end up building ad-hoc harness scripts or tying themselves into proprietary eval platforms that assume a single prompt-response LLM rather than a multi-agent state machine.

Having a portable, schema-driven spec like EvalPort for orchestrations would be very useful, especially when teams want to benchmark whether migrating a workflow from AutoGen/LangGraph to Semantic Kernel introduces regression.

A couple of architectural observations from running multi-agent evals in production:

1. Terminal Output vs. Trajectory Graders

For simple single-turn tasks, output = await res.get() works. But in GroupChatOrchestration, evaluating only the terminal string misses 80% of multi-agent failure modes (e.g., tool hallucinations that got glossed over by the summary agent, turn oscillations, or inefficient routing).

In Semantic Kernel, you can extract the full message history from the orchestration result:

# Extract the full conversation trajectory for multi-step grading
messages = await res.get_messages() 
trajectory = [
    {"role": m.role.value, "author": m.name, "content": m.content, "tool_calls": len(m.items)}
    for m in messages
]

If EvalPort's Result schema allows attaching metadata or an execution trace (trajectory: list[dict]), graders could evaluate:

• Turn efficiency (did it solve the task in <= 4 turns?)
• Tool selection precision (did it invoke the intended plugin?)
• Inter-agent consensus

2. Async Streaming / Cancellation Support

Production agent runs can stall or exceed timeouts. When wrapping orchestration.invoke(), ensure the adapter accepts a timeout or asyncio.timeout(tc.timeout_seconds) so a single wandering agent test case doesn't hang the entire suite.

3. Packaging Strategy

Keeping it in a standalone external package (e.g. evalport-semantic-kernel or pip install evalport[semantic-kernel]) is 100% the right move per the current SK contribution model. As long as the adapter depends only on semantic-kernel (or just adheres to the generic duck-typed invoke(task=...) callable protocol), it will be very easy for developers to drop into their CI/CD test pipelines.

[adhabnr-ux]

Really useful feedback — let me answer each point against EvalPort's actual schema rather than what I'd like it to support, since I don't want to promise anything the spec doesn't really have.

1. Trajectory vs. terminal-output grading. You're right that Result.actual_output won't do this on its own — it's typed as Optional[str], not a structured list, so it's meant for a single output string/summary, not a full message history. But both Result and each GraderResult carry a free-form metadata: Dict[str, Any] (see sdk/python/openeval/types.py), so the honest place for your trajectory list is result.metadata["trajectory"], with actual_output holding a short human-readable summary (e.g. the final message content) for anything that just wants a quick look. A GraderType of "custom" (or a descriptive non-standard type string — the spec explicitly allows that, it's treated like "custom" for validation purposes) then lets you write turn-efficiency / tool-selection / inter-agent-consensus graders that read metadata.trajectory directly. That logic genuinely belongs in the adapter, not the core spec — EvalPort doesn't (and I don't think should) prescribe multi-agent-specific grading semantics itself.

2. Async timeout/cancellation. Agreed this is a real adapter concern rather than something the spec needs to solve — but there's a nice hook for it already in the schema: TestCase.timeout_ms: Optional[int] exists today. So a Semantic Kernel adapter's invoke() wrapper can honestly read test_case.timeout_ms and enforce it with asyncio.wait_for(orchestration.invoke(...), timeout=test_case.timeout_ms / 1000) (or asyncio.timeout, Python 3.11+) per test case, rather than inventing a new field — one wandering agent then can't stall the whole suite, and the per-case override comes for free.

3. Packaging. Agreed, and it matches the convention every adapter in adapters/ already follows: a standalone <framework>-openeval-adapter package depending only on evalport-sdk plus the target framework (or nothing at all, when duck-typing the target's shape is enough — see agenteval-openeval-adapter/niceeval-openeval-exporter for two different takes on that trade-off). A generic duck-typed invoke(task=...) protocol is exactly the right shape for something like evalport-semantic-kernel.

I don't want to promise a full SK adapter here without actually building and testing it first (that's bitten worse proposals than mine), but the shape above — metadata.trajectory for the full history, TestCase.timeout_ms for the per-case timeout, a standalone package with a duck-typed invoke() — is concrete enough that I'd be glad to build it against GroupChatOrchestration specifically if there's interest. Would a first cut that handles the terminal-output + trajectory-in-metadata case, with the timeout wiring, be a useful starting point, or is there a more pressing orchestration pattern (e.g. SequentialOrchestration, HandoffOrchestration) worth targeting first?

[Harlock (capitaharlock)]

The str(output) line is where the apples-to-apples comparison starts to break down for multi-agent runs. Two orchestrations can produce equivalent final answers while differing materially in turn count, who spoke, tool calls, or termination reason, and none of that survives in actual_output.

I’d add a normalized trace slot to ResultSet, plus run metadata such as model/version/seed, and a grader mode that can report Improvement/Neutral/Regression against a baseline. Would you be open to making those first-class fields in the schema?

[solim (soul-sol)]

Portable suites for this are worth having, and I'd add one requirement that I only learned by
shipping a broken verifier: a suite that can only confirm the expected outcome is
indistinguishable from a function that returns a constant.

Concretely, three ways my own orchestration checks passed while being wrong:

1. Positive-only fixtures. I built a gate that judged whether an external listing existed.
   Eighteen fixtures, all green, and the real path was broken — every fixture was a case that
   should pass, so a checker hardcoded to "yes" would have scored identically. It only broke
   when I added a fixture that had to fail. For an orchestration suite this means each TestCase
   family needs a paired negative: an orchestration that must not satisfy X, run through the
   same grader.

2. The suite tests that the mapping executes, not that the mapping is right. I read an
   external field named activated as "in stock". Every test passed. The field actually meant
   "this option exists" — every size was activated: true and every size was sold out. No
   amount of schema conformance catches this; only a real-world contrast case does. Anywhere a
   grader interprets a runtime's field (turn count, tool-call status, a finish_reason), the
   spec should force the author to record how that meaning was established, because the same
   concept is named differently by each runtime — which is exactly the portability problem
   you're solving, one layer down.

3. A threshold silently discarded valid data. A grader with "output under 200 chars is a
   failure" turned 26 passing cases into 0, because short answers were correct answers. Worth a
   spec-level note that thresholds are assertions about the task, not about quality.

On the str(output) point raised upthread — agreed, and I'd go further: for multi-agent runs the
final string is the least comparable artifact. What survived contact for me was grading on the
terminal state of each participant, with three outcomes rather than two: finished (exit
status recorded by the launcher, never scraped from the transcript), needs-input (a question was
surfaced), unknown (alive, no output, no question). Two orchestrations that both "answered" but
where one left a worker in unknown are not equivalent runs, and a result-string grader scores
them identically.

One last distinction I'd bake into ResultSet if you can: FAILED ("provably didn't happen",
safe to re-run) versus UNVERIFIED ("may have happened, couldn't judge"). Collapsing them into
one failure bucket means a suite with retries re-executes side effects, and it also hides the
most interesting comparison between runtimes — how often each leaves you unable to tell.

The launcher/judgement scripts behind the above are MIT, in case a reference implementation of
the three-state part is useful: https://github.com/soul-sol/agent-watch

[adhabnr-ux]

Both of these are worth real answers rather than a thank-you and silence, so sorting by what actually changes vs. what's already covered:

Harlock (@capitaharlock) — normalized trace slot + run metadata + Improvement/Neutral/Regression grader mode. The trace slot and run metadata (model/version/seed) are exactly what metadata: Dict[str, Any] on both Result and ResultSet already exists for — I gave amasen02 the same answer above for trajectory specifically, and model/version/seed are the same shape of thing: adapter-specific, not universal across every EvalPort producer, so they belong in metadata rather than as new required top-level fields. That's a deliberate restraint the spec already applies elsewhere (e.g. it doesn't mandate a summary rollup formula either — producers compute what they need and put it where a consumer can find it).

The Improvement/Neutral/Regression grading mode is different in kind, though: that's a comparison against a baseline ResultSet, which is a consumer-side operation over two documents, not a property of one Result. EvalPort doesn't have (and I don't think should have) a schema-level notion of "this run vs. that run" — that's squarely adapter/tooling territory, the same reasoning as why group (an open RFC — Discussion #45) computes rollups on the consumer side rather than the schema mandating one.

solim (@soul-sol) — this one actually exposed a real gap, and I don't want to paper over it with metadata. The FAILED-vs-UNVERIFIED distinction doesn't fit today's schema cleanly: Result.passed=false conflates "provably didn't happen" with "couldn't tell," and Result.error is about execution/harness failures specifically (timeout/provider_error/runner_error), not "the agent ran, produced something, and I have no way to judge it." Those are different epistemic states with different consumer implications — you said it exactly right: collapsing them hides retry-safety and hides the most interesting cross-runtime comparison. That's the same shape of gap that became Discussion #47 (Result.constraint_violations, from a different maintainer's benchmark) — a real distinction the existing two-field model (grader_results + error) can't express, surfaced by someone actually running an eval against real fixtures, not by me guessing at the schema.

I don't want to bolt this onto a reply here — this project's own process is a proper two-week Discussion before anything gets implemented, the same path #45 and #47 went through. I'll open one on EvalPort specifically for the FAILED/UNVERIFIED terminal-state distinction, credit your agent-watch writeup and the three false-positive failure modes (positive-only fixtures, mapping-executes-vs-mapping-is-right, threshold-discards-valid-data) as the motivating case, and tag you on it. That last set of three, by the way, isn't a schema question at all — it's testing discipline any adapter (including whatever Semantic Kernel one comes out of this thread) should apply to its own conformance suite, and I'll hold myself to it when I build one.

Will link back here once that Discussion is up.

[adhabnr-ux]

Opened it: adhabnr-ux/evalport#49 — solim (@soul-sol) tagged and credited there directly.