Summary
aiperf handles simple request-response evals but does not support what we'd call "agentic evals": multi-step, stateful evaluations where the model issues a tool call, the tool executes, and the resulting world-state is fed back to the model in a loop. As aiperf matures toward being a single destination for model testing, this is the one category that currently forces users out to an external tool.
Context
aiperf already covers the testing needs that matter today and is maturing quickly:
- Performance testing — inference performance, acceptance length for speculative-decoding models.
- Accuracy / intelligence evals — point aiperf at a hosted inference endpoint and get an accuracy measurement for a given eval. Multiple-choice accuracy evals (e.g., MMLU) are available today, with more in the repo moving through PR review.
These accuracy evals have been genuinely useful for Model Optimizer / quantization work: validating that a quantized model stays within an accuracy range of the original. The tool's momentum and responsiveness are a big part of why it's the default place to go.
The gap
Today's evals are essentially request-response: send a multiple-choice question, get a response, parse the last line, match against the question-answer pair. Simple and minimal.
The evals that increasingly matter are agentic: a multi-step, stateful loop where:
- The model receives a question and decides it needs a tool call.
- The tool call executes somewhere.
- The results — the changed state of the world — are represented back to the model.
- The model decides the next step and issues another tool call.
- The cycle repeats.
aiperf cannot run this multi-step stateful loop. Under the assumption that aiperf wants to be the one-stop shop for model testing, this is the area where it doesn't solve the problem, which forces use of an external tool.
Current workaround
Two parts:
- Run the agentic eval externally via Harbor. Harbor handles the multi-step tool-call loop aiperf can't. The choice of Harbor was essentially a coin flip — it solved the problem first. Comparable agent-eval suites exist (e.g., Inspect AI), and Nemo Gym is noted as a potential Dynamo-built equivalent. No strong preference among them.
- Translate traces back into aiperf. A small custom script converts Harbor's request-response traces into the mooncake trace format, which aiperf can ingest to do request-response replay against a hosted inference endpoint. This enables testing realistic multi-step / multi-turn conversations against inference. The reporter is willing to upstream or contribute this translator.
Possible directions
Floated as options, not prescriptions — the implementation is entirely the team's call:
- Make Harbor an optional/extra Python dependency of aiperf (Harbor exposes a Python API), with optional pass-through to Harbor.
- Build a first-party agentic-eval capability within Dynamo (roughly what Nemo Gym is positioned as).
- Independently / additionally: accept the trace translator (Harbor → mooncake format) as an upstream contribution.
The team also floated simple Harbor ↔ aiperf interoperability so users wouldn't have to maintain a translation layer themselves; the reporter did not push back, but explicitly declined to prescribe any solution.
Summary
aiperf handles simple request-response evals but does not support what we'd call "agentic evals": multi-step, stateful evaluations where the model issues a tool call, the tool executes, and the resulting world-state is fed back to the model in a loop. As aiperf matures toward being a single destination for model testing, this is the one category that currently forces users out to an external tool.
Context
aiperf already covers the testing needs that matter today and is maturing quickly:
These accuracy evals have been genuinely useful for Model Optimizer / quantization work: validating that a quantized model stays within an accuracy range of the original. The tool's momentum and responsiveness are a big part of why it's the default place to go.
The gap
Today's evals are essentially request-response: send a multiple-choice question, get a response, parse the last line, match against the question-answer pair. Simple and minimal.
The evals that increasingly matter are agentic: a multi-step, stateful loop where:
aiperf cannot run this multi-step stateful loop. Under the assumption that aiperf wants to be the one-stop shop for model testing, this is the area where it doesn't solve the problem, which forces use of an external tool.
Current workaround
Two parts:
Possible directions
Floated as options, not prescriptions — the implementation is entirely the team's call:
The team also floated simple Harbor ↔ aiperf interoperability so users wouldn't have to maintain a translation layer themselves; the reporter did not push back, but explicitly declined to prescribe any solution.