Benchmark: ~49% median wall-time overhead vs upstream pi on a local Qwen3.8-27B (0.7.3)

[gltanaka]

Sharing a small third-party benchmark in case it is useful. Framing up front: this is about harness overhead, not about which agent is better. My sample is too small to support a correctness claim and I say so explicitly below.

Setup

• MacBook Pro, Apple M4 Max (16 cores), 128GB
• macOS 26.5.2, oMLX 0.6.1
• Model: Qwen3.8-27B-MLX-oQ8e running locally, 98,304-token context, 32,768 max output, temperature 0.6
• Both harnesses hit the same local oMLX OpenAI-compatible endpoint under the same 20-minute wall-clock cap
• Four OpenBench coding fixtures, two trials each, fresh macOS Seatbelt-sandboxed environment per run
• Versions: Prime Agent 0.7.3, upstream pi 0.73.1

Staleness note: v0.7.4 shipped 2026-08-19, after these runs. At your release cadence this may already be out of date.

Wall time

pi was faster on all three fixtures that produced signal:

Fixture	pi 0.73.1	Prime Agent 0.7.3	Delta
make-ci-green	195.6s	300.2s	+53%
add-feature	673.2s	860.2s	+28%
taskflow	878.2s	1,195.4s	+36%

Totals: 5,894.9s vs 7,112.3s (+20.7%). Median: 711.2s vs 1,060.5s (+49.1%). Completed-response output tokens: 24,905 vs 28,166 (+13.1%).

Three-for-three directional consistency is why I think the overhead signal is worth reporting even at this sample size. The extra tokens suggest at least part of it is additional model round-trips rather than harness-side latency, but I did not instrument that well enough to separate the two.

Correctness: no differentiation

External passes 6/8 vs 5/8, mean external score 0.825 vs 0.735, paired 1 win and 7 ties. That difference comes down to a single fixture and is within noise at n=4. I would not draw a quality conclusion from it, and I am reporting it only for completeness.

Both harnesses failed webcore

Both timed out at the 1,200s cap with no file changes. This was the result I found most interesting. Against a local 27B, an agentic loop means repeated long-context prefill, and prefill is where Apple Silicon local inference is weakest. This may say more about local model serving than about either harness, but if running against local endpoints is a use case you care about, the interaction seems worth knowing about.

Limitations

• 4 fixtures, 2 trials each. Small.
• Balanced but not fully randomized execution order, single session, so thermal and cache drift are possible.
• Token totals exclude partial streams from timed-out cells.
• Native tool schema differences between the two harnesses were not equalized.
• One retry of the final cell, triggered by a cleanup timeout rather than by score.

Method, raw results, and artifact hashes: https://github.com/promptdriven/pdd/blob/research/omlx-qwen38-oQ8e-benchmark/research/omlx-qwen38-pi-prime/README.md (disclosure: my company's repo).

Happy to rerun on 0.7.4+ with more fixtures and more trials. If there is a recommended configuration for local or OpenAI-compatible endpoints that I got wrong, I would rather fix the setup and re-measure than leave a misleading number sitting around.