[PENDING: #735] test: side-by-side offered-load parity against vllm bench

[Bslabe123]

Part of #481. Runs inference-perf and vllm bench serve against the same recording server and asserts the two tools offered the same workload, which nothing in this repo checks today.

Row in #606: "End-to-end / Peer-benchmark skew (#481)", type sim.

Stacked on #735 (test/load-shape-accuracy-633, 9b8c938); its commit shows in this diff until it merges. Rate and concurrency here are computed by #735's e2e/utils/load_shape.py, so the e2e tier has one definition of each. #735 reads the client's own timestamps; this reads the server's.

Why this is needed

Two recent skew reports were first read as inference-perf bugs and both decomposed into workload non-equivalence (the legacy range_ratio footgun, 0.0 meaning Uniform[0, len]; length-sampling, max-model-len and warmup deltas). Nothing could ask whether the two tools were even asked for the same thing.

This harness never merge-gates, by design. A delta can move from an inference-perf regression, a peer-tool change, or workload non-equivalence, and a gate cannot attribute which. The workload mapping is ours and asserted loudly; peer results are not ours and never asserted on.

What the test does

• The oracle is an absorber (e2e/utils/absorber.py), an OpenAI-compatible aiohttp server that records every request (arrival, reply-finished, body) while streaming at a configured TTFT and inter-chunk interval. The recorded requests are the offered workload; no stored baseline.
• Per tool, against expected.yaml: exact request count, max_tokens multiset, stream and ignore_eos flags, per-request prompt tokens under the shared tokenizer (with tolerance), and either delivered rate within rate_tolerance(n, arrival) or delivered concurrency via assert_delivered_concurrency (never above C, time-weighted plateau mean within half a slot).
• Tool against tool: the same quantities compared directly; a failure names the knob that leaked.
• No rate tolerance numbers. Each tool declares its spacing (tools.<tool>.arrival: constant|poisson); the tolerance follows from the request count. Tightening a case means raising num_requests.
• No reported-metric parity: no TTFT, ITL or throughput number is asserted.
• Warmup traffic is declared, not absorbed: leading_extra_requests trims the earliest N arrivals for one tool; a nonzero value is itself a parity finding.

The drop-in case contract (the reusable part)

e2e/tests/parity/cases/<name>/
  inference-perf.yaml   # verbatim config; harness overwrites only server.base_url + tokenizer path
  vllm-bench.args       # verbatim args, one per line, # comments ok; harness appends
                        # --base-url --model --tokenizer --save-result --result-filename
  expected.yaml         # invariants, absorber pacing, per-tool arrival + leading_extra_requests

Adding a case touches no Python. Starter cases: a_fixed_rate (40 QPS, 15s, 600 requests: the smallest count at which the Poisson side earns a useful 16% bound, matching #735) and b_fixed_concurrency (48 requests at concurrency 8).

Verified against the pinned vllm source, not by running it

Read from tag v0.10.0: benchmark() always runs one warmup prompt first (leading_extra_requests: 1); --random-range-ratio 0 gives fixed lengths; --max-concurrency is a semaphore; --random-input-len is reduced by the special-token count (hence prompt_tokens_rel_tol); --request-rate at default --burstiness 1.0 is Poisson (arrival: poisson); --backend is vestigial, so args files corrected to --endpoint-type openai; stream: true is hardcoded, so stream: false is not expressible on the vllm leg. transformers<5 is required (5.13 dropped all_special_tokens_extended).

Status

Verified locally: pdm run validate clean; pdm run test 898 passed, 16 skipped; pdm run pytest e2e/tests/parity 3 passed, 4 skipped. Against the absorber: case a rate 40.01/s on a 5% budget; case b plateau mean 7.88 of 8 on a half-slot budget (the server sees the client's turnaround gap; client-side timestamps do not); C-1 and C+1 both fail.

Not verified:

• The vllm leg has never been executed. No vllm in this environment; all 4 skips are vllm-side. Source reading does not prove the run completes, that the absorber satisfies vllm bench's parsing, or that both tools land inside tolerance.
• prompt_tokens_rel_tol: 0.15 is sized by argument, not observation. VLLM_PIN_DATE still warns, deliberately.
• No CI wiring (and never merge-blocking when wired). aiperf not started.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Bslabe123

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [Bslabe123]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment