[PENDING: #694] test: pin reported latency metrics to injected values

[Bslabe123]

Part of #726. Row in #606: Integration / "Metric accuracy gate, latency (#726)".

The gap

inference-perf reports four latency metrics per request: time to first token (TTFT), inter-token latency (ITL, the gap between streamed chunks), time per output token (TPOT) and request latency. The only test that checks these numbers against a known input is e2e/tests/test_metrics_fidelity.py (#614), which runs in the e2e tier against llm-d-inference-sim with fixed TTFT and inter-token latency, so it needs the sim binary and is not in the per-PR unit suite. Everything in tests/required either hands the latency math pre-made timestamps or only checks that a metric is present and plausible.

That is how #564 happened: #410 changed how streamed chunks were tokenized, ITL was under-reported, and nothing caught it for about two months.

What this adds

One new file, tests/required/integration/test_latency_metric_accuracy.py, 7 tests. Each:

1. Starts a fake OpenAI-compatible server (FakeOpenAIServer from fix: anchor TTFT to the first reasoning-channel token #694) that streams five word-chunks over real HTTP with scripted pauses: 250 ms to first token, then gaps of 400, 100, 400, 100 ms.
2. Runs a normal inference-perf request against it through the real client, streaming parser, metric collector and report generator. Only the model server is fake.
3. Reads TTFT, ITL, TPOT and request latency from the generated report and checks each against the injected pauses.

The reference values are the fake server's own per-chunk send timestamps. Server and client share one process and clock, so each reported number is compared to when the bytes actually left, not to how long asyncio.sleep was asked for. Tolerances (50 ms delivery, 100 ms scheduling) are documented in the file; 100 vs 400 ms gaps stay separable.

The seven checks:

• the fake server really pauses (so nothing below passes by accident)
• TTFT matches the 250 ms first-token delay
• changing the first-token delay moves reported TTFT by the same amount
• ITL min, max, mean and median match the 400/100 ms gaps
• TPOT excludes the first-token wait and is below normalized TPOT
• request latency is bounded both ways by the injected total
• the re-tokenization branch where [bug] Incorrect output length for neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8 #564 lived does not halve ITL or emit zero-length intervals

Compared with #614: runs in tests/required on every PR with no external binary; the reference is the send timestamp, not the configured sleep, so each value is bounded on both sides; the gaps are uneven (400/100 ms) so ITL min, max, mean and median are each checked, where #614 uses a constant gap and asserts only the median; and the report-time re-tokenization branch is exercised directly rather than sidestepped with use_server_output_tokens. The tokenizer here maps one word to one token, so the multi-token-per-chunk case #614 notes (zero-length gaps dragging the ITL mean down) is covered by neither test.

Breaking summarize_requests locally four ways (the #564 re-tokenization, TTFT anchored to end of stream, first-token wait folded into TPOT, flat average as ITL) fails the matching test each time. Product code is unchanged.

Status

Stacked on #694 (base 3b17a21), which supplies fake_openai_server.py. Once #694 merges the diff shrinks to this file and the [PENDING] prefix comes off.

Verified locally on #694: 7 new tests pass, pdm run validate clean, unit suite 892 passed / 15 skipped. About 10 seconds, almost all injected sleep.

Notes: the #606 row says this "requires giving MockModelServerClient a controllable response profile"; that mock is unary-only with no chunk timeline and would bypass the HTTP client and parser, so the row should read "uses the FakeOpenAIServer fake from #694". Coordinated-omission correction (adjusting latency by schedule_delay) is out of scope: a metric-definition question, open on #726 after v0.7.0.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Bslabe123

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [Bslabe123]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment