Problem
When investigating where time is spent during an LLMeter run, users currently need to write custom ad-hoc callbacks (as seen in dev notebooks) to capture per-request phase timings. There is no built-in way to:
-
Break down request time into server-side phases — specifically separating prefill (TTFT) from generation (decode), and distinguishing server time from client-side runner overhead.
-
Capture per-invocation metadata alongside timing — cache hit status, reasoning token counts, retry counts, and throughput metrics — in a structured format suitable for analysis.
-
Persist profiling data to disk for post-hoc analysis without needing the live callback or endpoint.
-
Visualize profiling results with ready-made plots (phase breakdown, request timeline/concurrency, throughput over time, TPOT distribution, etc.).
Expected Behavior
A built-in ProfileCallback that:
- Hooks into the existing callback lifecycle (
before_invoke / after_invoke / after_run)
- Captures detailed per-invocation profiles (timing phases, token counts, cache info, retries)
- Produces an aggregated report with time accounting and percentile statistics
- Saves structured data (JSON + JSONL) alongside run results
- Contributes summary stats to
result.stats for programmatic access
- Pairs with plotting utilities for common profiling visualizations
Use Case
Understanding where wall-clock time goes in a benchmark run — especially distinguishing server-side prefill and decode from client-side overhead — and identifying patterns like cache hit impact on TTFT, throughput degradation over time, or concurrency effects.
Problem
When investigating where time is spent during an LLMeter run, users currently need to write custom ad-hoc callbacks (as seen in dev notebooks) to capture per-request phase timings. There is no built-in way to:
Break down request time into server-side phases — specifically separating prefill (TTFT) from generation (decode), and distinguishing server time from client-side runner overhead.
Capture per-invocation metadata alongside timing — cache hit status, reasoning token counts, retry counts, and throughput metrics — in a structured format suitable for analysis.
Persist profiling data to disk for post-hoc analysis without needing the live callback or endpoint.
Visualize profiling results with ready-made plots (phase breakdown, request timeline/concurrency, throughput over time, TPOT distribution, etc.).
Expected Behavior
A built-in
ProfileCallbackthat:before_invoke/after_invoke/after_run)result.statsfor programmatic accessUse Case
Understanding where wall-clock time goes in a benchmark run — especially distinguishing server-side prefill and decode from client-side overhead — and identifying patterns like cache hit impact on TTFT, throughput degradation over time, or concurrency effects.