Skip to content

Add a built-in profiling callback for per-invocation timing analysis #90

Description

@acere

Problem

When investigating where time is spent during an LLMeter run, users currently need to write custom ad-hoc callbacks (as seen in dev notebooks) to capture per-request phase timings. There is no built-in way to:

  1. Break down request time into server-side phases — specifically separating prefill (TTFT) from generation (decode), and distinguishing server time from client-side runner overhead.

  2. Capture per-invocation metadata alongside timing — cache hit status, reasoning token counts, retry counts, and throughput metrics — in a structured format suitable for analysis.

  3. Persist profiling data to disk for post-hoc analysis without needing the live callback or endpoint.

  4. Visualize profiling results with ready-made plots (phase breakdown, request timeline/concurrency, throughput over time, TPOT distribution, etc.).

Expected Behavior

A built-in ProfileCallback that:

  • Hooks into the existing callback lifecycle (before_invoke / after_invoke / after_run)
  • Captures detailed per-invocation profiles (timing phases, token counts, cache info, retries)
  • Produces an aggregated report with time accounting and percentile statistics
  • Saves structured data (JSON + JSONL) alongside run results
  • Contributes summary stats to result.stats for programmatic access
  • Pairs with plotting utilities for common profiling visualizations

Use Case

Understanding where wall-clock time goes in a benchmark run — especially distinguishing server-side prefill and decode from client-side overhead — and identifying patterns like cache hit impact on TTFT, throughput degradation over time, or concurrency effects.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions