DEP: FPM-based modeling

[simone-chen]

Summary

Adopt Forward Pass Metrics (FPM) modeling as an engine-latency provider for AISimulate. FPM-based modeling is a blackbox statistical model to predict the wall clock time of model.forward(batch) in one iteration of the scheduler.

A user first checks whether an exact (model, hardware, backend, backend version, parallelism, quantization, engine-configuration identity) is covered. If coverage is missing, an agent-guided workflow collects the missing pure-prefill and pure-decode latency surfaces on the target GPUs, validates them, and publishes a sealed dataset for local prediction and optional sharing.

Motivation

AISimulate's current model is bottom-up: it microbenchmarks kernels (GEMM, MoE, attention, comms) and analytically composes them into an iteration. Three problems:

• Slow onboarding — new architectures need ops registered and shapes collected first, an engineering-weeks process when models matter within a week. FPM inverts this: the engine that runs the model also measures it, so onboarding is just a collection campaign.

• Lost iteration-level effects — composition misses CUDA-Graph padding, bucket transitions, fallback behavior, and scheduler/output time. A whole-forward measurement captures all of it by construction.

• Broken parallelism assumptions — AIC assumes even traffic across DP/EP ranks; real deployments skew. FPM aggregates by the slowest rank, the one that actually determines completion.

Proposal

AISimulate owns collection planning, deployment generation, validation, publication, coverage reporting, and the whole-forward estimator. Dynamo owns the canonical per-iteration ForwardPassMetrics telemetry and self-benchmark lifecycle. The existing op_level model and all feature-disabled serving paths remain unchanged.

Dimensions of FPM

FPM consists of 3 dimensions for prefill-only steps and 2 dimensions for decode-only steps.

Prefill:

• num_prefill_requests
• sum_prefill_tokens: Total number of freshly computed KV tokens
• sum_prefill_kv_tokens: Total number of cached KV tokens, including prefix-cache hits and KV from earlier chunks.

Decode:

• num_decode_requests
• sum_decode_kv_tokens: Total context length across the decode batch

Note on mixed steps: Mixed steps account for a significant portion of aggregated serving workloads and are characterized by all 5 dimensions above. Collecting a complete 5-D latency surface would require an impractically large search space. Instead, the model estimates a mixed step as the prefill-surface latency plus the decode surface’s marginal cost after subtracting the decode pass baseline.

Lifecycle and failure handling

Agent-guided self-helped FPM collection

1. Coverage: Resolve the requested deployment and report covered, partial, or missing cells, the unsupported settings, and an estimated collection cost.
2. Authorization: Freeze the plan and require user confirmation before the estimated or observed cost exceeds the stated GPU-hour budget.
3. Collection: Generate the deployment artifacts, run Dynamo self-benchmark points, and checkpoint each cell. Regular model serving is not exposed from a collection workload.
4. Validation: Verify schema, plan and run identities, rank agreement, point coverage, resolved engine markers, finite latency, cleanup status, and artifact integrity.
5. Publication: Publish passed cells atomically. Repeating the same attempt is idempotent. A different attempt cannot overwrite or mix with an already sealed cell; first publisher wins and the conflict is reported.
6. Partial readiness: The default publication gate requires all frozen cells. Explicit partial publication may expose passed cells, but the campaign remains incomplete, records all missing cells, and exits nonzero. Coverage stays partial.
7. Prediction: Load only validated cells. Every result reports the prediction source, exact cell identity, dataset digest, and whether the point was measured or interpolated.
8. Failure, cancellation, and cleanup: Stop at a safe iteration boundary when possible, terminate engine process groups, remove synthetic KV state and temporary resources, preserve checkpoints, and refuse publication when cleanup or result integrity is uncertain. Resume must not rerun sealed cells or depend on pruned raw artifacts.
9. Upgrade and rollback: Backend or schema changes create a new identity. Old data remains inspectable but does not satisfy new coverage. Rollback is selecting forward_model=op_level or a previously validated dataset; no serving or deployment migration is required.

Rollout and validation

• Phase 0 — foundation, complete: Use the merged graph-aware Dynamo self-benchmark, shared Generator/collector contract, pure-deployment generator, FPM collector, Python whole-forward model, and compiled Rust port as the implementation baseline.
• Phase 1 — product-gated MVP: Add coverage reporting and the agent-guided bootstrap journey. The proposed evaluation matrix is MiniMax M2.7 on vLLM/H200 with TP4, TEP4, and DEP4 at 1k/1k, 8k/1k, and 1k/8k; DeepSeek-V4-Pro on vLLM/B200 with TP8 and TEP8 at 1k/1k and 8k/1k; and a separately gated Kimi K3 vLLM/GB200 agentic case. Use self-collected end-to-end results or an exact InferenceX match. Defer the agentic case until AISimulate supports the required AgentX data. Report FPM-level measured-versus-predicted error and end-to-end TTFT/TPOT error separately. Do not mark a cell supported without both dataset validation and the product accuracy gate.
• Phase 2 — controlled expansion: Add approved multinode, disaggregated, backend, model-feature, and workload coverage one dimension at a time. Each expansion must add identity fields or a new collection profile, compatibility tests, failure-path tests, and an evaluation result before release.

Requirements

1. With whole-forward modeling disabled, AISimulate and Dynamo must preserve existing op_level, serving, Planner, router, runtime, backend, and deployment behavior.
2. A coverage check must report the resolved engine identity, collection profile, covered domain, missing cells, unsupported settings, and estimated GPU-hours before collection begins.
3. A collection run must not exceed the user-approved GPU-hour budget without a new confirmation.
4. Every FPM cell must identify the exact model, hardware, backend and version, quantization, parallelism, backend selections, and a canonical representation of all other latency-affecting engine settings.
5. Dynamo self-benchmark must execute the same point on every participating rank, produce exactly one valid FPM per rank and point, and use the maximum positive rank wall time as the published latency.
6. Benchmark startup, timeout, cancellation, success, and failure paths must perform bounded synthetic-state and process cleanup. A cell with uncertain cleanup or missing result publication must fail.
7. The collector must reject mismatched schemas, identities, digests, row counts, run IDs, duplicate keys, non-finite or non-positive latency, phase-invalid coordinates, and mixed backend versions.
8. Dataset publication must be atomic and idempotent. A sealed cell must not be overwritten or combined with a different run identity.
9. Partial publication must require an explicit option, record every missing cell, keep coverage partial, and return a nonzero incomplete-run result.
10. The estimator must return exact measured values for exact keys and must keep Python and Rust predictions within 1% relative difference for approved in-domain interpolation fixtures. Error behavior must be symmetric.
11. The estimator must reject out-of-domain, wrong-identity, unsupported-feature, and corrupt-dataset queries. It must not silently extrapolate or switch to op_level.
12. Mixed-step and generation-only composition must have focused unit tests and end-to-end validation against measured serving results for every released model/topology cell.
13. Every prediction result must expose the model source, dataset digest, cell identity, measurement or interpolation status, and a structured reason when the model is not ready.
14. MVP evaluation must report FPM-level error and end-to-end TTFT/TPOT error by model, topology, and workload shape, including p50, p95, maximum error, and missing-result counts.
15. A backend version, collection-profile version, or incompatible FPM schema change must invalidate coverage until compatible data is collected or explicitly migrated and revalidated.
16. Collection and sharing artifacts must exclude prompts, token IDs, model weights, secrets, credentials, and user traffic. Upload to a central dataset must be explicit opt-in.
17. Rollback to op_level or a previously validated FPM dataset must require no change to the deployed Dynamo serving graph.

Risks

• Single-sample measurement noise: A cell may preserve transient noise and distort interpolation. Mitigation: use warmups, stable clocks and exclusive nodes, record collection timing and provenance, and validate with held-out median-of-multiple measurements. Residual risk remains until signoff thresholds are defined.
• Configuration drift: Unrecorded scheduler, cache, kernel, CUDA-graph, or backend settings can invalidate a cell. Mitigation: make each setting an identity field or part of a checked collection-profile digest; fail coverage when it differs.
• FPM accuracy does not imply end-to-end accuracy: Queueing, routing, cache behavior, and mixed-step composition can dominate TTFT or TPOT. Mitigation: gate release on both FPM-level and end-to-end evaluation, with Dynamo components retaining ownership of their state models.
• Mixed-step approximation bias: Marginal-decode composition can under- or over-estimate some regimes. Mitigation: track mixed-step residuals separately and add measured mixed modeling only if the MVP gate fails.
• Coverage and GPU cost growth: User-driven collection can recreate the same matrix explosion. Mitigation: collect only missing cells, show cost first, enforce budgets, reuse exact sealed cells, and make sharing opt-in.
• Partial, stale, or corrupt data: Interrupted writes or overlapping campaigns can make invalid data appear current. Mitigation: schema and identity checks, digest-sealed atomic pairs, checkpoints, first-publisher-wins, explicit partial state, and fail-closed loading.
• Resource leakage after failure: Collection jobs can leave engines, Pods, synthetic KV state, or temporary data behind. Mitigation: bounded process-group termination, controller-aware cleanup, verified deletion, resumable checkpoints, and no publication on ambiguous cleanup.

Open decisions

• Define the FPM-level and end-to-end accuracy thresholds required for product signoff.
• Set the default GPU-hour budget and the confirmation threshold for collection.
• Decide which engine settings are explicit identity fields and which are covered by the collection-profile digest.
• Confirm the Phase 1 evaluation matrix and when the AgentX-dependent Kimi K3 case becomes eligible.
• Define the access, retention, and promotion policy for opt-in shared datasets.

References

• Dynamo collection implementation: Graph-aware vLLM self-benchmark, PR #11509
• AISimulate modeling implementation: Python whole-forward FPM model, PR #1384; compiled Rust FPM model, PR #1461
• Collection and deployment implementation: shared FPM contract, PR #1473; pure-deployment generator, PR #1474; FPM collection workflow, PR #1475
• Current contracts: Dynamo ForwardPassMetrics schema; FPM trace schema; whole-forward model; modeling plan