feat(sdk): analytical queueing model — distribution-level TTFT/ITL estimates - #1380
Draft
tianhaox wants to merge 58 commits into
Draft
feat(sdk): analytical queueing model — distribution-level TTFT/ITL estimates#1380tianhaox wants to merge 58 commits into
tianhaox wants to merge 58 commits into
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds a scheduler-derived Pass-Calendar queueing model with closed-form and closed-loop evaluators, distributional latency reporting, backend integrations, a DES oracle, validation tooling, tests, and design documentation. ChangesQueueing Model
Estimated code review effort: 5 (Critical) | ~120 minutes Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (3 passed)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 17
Note
Due to the large number of review comments, Critical, Major severity comments were prioritized as inline comments.
🟡 Minor comments (7)
docs/design/queueing_model.md-79-79 (1)
79-79: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winUse an actual validation date.
As of July 17, 2026, this heading records July 18, 2026, which is in the future. Replace it with the execution date or mark the results as planned.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/design/queueing_model.md` at line 79, Update the “Validation results (2026-07-18)” heading to use the actual validation execution date, July 17, 2026, or explicitly label the results as planned if validation has not occurred.tests/unit/sdk/queueing/test_queueing_model.py-111-114 (1)
111-114: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winTest the documented SGLang invariant instead of allowing a 10% regression.
The assertion passes when SGLang
itl.p99is up to 10% below vLLM’s value, even though the test describes a whole-prefill-batch spike. Assert against the expected prefill-batch/pass duration, or use a tighter documented tolerance.As per path instructions, tests should verify the changed behavior rather than mask regressions with a broad tolerance.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit/sdk/queueing/test_queueing_model.py` around lines 111 - 114, Update the SGLang ITL assertion in the alternating-calendar test to verify the documented whole-prefill-batch/pass-duration invariant rather than permitting a broad 10% comparison against vLLM. Use the expected prefill-batch duration or the documented tighter tolerance, while preserving the existing p99-greater-than-p50 assertion.Source: Path instructions
tests/unit/sdk/queueing/test_queueing_model.py-162-165 (1)
162-165: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winActually verify legacy
ttft/tpotcompatibility.Despite its name, this test compares new queueing columns to literal input constants; it never checks the legacy scalar fields or a reporting row. A regression could break
ttft/tpotwhile this test still passes. Add assertions throughstatic_reportor the reporting schema.As per path instructions, tests should cover the changed compatibility contract.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit/sdk/queueing/test_queueing_model.py` around lines 162 - 165, Update test_static_columns_equal_legacy_scalars to exercise the legacy compatibility contract through static_report or the reporting schema, asserting that legacy ttft and tpot fields equal the corresponding scalar inputs. Retain coverage of the generated static columns only as appropriate, but ensure the test validates an actual reporting row rather than comparing solely against literals.Source: Path instructions
tests/unit/sdk/queueing/test_queueing_model.py-148-152 (1)
148-152: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winAssert the tail metric named by the test.
test_more_prefill_workers_reduce_ttft_tailcompares means, so a worse p99 can still pass as long as the mean improves. Comparettft_steady.p99for the one-worker and four-worker cases.As per path instructions, tests should cover the behavior named by the test.
Suggested change
- assert four.ttft_steady.mean <= one.ttft_steady.mean * 1.01 + assert four.ttft_steady.p99 <= one.ttft_steady.p99 * 1.01🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit/sdk/queueing/test_queueing_model.py` around lines 148 - 152, Update test_more_prefill_workers_reduce_ttft_tail to compare four.ttft_steady.p99 against one.ttft_steady.p99 instead of their means, while preserving the existing 1% tolerance.Source: Path instructions
docs/design/queueing_model.md-10-12 (1)
10-12: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winKeep the compatibility story consistent.
Line 11 says reporting replaces the blended-mean-only TTFT/TPOT columns, while Line 56 says the new columns are additive and legacy
ttft/tpotremain untouched. This can mislead consumers about a breaking schema change.As per path instructions, docs must match changed behavior.
Suggested change
-| **Replaces** | the empirical `_ttft_queuing_factor` heuristic and, for reporting, the blended-mean-only TTFT/TPOT columns | +| **Replaces** | the empirical `_ttft_queuing_factor` heuristic; reporting adds distributional TTFT/ITL columns while preserving legacy `ttft`/`tpot` |🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/design/queueing_model.md` around lines 10 - 12, Update the “Replaces” statement in the queueing model documentation to describe the reporting metrics as additive, not as replacing the legacy blended-mean-only TTFT/TPOT columns. Keep the compatibility behavior documented at the later reporting section consistent: legacy ttft and tpot fields remain unchanged while the new columns are added.Source: Path instructions
tests/unit/sdk/queueing/test_queueing_model.py-143-146 (1)
143-146: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winInclude the first decode pass in the disaggregated TTFT assertion.
The documented formula is prefill wait + prefill service + KV handoff + first decode pass, but this test only checks prefill service plus handoff. A regression that emits the first token before decode service would still pass.
As per path instructions, tests should cover the complete changed behavior.
Suggested change
# TTFT covers at least prefill service + handoff + first_decode = TIMING.decode_ms(1, wl.effective_isl) - assert rep.ttft_steady.mean >= (TIMING.prefill_ms(1, 2048, 0) + rep.kv_transfer_ms) * 0.9 + assert rep.ttft_steady.mean >= ( + TIMING.prefill_ms(1, wl.effective_isl, 0) + + rep.kv_transfer_ms + + first_decode + ) * 0.9🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit/sdk/queueing/test_queueing_model.py` around lines 143 - 146, Update the disaggregated TTFT assertion in the queueing model test to include the first decode pass in addition to prefill service and KV handoff. Reuse the existing timing helper or symbol representing first-token decode duration, while preserving the current tolerance and ITL assertion.Source: Path instructions
tools/queueing_oracle/run.py-63-74 (1)
63-74: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winReject zero-valued scheduler capacities before simulation.
Values such as
--num-workers 0,--block-size 0, or--concurrency 0currently produce division-by-zero errors or deadlocks.Proposed fix
args = p.parse_args() + positive_options = { + "--num-gpu-blocks": args.num_gpu_blocks, + "--block-size": args.block_size, + "--max-num-seqs": args.max_num_seqs, + "--max-num-batched-tokens": args.max_num_batched_tokens, + "--num-workers": args.num_workers, + } + for option, value in positive_options.items(): + if value <= 0: + p.error(f"{option} must be greater than zero") + if args.concurrency is not None and args.concurrency <= 0: + p.error("--concurrency must be greater than zero") + engine_args = EngineArgs(🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tools/queueing_oracle/run.py` around lines 63 - 74, Validate scheduler capacity arguments immediately after parse_args in the run entrypoint before constructing EngineArgs or starting simulation; reject zero values for num-workers, block-size, and concurrency with a clear argument error, while preserving existing behavior for positive values.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/design/queueing_model.md`:
- Around line 52-54: Update the vLLM row in the queueing model table to limit
the “0.0%” validation claim to only the metrics reported as exactly zero, and
explicitly summarize the remaining Section 5 residuals, including itl_p99 at
0.1%, itl_mean deviations up to 8.6%, and the 12.6% outlier. Apply the same
correction to the corresponding validation statement around the additional
referenced section.
In `@src/aiconfigurator/sdk/backends/base_backend.py`:
- Around line 1382-1396: Update the queueing metrics flow around
operating_point_columns so no-decode aggregate points with osl <= 1 return zero
for all itl_* metrics instead of reporting mix-step duration. Preserve existing
calculations for osl > 1, and add a regression test covering the no-decode case.
- Around line 1382-1396: Update the shared operating_point_columns evaluator
contract and its callers so queueing TTFT derives the request-own-prefill term
from the pure prefill latency rather than mix_step_ms, while preserving
mixed-pass latency for mixed-step calculations. Add a regression case with
differing pure-prefill and mixed-pass values, including the affected backend
call around queueing_closed_form.operating_point_columns.
In `@src/aiconfigurator/sdk/queueing/__init__.py`:
- Around line 64-78: The open-loop path in the workload evaluation flow must
stop using evaluate_closed_loop with heuristic concurrency to synthesize its
result. Replace the closed_wl/base construction with the documented open-loop
single-mass report, ensuring throughput is arrival-limited when stable and queue
waiting incorporates the selected backend; otherwise explicitly reject
unsupported backends. Preserve the existing workload inputs and return
mode="agg" semantics while removing the /10.0 concurrency heuristic and
synthetic TTFT shifting.
In `@src/aiconfigurator/sdk/queueing/calendar.py`:
- Line 209: Update the convergence bound calculation at max_passes to account
for prefill chunk depth using effective_isl and the backend chunk budget, while
retaining coverage for decode passes; derive a conservative backend-aware bound
that supports workloads requiring many prefill passes rather than relying only
on warmup_generations, window_generations, and wl.osl.
In `@src/aiconfigurator/sdk/queueing/closed_form.py`:
- Around line 135-138: Update the ITL percentile calculation near itl_mean to
use the existing weighted-duration quantile logic, such as
Distribution.quantile, over sorted pass durations and their frequencies rather
than selecting t_gen or t_mix with branches. Ensure p50 and p99 follow the same
boundary semantics as Distribution.quantile, including the exact 1% case.
In `@src/aiconfigurator/sdk/queueing/disagg.py`:
- Around line 225-239: Update the disaggregated queueing calculation around
QueueingReport and its request-state tracking to measure prefill queue wait
separately from prefill service: record each request’s first prefill start,
compute start_ms minus arrival_ms, and aggregate that value over steady-state
requests. Use the resulting aggregate for prefill_queue_ms so it matches the
closed-form path’s w_p instead of deriving it from TTFT and handoff.
- Around line 51-52: Update _prefill_pass_ms and every caller in the chunked
prefill paths to carry the computed context across passes: iterate over
remaining tokens, pass the actual chunk size plus computed_before and
computed_before to timing.prefill_ms, and advance computed_before after each
pass. Ensure the final partial chunk is priced using its actual token count
rather than the full chunk_per_req value, including the flows around the
referenced call sites.
- Around line 208-225: The decode loop around run_decode_pass must not dispatch
replacement _Req instances before simulated time reaches end. Defer completion
accounting, replacement creation/dispatch, and steady-state metrics for each
completed request to an event processed at end; update event selection and
release ordering so these events occur before any work scheduled after that
timestamp, while preserving correct window_ms throughput accounting based on the
deferred completion times.
- Around line 34-40: Add validation to the frozen DisaggSpec dataclass so
num_prefill_workers and num_decode_workers must both be positive, and
prefill_inflight_cap must be None or a supported positive value. Reject invalid
inputs during construction while preserving None as the engine-batched mode.
- Line 58: Update both disaggregated evaluator decode paths to honor the
decode_eng EngineSpec when selecting decode batches, including its max_num_seqs
and token limits. Thread decode_eng through the relevant batch-selection calls
and preserve prefill-engine usage for prefill decisions; alternatively remove
decode_eng only if the design explicitly supports prefill-only limits.
- Around line 267-281: Update the prefill batch-size calculations in the
engine-batched branch and the m_batch occupancy calculation to cap batching by
per-worker concurrency rather than global c. Use the existing per-worker
concurrency symbol, preserving the current budget and inflight-cap constraints
while preventing timing evaluation of unattainable batch sizes.
In `@src/aiconfigurator/sdk/queueing/spec.py`:
- Around line 52-56: Extend __post_init__ validation to reject non-positive
concurrency/request_rate, out-of-range isl/osl prefix values, invalid capacity
settings, and absent KV capacity before any arithmetic or guaranteed_no_evict
logic runs. Keep the existing exactly-one arrival-setting check and raise clear
ValueErrors for each invalid workload or engine parameter.
In `@tools/queueing_oracle/validate_formula.py`:
- Around line 15-16: Update the validation gate in validate_formula.py to
explicitly reject non-finite formula results or residuals before applying max()
or the tolerance comparison. Use a finite-value check so NaN and infinities
cause validation to fail, while preserving existing behavior for finite results.
In `@tools/queueing_oracle/vllm_sim.py`:
- Around line 363-368: Update the preemption bookkeeping around scheduled victim
removal and the corresponding logic near the later running-list handling: record
each request’s complete pass contribution, including batch-count and timing
aggregates, and roll it back when a prefilling request is preempted rather than
refunding only budget. After FIFO removal mutates the running list, derive the
next iteration index from the updated list so the request shifted into the
removed position is not skipped.
- Around line 217-234: Update the sequence state around allocate_blocks,
match_prefix, pass completion, and preemption so newly allocated prompt hashes
remain provisional and cannot be returned as cache hits. Track provisional
hashed blocks separately, publish each hash only after its complete block has
executed, and remove or restore provisional state correctly when requests are
preempted or allocation is abandoned.
- Around line 280-285: Update VllmSimCore.__init__ so KvManager’s enable_sharing
is enabled only when block KV mode is selected and args.enable_prefix_caching is
true. Preserve the existing non-block behavior and pass the resulting condition
to KvManager without changing other initialization logic.
---
Minor comments:
In `@docs/design/queueing_model.md`:
- Line 79: Update the “Validation results (2026-07-18)” heading to use the
actual validation execution date, July 17, 2026, or explicitly label the results
as planned if validation has not occurred.
- Around line 10-12: Update the “Replaces” statement in the queueing model
documentation to describe the reporting metrics as additive, not as replacing
the legacy blended-mean-only TTFT/TPOT columns. Keep the compatibility behavior
documented at the later reporting section consistent: legacy ttft and tpot
fields remain unchanged while the new columns are added.
In `@tests/unit/sdk/queueing/test_queueing_model.py`:
- Around line 111-114: Update the SGLang ITL assertion in the
alternating-calendar test to verify the documented
whole-prefill-batch/pass-duration invariant rather than permitting a broad 10%
comparison against vLLM. Use the expected prefill-batch duration or the
documented tighter tolerance, while preserving the existing p99-greater-than-p50
assertion.
- Around line 162-165: Update test_static_columns_equal_legacy_scalars to
exercise the legacy compatibility contract through static_report or the
reporting schema, asserting that legacy ttft and tpot fields equal the
corresponding scalar inputs. Retain coverage of the generated static columns
only as appropriate, but ensure the test validates an actual reporting row
rather than comparing solely against literals.
- Around line 148-152: Update test_more_prefill_workers_reduce_ttft_tail to
compare four.ttft_steady.p99 against one.ttft_steady.p99 instead of their means,
while preserving the existing 1% tolerance.
- Around line 143-146: Update the disaggregated TTFT assertion in the queueing
model test to include the first decode pass in addition to prefill service and
KV handoff. Reuse the existing timing helper or symbol representing first-token
decode duration, while preserving the current tolerance and ITL assertion.
In `@tools/queueing_oracle/run.py`:
- Around line 63-74: Validate scheduler capacity arguments immediately after
parse_args in the run entrypoint before constructing EngineArgs or starting
simulation; reject zero values for num-workers, block-size, and concurrency with
a clear argument error, while preserving existing behavior for positive values.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 6f1d28b2-4d1c-4da0-a792-2ff957527497
📒 Files selected for processing (20)
docs/design/queueing_model.mdsrc/aiconfigurator/sdk/backends/base_backend.pysrc/aiconfigurator/sdk/common.pysrc/aiconfigurator/sdk/picking.pysrc/aiconfigurator/sdk/queueing/__init__.pysrc/aiconfigurator/sdk/queueing/calendar.pysrc/aiconfigurator/sdk/queueing/closed_form.pysrc/aiconfigurator/sdk/queueing/disagg.pysrc/aiconfigurator/sdk/queueing/spec.pysrc/aiconfigurator/sdk/queueing/timing.pysrc/aiconfigurator/sdk/sweep.pytests/unit/sdk/queueing/__init__.pytests/unit/sdk/queueing/test_queueing_model.pytools/queueing_oracle/README.mdtools/queueing_oracle/compare_mocker.pytools/queueing_oracle/metrics.pytools/queueing_oracle/run.pytools/queueing_oracle/validate_formula.pytools/queueing_oracle/vllm_sim.pytools/queueing_oracle/workload.py
📜 Review details
🧰 Additional context used
📓 Path-based instructions (7)
**/*
📄 CodeRabbit inference engine (.claude/rules/repo-guide.md)
**/*: Treat this file as the only always-injected rule file; do not add new always-on rules here without human approval. New rule files must includepaths:frontmatter.
When reviewing changes in a governed area, read that area's rule files first, even if path-based auto-loading does not occur during a read-only review. Rule violations are review findings even when the code works.
A task must remain within its module: collector tasks may change onlycollector/and its tests, generator tasks onlysrc/aiconfigurator/generator/and its tests, and SDK tasks must not modify either. Cross-module contract changes require explicit human approval.
Files:
tools/queueing_oracle/README.mdtools/queueing_oracle/workload.pysrc/aiconfigurator/sdk/sweep.pysrc/aiconfigurator/sdk/queueing/timing.pytools/queueing_oracle/metrics.pytools/queueing_oracle/run.pydocs/design/queueing_model.mdsrc/aiconfigurator/sdk/queueing/__init__.pysrc/aiconfigurator/sdk/backends/base_backend.pysrc/aiconfigurator/sdk/picking.pytools/queueing_oracle/compare_mocker.pytests/unit/sdk/queueing/test_queueing_model.pysrc/aiconfigurator/sdk/queueing/spec.pysrc/aiconfigurator/sdk/common.pysrc/aiconfigurator/sdk/queueing/closed_form.pysrc/aiconfigurator/sdk/queueing/disagg.pysrc/aiconfigurator/sdk/queueing/calendar.pytools/queueing_oracle/vllm_sim.pytools/queueing_oracle/validate_formula.py
⚙️ CodeRabbit configuration file
**/*: - Prefer applicable inline comments. When the correct fix is clear, small, and limited to the commented diff hunk, include it as a GitHub Suggested Change so the author can apply it with one click.
- Do not use a suggested change when the fix requires broader design choices, multiple files, generated artifacts, unavailable context, or validation that cannot be inferred from the diff.
- If a comment is not directly applicable, state the smallest concrete next step and why a one-click suggestion is not safe.
Files:
tools/queueing_oracle/README.mdtools/queueing_oracle/workload.pysrc/aiconfigurator/sdk/sweep.pysrc/aiconfigurator/sdk/queueing/timing.pytools/queueing_oracle/metrics.pytools/queueing_oracle/run.pydocs/design/queueing_model.mdsrc/aiconfigurator/sdk/queueing/__init__.pysrc/aiconfigurator/sdk/backends/base_backend.pysrc/aiconfigurator/sdk/picking.pytools/queueing_oracle/compare_mocker.pytests/unit/sdk/queueing/test_queueing_model.pysrc/aiconfigurator/sdk/queueing/spec.pysrc/aiconfigurator/sdk/common.pysrc/aiconfigurator/sdk/queueing/closed_form.pysrc/aiconfigurator/sdk/queueing/disagg.pysrc/aiconfigurator/sdk/queueing/calendar.pytools/queueing_oracle/vllm_sim.pytools/queueing_oracle/validate_formula.py
**/*.{py,pyi}
📄 CodeRabbit inference engine (AGENTS.md)
Run
ruff check .andruff format --check .to validate Python linting and formatting.
Files:
tools/queueing_oracle/workload.pysrc/aiconfigurator/sdk/sweep.pysrc/aiconfigurator/sdk/queueing/timing.pytools/queueing_oracle/metrics.pytools/queueing_oracle/run.pysrc/aiconfigurator/sdk/queueing/__init__.pysrc/aiconfigurator/sdk/backends/base_backend.pysrc/aiconfigurator/sdk/picking.pytools/queueing_oracle/compare_mocker.pytests/unit/sdk/queueing/test_queueing_model.pysrc/aiconfigurator/sdk/queueing/spec.pysrc/aiconfigurator/sdk/common.pysrc/aiconfigurator/sdk/queueing/closed_form.pysrc/aiconfigurator/sdk/queueing/disagg.pysrc/aiconfigurator/sdk/queueing/calendar.pytools/queueing_oracle/vllm_sim.pytools/queueing_oracle/validate_formula.py
src/aiconfigurator/sdk/**
⚙️ CodeRabbit configuration file
src/aiconfigurator/sdk/**: - Verify SDK API changes remain compatible with generator inputs, profiler data flow, and documented examples.
- Flag silent schema or field-name drift between SDK models and generator/module bridge code.
Files:
src/aiconfigurator/sdk/sweep.pysrc/aiconfigurator/sdk/queueing/timing.pysrc/aiconfigurator/sdk/queueing/__init__.pysrc/aiconfigurator/sdk/backends/base_backend.pysrc/aiconfigurator/sdk/picking.pysrc/aiconfigurator/sdk/queueing/spec.pysrc/aiconfigurator/sdk/common.pysrc/aiconfigurator/sdk/queueing/closed_form.pysrc/aiconfigurator/sdk/queueing/disagg.pysrc/aiconfigurator/sdk/queueing/calendar.py
docs/**
⚙️ CodeRabbit configuration file
docs/**: - Check that docs match changed CLI, SDK, generator, backend, and support-matrix behavior.
- Flag docs that describe unsupported runtimes, stale command names, or behavior not covered by tests or support-matrix evidence.
Files:
docs/design/queueing_model.md
tests/**/*.{py,yaml,txt,sh}
📄 CodeRabbit inference engine (.claude/rules/generator/testing.md)
Use integration tests for the full input-to-artifacts pipeline, comparing output against golden snapshots without external dependencies.
Files:
tests/unit/sdk/queueing/test_queueing_model.py
tests/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Run unit tests with
pytest -m unit; usepytest -m "unit or build"for the build-test subset when required data is available.
Files:
tests/unit/sdk/queueing/test_queueing_model.py
tests/**
⚙️ CodeRabbit configuration file
tests/**: - Check that tests cover the changed behavior rather than only the happy path.
- Watch for fixtures or golden outputs that mask backend drift, support-matrix ordering changes, or CLI output regressions.
Files:
tests/unit/sdk/queueing/test_queueing_model.py
🧠 Learnings (2)
📚 Learning: 2026-05-01T00:39:37.334Z
Learnt from: simone-chen
Repo: ai-dynamo/aiconfigurator PR: 956
File: src/aiconfigurator/sdk/perf_database.py:4144-4151
Timestamp: 2026-05-01T00:39:37.334Z
Learning: In src/aiconfigurator/sdk/**/*.py, preserve upstream metadata for PerformanceResult.source: since PerformanceResult defaults source to "silicon", callers should not force-set result.source (e.g., PerfDatabase._query_silicon_or_hybrid should rely on the default and keep any existing source information rather than overwriting it). Only set source explicitly when you truly intend to change it.
Applied to files:
src/aiconfigurator/sdk/sweep.pysrc/aiconfigurator/sdk/queueing/timing.pysrc/aiconfigurator/sdk/queueing/__init__.pysrc/aiconfigurator/sdk/backends/base_backend.pysrc/aiconfigurator/sdk/picking.pysrc/aiconfigurator/sdk/queueing/spec.pysrc/aiconfigurator/sdk/common.pysrc/aiconfigurator/sdk/queueing/closed_form.pysrc/aiconfigurator/sdk/queueing/disagg.pysrc/aiconfigurator/sdk/queueing/calendar.py
📚 Learning: 2026-03-17T07:41:41.843Z
Learnt from: YijiaZhao
Repo: ai-dynamo/aiconfigurator PR: 600
File: src/aiconfigurator/sdk/backends/vllm_backend.py:42-43
Timestamp: 2026-03-17T07:41:41.843Z
Learning: In src/aiconfigurator/sdk/backends/vllm_backend.py, do not include seq_imbalance_correction_scale or gen_seq_imbalance_correction_scale in the AGG cache key isl/osl/b/ctx_tokens. These correction scales are per-instance and constant for a given usage of VLLMBackend, so they do not affect cache hits. Update the cache key computation to exclude these fields and adjust any tests that validate the cache key content. This guidance targets this file; apply similar logic to other backends only if they share the same per-instance scale semantics.
Applied to files:
src/aiconfigurator/sdk/backends/base_backend.py
🪛 ast-grep (0.44.1)
tools/queueing_oracle/workload.py
[info] 30-30: use secrets package over random package
Context: random.Random(seed)
Note: [CWE-330] Use of Insufficiently Random Values.
(avoid-random-python)
[warning] 65-65: File path is request-/variable-derived; validate and normalize to prevent path traversal.
Context: open(path)
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(open-filename-from-request)
tools/queueing_oracle/metrics.py
[warning] 71-71: File path is request-/variable-derived; validate and normalize to prevent path traversal.
Context: open(path, "w")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(open-filename-from-request)
tools/queueing_oracle/compare_mocker.py
[warning] 59-59: File path is request-/variable-derived; validate and normalize to prevent path traversal.
Context: open(args.replay_json)
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(open-filename-from-request)
🪛 LanguageTool
docs/design/queueing_model.md
[grammar] ~88-~88: Ensure spelling is correct
Context: ...C64) - disagg (1P1D tandem): TTFT 0.1%, ITL 0.0%, throughput 7.3% End-to-end cross-check against dynamo mo...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🔇 Additional comments (22)
src/aiconfigurator/sdk/backends/base_backend.py (3)
749-752: LGTM!
1382-1396: 🎯 Functional CorrectnessConfirm whether aggregate queueing TTFT includes encoder time.
Legacy
ttftincludesencoder_latency_ms(Line 1245), butoperating_point_columnsreceives no encoder latency. Vision-model aggregate rows will therefore report queueingttft_*values below the legacy TTFT by the encoder phase, unlike the static/disaggregated mappings. If these columns are intended to be end-to-end TTFT, include encoder latency; otherwise document them as LLM-only metrics.
18-18: 📐 Maintainability & Code QualityNo Python lint/format action needed.
ruff check .andruff format --check .are clean.> Likely an incorrect or invalid review comment.src/aiconfigurator/sdk/common.py (1)
675-686: LGTM!Also applies to: 735-749, 817-832
src/aiconfigurator/sdk/picking.py (1)
161-181: LGTM!src/aiconfigurator/sdk/sweep.py (1)
203-214: LGTM!src/aiconfigurator/sdk/queueing/spec.py (2)
1-51: LGTM!Also applies to: 57-62, 77-192
1-192: 📐 Maintainability & Code QualityNo changes needed.
src/aiconfigurator/sdk/queueing/timing.py (1)
1-69: LGTM!src/aiconfigurator/sdk/queueing/__init__.py (1)
1-63: LGTM!Also applies to: 81-117
src/aiconfigurator/sdk/queueing/calendar.py (1)
1-208: LGTM!Also applies to: 210-266
src/aiconfigurator/sdk/queueing/closed_form.py (1)
1-134: LGTM!Also applies to: 139-201
src/aiconfigurator/sdk/queueing/disagg.py (1)
1-341: 📐 Maintainability & Code QualityNo changes needed.
tests/unit/sdk/queueing/test_queueing_model.py (1)
3-8: 📐 Maintainability & Code QualityRun the pytest and golden-snapshot checks in an environment that has them available. Ruff already passes here, but the test-file change still needs the remaining coverage run.
tools/queueing_oracle/README.md (1)
1-99: LGTM!tools/queueing_oracle/vllm_sim.py (2)
1-25: LGTM!Also applies to: 35-216, 235-279, 286-362, 369-392, 404-672
26-34: 📐 Maintainability & Code QualityNo changes needed
tools/queueing_oracle/workload.py (1)
1-96: LGTM!tools/queueing_oracle/metrics.py (1)
1-74: LGTM!tools/queueing_oracle/run.py (1)
1-62: LGTM!Also applies to: 75-113
tools/queueing_oracle/compare_mocker.py (1)
1-126: LGTM!tools/queueing_oracle/validate_formula.py (1)
1-14: LGTM!Also applies to: 17-112, 124-187
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/aiconfigurator/sdk/queueing/calendar.py`:
- Around line 202-206: Update both calendar scheduler paths containing
prompt_before, prompt_tokens, and agg.add so latency charging includes
recomputation of generated-token history beyond wl.isl. Remove the wl.isl cap
from the timing delta while preserving the existing before/after computed-token
accounting and aggregation behavior; apply the same change consistently at both
sites.
- Around line 358-362: Update the kv_capacity_tokens validation in the
resident-sequence check to use the full required sequence length, up to wl.isl +
wl.osl - 1, rather than wl.isl + 1. Preserve the existing block-size ceiling and
capacity comparison so osl=1 boundaries remain valid while longer decodes are
rejected early.
- Around line 235-240: Update the queue transition in the scheduling flow around
_schedule so the admitted slot s is removed from state.waiting by identity,
rather than always popping index 0 after preemption. Preserve the existing
blocked/self-preempted break behavior, then append only s to state.running
without removing any preempted victim.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 524c4f6a-b148-4121-8c69-9d1e0266bea8
📒 Files selected for processing (6)
docs/design/queueing_model.mdsrc/aiconfigurator/sdk/queueing/calendar.pysrc/aiconfigurator/sdk/queueing/disagg.pysrc/aiconfigurator/sdk/queueing/spec.pytests/unit/sdk/queueing/test_queueing_model.pytools/queueing_oracle/validate_formula.py
🚧 Files skipped from review as they are similar to previous changes (4)
- docs/design/queueing_model.md
- src/aiconfigurator/sdk/queueing/spec.py
- tools/queueing_oracle/validate_formula.py
- src/aiconfigurator/sdk/queueing/disagg.py
📜 Review details
🧰 Additional context used
📓 Path-based instructions (6)
tests/**/*.{py,yaml,txt,sh}
📄 CodeRabbit inference engine (.claude/rules/generator/testing.md)
Use integration tests for the full input-to-artifacts pipeline, comparing output against golden snapshots without external dependencies.
Files:
tests/unit/sdk/queueing/test_queueing_model.py
**/*
📄 CodeRabbit inference engine (.claude/rules/repo-guide.md)
**/*: Treat this file as the only always-injected rule file; do not add new always-on rules here without human approval. New rule files must includepaths:frontmatter.
When reviewing changes in a governed area, read that area's rule files first, even if path-based auto-loading does not occur during a read-only review. Rule violations are review findings even when the code works.
A task must remain within its module: collector tasks may change onlycollector/and its tests, generator tasks onlysrc/aiconfigurator/generator/and its tests, and SDK tasks must not modify either. Cross-module contract changes require explicit human approval.
Files:
tests/unit/sdk/queueing/test_queueing_model.pysrc/aiconfigurator/sdk/queueing/calendar.py
⚙️ CodeRabbit configuration file
**/*: - Prefer applicable inline comments. When the correct fix is clear, small, and limited to the commented diff hunk, include it as a GitHub Suggested Change so the author can apply it with one click.
- Do not use a suggested change when the fix requires broader design choices, multiple files, generated artifacts, unavailable context, or validation that cannot be inferred from the diff.
- If a comment is not directly applicable, state the smallest concrete next step and why a one-click suggestion is not safe.
Files:
tests/unit/sdk/queueing/test_queueing_model.pysrc/aiconfigurator/sdk/queueing/calendar.py
**/*.{py,pyi}
📄 CodeRabbit inference engine (AGENTS.md)
Run
ruff check .andruff format --check .to validate Python linting and formatting.
Files:
tests/unit/sdk/queueing/test_queueing_model.pysrc/aiconfigurator/sdk/queueing/calendar.py
tests/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Run unit tests with
pytest -m unit; usepytest -m "unit or build"for the build-test subset when required data is available.
Files:
tests/unit/sdk/queueing/test_queueing_model.py
tests/**
⚙️ CodeRabbit configuration file
tests/**: - Check that tests cover the changed behavior rather than only the happy path.
- Watch for fixtures or golden outputs that mask backend drift, support-matrix ordering changes, or CLI output regressions.
Files:
tests/unit/sdk/queueing/test_queueing_model.py
src/aiconfigurator/sdk/**
⚙️ CodeRabbit configuration file
src/aiconfigurator/sdk/**: - Verify SDK API changes remain compatible with generator inputs, profiler data flow, and documented examples.
- Flag silent schema or field-name drift between SDK models and generator/module bridge code.
Files:
src/aiconfigurator/sdk/queueing/calendar.py
🧠 Learnings (1)
📚 Learning: 2026-05-01T00:39:37.334Z
Learnt from: simone-chen
Repo: ai-dynamo/aiconfigurator PR: 956
File: src/aiconfigurator/sdk/perf_database.py:4144-4151
Timestamp: 2026-05-01T00:39:37.334Z
Learning: In src/aiconfigurator/sdk/**/*.py, preserve upstream metadata for PerformanceResult.source: since PerformanceResult defaults source to "silicon", callers should not force-set result.source (e.g., PerfDatabase._query_silicon_or_hybrid should rely on the default and keep any existing source information rather than overwriting it). Only set source explicitly when you truly intend to change it.
Applied to files:
src/aiconfigurator/sdk/queueing/calendar.py
🔇 Additional comments (4)
tests/unit/sdk/queueing/test_queueing_model.py (1)
195-232: 📐 Maintainability & Code QualityRun the required Python validation checks.
No code-level concerns found in these tests. Please confirm
ruff check .,ruff format --check ., andpytest -m unitpass.As per coding guidelines, Python changes require both Ruff checks. As per path instructions, AGENTS.md requires the unit-test suite.
Sources: Coding guidelines, Path instructions
src/aiconfigurator/sdk/queueing/calendar.py (3)
379-379: The existing convergence-bound finding remains unresolved.With
isl=2000,osl=1, concurrency 1, and budget 1, the default eight-request run needs about 16,000 passes, while this cap permits only 3,200.
10-20: LGTM!Also applies to: 29-29, 47-201, 207-234, 241-249, 272-303, 307-326, 355-357, 364-366, 380-388, 418-420, 437-438
41-44: 📐 Maintainability & Code QualityRun
pytest -m unitin a Python environment withpytestinstalled.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/design/queueing_model.md`:
- Around line 28-29: The screening description in the design table overclaims
that shared workload bias preserves candidate ranking. Replace that assertion
with an empirically validated qualification, or document the specific conditions
and evidence required for ranking preservation, consistent with the
family-dependent bias described in §5; do not present the one-click screening
path as ranking-safe without that evidence.
In `@tools/queueing_oracle/validate_formula.py`:
- Around line 169-174: Update _Timing.prefill_ms to retain the nonnegative
prefill clamp and update _Timing.decode_ms to clamp ctx to a minimum of 1 before
calling f_decode, matching CallbackPerfModel’s timing behavior so both paths
consume identical timings.
- Line 237: Update the final status reporting around the validation script’s
`all_failures` result so the success message is emitted only when evaluator
tolerance checks pass, while closed-form failures remain exempt from this gate.
Make the summary explicitly distinguish evaluator tolerance failures from
exempted closed-form failures.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: c0a3f681-003b-451a-92c5-53b2382f5d62
📒 Files selected for processing (8)
docs/design/queueing_model.mdsrc/aiconfigurator/sdk/queueing/__init__.pysrc/aiconfigurator/sdk/queueing/calendar.pysrc/aiconfigurator/sdk/queueing/closed_form.pysrc/aiconfigurator/sdk/queueing/spec.pytests/unit/sdk/queueing/test_queueing_model.pytools/queueing_oracle/README.mdtools/queueing_oracle/validate_formula.py
💤 Files with no reviewable changes (1)
- tests/unit/sdk/queueing/test_queueing_model.py
🚧 Files skipped from review as they are similar to previous changes (2)
- tools/queueing_oracle/README.md
- src/aiconfigurator/sdk/queueing/closed_form.py
📜 Review details
🧰 Additional context used
📓 Path-based instructions (4)
**/*
📄 CodeRabbit inference engine (.claude/rules/repo-guide.md)
**/*: Treat this file as the only always-injected rule file; do not add new always-on rules here without human approval. New rule files must includepaths:frontmatter.
When reviewing changes in a governed area, read that area's rule files first, even if path-based auto-loading does not occur during a read-only review. Rule violations are review findings even when the code works.
A task must remain within its module: collector tasks may change onlycollector/and its tests, generator tasks onlysrc/aiconfigurator/generator/and its tests, and SDK tasks must not modify either. Cross-module contract changes require explicit human approval.
Files:
src/aiconfigurator/sdk/queueing/__init__.pytools/queueing_oracle/validate_formula.pysrc/aiconfigurator/sdk/queueing/spec.pysrc/aiconfigurator/sdk/queueing/calendar.pydocs/design/queueing_model.md
⚙️ CodeRabbit configuration file
**/*: - Prefer applicable inline comments. When the correct fix is clear, small, and limited to the commented diff hunk, include it as a GitHub Suggested Change so the author can apply it with one click.
- Do not use a suggested change when the fix requires broader design choices, multiple files, generated artifacts, unavailable context, or validation that cannot be inferred from the diff.
- If a comment is not directly applicable, state the smallest concrete next step and why a one-click suggestion is not safe.
Files:
src/aiconfigurator/sdk/queueing/__init__.pytools/queueing_oracle/validate_formula.pysrc/aiconfigurator/sdk/queueing/spec.pysrc/aiconfigurator/sdk/queueing/calendar.pydocs/design/queueing_model.md
**/*.{py,pyi}
📄 CodeRabbit inference engine (AGENTS.md)
Run
ruff check .andruff format --check .to validate Python linting and formatting.
Files:
src/aiconfigurator/sdk/queueing/__init__.pytools/queueing_oracle/validate_formula.pysrc/aiconfigurator/sdk/queueing/spec.pysrc/aiconfigurator/sdk/queueing/calendar.py
src/aiconfigurator/sdk/**
⚙️ CodeRabbit configuration file
src/aiconfigurator/sdk/**: - Verify SDK API changes remain compatible with generator inputs, profiler data flow, and documented examples.
- Flag silent schema or field-name drift between SDK models and generator/module bridge code.
Files:
src/aiconfigurator/sdk/queueing/__init__.pysrc/aiconfigurator/sdk/queueing/spec.pysrc/aiconfigurator/sdk/queueing/calendar.py
docs/**
⚙️ CodeRabbit configuration file
docs/**: - Check that docs match changed CLI, SDK, generator, backend, and support-matrix behavior.
- Flag docs that describe unsupported runtimes, stale command names, or behavior not covered by tests or support-matrix evidence.
Files:
docs/design/queueing_model.md
🧠 Learnings (1)
📚 Learning: 2026-05-01T00:39:37.334Z
Learnt from: simone-chen
Repo: ai-dynamo/aiconfigurator PR: 956
File: src/aiconfigurator/sdk/perf_database.py:4144-4151
Timestamp: 2026-05-01T00:39:37.334Z
Learning: In src/aiconfigurator/sdk/**/*.py, preserve upstream metadata for PerformanceResult.source: since PerformanceResult defaults source to "silicon", callers should not force-set result.source (e.g., PerfDatabase._query_silicon_or_hybrid should rely on the default and keep any existing source information rather than overwriting it). Only set source explicitly when you truly intend to change it.
Applied to files:
src/aiconfigurator/sdk/queueing/__init__.pysrc/aiconfigurator/sdk/queueing/spec.pysrc/aiconfigurator/sdk/queueing/calendar.py
🪛 LanguageTool
docs/design/queueing_model.md
[grammar] ~82-~82: Ensure spelling is correct
Context: ... benchmark length (same deployment: mean 456ms at N=200, 223ms at N=1000). ## 5. Vali...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[grammar] ~83-~83: Ensure spelling is correct
Context: ... (same deployment: mean 456ms at N=200, 223ms at N=1000). ## 5. Validation (2026-07-...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[style] ~102-~102: Consider an alternative for the overused word “exactly”.
Context: ...ng is initial-condition dependent) is exactly what the evaluator exists to capture....
(EXACTLY_PRECISELY)
🔇 Additional comments (10)
docs/design/queueing_model.md (1)
6-26: LGTM!Also applies to: 31-51, 53-83, 85-145
src/aiconfigurator/sdk/queueing/spec.py (3)
52-76: Existing constructor-validation finding still applies.
WorkloadSpecandEngineSpecstill permit the invalid values identified in the prior review.
143-192: LGTM!
16-32: 📐 Maintainability & Code QualityNo changes needed.
src/aiconfigurator/sdk/queueing/calendar.py (2)
209-209: Existing convergence-bound finding still applies.Line 209 still ignores prefill chunk depth, so valid long-prefill workloads can exhaust
max_passes. Derive a conservative backend-aware bound; a one-click change is unsafe because progress differs by calendar.
37-60: LGTM!src/aiconfigurator/sdk/queueing/__init__.py (2)
3-50: LGTM!
53-81: LGTM!tools/queueing_oracle/validate_formula.py (2)
153-162: The non-finite validation gap remains.
NaNerrors can still bypass both the tolerance comparison andfailures. This was already reported on an earlier commit.
20-29: 📐 Maintainability & Code QualityNo change needed
… from scheduler semantics
Adds sdk/queueing: an algorithm-derived (zero fitted constants) queueing
model for continuous batching. Three tiers: O(1) closed form on the
run_agg hot path, a deterministic limit-cycle evaluator for full
TTFT/ITL/TPOT distributions, and a P/D tandem recursion with the KV
handoff term. Backend calendars: vllm (validated to 0.0% on TTFT
p50/p99/transient and ITL quantiles vs the mocker-parity DES oracle),
sglang/trtllm (structural).
New additive summary columns (legacy ttft/tpot untouched):
ttft_steady_{mean,p50,p90,p99}, ttft_transient_{mean,max},
itl_{mean,p50,p99} across ColumnsAgg/Static/Disagg. Static mode maps
degenerately; disagg follows prefill (TTFT) / decode (ITL single mass).
Lands tools/queueing_oracle (DES verified 0.0% vs dynamo mocker) plus
validate_formula.py as the parity gate, unit tests, and the design doc
with term provenance and failure-mode catalog.
End-to-end vs dynamo mocker with real AIC timing (Llama-3.1-8B/h200):
legacy ttft heuristic -30%; this model -6.9% blended, itl_p50 exact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
Adds block-counted KV state to the pass-calendar evaluator: capacity gates admission and decode growth, exhaustion triggers preemption-with- recompute (LIFO/FIFO), covering the memory-limited waiting-queue regime. EngineSpec gains kv_capacity_tokens (all calendars), block_size, and preemption_mode so external inputs fully characterize the scheduler. QueueingReport.kv_thrashing flags the deep-preemption regime, which is detected rather than predicted (treat as infeasible, like OOM). Fixes: disagg tandem pool clock could lag behind arrivals (pass started before its request existed); decode-token allocation failure now skips the token instead of blocking the rest of the running set (core.rs emit semantics); budget exhaustion exits the running loop like the oracle. Validation battery grows 6 -> 13 cases (KV-mild, KV-thrash detection, max_num_seqs-capped queueing, prefix, short-osl, C=1, deep staircase): TTFT p50/p99/transient 0.0% in every quantitative case; KV-mild steady p50/p99 0.0%; thrash detection agrees on both sides. Design doc failure modes updated accordingly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
Scopes the package to the analytical correction. sdk/queueing now holds exactly two precision tiers of the same pass-calendar model: - closed form (screening): O(1) arithmetic populating the summary columns on the sweep hot path; per-workload bias is shared across sweep candidates, preserving ranking - limit-cycle evaluator (quantitative): the model's recursion evaluated numerically, capturing the cohort effects the closed form approximates; gated within 10-15% (mostly 0.0%) vs the DES oracle across 9 config families Moves out of scope (preserved on the queueing-sim-layer branch): KV-capacity/preemption state, the disagg tandem recursion, and the thrashing detector — simulation-level concerns beyond the correction. Disagg summary columns keep their degenerate composition. Improves the closed form's ITL gap weighting ((c-1)/c: a mix pass stalls only requests not being prefilled in it) and restructures the validation gate: evaluator enforced tight, closed-form screening tier reported with sanity assertions and documented per-family bias. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
DatabaseTimingModel no longer re-implements the per-op query loop — it delegates to BaseBackend._run_context_phase/_run_generation_phase (the estimators behind run_static), so backend-specific overrides, correction scales, and future phase-runner evolution apply automatically. Verified numerically identical on the reference case; end-to-end evaluator vs dynamo.replay ground truth improves the blended TTFT mean to -2.5% (itl_p99 0.1%). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
Re-anchors the oracle's correctness basis from a development-time replay baseline to the vLLM v1 scheduler source itself, with a clause-by-clause file:line provenance table in vllm_sim.py (unified budget :334, running-set-first :346, chunked prefill cap :372, admission cap :534, waiting-admission failure :716-723, running-path preemption :437-471). Fixes a semantic divergence surfaced by the source audit: a WAITING admission whose block allocation fails is now put back intact and stops admission (scheduler.py:716-723) instead of preempting running requests. Out of reach for the model's validated domain (no KV pressure there), but makes the oracle faithful for future capacity work. TimingModel docs now cite the SDK's own phase-runner parameterization (_run_context_phase(batch_size, isl, prefix)) as the interface's origin. Drops the replay-comparison script from the correction PR (kept on the simulation-layer branch where that experiment lives). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
…acle tooling The DES oracle and its gate script were development tooling for deriving and validating the correction; the PR does not need to ship them. The validation record — the vLLM v1 source clause-audit table (file:line) and the measured accuracy across 9 config families — moves into docs/design/queueing_model.md §5. The tooling itself is preserved on a development branch for future re-validation. PR content is now the correction proper: sdk/queueing (closed form + limit-cycle evaluator + timing delegation), the nine additive summary columns, unit tests, and the design doc. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
SLA targets become (value, percentile) pairs on the steady-state
distributions. RuntimeConfig gains ttft/tpot/itl/request_latency
percentile fields (0.5/0.75/0.9/0.95/0.99/0.999) and an optional itl
target — the streaming-smoothness SLA, the metric where the calendar is
most accurate and where agg vs disagg differ most. Defaults: p50 for
ttft/tpot/request_latency, p99 for itl. Percentiles are constraint
parameters, not columns: arbitrary quantiles are read off the model
distributions at filter time, keeping the schema fixed.
TTFT feasibility on the sweep path is resolved by a two-stage funnel:
- wide-keep on the cohort bracket (new ttft_steady_p99_{lo,hi} columns:
structural bounds on the steady distribution support — solo prompts vs
full-budget packing — so screening bias can never falsely reject and
one bracket serves every percentile);
- lazy quantitative resolution: straddlers are re-scored with the
limit-cycle evaluator in throughput order until top_k feasible rows
are confirmed, enforcing all requested constraints at their
percentiles; unresolved rows are kept (conservative) with the new
queueing_tier column making every row's precision class visible.
The legacy ttft scalar (equivalent to a blended mean at an N implicitly
baked into its fitted constants, ~420 on the reference workload) remains
emitted for reference but no longer gates sweep feasibility. The legacy
path (pareto_analysis / find_best_*) keeps legacy semantics.
Evaluator additionally reports the e2e distribution; refine hot path is
bounded by timing quantization (64-token grain), per-operating-point
report caching on the backend instance, short evaluation windows, and a
per-call refine budget with logged (never silent) skips. Measured on the
reference case: loose SLA ~2x sweep time, boundary-tight SLA ~7x; the
tight-SLA run rescued 16 falsely-rejected configs including the
top-throughput one (legacy 283ms vs quantitative p50 123ms at a 210ms
target).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
The rows a human reads (best_config_topn) are upgraded to quantitative-tier numbers once, at the report boundary — rebuilt self-contained from each row's own metadata (model/backend/version/ system + parallelism columns), so it runs where sweep-time objects are gone and can never crash the report path (per-group failures log and skip). Nothing is dropped: for certain-pass rows the bracket guarantees the refined p99 also complies. Also fixes a timing-quantization edge: quantizing isl and prefix independently could collapse the effective length to zero for small trailing chunks; the effective difference is quantized instead. Measured: upgrade of a 4-row top-N adds ~0.3s; warm evaluator cost is ~150-350ms per operating point. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
…gs to debug CLI gains --ttft-percentile / --tpot-percentile / --request-latency- percentile (p50/p75/p90/p95/p99/p999, default p50) and a new optional --itl target with --itl-percentile (default p99), threaded through Task and RuntimeConfig into the sweep funnel. The top-config tables print the constrained percentile's steady value in parentheses next to the legacy TTFT (nearest stored percentile column, honestly labeled; enforcement uses the exact requested quantile). Rows without queueing distributions (disagg composed / static) print unchanged. Refine bookkeeping logs demoted from info to debug. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
Decouples two orthogonal switches (both default OFF, keeping the default run byte-compatible with legacy filtering at zero added cost — CI-safe, verified back at baseline runtime): - percentile mode (implicit when any --*-percentile or --itl is given): constraints are enforced at the requested percentile using the closed form's stored quantiles — still O(1), no evaluator. The stored TTFT quantile set is completed (p50/p75/p90/p95/p99/p999) so every supported percentile has an exact column; itl screening reads its two-mass tail. tpot/request_latency quantile enforcement requires the evaluator and is documented as such. - --sla-refine: additionally resolve bracket-straddling candidates with the limit-cycle evaluator and upgrade reported rows to quantitative tier. Report tables now name what they show: the TTFT header is TTFT(avg) in legacy mode (the blended-mean estimate) and TTFT(P<q>) in percentile mode with the exact matching quantile data — headers and data can no longer disagree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
Adds an ITL(P99) column (ITL(P50) when --itl-percentile p50) to the agg and disagg top-config tables, sourced from the always-emitted queueing columns. The ITL mean is deliberately not displayed: for a bimodal distribution it is a value that rarely occurs, and with fixed osl it equals the TPOT mean already shown via tokens/s/user. This surfaces the agg-vs-disagg smoothness signature (mix-pass stutter tail vs a single mass) directly in the report. Display helpers are type-guarded so mock tasks in tests fall back to legacy labels. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
Latency trio now reads TTFT / TPOT / ITL left to right. TPOT shows the mean with an explicit TPOT(avg) header: unlike ITL (bimodal — mean is unrepresentative), per-request averaging makes the TPOT distribution narrow, so the mean is the right summary for it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
Elimination semantics are set exclusively by the constraint args (--*-percentile / --itl; default: the legacy avg filter). --sla-refine alone no longer switches filtering or headers to percentile mode — it keeps the identical candidate set and only upgrades the reported top-N rows to quantitative-tier numbers. Combined with percentile constraints it resolves SLA-boundary candidates with the evaluator at the requested percentile, as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
Replaces a stale reference to a development-time artifact with the actual provenance (vllm/v1/core/sched/scheduler.py:652). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
…, screening-scope docs, multimodal refine - exp YAML mode: Task.from_yaml already accepts the new SLA fields (verified); example.yaml documents them. - Removes the unshipped open-loop helpers (M/D/1 wait and standalone mean estimates) — no product caller and no validation; open-loop queueing returns with track-two work rather than sitting as a false affordance. Scope statements updated accordingly. - CLI help and design doc now state the screening-tier scope explicitly: without --sla-refine, percentile enforcement exists only for TTFT (full stored quantile set) and ITL (two-mass anchors); TPOT and request-latency percentiles require the evaluator. - Multimodal rows become refinable when the runtime image context is available: vision tokens join the prefill length and the encoder latency shifts the TTFT/e2e distributions additively (run_agg's own composition), applied on copies so cached reports stay clean. Without runtime context they remain visibly at screening tier. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
…engine Review fixes on top of the pass-calendar change: Model/engine alignment (verified against vLLM v0.24.0 source): - pin scheduler line anchors to tag v0.24.0 (they had drifted ~120-170 lines); add prefix-budget and fused-execution clauses to the provenance table; note anchors are version-specific - closed form: chunk counts now use the EFFECTIVE prompt length (isl - prefix) - cached tokens do not consume the token budget (scheduler.py:710-712). Restores the funnel invariant "wide-keep never falsely rejects" for prefix rows (before the fix: lo 73.98 > evaluator p99 57.48 on the regression case) - closed form: encoder latency and per-request dispatch overhead now reach the screening TTFT columns (the legacy ttft already carried both); the evaluator reports get the same additive shift in refine - percentile screens and displayed TTFT(Pxx) were silently permissive for multimodal rows - fused calendar: prefill completers are no longer billed as decode rows (the fused pass samples their first token off the final chunk's logits); C=1 steady TTFT now equals pure prefill - sglang calendar: mixed-chunk mode added and made the default, matching AIC's own deployment rule (rule_plugin/sglang.rule sets enable_mixed_chunk=true); alternating mode remains for mixed-chunk-off Failure-mode documentation (queueing_model.md par.6): speculative decoding (amortized ITL for nextn rows), attention-DP prefill cadence, the closed-loop admission boundary convention (shared with the DES oracle, so par.5 residuals cannot see it), async-scheduling watch item. Implementation hygiene: - _Slot is eq=False (slots.index must match identity, not field-equal cohort peers) - test_queueing_model.py gains the unit marker - CI gates on -m "unit or build", so the queueing tests were silently skipped - drop dead branch in the funnel lazy loop; None-check instead of falsy `or` fallback in the sweep TTFT screen; remove dangling sdk.queueing.evaluate_disagg comment references - new tests: additive-stage shift, prefix effective-length, prefix bracket invariant vs evaluator, C=1 TTFT, sglang mixed default, percentile CLI args, report header semantics Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
…ecord - Replace line-number anchors with conceptual clause descriptions (code comments and the design-doc provenance table). Line pins rot with every upstream release; the clauses are semantic, and drift is caught by the validation gate, not by re-reading pinned lines. The audited engine version stays recorded once in the validation record. - Record the 2026-07-19 re-validation: after the completer-billing and effective-isl corrections, the DES oracle (which shared the completer-as-decode-row convention — both sides wrong together, invisible to the gate) was corrected against the engine first, then all 9 families re-gate within tolerance; C1 is now exact (0.0% on every metric). Oracle fix lives on the oracle dev branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
Timing hook (closes the largest screening/quantitative inconsistency): - DatabaseTimingModel.mixed_pass_ms delegates a fused prefill+decode pass to BaseBackend._get_mix_step_latency — the same runner behind run_agg's t_mix, batching efficiency included. The previous prefill_ms + decode_ms sum double-counted the shared non-attention cost (weights are loaded from HBM once for the combined batch). - The calendar prefers the hook for genuinely mixed passes and falls back to the sum for timing models without it, so the DES validation gate (plain prefill/decode callbacks, timing-independent by design) is unaffected. Pure-prefill and pure-decode passes keep the dedicated phase estimators. - Applies to the fused (vllm/trtllm) calendar and the sglang mixed-chunk path. End-to-end smoke (Llama-3.1-8B / h200_sxm / vLLM 0.24.0, --ttft 210 p50 --sla-refine): funnel refines 19/59 pareto rows, top-N fully quantitative-tier. Docs: cli_user_guide gains a "Percentile SLA targets" subsection (three modes, flag table, header semantics, cost figures, exp-YAML pointer) and the defaults-table/pointer-line mentions; design doc records the timing-delegation note (the recorded end-to-end figure predates it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
tools/queueing_oracle/: a stdlib-only discrete-event simulation of vLLM v1 iteration-level scheduling, plus the 9-family validation gate that drives sdk.queueing and the DES with identical timing callbacks — residuals isolate scheduling semantics, so the gate is the executable form of the model's source-anchoring policy. - vllm_sim.py: engine core anchored clause-by-clause to the vLLM v1 scheduler (unified token budget, running-set-first, chunked prefill, put-back-no-preempt waiting admission, LIFO preemption with recompute); KV block accounting mirrors the vLLM BlockPool lifecycle (refcounted hashed blocks, eviction-ordered free queue, prefix re-activation, cache hit capped at length-1 for logits). - Disagg (P/D) follows the serving flow: the prefill worker produces the first token (the TTFT token), KV handoff defers the decode-side continuation — the transfer delay lands in the first ITL gap, not TTFT; on decode workers the transferred KV counts as computed tokens (the KV-connector convention), so decode passes never bill prefill compute. - workload.py / metrics.py / run.py: synthetic + mooncake-trace workloads, percentile summaries, standalone CLI; SyntheticPerfModel is an arbitrary roofline-shaped default for standalone runs — studies inject their own timing callbacks. - validate_formula.py: the gate — evaluator tier GATED at 10-15% tolerances across 9 config families, closed-form screening tier report-only with sanity checks. - tests/unit/sdk/queueing/test_oracle_gate.py: the gate runs in CI (~1s), making scheduler-semantics drift a failing check instead of a stale doc claim; plus disagg serving-flow assertions (prefill-side TTFT, handoff in the first gap, osl=1 completing on the prefill worker). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
Disagg handoffs are no longer independent constant delays. Each transfer is a fluid flow on a TransferFabric: concurrent flows share per-worker NIC bandwidth max-min fairly, so fan-out (a clump of completions leaving one prefill worker's egress) and fan-in (several prefill workers landing on one decode worker's ingress) slow each other down by the computed fair share. The destination decode worker is chosen when the transfer starts — that is when the flow joins the ingress contention. Bandwidths come from the AIC system spec instead of a hand-configured constant: sysspec.transfer_spec_from_system reads node.inter_node_bw / node.intra_node_bw (Byte/s per GPU, rank-local arithmetic so TP sharding cancels) and de-rates by bw_efficiency (default 0.8, following the spec's own mem_bw_empirical_scaling_factor convention). - vllm_sim.py: TransferSpec + TransferFabric (max-min water-filling, event-driven fluid recompute); EngineArgs loses the per-request transfer fields; DisaggSimulator drives the fabric (xfer_tick wake-ups, dst pinned at submit) - sysspec.py: system-spec wiring helper (core sim stays stdlib-only) - tests: fabric sharing math (fan-in, fan-out, disjoint pairs, staggered leftover-bandwidth, efficiency de-rate), sys-spec wiring incl. the real h200_sxm yaml, and an end-to-end fan-in-doubles-handoff assertion Known scope limits documented in the README: per-worker NICs only (no shared-fabric topology), no transfer/compute interference on the GPU. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
# Conflicts: # src/aiconfigurator/sdk/backends/base_backend.py # src/aiconfigurator/sdk/common.py # src/aiconfigurator/sdk/config.py
…merge Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
tianhaox
force-pushed
the
queueing-pass-calendar
branch
from
July 19, 2026 03:22
8f03832 to
80a0a84
Compare
The P/D tandem pass-calendar recursion (sdk/queueing/disagg.py) replaces the composed static scalars for reported disagg rows: - Serving-flow semantics: the prefill pool emits the first (TTFT) token; the KV handoff is a flow on a max-min-fair per-NIC transfer fabric (fan-in/fan-out contention computed, bandwidths from the system spec) and lands in the first ITL gap; decode workers pay no prefill compute (KV-connector convention). Rate matching is an output: pool imbalance surfaces as prefill queueing or decode saturation instead of scalar derates. - Heterogeneous P/D: each pool is priced by its own timing model. - Cohort-phase multi-stability handled structurally: the tandem system has multiple valid limit cycles selected by initial arrival phase (large-batch vs pipeline cycles, TTFTs differing by multiples); evaluate_disagg_mixed reports a deterministic equal-weight mixture over initial-arrival staggers (no RNG). Documented as failure mode 15. - Report boundary: --sla-refine upgrades composed disagg rows to quantitative via refine_report_rows (rows rebuilt from (p)/(d) metadata; multimodal rows stay visibly composed). Validated: four new gated disagg families in the oracle gate (1P1D, fan-in 2P1D, bandwidth-tight 2P1D, 2P2D) — TTFT mean/p50/p99 and ITL p50/p99 essentially exact (mostly 0.0%, itl_mean <= 2.6%), including the fan-in transfer spike reproduced to 0.0%. Getting there surfaced three driver/fabric bugs, all fixed on both sides: - a dispatch-time busy_until bump froze a worker until a future arrival, swallowing every pass it could still run (the dominant error: half the steady TTFT mass landed one full prefill pass too late) - passes must never consume requests that have not arrived by pass start (pool_arrival_ms filtering; next_start = max(busy, earliest arrival)) - the transfer fabric now advances piecewise, stopping at every internal flow completion (rates change there; a linear step mis-shared bandwidth and, on future-timestamped submits, dragged completions to the submit time). Completions carry their true finish times. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
Contributor
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '=== cited commits ==='
git show --stat --oneline --decorate --no-renames b0e4c6baf d983a7ef4 || true
echo '=== queueing module map ==='
ast-grep outline src/aiconfigurator/sdk/queueing --items all --type class,function --view expanded
ast-grep outline tools/queueing_oracle --items all --type class,function --view expanded
ast-grep outline tests/unit/sdk/queueing --items all --type function --view expanded
echo '=== guards, conventions, and regression references ==='
rg -n -C 4 --glob '*.py' \
'(guaranteed_no_evict|kv_capacity_tokens|DisaggSpec|isfinite|non.finite|no-enable-prefix-caching|enable_prefix_caching|itl_.*osl|osl.*itl|import.contract|import_contract)' \
src/aiconfigurator/sdk/queueing tools/queueing_oracle tests/unit/sdk/queueing
echo '=== changed hunks in cited commits ==='
git diff b0e4c6baf^ b0e4c6baf -- \
src/aiconfigurator/sdk/queueing \
tools/queueing_oracle \
tests/unit/sdk/queueing \
tests | sed -n '1,1800p'
git diff d983a7ef4^ d983a7ef4 -- \
src/aiconfigurator/sdk/queueing \
tools/queueing_oracle \
tests/unit/sdk/queueing \
tests | sed -n '1,1200p'Length of output: 32864 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '=== exact implementation diffs ==='
git diff b0e4c6baf^ b0e4c6baf -- \
src/aiconfigurator_core/sdk/queueing/closed_form.py \
src/aiconfigurator_core/sdk/queueing/spec.py \
src/aiconfigurator_core/sdk/queueing/disagg.py
echo '=== source maps ==='
ast-grep outline src/aiconfigurator_core/sdk/queueing/closed_form.py --items all --type class,function --view expanded
ast-grep outline src/aiconfigurator_core/sdk/queueing/spec.py --items all --type class,function --view expanded
ast-grep outline src/aiconfigurator_core/sdk/queueing/disagg.py --items all --type class,function --view expanded
ast-grep outline tests/cross_package/test_import_contract.py --items all --type class,function --view expanded
echo '=== relevant implementation and import-contract context ==='
rg -n -C 8 --glob '*.py' \
'(def operating_point_columns|class WorkloadSpec|class EngineSpec|def __post_init__|class DisaggSpec|itl_mean|itl_p50|itl_p99|CORE_SDK_LEAF_MODULES|import_module)' \
src/aiconfigurator_core/sdk/queueing tests/cross_package/test_import_contract.py
echo '=== tandem rewrite and prior referenced fixes (metadata only) ==='
git show --stat --oneline --no-renames e3ebd31 d70dd62 0b7eb21 ae02524 2>/dev/null || trueLength of output: 9449 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '=== tracked queueing implementation paths ==='
git ls-tree -r --name-only HEAD | grep -E '(^|/)(closed_form|spec|disagg)\.py$|queueing' | sed -n '1,500p'
echo '=== filesystem filename discovery ==='
fd -HI -t f '^(closed_form|spec|disagg)\.py$' . | sed -n '1,300p'
echo '=== b0e4c6b changed paths exactly ==='
git diff-tree --no-commit-id --name-only -r b0e4c6baf
echo '=== source content from the cited commit (path independent) ==='
for path in $(git diff-tree --no-commit-id --name-only -r b0e4c6baf | grep -E '(^|/)(closed_form|spec|disagg)\.py$' || true); do
echo "--- $path ---"
git show "b0e4c6baf:$path" | nl -ba | sed -n '1,360p'
doneLength of output: 2989 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '=== canonical implementation diff ==='
git diff --unified=20 b0e4c6baf^ b0e4c6baf -- \
aic-core/src/aiconfigurator_core/sdk/queueing/closed_form.py \
aic-core/src/aiconfigurator_core/sdk/queueing/spec.py \
aic-core/src/aiconfigurator_core/sdk/queueing/disagg.py
echo '=== definition maps ==='
ast-grep outline aic-core/src/aiconfigurator_core/sdk/queueing/closed_form.py --items all --type class,function --view expanded
ast-grep outline aic-core/src/aiconfigurator_core/sdk/queueing/spec.py --items all --type class,function --view expanded
ast-grep outline aic-core/src/aiconfigurator_core/sdk/queueing/disagg.py --items all --type class,function --view expanded
echo '=== focused implementation contexts ==='
rg -n -C 10 --glob '*.py' \
'(def operating_point_columns|if osl <= 1|class WorkloadSpec|class EngineSpec|class DisaggSpec|def __post_init__|guaranteed_no_evict|kv_capacity_tokens|num_prefill_workers|num_decode_workers|prefill_inflight_cap)' \
aic-core/src/aiconfigurator_core/sdk/queueing
echo '=== compatibility wrapper mapping ==='
sed -n '1,120p' src/aiconfigurator/sdk/queueing/closed_form.py
sed -n '1,120p' src/aiconfigurator/sdk/queueing/spec.py
sed -n '1,120p' src/aiconfigurator/sdk/queueing/disagg.pyLength of output: 49767
Verified by static source and test inspection; tests were not executed in the sandbox. ✏️ Learnings added
|
The flag encodes admission visibility, not engine overlap in general. Measured on h20e_sxm/trtllm 1.3.0rc20 (Qwen3-32B tp4, isl4096/osl256): async=True overpredicts steady TTFT by 25-30% at C>=32; async=False lands within ~4%. Recorded in the EngineSpec comment and the design-doc provenance row. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tandem dispatched closed-loop replacements at the completion instant with immediate pool visibility, ignoring the client/frontend turnaround the agg calendar already models. Replacements now become visible at completion + turnaround_ms (TTFT origin stays at the dispatch instant, matching both the calendar and what a real client measures). Validated against h20e 2P1D (trtllm 1.3.0rc20, Qwen3-32B tp1): kappa=1 low-load TTFT moves to -3.2% of measured. Note: the eps-scale visibility delay does NOT (and should not) reproduce the jitter-selected queued attractor at the saturation knee — that remains a documented failure-mode-15 boundary, not a fitting target. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…or boundary DisaggSpec.prefill_inflight_cap: measured mapping — TRT-LLM native disaggregated serving is solo-serial ctx prefills behind a static round-robin router, i.e. kappa=1 (h20e 2P1D evidence, low-load TTFT within ~3%). Failure mode 15 gains the measured saturation-knee instance: deterministic recursion locks the zero-wait pipeline attractor at rho->1 for any kappa/phase/turnaround while real jitter selects a queued attractor; below and beyond the knee the recursion is accurate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verified against sglang 0.5.14 source (schedule_policy.py): the per-iteration extend budget is chunked_prefill_size SHARED across the batch (alloc = min(extend_input_len, rem_chunk_tokens)) — not a per-request chunk cap under a max_prefill_tokens batch budget as the calendar had it — and mixed-chunk decode rows debit the budget (num_mixed_decode_tokens), the opposite of the previous comment. max_prefill_tokens stays as the admitted-input cap (binding only when smaller). At chunked_prefill_size=8192 / max_prefill_tokens=16384 this halves the modeled mixed-pass size (ITL spike ~4s not ~7.7s). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… evidence TrtllmCalendar: chunked-off ITL bimodal signature, chunked-on budget spike, KV-capped deep queueing +-3%, TPOT <=2% at C>=32 (trtllm 1.3.0rc20, Qwen3-32B tp4). AlternatingCalendar: both branches — mixed spike tracks the shared chunk budget (+-1.7%), alternating TPOT <1% / X exact at C=32 with cohort-locked prefill waves reproduced (sglang 0.5.14, tp1, clean thread-per-slot client; aiperf triggers a deterministic +1x-prefill TTFT artifact in non-mixed mode, tool-side). Remaining TTFT residuals are failure-mode-16 timing-bias amplification, not calendar structure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…shape streams WorkloadSpec gains isl_quantiles/osl_quantiles (inverse-CDF strata, see stratified_quantiles); slots draw their own (isl, osl) from a golden-ratio-stride rotation over the strata — zero RNG, exactly reproducible, degenerate streams reproduce the fixed-shape recursion bit-for-bit (parity-tested). Heterogeneity lives INSIDE the batch: a mixture of homogeneous fixed-shape runs keeps each component's convoy structure and cannot represent the desynchronization that shape diversity causes. Validated against h20e trtllm 1.3.0rc20 tp4 (isl 4096 cv 0.25, osl 256 cv 0.5, C 8/32): steady TTFT error drops from +159%/+109% (fixed-shape prediction) to +2%/+31%; TPOT within 4%, throughput within 6%; ITL spike band smearing reproduced (fixed single point -> wide band), amplitudes ~1.7x high pending pass-packing refinement. Scope: closed-loop agg calendars; the disagg tandem stays fixed-shape. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…le arrivals evaluate_open_loop consumes WorkloadSpec.request_rate (previously a declared-but-unconsumed field): arrivals come from a 64-stratum exponential inverse-CDF stream normalized to an exact 1/rate mean with a golden-ratio-stride rotation — the zero-RNG counterpart of Poisson arrivals, same construction as the shape streams, and composable with them. The in-flight population floats, arrivals beyond the admission cap wait in FIFO, TTFT includes the queue wait, and a diverging waiting queue raises (no steady state at or beyond capacity) instead of hanging. No Little's-law TTFT anchor: the closed-loop cycle identity does not apply to open arrivals. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…near-critical bias amplification h20e trtllm tp4 open-loop sweep (lambda 0.5-0.95x capacity, aiperf poisson): TTFT p50 within 4-8% and ITL spike position exact at rho<=0.7; measured p99 tails ~2x heavier than the anti-clustered deterministic stream predicts; at rho>=0.85 the known timing-layer capacity bias (+12% optimistic) is amplified by 1/(1-rho) into 2-2.7x errors. Documented as usage bounds in the evaluate_open_loop docstring. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ports Formalizes the tiered interface that grew across the validation campaign. Input tiers: W0 fixed shape, W1 +open-loop rate, W2 +shape marginals, W3/W4 contract placeholders (joint shape/prefix streams, temporal structure). QueueingReport.workload_fidelity declares the tier each evaluation consumed (workload_fidelity() helper, set by all evaluators including static_report); evaluators that do not consume a tier reject its inputs loudly — the fixed-shape disagg tandem now raises on shape quantiles instead of silently pricing a homogeneous tandem. Design doc gains §3.1 (input tiers x evaluator cost ladder x measured error-bar clauses) and the §3 backend table is refreshed to the validated status. stratified_quantiles/workload_fidelity exported. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
WorkloadSpec.shape_tuples: (isl, prefix, osl) triples drawn as a unit from the deterministic stratified stream (stratified_shape_tuples: one stratum = one real trace record at a work-ordered quantile midpoint), so isl<->osl correlation and PER-REQUEST prefix hits survive — marginal streams can carry neither. Slots gain a prefix field consumed by all calendars. WorkloadSpec.arrival_quantiles: raw empirical inter-arrival strata for the open loop (zeros express batched arrivals), exact-mean normalized to 1000/request_rate. Fidelity tier W3 goes live; the fixed-shape disagg tandem rejects W3 inputs loudly. Motivated by the Mooncake conversation trace: 3s-bucketed batch arrivals, heavy-tailed isl (p50 6.9k / p99 85k), and 37% of input tokens arriving prefix-cached — none expressible at W2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
evaluate_open_loop gains arrival_trace: a verbatim (arrival_ms, isl, prefix, osl) sequence evaluated with pairing and ordering preserved — millisecond-cost analytical trace replay, no simulator. Motivation (Mooncake window, isolated by successive elimination): the deterministic quantile streams reproduce marginals but destroy the trace's temporal joint structure (which request lands in which burst; multi-turn follow-ups arriving prefix-hot right after their parent). Shape strata K=32->600 and arrival strata K=32->599 both left TTFT p50 at ~+170% vs the live replay; the exact-trace mode alone closes it to +59% (mean +126%->+22%, p99 +63%->-15%). The earlier 'burst-internal chunk scheduling' hypothesis was disconfirmed by a controlled probe (an AIC-timed reference simulator produces the same serial ladder as the calendar on uniform bursts). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Mooncake cached_tokens discrimination campaign exonerated the cache oracle (offline page-LRU matches the engine's own per-request accounting to ~1%; causality and capacity refinements immaterial) and localized the remaining congestion residual to the fused mixed-pass discount: at long-context continuous-prefill load, disabling the discount moves TPOT -46%->-24% and lands ITL/TTFT p99 within 4%/0.4% of the live engine. Documented as a regime-validity clause on mixed_pass_ms; the median-TTFT remainder is the same prefill-drain-speed mechanism as the fixed-8192 controlled experiment (single named item, two symptoms). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…request trace diagnostics Closes the Mooncake trace-replay residual (TTFT p50 +59% / TPOT -46%) by fixing two general modeling problems found via per-request three-way diff (calendar vs AIC-mocker vs live engine) and live scheduler probes: 1. Arrival plane: trace timestamps record client dispatch, but the engine's FCFS queue orders by scheduler arrival. The ingest path (serialize -> HTTP -> tokenize) has a size-dependent slope, so near-simultaneous dispatches serve shortest-first. Live probe: 23k+1k prompts dispatched together always serve small-first, flip at 60-100ms lag => ~3.6us/token; replay data: 96% same-bucket big-then-small pairs invert while dispatch order is 0% inverted. New WorkloadSpec.ingest_us_per_token (default 0 = old behavior), applied in evaluate_open_loop; stable sort keeps exact ties in dispatch order. 2. Mixed-pass hook regime: run_agg's t_mix prices chunk attention via prefix*floor(ctx_tokens/isl) - the workload-average shape, exact only for whole-prompt passes. Mid-prompt chunks resume at growing past (8k chunk at 16k past ~1.45x fresh) which the signature cannot express; per-pass optimism shifted the in-flight equilibrium (N 7.8 vs 15) and compounded to TPOT -38%. Calendars now route passes carrying a resumed or budget-clipped chunk to the prefill+decode sum, which prices computed_before as per-pass past. Whole-prompt regime unchanged (fixed-8192: hook off moved TTFT p99 +0.4%). Chunking semantics themselves are CONFIRMED correct: live iter-log probes show FCFS greedy head chunks with last-chunk leftover co-scheduling, exactly FusedCalendar.step (a queued 1k prompt rides the 23k prompt's final 6.8k chunk: ctx_req=2, ctx_tok=7823). Mooncake replay after both fixes (h20e trtllm tp4, 600 reqs, rate 0.6/s): TTFT p50 -11%, p99 -7%, TPOT -5%, X +4%, in-flight N 13.3 vs 15.0; work- banded TTFT p50 small/mid/large/xl = -20%/-7%/-7%/+2% (was +146%/+64%/ -15%/-30%). Beats the AIC-backend dynamo mocker on the same trace (TTFT p50 -13%, TPOT -14%) at zero GPU cost. Also: QueueingReport.per_request diagnostics for arrival_trace mode (the tool that localized this), regression anchors re-run (step0 C=32 chunked-ON moved closer to measurement: 3204->3301 vs ~3300 measured; chunked-OFF arms structurally unchanged), 66 unit tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…erification Generalization campaign on three held-out traces (Mooncake toolagent, Mooncake synthetic, InferenceX cc-traces Claude Code sessions at 131k yarn) validated the calendar structure unchanged across prefix reuse 5.9-92.7%, bucketed/exact/session-causal arrivals, isl 6k-68k. Two measured refinements to the ingest_us_per_token documentation: - The slope is mildly superlinear in isl: 3.6 us/tok at <=23k (flip-point differential) vs 6.4/7.5 us/tok at 50k/100k (fully-cached solo TTFT floor ~750ms at 105k, cached_tokens = isl-1). Measuring c at the trace's dominant isl scale moved the cc-traces ~66k-prefix small-work band from -49% to -9%; magnitude-insensitivity holds except for prefix-dominated workloads where TTFT ~ the overhead. - Placement verified by a queued-probe experiment: a fully-cached 105k request dispatched into a 100k prefill backlog gains ZERO post-queue delay (its first token co-schedules with the backlog's last-chunk leftover, four trials, 0 ms spread) - the overhead is entirely pre-queue and absorbed by queue wait exactly as the arrival-plane term assumes. No in-service block-match/KV-attach cost term is needed in the calendar. Also quantified (documented in experiment records, not patched, per the timing-layer boundary policy): DB decode pricing at bs<=3 runs -13~-14% fast across ctx 8k-105k (flat ~1.2-2ms/step host component not hidden by overlap at tiny batch), which the in-flight equilibrium amplifies to the cc-traces system-level TPOT -28%. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The only modeled admission-under-KV-pressure behavior is the GUARANTEED_NO_EVICT gate (trtllm calendar, validated in the KV-capped deep-queueing arm). vLLM recompute-preemption and SGLang retract are different dynamics no calendar reproduces, yet the vllm/sglang calendars silently ignored kv_capacity_tokens (and trtllm silently ignored it without the no-evict flag, i.e. MAX_UTILIZATION) — returning optimistic numbers for KV-tight deployments. That violates the same loud-rejection contract the workload-fidelity tiers follow. BaseCalendar.admission_cap now raises on kv_capacity_tokens or guaranteed_no_evict for backends without modeled semantics; the trtllm calendar raises on kv_capacity_tokens without guaranteed_no_evict. The screening funnel (refine.py) never sets these fields, so no production path changes. Backend table in the design doc carries the scope note. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s p99 anti-clustering) Poisson inter-arrivals are i.i.d. — serial correlation zero. The golden-stride rotation is negatively correlated by construction (windowed count dispersion 0.25 vs Poisson's 1.0): it anti-clusters, starving queueing tails. Block permutations overshoot the other way (positive correlation, dispersion 1.8-14). The correct deterministic order is correlation-FREE: a per-period counter-seeded shuffle (Random(period*2654435761+97003) — a pure function of the period index, bit-for-bit reproducible; the zero-RNG rule bans runtime nondeterminism, not pseudorandomness). Arrival strata widen 64 -> 256 so the without-replacement correlation (~ -1/k) stays small while the 512-request default window still spans two full periods (windowed rate exact). M/D/1 calibration vs true Poisson: dispersion 0.96, queue p99 within 8% at rho<=0.7, -18% at rho 0.83 (was -80%); residual = finite- period truncation of bursts rarer than 1/k, documented. W3 empirical arrival_quantiles keep the validated golden-stride path unchanged (Mooncake stream-mode arm was produced with it); exact-trace replay is untouched. Acceptance vs exp6 live measurements (h20e trtllm tp4, aiperf --arrival-pattern poisson, isl4096/osl256, chunked off): rate 0.83: TTFT mean/p50/p99 was -19%/-4%/-49% -> -10%/-3%/-24% rate 1.15: was -25%/-28%/-54% -> -3.8%/-3.2%/-16% Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Promotes the accuracy-bearing pieces of the trace validation pipelines out of per-experiment scripts into a tested module (sdk/queueing/trace.py): - prefix_hits: leading-hit page-LRU oracle (engine page pool capacity, global LRU, leading full pages only) — the semantics verified against live-engine cached_tokens self-reporting to +14/+15 tokens mean at 34-93% reuse during the validation campaign. - TraceRecord.pages: position-aligned (namespace, block_hash, page_idx) ids from content-block hashes; trailing partial page dropped like the engine; session namespacing for locally-scoped hashes. - load_mooncake_jsonl / load_cc_sessions_jsonl: format loaders (cc subagent groups flattened by their own timestamps; oversized sessions dropped whole to keep multi-turn prefix chains intact). - workload_from_trace: both W-contract consumption paths from one window — exact arrival_trace tuples and the sweepable W3 stream form (joint shape tuples + empirical inter-arrival quantiles), plus audit numbers (reuse fraction, replay-clock rate). Identity contract documented: prefixes are computed on the trace's own token identity (models the system that produced the trace); tokenizer round-trip inflation is a replay-harness artifact and stays out of the adapter. Cross-checked against the experiment pipelines on real traces: conversation 17.2% vs 16.9%, toolagent 49.8% vs 49.3%, cc-traces 94.3% vs 92.7% reuse (residuals = the retokenization artifact + session selection). 14 new unit tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Agentic workloads have ENDOGENOUS arrivals: turn k+1 dispatches at completion_k + think gap, so the arrival process depends on the serving speed being predicted — inexpressible in the closed loop (immediate replacement) or the open loop (exogenous timestamps); exact-trace replay sidesteps it only by consuming measured dispatch times. evaluate_sessions(sessions, ...) runs the same backend calendars over session LANES (one outstanding request per lane, sequential turns with per-turn isl/prefix/osl and think_ms); dispatch instants are outputs of the recursion, mapped through the same arrival plane (epsilon + ingest slope). per_request carries session/turn ids and the EMERGENT arrival_ms so session dynamics themselves can be audited against a replay, not just latency-given-arrivals. A-priori validation (cc-traces, 10 sessions / 315 turns, think gaps from recorded t/api_time capped at 15 s, prefixes from the SDK page oracle, ZERO measured arrivals fed): TTFT work-bands -11~-35%, p90/p99 +8/+13%; emergent dispatch drift |dt| p50 81 s over 6-40 min sessions with -105 s bias — the frozen decode-corner timing bias advances the model timeline and desynchronizes prefill storms, which is the named W4 error-propagation path (token-cadence metrics degrade through desync while prefill-dominated TTFT bands survive). Design-doc W-tier table updated (W4 row implemented + W1 p99 record refreshed for the correlation-free arrival stream). 6 new unit tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- ruff check/format clean across the queueing module and its tests (import sorting, zip strict=, context-managed file reads, dead local, pairwise, __all__ updated with the trace/session exports). - calendar.py module docstring no longer claims trtllm/sglang are 'not yet validated' (both carry live-validation records) and now describes both loop modes; package __init__ docstring no longer declares open-loop/trace inputs out of scope (they are entry points). - design doc §1 points shape/trace/session inputs at the §3.1 fidelity contract instead of the stale fixed-shape-only scope sentence. No behavior changes; 90 tests pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…xact trace replay) The tandem previously accepted only fixed-shape closed-loop workloads, so none of the trace-validated workload axes (shape marginals/joints, per-request prefix, empirical arrivals, verbatim replay) reached disagg. - spec: factor the deterministic shape draw (_shape_drawer) and the open-loop inter-arrival construction (_interarrival_stream: empirical golden-stride rotation / correlation-free per-period shuffle) out of calendar.py into shared helpers; agg and disagg now consume identical workload streams by construction. Agg behavior is bit-identical (90-test suite + 9-family oracle gate unchanged). - disagg: per-request (isl, prefix, osl) throughout the recursion; KV handoff priced per request (isl x kv_bytes_per_token — full context, cached prefix saves prefill compute, not transfer bytes); open-loop arrivals routed static-RR at the scheduler-visibility instant (same arrival-plane mapping: turnaround + isl x ingest slope) with a prefill backlog divergence guard; arrival_trace exact replay with per_request diagnostics incl. xfer_ms; evaluate_disagg_mixed passes open-loop workloads through (no initial-cohort phase to mix over). - honesty: the tandem now rejects kv_capacity_tokens/guaranteed_no_evict loudly (no KV admission gate or hold-until-transfer accounting is modeled) instead of silently ignoring them. Fixed-shape closed-loop path (the DES-gated core) is unchanged; the gate families pass as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Factor the tandem event loop into disagg._run_tandem — ONE copy of the pass/handoff semantics driven by every disagg entry point (closed loop, open loop, trace replay, session lanes) through an on_complete hook and a dynamic pending feed. evaluate_disagg becomes a thin wrapper (fixed-shape closed-loop path bit-identical: oracle gate unchanged). evaluate_sessions_disagg: same lane semantics as the agg evaluator (turn k+1 dispatches at completion_k + think, same arrival plane), tandem serving flow (static-RR prefill router, per-turn KV handoff on the max-min-fair fabric, first token prefill-side); per_request carries session/turn, the emergent arrival_ms, and xfer_ms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…submits DES DisaggSimulator gains trace mode (concurrency=None: arrivals dispatched at their own arrival_ms, mirroring the agg Simulator); workload.from_tuples builds heterogeneous open-loop request lists. Two new GATED families feed the same verbatim (arrival, isl, osl) tuples to the DES and to evaluate_disagg(arrival_trace=...) and join per request. The DF family (variable-shape fan-in 2P1D) caught a real defect: tandem passes are computed with FUTURE end timestamps in worker-index order, and submitting KV flows straight into the fluid fabric advanced its clock out of time order — later-indexed workers' earlier flows were clamped to a later start, inflating small transfers to tens of ms (itl_p99 +15.7% vs the DES; invisible to the closed-loop families, whose phase-locked cycles keep pass ends monotone). Handoffs now queue in a time-ordered outbox and join the fabric through the event loop. After the fix both open-loop families gate clean with per-request |dTTFT| median/p90 = 0.0%. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
enable_chunked_prefill=False on the prefill engine now stops admission once a whole prompt no longer fits the remaining budget (the agg FusedCalendar rule) instead of always splitting — TRT-LLM disagg ctx workers deploy chunked-off, so the tandem was co-scheduling head chunks the real engine serializes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…_ttft) The two real disagg flows place the handoff differently, and only a slow fabric can tell them apart: the DES-idealized flow streams the prefill worker's first token immediately (handoff in the FIRST ITL GAP — the gate families pin this default), while TRT-LLM native disagg and the dynamo frontend are decode-attach (first token user-visible only after the KV transfer: measured h20e slow-link A/B, UCX_TLS=cuda_copy,tcp — c=1 TTFT absorbed the full 2.2 s transfer while the decode ITL spike stayed unchanged). DisaggSpec.handoff_in_ttft=True selects the decode-attach flow: first-token emission moves to transfer completion, gap 1 is a clean decode gap, e2e is invariant. On NVLink the flows differ by ~3 ms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tandem's prefill admission was budget/kappa-gated only; a ctx worker deployed at bs4 co-schedules at most 4 prompts per pass regardless of the token budget. Measured (cc-traces 48k window on live 1P1D): kappa-only brackets both miss under deep queueing; with the bs cap and per-request engine-reported prefixes the exact-replay arm lands at per-request |dTTFT| median 2% (p90 14%), distribution -5/-3/-2%. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Closes the oracle-vs-engine reuse gap found in the cc-traces disagg round (oracle 58.4% vs engine cached_tokens 62.7%, which queueing amplified into TTFT +44% at rho=0.72). Three named components, two of them modeled: - prefix_hit_tokens: token-granular oracle with the engine's partial-tail block reuse (TRT-LLM enable_partial_reuse) — the prompt's trailing partial page reuses when the full leading run hit; matched, never inserted. TraceRecord.pages_and_tail exposes the tail pair. - eviction="leaf-lru": radix-tree leaf-first eviction with LRU tiebreak (TRT-LLM block-reuse semantics). The flat page-LRU hole-punches chains mid-prefix under pressure, collapsing leading runs to zero on turns the engine serves ~100% cached; leaf-lru shrinks chains tail-to-root. Measured (cc window, 97 turns, 132k pool): with capacity = pool minus expected in-flight KV the oracle lands +257 tokens/turn of the engine and the exact-replay arm goes from TTFT +44% to mean -1%/p99 +3%. - The remaining unit mismatch (trace tokens vs the replay's re-tokenized server tokens, ~5.3% inflation) is a replay artifact handled by the comparison layer, per the module's identity contract. prefix_hits and workload_from_trace defaults are unchanged (bit-identical to every validated round); workload_from_trace gains partial_tail_reuse and eviction passthroughs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
Sharing measurements from a disagg investigation (context: #1396) that this model would be well placed to explain — but, as currently scoped, would not predict. Offering them as a possible validation case for the disagg tandem, plus three concrete gaps. Setup. 8×B200, Kimi-K2.5-NVFP4, dummy weights ( Finding 1 — decode-side KV pool bounds concurrent transfers, and that is what sets throughputIn vLLM's NixlConnector the consumer pulls: a hard cap on in-flight handoffs. Sweeping decode
Controlled test — same
Headroom is causal. Note also that at low headroom mean TPOT 45.2 ms vs median ITL 16.6 ms — the loss is a stall/preempt long tail, not slower steps. Two things in the design doc keep this out of reach today:
Finding 2 — with a pull-based connector the prefill slot is held until the transfer completes, not until prefill completes
Direct test — raise prefill
+67% at conc 96 from a prefill-side knob that the model treats as non-binding. With κ=1 the model would put prefill capacity at 4 / 0.32 s = 12.5 req/s and never flag it — over-predicting in the same direction as the current estimator. The conc-144 regression is the interesting part: the two limiters are coupled. More prefill supply only helps while decode staging headroom lasts; past that, the extra in-flight requests re-flood the pool and throughput falls below the κ=1 baseline. A model with both effects would reproduce this non-monotonicity; a model with neither predicts monotone improvement. Finding 3 — a NIC-bandwidth fabric is the wrong cost model for intra-node handoffs
Suggested minimal changes
Separately, this matters for config picking, not just latency reporting: the current sweep sizes the decode pool as though the running batch were its only occupant, so a recommended Happy to re-run any of these, share the raw logs, or test a patched branch. |
Contributor
Author
|
Correction to my comment above, plus a direct measurement of the mechanism. I attributed the mean-TPOT-45.2 ms / median-ITL-16.6 ms tail to vLLM's preempt-and-recompute. That is wrong — there were zero preemptions in every run. vLLM V1 appends The same log line carries
Headroom +11.1 → +4.2 halves the staging pipeline depth (7.5 → 3.0 concurrent transfers) while the pool saturates at 99.6%. Nothing is evicted; the transfer pipeline is starved of admission. (The third row has zero This makes the modelling ask simpler and more mechanical than what I wrote before. There is no eviction dynamic to reproduce, so the calendar does not need vLLM's preemption semantics at all. It needs one state variable: with the tandem's transfer stage serialized as that bound approaches 0. For reference on the engine side: the scheduler's |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
sdk/queueing(shipped inaiconfigurator_core.sdk.queueing, compat aliasaiconfigurator.sdk.queueing): an analytical queueing correction for continuous batching, percentile SLA semantics resolved by a two-stage sweep funnel, a disagg P/D tandem model with computed KV-transfer contention, and the DES oracle + validation gate that anchor all of it, running in CI. Every term traces to a scheduling-loop mechanism or a queueing-theory identity; there are no fitted constants (provenance table indocs/design/queueing_model.md).Fifteen additive summary columns across
ColumnsAgg/ColumnsStatic/ColumnsDisagg(legacyttft/tpotuntouched):ttft_steady_{mean,p50,p75,p90,p95,p99,p999}— steady-state TTFT distribution (SLA semantics)ttft_steady_p99_{lo,hi}— the cohort bracket: structural bounds on the steady distribution's support, used by the sweep funnelttft_transient_{mean,max}— initial-burst admission-staircase envelope (cold start / synchronized burst)itl_{mean,p50,p99}— inter-token-latency distributionqueueing_tier— precision class of each row's numbers ("screening" | "quantitative" | "static" | "composed")Why
The agg TTFT estimate used an empirical queuing factor whose blended mean underestimated dynamic TTFT by ~30% on a reference workload and depends on benchmark length N (same deployment: 456ms at 200 requests, 223ms at 1000) — neither a steady-state nor a percentile semantic; as an SLA gate it both falsely accepts and falsely rejects. Disagg had it worse: the composed tier reports the prefill worker's single static-batch latency as TTFT, invisible to closed-loop queueing — on a rate-matched 5P3D deployment at C=252 the composed value is 183.8ms while the tandem p50 is 1185.7ms (prefill-pool queueing dominates).
How
One model, two precision tiers — screening (O(1) closed form on quantities
run_aggalready computes: effective-isl chunk counts, encoder/dispatch additive stages; zero extra perf-DB queries) and quantitative (deterministic limit-cycle evaluation of the same model; timing delegates to the SDK's own phase runners, with amixed_pass_mshook onto the mix-step runner so fused passes pay the shared non-attention cost once).Backend calendars: vLLM fused (validated), TRT-LLM fused +
GUARANTEED_NO_EVICTcap, SGLang mixed-chunk by default — matching AIC's own deployment rule — with the alternating (dedicated prefill batch) structure available.Percentile SLA semantics, one rule on every surface: targets become (value, percentile) pairs on the steady distributions. Setting any
--*-percentileflag or an--itltarget (CLI) — or the corresponding YAML field — activates percentile filtering; unset metrics use p50 (ttft/tpot/request-latency) and p99 (itl). The only explicit switch is--sla-refine/sla_refine: a precision upgrade that never changes elimination semantics.Two-stage sweep funnel (
sdk/queueing/refine.py) on the sweep path: (1) wide-keep on the bracket lower bound — screening bias can never falsely reject; (2) lazy quantitative resolution of straddlers in throughput order, all constraints enforced at their percentiles, unresolved rows kept with tier visible. The legacy path (pareto_analysis/find_best_*, webapp) keeps legacy semantics. Report boundary upgrades the rows a human reads to quantitative tier.Disagg tandem model (
sdk/queueing/disagg.py): serving-flow semantics — the prefill pool emits the first (TTFT) token; the KV handoff is a flow on a max-min-fair per-NIC transfer fabric (fan-in/fan-out contention computed from concurrent flows; bandwidths from the system specnode.inter_node_bw× efficiency), landing in the first ITL gap; decode workers pay no prefill compute (KV-connector convention). Rate matching becomes an output: pool imbalance surfaces as prefill queueing or decode saturation. Heterogeneous P/D pools are priced by their own timing models. The tandem system is multi-stable in cohort phase;evaluate_disagg_mixedreports a deterministic phase mixture (no RNG).DES oracle in-tree (
tools/queueing_oracle/, stdlib-only): a discrete-event simulation of vLLM v1 iteration-level scheduling (unified token budget, running-set-first, chunked prefill, put-back-no-preempt admission, LIFO preemption, BlockPool-style KV accounting with prefix reuse, pull-model KV hold until the transfer completes) plus agg and disagg drivers with the same transfer fabric.validate_formula.pyis the gate: identical timing on both sides isolates scheduling semantics; it runs in CI as a unit test (~1s), so upstream scheduler drift is a failing check.Validation (recorded in
docs/design/queueing_model.md§5)itl_p99reproduced to 0.0%.--ttft 210the funnel rescued 16 configs — including the top-throughput one — that the legacy scalar would have falsely rejected. Funnel cost: ~2x sweep time at loose SLAs, ~7x boundary-tight.Testing
tests/unit/sdk/queueing/— 48 unit tests (structural assertions, funnel semantics, disagg tandem, transfer-fabric sharing math, oracle gate + disagg serving-flow/KV-hold checks)aic defaultagg + disagg runs with--sla-refineon the real perf DB; tier distribution verified in saved artifacts🤖 Generated with Claude Code