From 2e68f225c19fced981938c11bd877e0e63bda9e8 Mon Sep 17 00:00:00 2001 From: functionstackx <47992694+functionstackx@users.noreply.github.com> Date: Sat, 25 Jul 2026 00:05:18 -0400 Subject: [PATCH] docs(golden-al): document SGLang and TRT-LLM synthetic acceptance knobs The golden AL README only showed vLLM's synthetic_acceptance_length. Add the SGLang env vars (SGLANG_SIMULATE_ACC_LEN / _METHOD / _TOKEN_MODE) and TRT-LLM's TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS, including the off-by-one: the TRT-LLM value counts accepted draft tokens only and excludes the bonus token, so it must be set to golden AL minus 1 (fractional allowed, e.g. AL 3.5 -> 2.5). Also teach the codeowner sign-off verify prompt (Check 10) to accept TRT-LLM configs and to compare the pinned TRT-LLM value plus 1 against the golden curve. Keep README_zh.md in parity. Requested in https://github.com/SemiAnalysisAI/InferenceX/pull/2309#pullrequestreview-4778348624 Co-Authored-By: Claude Fable 5 --- .github/codeowner-signoff-verify-prompt.md | 8 +++++++- golden_al_distribution/README.md | 14 ++++++++++++++ golden_al_distribution/README_zh.md | 14 ++++++++++++++ 3 files changed, 35 insertions(+), 1 deletion(-) diff --git a/.github/codeowner-signoff-verify-prompt.md b/.github/codeowner-signoff-verify-prompt.md index 5f400f6ea9..7b750f8eb2 100644 --- a/.github/codeowner-signoff-verify-prompt.md +++ b/.github/codeowner-signoff-verify-prompt.md @@ -323,13 +323,19 @@ Verify BOTH: YAMLs, or exported before launch in benchmark scripts). - vLLM: the `--speculative-config` JSON contains BOTH `"rejection_sample_method": "synthetic"` and `"synthetic_acceptance_length": `. + - TRT-LLM: env var `TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS: ` in the + server/decode environment. The value counts accepted DRAFT tokens only and + EXCLUDES the bonus/verification token, so it must be the golden AL minus 1 + (fractional values allowed — e.g. golden AL 3.5 -> 2.5). FAIL if an agentic spec-decode config runs real (unsimulated) acceptance — name the config/script and line. - (b) AL VALUE MATCHES THE GOLDEN CURVE. Read the committed golden AL YAML for the model in `golden_al_distribution/` (default-branch checkout; e.g. `qwen3.5_mtp.yaml`, `kimik2.5_eagle3.yaml`) and confirm the pinned AL equals the golden value for that model, thinking mode, and the config's `num_speculative_tokens` / MTP level (e.g. - qwen3.5 thinking_on with 3 speculative tokens -> 3.39). A submission may choose any + qwen3.5 thinking_on with 3 speculative tokens -> 3.39). For TRT-LLM configs, compare + the pinned `TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS` value PLUS 1 against the + golden AL (the env var excludes the bonus token). A submission may choose any supported draft length, but it may NOT substitute a different acceptance target. FAIL on a mismatch — name the config, the pinned value, and the expected golden value. If the model has no committed golden curve yet, do not guess: the sign-off's diff --git a/golden_al_distribution/README.md b/golden_al_distribution/README.md index c1a4f64a33..93666c4fba 100644 --- a/golden_al_distribution/README.md +++ b/golden_al_distribution/README.md @@ -39,6 +39,20 @@ vllm serve MODEL \ The option was unified across vLLM model runners in [vllm-project/vllm#40662](https://github.com/vllm-project/vllm/pull/40662). +SGLang supports the same policy through its simulated-acceptance environment variables, set in the server environment (the `aggregated_environment` / `decode_environment` sections of srt-slurm YAMLs, or exported before launch in benchmark scripts): + +```yaml +SGLANG_SIMULATE_ACC_LEN: '3.24' # AL from the committed golden YAML +SGLANG_SIMULATE_ACC_METHOD: match-expected +SGLANG_SIMULATE_ACC_TOKEN_MODE: real-draft-token +``` + +TensorRT-LLM supports it through [`TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS`](https://github.com/NVIDIA/TensorRT-LLM/blob/2cbdaa0ffa36fbef7960a0ad9f0458373025fa9f/tensorrt_llm/_torch/speculative/interface.py#L1065-L1078). **Note the off-by-one:** this variable counts accepted *draft* tokens only and excludes the bonus/verification token, so set it to the golden AL **minus 1**. Fractional values are supported — the integer part is accepted every iteration and the fractional part is the probability of accepting one additional draft token. For example, a golden AL of `3.5` becomes: + +```bash +TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS=2.5 +``` + This policy follows the same broad principle as MLPerf Inference: prescribe the workload rules needed for comparable system measurements. InferenceX is evaluating inference-system performance, not the ability to fine-tune a benchmark-specific speculative head. ## How a golden AL curve is collected diff --git a/golden_al_distribution/README_zh.md b/golden_al_distribution/README_zh.md index a62a77e227..e4231027e5 100644 --- a/golden_al_distribution/README_zh.md +++ b/golden_al_distribution/README_zh.md @@ -39,6 +39,20 @@ vllm serve MODEL \ 该选项通过 [vllm-project/vllm#40662](https://github.com/vllm-project/vllm/pull/40662) 在 vLLM 的不同模型运行器中实现统一。 +SGLang 通过其模拟接受环境变量支持同一策略。这些变量设置在服务环境中（srt-slurm YAML 的 `aggregated_environment` / `decode_environment` 部分，或在基准脚本中于启动前导出）： + +```yaml +SGLANG_SIMULATE_ACC_LEN: '3.24' # 来自已提交黄金 YAML 的 AL +SGLANG_SIMULATE_ACC_METHOD: match-expected +SGLANG_SIMULATE_ACC_TOKEN_MODE: real-draft-token +``` + +TensorRT-LLM 通过 [`TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS`](https://github.com/NVIDIA/TensorRT-LLM/blob/2cbdaa0ffa36fbef7960a0ad9f0458373025fa9f/tensorrt_llm/_torch/speculative/interface.py#L1065-L1078) 支持该策略。**注意存在差一（off-by-one）：** 该变量只统计被接受的*草稿* token，不包括 bonus/验证 token，因此应设置为黄金 AL **减 1**。支持小数值——整数部分在每次迭代中始终被接受，小数部分是额外接受一个草稿 token 的概率。例如，黄金 AL 为 `3.5` 时应设置： + +```bash +TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS=2.5 +``` + 这一策略遵循与 MLPerf Inference 相同的广义原则：规定可比较系统测量所需的工作负载规则。InferenceX 评估的是推理系统性能，而不是微调特定基准推测头的能力。 ## 黄金 AL 曲线如何收集