Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion .github/codeowner-signoff-verify-prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -323,13 +323,19 @@ Verify BOTH:
YAMLs, or exported before launch in benchmark scripts).
- vLLM: the `--speculative-config` JSON contains BOTH
`"rejection_sample_method": "synthetic"` and `"synthetic_acceptance_length": <AL>`.
- TRT-LLM: env var `TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS: <AL - 1>` in the
server/decode environment. The value counts accepted DRAFT tokens only and
EXCLUDES the bonus/verification token, so it must be the golden AL minus 1
(fractional values allowed — e.g. golden AL 3.5 -> 2.5).
FAIL if an agentic spec-decode config runs real (unsimulated) acceptance — name the
config/script and line.
- (b) AL VALUE MATCHES THE GOLDEN CURVE. Read the committed golden AL YAML for the
model in `golden_al_distribution/` (default-branch checkout; e.g. `qwen3.5_mtp.yaml`,
`kimik2.5_eagle3.yaml`) and confirm the pinned AL equals the golden value for that
model, thinking mode, and the config's `num_speculative_tokens` / MTP level (e.g.
qwen3.5 thinking_on with 3 speculative tokens -> 3.39). A submission may choose any
qwen3.5 thinking_on with 3 speculative tokens -> 3.39). For TRT-LLM configs, compare
the pinned `TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS` value PLUS 1 against the
golden AL (the env var excludes the bonus token). A submission may choose any
supported draft length, but it may NOT substitute a different acceptance target.
FAIL on a mismatch — name the config, the pinned value, and the expected golden
value. If the model has no committed golden curve yet, do not guess: the sign-off's
Expand Down
14 changes: 14 additions & 0 deletions golden_al_distribution/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,20 @@ vllm serve MODEL \

The option was unified across vLLM model runners in [vllm-project/vllm#40662](https://github.com/vllm-project/vllm/pull/40662).

SGLang supports the same policy through its simulated-acceptance environment variables, set in the server environment (the `aggregated_environment` / `decode_environment` sections of srt-slurm YAMLs, or exported before launch in benchmark scripts):

```yaml
SGLANG_SIMULATE_ACC_LEN: '3.24' # AL from the committed golden YAML
SGLANG_SIMULATE_ACC_METHOD: match-expected
SGLANG_SIMULATE_ACC_TOKEN_MODE: real-draft-token
```

TensorRT-LLM supports it through [`TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS`](https://github.com/NVIDIA/TensorRT-LLM/blob/2cbdaa0ffa36fbef7960a0ad9f0458373025fa9f/tensorrt_llm/_torch/speculative/interface.py#L1065-L1078). **Note the off-by-one:** this variable counts accepted *draft* tokens only and excludes the bonus/verification token, so set it to the golden AL **minus 1**. Fractional values are supported — the integer part is accepted every iteration and the fractional part is the probability of accepting one additional draft token. For example, a golden AL of `3.5` becomes:

```bash
TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS=2.5
```

This policy follows the same broad principle as MLPerf Inference: prescribe the workload rules needed for comparable system measurements. InferenceX is evaluating inference-system performance, not the ability to fine-tune a benchmark-specific speculative head.

## How a golden AL curve is collected
Expand Down
14 changes: 14 additions & 0 deletions golden_al_distribution/README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,20 @@ vllm serve MODEL \

该选项通过 [vllm-project/vllm#40662](https://github.com/vllm-project/vllm/pull/40662) 在 vLLM 的不同模型运行器中实现统一。

SGLang 通过其模拟接受环境变量支持同一策略。这些变量设置在服务环境中(srt-slurm YAML 的 `aggregated_environment` / `decode_environment` 部分,或在基准脚本中于启动前导出):

```yaml
SGLANG_SIMULATE_ACC_LEN: '3.24' # 来自已提交黄金 YAML 的 AL
SGLANG_SIMULATE_ACC_METHOD: match-expected
SGLANG_SIMULATE_ACC_TOKEN_MODE: real-draft-token
```

TensorRT-LLM 通过 [`TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS`](https://github.com/NVIDIA/TensorRT-LLM/blob/2cbdaa0ffa36fbef7960a0ad9f0458373025fa9f/tensorrt_llm/_torch/speculative/interface.py#L1065-L1078) 支持该策略。**注意存在差一(off-by-one):** 该变量只统计被接受的*草稿* token,不包括 bonus/验证 token,因此应设置为黄金 AL **减 1**。支持小数值——整数部分在每次迭代中始终被接受,小数部分是额外接受一个草稿 token 的概率。例如,黄金 AL 为 `3.5` 时应设置:

```bash
TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS=2.5
```

这一策略遵循与 MLPerf Inference 相同的广义原则:规定可比较系统测量所需的工作负载规则。InferenceX 评估的是推理系统性能,而不是微调特定基准推测头的能力。

## 黄金 AL 曲线如何收集
Expand Down