Blog: Add Office Hours #51 recap — Speculators v0.5 and sparse MLA#247
Blog: Add Office Hours #51 recap — Speculators v0.5 and sparse MLA#247soyr-redhat wants to merge 1 commit into
Conversation
c8158f5 to
6986559
Compare
6986559 to
cbe6091
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c8158f5d19
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
|
||
| Training speculator models requires hidden states from the target model. Speculators now supports a new extraction system with two primary modes: | ||
|
|
||
| **Online training:** The vLLM server extracts hidden states and sends them directly to the training process in real time via NCCL. This keeps the pipeline simple and avoids disk I/O. |
There was a problem hiding this comment.
Readers trying to reproduce Speculators v0.5 online training will look for direct NCCL transfer and assume disk I/O is avoided, but the v0.5 release post in this repo describes the current path as REST requests to vLLM with hidden states temporarily written to disk/ram disk (_posts/2026-05-28-speculators-v050.md lines 91-99), and the public tutorial uses --vllm-endpoint, --on-missing generate, and --on-generate delete. Please describe this as on-demand file-backed extraction unless NCCL support has actually landed and the rest of the docs are being updated too.
Useful? React with 👍 / 👎.
|
|
||
| ### How MLA works | ||
|
|
||
| MLA compresses the KV cache by projecting keys and values into a shared latent space. In DeepSeek models, this reduces the cached representation from dimension 16,384 down to 512, a 57x reduction. There are two mathematically equivalent ways to compute MLA: |
There was a problem hiding this comment.
The numbers in this sentence do not yield the claimed reduction: 16,384 / 512 is 32x, not 57x. If the intent is the usual DeepSeek MLA KV-cache comparison, it needs to count both K and V plus the decoupled RoPE key, i.e. 32,768 elements down to 576 elements (512 + 64), which is about 57x; otherwise the published post will give readers internally inconsistent dimensions.
Useful? React with 👍 / 👎.
cbe6091 to
adc748f
Compare
adc748f to
9054b76
Compare
Covers the Speculators v0.5 update (DFlash, P-Eagle, Fast MTP, hidden states extraction) and Matt Bonanni's engineering spotlight on accelerating sparse MLA via masked MHA in DeepSeek models. Signed-off-by: Sawyer Bowerman <sbowerma@redhat.com>
9054b76 to
6262bc1
Compare
1 participant
Summary
Authors
Sawyer Bowerman (@soyr-redhat)
Based on presentations by Michael Goin, Fynn Schmitt-Ulms, Helen Zhao, and Matt Bonanni.