Feature: Support Multi-Token Prediction (MTP) Training for Language and Speech Modalities
Motivation
DeepSeek-V3 introduced Multi-Token Prediction (MTP) — a training paradigm where the model predicts multiple future tokens simultaneously using shared transformer trunk with independent output heads. This approach has been shown to improve both training efficiency and inference throughput via speculative decoding.
More recently, StepAudio 2.5 (StepFun, May 2026) extended MTP to the speech domain with MTP-5, demonstrating that a depth-5 MTP module can accelerate ASR inference to ~500 tokens/s while reducing inference cost by 80%. This suggests MTP is a modality-agnostic technique with broad applicability.
Requested Features
-
MTP Training Support for Language Models — Implement the MTP training objective described in DeepSeek-V3 Sec 3.4, where K auxiliary prediction heads (sharing the main trunk) predict the next K tokens in parallel. The loss is computed as the weighted sum of the main cross-entropy loss and the MTP auxiliary losses.
-
MTP-5 Training for Speech/Audio Models — Extend MTP training to speech recognition/understanding tasks, following the StepAudio 2.5 recipe: an audio encoder → linear adapter → LLM backbone → MTP-5 module pipeline. This includes designing appropriate speech tokenization and alignment strategies so that the MTP heads can predict multiple future speech tokens per step.
-
Configurable MTP Depth and Architecture — Allow users to specify:
- Number of MTP layers (K, e.g., 1–5)
- Whether MTP heads share the transformer trunk (DeepSeek-V3 style) or use independent lightweight modules
- Loss weighting coefficient for auxiliary MTP losses
-
Speculative Decoding Integration — After MTP training, export the MTP module as a draft model for speculative decoding to improve inference throughput without quality degradation.
References
Feature: Support Multi-Token Prediction (MTP) Training for Language and Speech Modalities
Motivation
DeepSeek-V3 introduced Multi-Token Prediction (MTP) — a training paradigm where the model predicts multiple future tokens simultaneously using shared transformer trunk with independent output heads. This approach has been shown to improve both training efficiency and inference throughput via speculative decoding.
More recently, StepAudio 2.5 (StepFun, May 2026) extended MTP to the speech domain with MTP-5, demonstrating that a depth-5 MTP module can accelerate ASR inference to ~500 tokens/s while reducing inference cost by 80%. This suggests MTP is a modality-agnostic technique with broad applicability.
Requested Features
MTP Training Support for Language Models — Implement the MTP training objective described in DeepSeek-V3 Sec 3.4, where K auxiliary prediction heads (sharing the main trunk) predict the next K tokens in parallel. The loss is computed as the weighted sum of the main cross-entropy loss and the MTP auxiliary losses.
MTP-5 Training for Speech/Audio Models — Extend MTP training to speech recognition/understanding tasks, following the StepAudio 2.5 recipe: an audio encoder → linear adapter → LLM backbone → MTP-5 module pipeline. This includes designing appropriate speech tokenization and alignment strategies so that the MTP heads can predict multiple future speech tokens per step.
Configurable MTP Depth and Architecture — Allow users to specify:
Speculative Decoding Integration — After MTP training, export the MTP module as a draft model for speculative decoding to improve inference throughput without quality degradation.
References