What is the difference between multi-token prediction and speculative decoding?

Multi-token prediction is a property of the model itself: additional heads trained alongside it to predict several future tokens. Speculative decoding is the inference technique that exploits them together with a verification mechanism: when MTP heads act as their own proposer, that is self-speculative decoding, a particular case covered in the dedicated speculative decoding entry.

Can any inference engine exploit a model's MTP heads?

No: the capability exists in the model's weights, but it must be explicitly implemented in the serving engine to be used as a speculative decoding proposal. An engine that does not support it simply discards the MTP heads and generates in standard autoregressive fashion.

What is multi-token prediction in LLMs?

A technique that adds multi-token prediction heads to an LLM, originally built for training and reused to speed up inference.

Multi-token prediction is an architectural change to a language model that adds extra prediction heads capable of forecasting several future tokens at each step, instead of only the next one. It originates not as an inference trick but as a richer training signal: forcing the model to also predict the second and third token ahead pushes it to build more coherent long-range internal representations, a sample-efficiency gain that Meta's original 2024 paper measures as 12% more problems solved on HumanEval and 17% more on MBPP for a 13-billion-parameter model, in relative terms and not in percentage points. The same idea can then be reused to speed up generation: that is the case of DeepSeek-V3, where the MTP module proposes a token that the model itself verifies, a form of speculative decoding that needs no separate second model.

Where MTP heads come from and what changes when they run at inference

A standard model has a single prediction head, trained to estimate the probability of the immediately following token given the entire preceding context. The Meta paper adds n-1 additional heads (n=4 in the studied version, so three beyond the standard one), each conditioned on the same shared trunk but trained to predict the token 2, 3 and 4 steps ahead. In many architectures those extra heads are discarded at the end of training: they only exist to provide a denser learning signal that pushes the model to plan past the immediate token. They only become an inference tool when they are kept after training, as in DeepSeek-V3: the model proposes its own next token with its own MTP module and verifies it with the standard prediction head, a case of self-speculative decoding covered in detail in the entry on speculative decoding, not repeated here: the difference is that no separate draft model needs to be kept around, the proposing capability already lives inside the model's own weights. Frameworks like Medusa and EAGLE work the same territory by grafting similar heads onto an already-trained model, without retraining it from scratch, but they remain variants of the same principle. The operational point for anyone choosing an inference engine: a model with native MTP heads only uses that capability if the runtime implements it explicitly, and the leading serving engines today expose it as a speculative decoding method that has to be switched on in configuration.

An enterprise example

The best-documented case is exactly the one in the DeepSeek-V3 technical report: the model reaches an 85-90% acceptance rate for the second token proposed by its own MTP module, and that translates into 1.8 times the decoding speed, measured in tokens per second per request. The number is not universal: it depends entirely on the acceptance rate of the proposed tokens, which in turn depends on how closely the MTP module's distribution matches what the model would have produced token by token during the decode phase. The gain peaks on an interactive, low-concurrency workload, where the time of a single response is what counts, and it thins out as parallel requests saturate the GPU: anyone weighing it for a batch workload should expect less than that number, not that number. The MTP module shares the embedding and the output head with the main model, but it has a Transformer block of its own, so it carries an added, non-zero memory cost for its KV cache.

Why it matters for decision makers

Multi-token prediction moves the acceleration lever inside the model rather than into a satellite component: there is no second model to version, update and keep in sync with the main one, unlike speculative decoding with an external draft. The flip side is that the capability is native: a company cannot bolt it onto a model that lacks it simply by switching inference engines; it has to choose a model that includes it from the start, or a framework like Medusa/EAGLE to graft it onto an existing model. Anyone evaluating an open-weight model for self-hosted inference should check two things together: whether the model exposes native MTP heads, and whether the chosen serving engine actually knows how to exploit them, because the second condition without the first is useless and the first without the second remains wasted capability.