What is speculative decoding in LLMs?

Speculative decoding is a technique that speeds up a language model's text generation without changing the final output's quality in the slightest. Instead of generating one token at a time with the large, expensive model (the "target" model), a small and fast model (the "draft" model) proposes a short sequence of candidate tokens in advance. The target model then verifies those candidates in a single parallel pass, instead of generating them one by one in sequence: if they match what it would have produced itself, it accepts them all at once, otherwise it corrects from the first point where they diverge. The key point, often misunderstood, is that the final output is mathematically identical to what the target model would have generated on its own, token by token: it is not an approximation nor a quality trade-off, it is a different and more efficient way of arriving at the exact same result, by exploiting the fact that verifying in parallel costs much less than generating in sequence.

See also: Diffusion language model · A model that generates text by refining an entire masked sequence in parallel, instead of producing it one token at a time.

Why quality does not change

The mathematical guarantee sits in the verification mechanism: the target model does not simply "trust" the draft model, it recomputes the probability distribution it would have used itself for every proposed token and accepts or rejects candidates by comparing them against that distribution, not against a rough judgment call. When a token proposed by the draft is rejected, the system resamples from that point exactly as the target model would have done with no draft at all, so the final inference sequence never loses accuracy. The gain comes from the fact that verifying several tokens in a single forward pass costs almost as much as verifying one, because decode is limited by GPU memory bandwidth rather than by compute: speculative decoding uses arithmetic capacity that plain decode leaves idle, without touching the result.

A concrete enterprise example

A customer-facing conversational assistant is the case where perceived latency matters more than any other metric: a user who waits three seconds before seeing the answer start to appear perceives the service as slow, regardless of how accurate the final answer turns out to be. A company serving this assistant with an expensive flagship model can pair it with a cheaper, smaller model as the draft: the latter proposes the most predictable words in the response (greetings, courtesy phrases, common sentence structures), the flagship model verifies them in bulk, and the customer sees the response stream faster without the provider giving up a single bit of the main model's quality.

Why it matters for decision makers

Speculative decoding lowers latency at the exact same output quality, which sets it apart from other optimization techniques that instead trade quality for speed. The cost is not zero: it requires two models instead of one, a draft model chosen and maintained alongside the target one, with a more complex serving pipeline to operate and monitor than a single model would need. Some models avoid the second model altogether because the proposing capability lives in their own weights, through multi-token prediction. The real payoff depends on how well the draft "guesses" the target's tokens: a draft that is too different or poorly matched generates overhead instead of savings. Anyone evaluating an inference provider should ask whether and how it uses speculative decoding, because it is today one of the concrete levers for cutting the latency bill without renegotiating the main model.