Speculative Decoding: How LLMs Generate Faster

Large Language Models (LLMs) have demonstrated remarkable capabilities, but their real-time application is often hampered by a significant bottleneck: inference speed. Generating text token by token is a slow, sequential process that can lead to frustrating delays in interactive applications like chatbots and coding assistants. This latency doesn't just degrade the user experience; it limits the feasibility of more complex, multi-step agentic workflows. Speculative decoding is a powerful optimization technique that directly addresses this problem, often delivering a 2x to 3x reduction in generation time.

This technique works by using a small, fast "draft" model to generate a chunk of candidate tokens, which the larger, more accurate "target" model then validates in a single, parallel step. The result is a system that achieves much of the speed of the small model while retaining the full quality of the large one. For engineers and technical leaders building with LLMs, understanding speculative decoding is critical for optimizing performance, managing infrastructure costs, and making informed architectural decisions.

The Bottleneck of Auto-Regressive Generation

To understand why speculative decoding is so effective, we first need to pinpoint why standard LLM generation is slow. The core reason lies in its auto-regressive^[1] nature. An LLM generates text one token at a time, with each new token depending on all the tokens that came before it. This creates a sequential dependency chain: you cannot compute the fifth token until you have finalized the fourth, which requires the third, and so on.

This sequential process clashes with the parallel processing power of modern hardware like GPUs. While a Transformer^[2] model can process all tokens of a prompt in parallel during the initial pass, generation is a loop. Inside that loop, for every single token, the model must perform a full forward pass. This involves loading the model's enormous weights, which can be tens or hundreds of gigabytes, from the GPU's high-bandwidth memory (HBM) into the on-chip SRAM of the processing units.

This memory transfer is the dominant bottleneck. The actual mathematical computations (FLOPs^[3]) are fast, but the process is memory-bound. As detailed in performance analyses, the arithmetic intensity (the ratio of compute to memory access) of a single-token forward pass is very low. The processor spends most of its time waiting for data to arrive from HBM. Generating a 100-token response requires 100 separate, expensive memory-loading cycles. This is why even powerful hardware can seem sluggish when generating text from a large model.

How Speculative Decoding Works: The Core Mechanism

Speculative decoding breaks this one-token-at-a-time cycle. The technique was formally detailed in a late 2022 paper from Google, "Fast Inference from Transformers via Speculative Decoding^[4]", and concurrent work from DeepMind, "Accelerating Large Language Model Decoding with Speculative Sampling^[5]", building on precursors like the 2018 paper "Blockwise Parallel Decoding for Deep Autoregressive Models^[6]". The central idea is to use two models:

1. The Target Model: The large, powerful, high-quality LLM you actually want to use (e.g., Llama 3 70B, GPT-4).
2. The Draft Model: A much smaller, faster model (e.g., Llama 3 8B, or a distilled version of the target model). While standard speculative decoding requires a shared tokenizer, this is no longer a strict requirement, as newer implementations like Hugging Face's Universal Assisted Generation allow the models to use different tokenizers.

The process replaces a single, slow target model step with a three-part cycle: drafting, verifying, and correcting.

Step 1: Drafting

The draft model takes the current sequence of confirmed tokens and, using fast auto-regressive generation, produces a short continuation of k tokens. Let's say k=5. Because the draft model is small, this step is very quick and cheap.

Step 2: Verifying

This is the crucial insight. The target model is given the original sequence plus the k draft tokens as a single, combined input. It then performs one forward pass. Because the Transformer architecture can attend to all positions simultaneously, the target model computes the probability distribution for the next token at each position in the draft, all at once.

For example, if the input is (c_1, c_2, ..., c_n) and the draft is (d_1, d_2, d_3, d_4, d_5), the target model processes (c_1, ..., c_n, d_1, ..., d_5) and outputs logit vectors for the token that should follow c_n, the token that should follow d_1, and so on.

We now have the draft model's choices and the target model's preferences. We compare them token by token:

• Check if the target model's most likely next token after the original sequence c_n is indeed d_1. If they match, d_1 is accepted.
• If d_1 was accepted, we then check if the target model's most likely token after (c_n, d_1) is d_2. If they match, d_2 is accepted.
• This continues until the first mismatch. If the target model's prediction at any step i does not match the draft token d_i, then d_i and all subsequent draft tokens (d_i+1 to d_k) are rejected.

In a single forward pass of the large model, we have potentially validated and accepted multiple tokens, effectively parallelizing the memory-bound operation.

Step 3: Correcting and Sampling

After the verification loop, we have accepted some number of tokens, n, where 0 <= n <= k.

If we find a mismatch at position i, the simple approach of just taking the target model's preferred token would introduce a statistical bias, making the output different from what the target model would have produced alone. To ensure the output distribution is mathematically identical to the target model, a more sophisticated correction is needed. This method has intellectual roots in statistical sampling techniques like the Metropolis-Hastings algorithm^[7].

For each token d_i proposed by the draft model, it is accepted if it is the argmax^[8] of the target model's distribution. If it is not, we can still potentially accept it based on a probabilistic check. The token d_i is accepted with probability min(1, P_target(d_i) / P_draft(d_i)), where P is the probability assigned to that token by each model. If the token is rejected, we must sample a new token from a corrected probability distribution (P_target - P_draft)+ (the normalized positive difference between the two distributions). This procedure guarantees that, over many generations, the sequence of tokens produced is statistically indistinguishable from sampling directly from the target model. It is a lossless optimization from a quality perspective.

Once a token is rejected and a new one is sampled from the corrected distribution, the process repeats: the draft model generates a new draft of k tokens starting from the newly appended sequence.

A Concrete Example

Let's walk through a simplified scenario. Assume k=4.

• Prompt: The Statue of Liberty is in
• Target Model: A large, accurate model.
• Draft Model: A smaller, faster model.

Cycle 1:

1. Draft: The draft model takes the prompt and quickly generates a 4-token draft: New York City.

2. Verify: The target model receives the combined sequence: The Statue of Liberty is in New York City. It performs a single forward pass and computes the next-token probabilities at each position.

Input Position | Draft Token | Target Model's Top Prediction
----------------------------------------------------------------
... is in | " New" | " New" --> Match, accept.
... in New | " York" | " York" --> Match, accept.
... New York | " City" | " City" --> Match, accept.
... York City | "." | "," --> Mismatch! Reject.

3. Correct:

• We have accepted three tokens: New, York, City.
• At the point of mismatch, the draft proposed . but the target model preferred ,. We reject . and all subsequent draft tokens (none in this case).
• We now sample from the target model's corrected distribution for the token following City. Let's say we sample the most likely token, which is ,.

Result: In a single pass of the expensive target model, we successfully generated four tokens ( New, York, City, ,). Without speculative decoding, this would have required four separate, sequential forward passes. The speedup is substantial.

The process now repeats from the new sequence: The Statue of Liberty is in New York City,.

The Real Tradeoffs and Decision Lens

While speculative decoding offers significant speedups, it's not a free lunch. Technical leaders must weigh its benefits against its costs.

Gains

• Reduced Latency: This is the primary advantage. By processing multiple tokens per target model iteration, the wall-clock time for generating a response is significantly reduced. The original paper reported 2x to 2.5x speedups, and this is consistent with figures seen in production implementations. This is most impactful for interactive, latency-sensitive applications.
• Mathematically Lossless Output: When implemented correctly with the proper sampling correction, speculative decoding does not change the output. The generated text has the exact same statistical properties as if it were generated by the target model alone. This is not an approximation; it's an algorithmic optimization.

Costs and Considerations

• Increased Memory Usage: The most significant drawback is the need to have two models loaded into VRAM^[9] simultaneously. This can be a prohibitive cost, especially when the target model already pushes the limits of available hardware. An 8B parameter draft model alongside a 70B target model requires a substantial amount of additional memory.
• Draft Model Selection is Key: The performance gain is highly dependent on the draft model. The goal is to maximize the acceptance rate.
• If the draft model is too weak, its predictions will rarely match the target model's, resulting in a low acceptance rate and minimal speedup. The overhead might even make it slower.
• If the draft model is too large or slow, the time spent drafting negates the gains from parallel verification.
• The ideal draft model is often a smaller model from the same family or a specially trained one using techniques like knowledge distillation^[10] or quantization to create a cheap-to-run miniature of the target model.
• Implementation Complexity: The logic is more complex than a standard generation loop. It requires careful state management, synchronization between the two models, and correct implementation of the sampling correction algorithm.
• Variable Speedup: The 2x-3x speedup is an average, not a guarantee. The actual performance depends on the predictability of the text. For simple, repetitive tasks (like writing boilerplate code), the acceptance rate will be high and the speedup will be large. For highly creative or complex reasoning tasks, the acceptance rate may be lower.

Decision Lens for Technical Leaders

When should you use speculative decoding?

• Use it when: Your primary metric is single-stream latency. This is crucial for user-facing applications like chatbots, virtual assistants, and real-time code completion where every millisecond of delay hurts the user experience. Use it if you have sufficient VRAM to accommodate both models without compromising your batch size too severely.
• Reconsider when: Your primary metric is maximum throughput. For offline, batch-processing jobs where latency for any single job is less important than the total number of jobs completed per hour, other techniques may be more effective. For example, continuous batching is designed to maximize GPU utilization for high-throughput serving and might be a better focus. Also, reconsider if you are severely VRAM-constrained, as the memory overhead is its biggest cost.

Common Misconceptions

Several misconceptions have arisen around speculative decoding. It's important to correct them.

Misconception 1: It degrades output quality.
This is false. As explained, the verification and correction mechanism is designed to be mathematically lossless. The final output is drawn from the exact same probability distribution as the target model. It is not an approximation or a shortcut that trades quality for speed. Any implementation that results in different output has a bug.

Misconception 2: It's just a complicated way of using a smaller model.
This misses the point. You are not getting the output of the small model. You are getting the high-quality output of the large target model, but with a latency that approaches that of the small draft model. It's a method for accelerating the large model, not a compromise to use a smaller one.

Misconception 3: The speedup is constant.
This is not true. The acceleration is dynamic and depends heavily on the agreement between the draft and target models. If the draft model is a poor predictor for a given task, the acceptance rate will be low, and the speedup may be negligible. The choice of draft model and the k value (the draft length) are important tuning parameters.

History and Current State

The modern formulation of speculative decoding for Transformers gained prominence in late 2022. While the core ideas have precedents in statistical computing, its application to LLM inference was a key innovation for addressing the memory bandwidth bottleneck.

Today, speculative decoding is moving from a research concept to a standard feature in mainstream LLM serving infrastructure.

• Library Support: The Hugging Face transformers library, a de facto standard for working with LLMs, has built-in support for speculative decoding in its generate method. Developers can enable it by providing an assistant_model (the draft model) during generation. This has made the technique widely accessible.
• Inference Engines: High-performance inference engines have also adopted it. NVIDIA's TensorRT-LLM^[11] includes a highly optimized implementation^[12] designed to maximize performance on NVIDIA GPUs. Other serving frameworks like vLLM and TGI also have support or are actively developing it.
• Architectural Variants: Research is ongoing to mitigate the primary cost of speculative decoding: the memory overhead of a second model. A notable variant is Medusa, proposed in early 2024. The Medusa approach^[13] avoids a separate draft model entirely. Instead, it adds multiple, lightweight "decoding heads" to the final layers of the target model itself. Each head is trained to predict the next few tokens. This allows for drafting and verifying within a single model, drastically reducing the memory overhead while still providing significant latency reduction. Such techniques represent the next step in making fast inference more accessible and memory-efficient.

As LLMs continue to grow in size and capability, optimizations like speculative decoding and its derivatives will become less of a niche trick and more of a default requirement for building responsive and cost-effective AI systems.