Prompting & Inference

Speculative Decoding

An inference speedup where a small fast model drafts several tokens and the big model checks them in parallel.

Speculative decoding attacks the slow, one-token-at-a-time nature of generation by having a small draft model guess several tokens ahead, then having the large target model verify those guesses in a single parallel pass and keep the ones it agrees with. Because verifying in bulk is cheaper than generating one by one, it typically runs two to three times faster with identical output. It is built into popular serving frameworks and needs no change to the model itself.