Prompting & Inference
Speculative Decoding
An inference speedup where a small fast model drafts several tokens and the big model checks them in parallel.
Speculative decoding attacks the slow, one-token-at-a-time nature of generation by having a small draft model guess several tokens ahead, then having the large target model verify those guesses in a single parallel pass and keep the ones it agrees with. Because verifying in bulk is cheaper than generating one by one, it typically runs two to three times faster with identical output. It is built into popular serving frameworks and needs no change to the model itself.
Related terms
Inference
Actually running a trained model to get an answer, as opposed to training it in the first place.
Knowledge Distillation
Training a smaller model to imitate a larger one, keeping much of the quality at a fraction of the cost.
KV Cache
A memory optimization that stores earlier work during generation so the model does not recompute it for every new token.
Model Serving
The infrastructure that runs a trained model in production and answers live requests reliably and at scale.