Infrastructure
KV Cache
A memory optimization that stores earlier work during generation so the model does not recompute it for every new token.
When a language model generates text, each new token would normally require reprocessing the entire sequence so far. The key-value cache saves the intermediate attention results for tokens already produced, so the model only computes the new one, which hugely speeds up generation. The catch is that this cache grows with context length and eats GPU memory, making it a central concern in serving long contexts efficiently.
Related terms
Attention
A mechanism that lets a model focus on the most relevant parts of its input when producing each piece of output.
Inference
Actually running a trained model to get an answer, as opposed to training it in the first place.
Context Window
The maximum amount of text, measured in tokens, that a model can consider at once.
Model Serving
The infrastructure that runs a trained model in production and answers live requests reliably and at scale.