Infrastructure

KV Cache

A memory optimization that stores earlier work during generation so the model does not recompute it for every new token.

When a language model generates text, each new token would normally require reprocessing the entire sequence so far. The key-value cache saves the intermediate attention results for tokens already produced, so the model only computes the new one, which hugely speeds up generation. The catch is that this cache grows with context length and eats GPU memory, making it a central concern in serving long contexts efficiently.