Infrastructure
Model Serving
The infrastructure that runs a trained model in production and answers live requests reliably and at scale.
Model serving is the engineering of putting a model behind an API so applications can send inputs and get predictions, handling batching, scaling, latency, and reliability under real traffic. It is a distinct discipline from training, with its own tools and tradeoffs, and it is where most of the ongoing cost of an AI product lives. Serving large language models well involves techniques like batching, the KV cache, and speculative decoding.
Related terms
Inference
Actually running a trained model to get an answer, as opposed to training it in the first place.
KV Cache
A memory optimization that stores earlier work during generation so the model does not recompute it for every new token.
Speculative Decoding
An inference speedup where a small fast model drafts several tokens and the big model checks them in parallel.
MLOps
The practice of reliably building, deploying, monitoring, and maintaining machine learning models in production, like DevOps for ML.