Infrastructure

Model Serving

The infrastructure that runs a trained model in production and answers live requests reliably and at scale.

Model serving is the engineering of putting a model behind an API so applications can send inputs and get predictions, handling batching, scaling, latency, and reliability under real traffic. It is a distinct discipline from training, with its own tools and tradeoffs, and it is where most of the ongoing cost of an AI product lives. Serving large language models well involves techniques like batching, the KV cache, and speculative decoding.