Prompting & Inference

Inference

Actually running a trained model to get an answer, as opposed to training it in the first place.

Inference is the use phase of a model: you give it an input and it produces an output using what it learned during training. Every time you send a message to a chatbot, you are paying for inference. Training is a large one-time cost, but inference is the recurring one, so making inference fast and cheap is a major focus for anyone running AI at scale.