Prompting & Inference
Inference
Actually running a trained model to get an answer, as opposed to training it in the first place.
Inference is the use phase of a model: you give it an input and it produces an output using what it learned during training. Every time you send a message to a chatbot, you are paying for inference. Training is a large one-time cost, but inference is the recurring one, so making inference fast and cheap is a major focus for anyone running AI at scale.
Related terms
Large Language Model
A model trained on massive amounts of text that predicts and generates language, powering tools like ChatGPT and Claude.
Token
The small chunk of text a language model actually reads and writes, usually a word or part of a word.
Compute
The raw processing power used to train or run a model, a key driver of cost and capability.
Quantization
Shrinking a model by storing its numbers at lower precision, so it runs faster and on smaller hardware.