Prompting & Inference
Token
The small chunk of text a language model actually reads and writes, usually a word or part of a word.
Language models do not see whole words or letters, they see tokens, which are pieces of text that often map to roughly three-quarters of a word in English. A model reads your prompt as a sequence of tokens and generates its reply one token at a time. Tokens are also the unit of pricing and context limits, so the length of a conversation is measured in tokens, not characters.
Related terms
Tokenization
The step that chops text into tokens before a model can process it.
Context Window
The maximum amount of text, measured in tokens, that a model can consider at once.
Large Language Model
A model trained on massive amounts of text that predicts and generates language, powering tools like ChatGPT and Claude.
Inference
Actually running a trained model to get an answer, as opposed to training it in the first place.