Prompting & Inference

Token

The small chunk of text a language model actually reads and writes, usually a word or part of a word.

Language models do not see whole words or letters, they see tokens, which are pieces of text that often map to roughly three-quarters of a word in English. A model reads your prompt as a sequence of tokens and generates its reply one token at a time. Tokens are also the unit of pricing and context limits, so the length of a conversation is measured in tokens, not characters.