Prompting & Inference
Tokenization
The step that chops text into tokens before a model can process it.
Tokenization is the preprocessing that breaks raw text into the tokens a model works with, using a fixed vocabulary learned ahead of time. Common words become single tokens while rarer ones get split into pieces, which is why unusual words and other languages can cost more tokens. How text is tokenized affects both efficiency and, occasionally, how well a model handles certain inputs.
Related terms
Token
The small chunk of text a language model actually reads and writes, usually a word or part of a word.
Embedding
A way of turning words, images, or other data into lists of numbers that capture their meaning, so a computer can compare them.
Large Language Model
A model trained on massive amounts of text that predicts and generates language, powering tools like ChatGPT and Claude.