Prompting & Inference

Tokenization

The step that chops text into tokens before a model can process it.

Tokenization is the preprocessing that breaks raw text into the tokens a model works with, using a fixed vocabulary learned ahead of time. Common words become single tokens while rarer ones get split into pieces, which is why unusual words and other languages can cost more tokens. How text is tokenized affects both efficiency and, occasionally, how well a model handles certain inputs.