Training & Learning

Pretraining

The first, largest training phase where a model learns general patterns from a huge amount of broad data.

Pretraining is where a model builds its general competence by learning from a massive, broad dataset, usually by predicting missing or next pieces of the data. For a language model this is the expensive phase that teaches grammar, facts, and reasoning patterns from internet-scale text. What comes out is a capable but raw base model, which is then refined through fine-tuning for specific uses.