Training & Learning
Pretraining
The first, largest training phase where a model learns general patterns from a huge amount of broad data.
Pretraining is where a model builds its general competence by learning from a massive, broad dataset, usually by predicting missing or next pieces of the data. For a language model this is the expensive phase that teaches grammar, facts, and reasoning patterns from internet-scale text. What comes out is a capable but raw base model, which is then refined through fine-tuning for specific uses.
Related terms
Fine-Tuning
Taking a pretrained model and training it further on a smaller, focused dataset to specialize it for a task or style.
Foundation Model
A large, general-purpose model trained on broad data that many different applications can be built on top of.
Training Data
The examples a model learns from, whose size, quality, and bias largely determine how the model turns out.
Transfer Learning
Reusing what a model learned on one task as a head start for a different but related task.