Training & Learning
Scaling Laws
The empirical finding that model performance improves predictably as you increase model size, data, and compute together.
Scaling laws describe a smooth, predictable relationship: give a model more parameters, more training data, and more compute in balance, and its performance improves along a steady curve. These observations are a big reason labs invest so heavily in ever-larger models, since the payoff can be forecast in advance. They also inform how best to split a fixed compute budget between model size and data.
Reference: Wikipedia: Scaling Laws
Related terms
Compute
The raw processing power used to train or run a model, a key driver of cost and capability.
Parameters
The adjustable numbers inside a model that get tuned during training and store what the model has learned.
Training Data
The examples a model learns from, whose size, quality, and bias largely determine how the model turns out.
Large Language Model
A model trained on massive amounts of text that predicts and generates language, powering tools like ChatGPT and Claude.