Training & Learning

Scaling Laws

The empirical finding that model performance improves predictably as you increase model size, data, and compute together.

Scaling laws describe a smooth, predictable relationship: give a model more parameters, more training data, and more compute in balance, and its performance improves along a steady curve. These observations are a big reason labs invest so heavily in ever-larger models, since the payoff can be forecast in advance. They also inform how best to split a fixed compute budget between model size and data.

Reference: Wikipedia: Scaling Laws