Training & Learning
Learning Rate
The size of the steps a model takes while learning, and one of the most important training settings to get right.
The learning rate controls how big an adjustment the model makes to its weights on each training step. Too high and training overshoots and becomes unstable, too low and it crawls or gets stuck. Finding a good value, and often decreasing it over the course of training, is one of the most consequential parts of tuning a model.
Reference: Wikipedia: Learning Rate
Related terms
Gradient Descent
The optimization method that gradually tweaks a model to lower its error, like walking downhill to the lowest point.
Hyperparameter
A setting you choose before training that controls how the model learns, as opposed to what it learns.
Optimizer
The algorithm that decides how to adjust a model weights from its errors, with Adam being the common default.
Backpropagation
The algorithm that figures out how to adjust each weight in a network to reduce its errors.