Training & Learning
Gradient Descent
The optimization method that gradually tweaks a model to lower its error, like walking downhill to the lowest point.
Gradient descent improves a model by repeatedly taking small steps in the direction that reduces its error the fastest. The gradient points uphill toward more error, so the method steps the opposite way, downhill, over and over. The size of each step is called the learning rate, and choosing it well is one of the everyday arts of training models.
Reference: Wikipedia: Gradient Descent
Related terms
Backpropagation
The algorithm that figures out how to adjust each weight in a network to reduce its errors.
Loss Function
The formula that scores how wrong a model is, giving training a single number to push downward.
Hyperparameter
A setting you choose before training that controls how the model learns, as opposed to what it learns.
Weights
The individual strengths of the connections in a neural network, which are what actually change as a model learns.