Models & Architecture

Dropout

A training trick that randomly switches off some neurons each step to stop a network from overfitting.

Dropout fights overfitting by randomly ignoring a fraction of neurons on each training step, so the network cannot lean too hard on any single unit and has to spread what it learns. The effect is a bit like training many slightly different networks and averaging them. At inference time all neurons are active, giving a more robust model.

Reference: Wikipedia: Dropout