Infrastructure
Quantization
Shrinking a model by storing its numbers at lower precision, so it runs faster and on smaller hardware.
Quantization reduces the precision of a models weights, for instance from 16-bit numbers down to 8-bit or 4-bit, which cuts memory use and speeds up inference. The trade-off is a usually small loss in accuracy in exchange for a model that is cheaper to run and can fit on modest hardware, even a laptop or phone. It is one of the main techniques for deploying large models affordably.
Related terms
Knowledge Distillation
Training a smaller model to imitate a larger one, keeping much of the quality at a fraction of the cost.
Inference
Actually running a trained model to get an answer, as opposed to training it in the first place.
Edge AI
Running AI directly on a local device like a phone or camera, instead of sending data to the cloud.
Weights
The individual strengths of the connections in a neural network, which are what actually change as a model learns.