Infrastructure

Quantization

Shrinking a model by storing its numbers at lower precision, so it runs faster and on smaller hardware.

Quantization reduces the precision of a models weights, for instance from 16-bit numbers down to 8-bit or 4-bit, which cuts memory use and speeds up inference. The trade-off is a usually small loss in accuracy in exchange for a model that is cheaper to run and can fit on modest hardware, even a laptop or phone. It is one of the main techniques for deploying large models affordably.