Infrastructure

Knowledge Distillation

Training a smaller model to imitate a larger one, keeping much of the quality at a fraction of the cost.

Knowledge distillation transfers what a large, capable teacher model knows into a smaller student model by training the student to mimic the teachers outputs. The result is a compact model that runs faster and cheaper while retaining a lot of the original performance. It is a common way to turn an expensive frontier model into something practical to deploy at scale.

Reference: Wikipedia: Knowledge Distillation