Knowledge distillation

What it is

Knowledge distillation transfers the knowledge of a large model, or of an ensemble of models, into a smaller model, so that predictive performance is largely retained at lower compute and deployment cost ^[1]. The direction usually runs from big to small. The reverse — a small model teaching a large one — is called reverse knowledge distillation, and it is rare ^[1].

How it works

A trained teacher model is run over the training inputs, and for each input it produces a probability distribution over classes rather than a single label. The student is trained to match that distribution. A temperature parameter is applied to the softmax, which makes the teacher's output distribution softer and higher in entropy ^[1]. These soft targets carry more information than hard labels, because they record how the classes relate to one another, and they reduce gradient variance, which in turn allows higher learning rates ^[1]. The loss is the cross-entropy between the teacher's outputs and the student's outputs, weighted by the square of the temperature when true labels are also used ^[1].

A small illustration makes the point concrete. Suppose a teacher is shown an image whose label is a particular breed of dog. A hard label says only 「this is breed A」. A soft distribution can say that the image is very likely breed A, somewhat likely breed B, and almost certainly not a car. That extra detail about which classes sit close to each other is exactly what the student learns from, and it is information the hard label throws away.

Where it came from

Bucilua, Caruana and Niculescu-Mizil proposed model compression in 2006: a small model was trained on pseudo-data labelled by an ensemble, and matched the ensemble's logits ^[1]. Hinton, Vinyals and Dean formalised the idea as knowledge distillation in 2015, in arXiv:1503.02531 ^[1]. That paper, first submitted in March 2015, starts from an observation: averaging the predictions of many models improves performance, but deploying a whole ensemble is cumbersome and expensive ^[2]. It builds on work by Caruana and collaborators showing that the knowledge in an ensemble can be compressed into a single model, and develops the approach with a different compression technique ^[2]. The authors also introduce a new type of ensemble, made of one or more full models plus many specialist models that learn to distinguish the fine-grained classes the full models confuse ^[2].

Where it is used today

Variants of the method are used well beyond image classifiers. They include distilling graph neural networks and distilling large language models: Stanford's Alpaca in 2023 was fine-tuned from LLaMA, and in January 2025 DeepSeek distilled six small models from R1 based on Qwen and Llama ^[1].

Limits

Distillation does not guarantee that the student acquires what the teacher knows in a reliable form. Distilled imitators can learn style rather than factuality and generalise poorly, as Gudibande and Wallace argued in 2023 ^[1]. There is also a contractual side to it: large developers' terms of service forbid training competitors on their outputs, which has created disputes over so-called distillation attacks ^[1].