Distilling the Knowledge in a Neural Network

Big models and ensembles generalize better, but they are expensive to run. Hinton, Vinyals and Dean compare it to insects: a larval form built for feeding and an adult form built for travel. Training can afford a cumbersome model; deployment needs a small one. Distillation moves what the cumbersome model learned into the small one.

The key observation is about what a trained classifier knows. Besides the right answer, it assigns probabilities to every wrong answer, and those small probabilities differ in informative ways: an image of a BMW might get a tiny chance of being a garbage truck but a much smaller chance of being a carrot. That pattern encodes how the model generalizes. Training a small model only on correct labels throws it away.

The method is to train the small model to match the big model's probabilities, its soft targets, instead of or as well as the labels. Most of the useful information sits in probabilities so small they barely affect the loss, so the softmax is "heated": logits are divided by a temperature T before the softmax, which flattens the distribution and makes the small probabilities count. The student is trained at the same temperature on these soft targets, usually with a smaller second loss on the true labels, and then used at temperature 1. The soft-target loss is multiplied by T² so its gradients keep the same scale as T changes. At very high temperature, distillation reduces to matching the teacher's logits, which an earlier method by Caruana and colleagues did directly.

On MNIST, a small net trained on soft targets nearly matched the large regularized net (74 errors against 67), against 146 errors when trained on labels. It even learned to recognize the digit 3 without seeing a single 3 in its transfer set. In Google's speech recognition, one distilled acoustic model captured most of a 10-model ensemble's improvement. The paper also trains specialist models for confusable classes on a 100-million-image dataset.

Distillation is the standard way to turn a capable but expensive model into one a customer can afford to run, and modern LLM distillation descends from this paper.

Use the big model to label, the small model to serve. A common pattern: an expensive model produces outputs or probabilities on the customer's data, and a small model is trained on them. This paper shows why that works better than training the small model on labels alone: the teacher's distribution over alternatives carries extra information per example.

Soft targets stretch small datasets. With only 3% of the speech training data, a model trained on soft targets reached 57.0% frame accuracy against 44.5% with labels, close to the 58.9% of a model trained on everything. When a customer has little labeled data but access to a strong model, distillation is often the cheapest path to a good small model.

Distillation transfers part of the gain, not all of it. One distilled model kept about 86% of an ensemble's frame-accuracy improvement. Set expectations accordingly: the student approaches the teacher, it does not reliably match it.