What is model distillation?

Distillation is the technique by which a small model, the student, is trained to imitate the answers of a large model, the teacher. Instead of training the student on the original raw data, it is trained to reproduce the teacher's output on a wide set of examples: the final model inherits much of the large model's behavior while having a fraction of its parameters. The technique originated in classic neural network research, well before the wave of large language models, but it became central precisely with LLMs because it lets you compress the capabilities of a model with hundreds of billions of parameters into one with only a few billion, keeping much of the performance on the tasks it was distilled for, while losing some generality on cases not represented in the student's training data.

How it works

A large number of teacher-model answers are generated on representative questions, and that set of question-answer pairs is used as training data for the student model. Some variants also use the teacher's internal probabilities, not just its final answer, to transfer nuances that plain text would miss. On the tasks covered by training the gap with the teacher is often small enough not to matter in production, but it is not zero: it widens on out-of-distribution inputs, on long reasoning chains and on the model's ability to know when it does not know. That is why a distillation is accepted only after measuring it on the real task, never on the vendor's benchmarks.

Why it matters for your business

Distillation is one of the main ways SLMs are produced: many of today's small models were born by distilling a flagship model, not by training from scratch. For a business, this means you can approach the performance of an expensive model with a cheaper, faster model to run in production, or distill your own proprietary model starting from an external provider to reduce dependence on it over time. Watch the license, though: many flagship model providers explicitly forbid, in their terms of use, using their model's output to train a competing one. That is a contractual clause to check before starting a distillation project, not a technical detail.