The takeaway: "Distillation" is the technique behind a lot of the small, cheap AI models you can now run on a laptop or phone — and it's also the word at the centre of two of the year's biggest AI disputes. It means training a small "student" model to imitate a big "teacher" model. Here's what it actually is, how it differs from the other tricks for shrinking models, and why it keeps turning up in courtrooms.

The core idea: learning from a teacher's confidence, not just its answers

The method dates to a 2015 paper by Geoffrey Hinton, Oriol Vinyals and Jeff Dean, then all at Google, titled "Distilling the Knowledge in a Neural Network." The insight is subtle. When a large model classifies an image of a dog, it doesn't just output "dog" — it produces a full distribution of probabilities: dog 90%, wolf 8%, cat 1%, and so on. Those "wrong" answers aren't noise. The fact that the model thinks a dog is much more wolf-like than cat-like is real knowledge about how the world is structured — Hinton informally called it the model's "dark knowledge."

A hard label ("this is a dog," full stop) throws that away. Distillation keeps it: the student is trained to match the teacher's entire softened probability distribution, not just the single right answer. A "temperature" setting controls how much the distribution is softened — turned up, the small differences between the wrong answers become more pronounced and more informative. The student learns not just what the teacher concludes, but how confident it is and what it confuses things with. That richer signal is why a well-distilled small model can punch well above the size you'd expect.

In practice, distillation comes in a few flavours: matching the teacher's final outputs (the classic method above), matching its internal representations layer by layer, or — the dominant approach for today's large language models — simply having the teacher generate a mountain of example outputs and training the student on those as ordinary examples.

How it differs from quantization and pruning

Three techniques often get lumped together as "making models smaller," but they do genuinely different things:

• Distillation trains a brand-new, smaller model to mimic a bigger one. The result is a different, smaller network.
• Quantization keeps the same model but stores its weights at lower numerical precision — 16-bit numbers rounded to 8-bit or 4-bit — to shrink memory and speed up inference.
• Pruning keeps the same model but removes weights or neurons judged unimportant, leaving a sparser version of the original.

The clean way to remember it: distillation builds a new student, quantization rounds the same weights, pruning deletes some of them. (The techniques are frequently combined — a model can be distilled and quantized, which is roughly what powers a lot of on-device AI.)

What it looks like in real models

The first widely-used example was DistilBERT, released by Hugging Face in 2019. Its paper reports that distillation could "reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster," using "a triple loss combining language modeling, distillation and cosine-distance losses" — a neat illustration of matching both outputs and internal features at once.

The most consequential recent example is DeepSeek-R1, released in January 2025. Alongside its large reasoning model, DeepSeek open-sourced a family of small distilled models — 1.5B, 7B, 8B, 14B, 32B and 70B, built on top of existing Qwen and Llama models — trained on hundreds of thousands of reasoning examples the big model generated. The paper's finding is the reason distillation matters so much right now: "the emergent reasoning patterns exhibited by these large-scale models can be systematically harnessed to guide and enhance the reasoning capabilities of smaller models." In plain terms: you can teach a small model to reason by having a big one show its work. It's the same principle behind the wave of tiny, capable models — including the sub-4GB builds now being squeezed onto phones.

Why the word shows up in courtrooms

Here's where the pedagogy meets the news. Because data-level distillation just needs the teacher's outputs, you can distil a model you don't own — by querying it through its public interface and training on what comes back. That is exactly what two high-profile disputes are about.

OpenAI has accused DeepSeek of distilling its models' outputs to train DeepSeek's own — which OpenAI says breaches terms of service that bar extracting outputs to build competing models. And in June, Anthropic accused Alibaba's Qwen lab of a large-scale "adversarial distillation" campaign against Claude, telling US senators it had detected roughly 28 million exchanges through around 25,000 fraudulent accounts. No lawsuit has been filed in either case, and the technical facts are contested — but the pattern is worth understanding: the fight isn't about whether distillation works. It's about whose outputs you're allowed to learn from.

The limits

Distillation is a transfer technique, not a magic one. A student generally can't exceed its teacher's ceiling — it inherits what the teacher knows, including the teacher's biases and blind spots, and can't invent capability the teacher never had. Genuine leaps in what AI can do still come from bigger and better teacher models, new architectures, and more training; distillation is how those gains get spread to models small enough to run cheaply. And the word gets stretched in marketing — "distilled" sometimes just means "a smaller model we trained on a bigger one's output," and occasionally gets used loosely enough to blur into quantization or pruning. When you see it, the useful question is: distilled from what, and with whose permission?