There is a looming awkwardness at the heart of how AI is built. These models learn from vast amounts of human-written text — and the supply of high-quality, public, human-written text is not infinite. A lot of the good stuff has already been used. So labs have increasingly turned to an idea that sounds either ingenious or circular depending on your mood: train the next AI partly on data generated by the current one. This is synthetic data, and it is now one of the most important — and most misunderstood — ingredients in modern AI.

What it is, and what it's good for

Synthetic data is simply training material produced by a model rather than collected from people: generated text, images, question-and-answer pairs, step-by-step reasoning, code. It earns its place in a few genuinely useful ways.

Distillation is the biggest. A large, capable "teacher" model generates outputs, and a smaller "student" model is trained to imitate them — a way of compressing expensive capability into a cheaper package. (It's also exactly the technique at the centre of this month's accusation against an Alibaba-linked operation: distillation done to a rival's model, without permission.)

Filling gaps is the next. If a model is weak on a rare language, an unusual document format, or an edge case that barely appears in real data, you can generate lots of targeted examples to shore it up — cheaper and faster than hunting for scarce real ones.

Privacy is a third: synthetic stand-ins for sensitive medical or financial records let teams build systems without exposing the real thing.

But the most powerful use is the one that sounds most like cheating: a model improving itself.

The trick that makes it work: a verifier

Here is the key distinction the hype usually skips. In domains where you can check whether an answer is right — mathematics, code, formal logic, games — a model can generate thousands of attempts at a problem, keep only the ones that pass the check, and then train on those winners. It is bootstrapping: the model pulls itself up by keeping its own best work and learning from it. Techniques in this family have driven much of the recent leap in AI reasoning and coding.

The thing doing the heavy lifting is the verifier — the maths checker, the code that either runs or doesn't, the game you either win or lose. The synthetic data is only trustworthy because something outside the model can confirm it's correct. Take the verifier away and the magic largely goes with it.

The thing that goes wrong: model collapse

Which brings us to the failure that haunts synthetic data. If you train a model on the unfiltered output of another model, and then train the next one on that model's output, and so on, quality doesn't hold — it degrades. Researchers call it model collapse. A widely-cited 2024 study in Nature by Shumailov and colleagues showed that naively recycling AI-generated data across generations causes models to progressively forget the rare and unusual — the tails of the distribution — and drift toward bland, repetitive, increasingly wrong averages. Errors don't wash out; they compound, like a photocopy of a photocopy of a photocopy — and once the rot sets in, it is hard to reverse.

Crucially, collapse is a consequence of doing it badly — closing the loop blindly on unfiltered, unverified output. It is avoidable, and the avoidance is exactly what serious labs do: keep a healthy share of real human data as an anchor, filter and verify the synthetic portion rather than trusting it wholesale, and never let a model feed unchecked on its own exhaust.

The honest limit

The most important thing to understand about synthetic data is what it can't do. It cannot conjure genuinely new knowledge about the world. A model can't teach a smaller one a fact neither of them ever had; it can't invent the true result of an experiment nobody has run. Distillation transfers skill but is capped by the teacher — the student rarely exceeds the source. Where synthetic data shines is in skills, reasoning, and format — things a verifier can score or a transformation can sharpen — far more than in facts, where it risks confidently manufacturing plausible nonsense.

So the right way to hold it: synthetic data is not a perpetual-motion machine that lets AI train itself to infinity, and it is not the fraud that "AI learning from AI" can sound like. It is a powerful, double-edged tool — extraordinary when grounded by a verifier or anchored to real data, quietly corrosive when it just eats itself. As the human-written web gets used up, getting that balance right is becoming one of the central problems in building the next generation of models.