One of the tidier scare stories about AI goes like this: the internet is filling up with AI-generated text, the next generation of models will be trained on that text, and so models will slowly poison themselves — each one a photocopy of a photocopy — until the whole enterprise degrades into noise. AI eating itself. It has a name in the research literature, model collapse, and unlike a lot of AI doom, the underlying result is real, peer-reviewed and reproducible.

The scare is still mostly wrong. Not because the research is shaky — it's solid — but because the viral version quietly swaps the experiment's setup for a different one. Here's what model collapse actually is, and the one word that decides whether it's a real threat or a lab curiosity.

What the research actually showed

The foundational paper is Shumailov and colleagues, published in Nature in July 2024. They ran a simple, brutal loop: train a model, use it to generate data, train the next model mostly on that generated data, and repeat — each generation learning from the last one's output. What they found is that quality degrades across generations in a specific pattern. First the model loses the tails of the distribution: the rare words, the unusual cases, the long-shot events. Then, later, it converges toward bland, low-variance output that eventually stops resembling the original data at all.

Their most-quoted example is a language model trained this way on Wikipedia text about English church towers. By the ninth generation, its output had wandered into repeating nonsense about jackrabbits of every colour. It's a vivid image, and it's real — but notice what produced it: a small model, trained over and over on its own output, with the real training data thrown away each round.

That last detail is the whole ballgame. The paper's headline result is about replacement — each generation trained on synthetic data instead of the real data. The researchers also tried keeping ten percent of the original data each round; that slowed the decline but didn't stop it, because ten percent of an ever-shrinking real signal is still a shrinking signal.

The one word that changes everything: accumulate

So the natural question is: what if you don't throw the real data away? What if each generation's synthetic data gets added to the original, rather than substituted for it — which is much closer to how real training corpora, and the real web, actually grow?

A 2024 study by Gerstgrasser and colleagues tested exactly that, across several model families — language models, image generators, and more. Their finding, in their own words: replacing real data with synthetic data does trend toward collapse, but accumulating real and synthetic data together avoids it. They even show mathematically that when data accumulates, the error stays bounded no matter how many generations you run. Collapse stops being inevitable and becomes conditional — a consequence of one assumption, that new data replaces old, which the real world mostly doesn't honour. Old data doesn't vanish from the web when new data appears; the archive keeps growing.

That's the correction the doom headline skips. Model collapse isn't a law of nature that kicks in the moment AI text touches a training set. It's what happens in a closed loop with no fresh real data — a setup practitioners already know to avoid.

This study is a preprint rather than a peer-reviewed publication, so it carries a little less weight than the Nature paper it answers. But its result is exactly the mirror image of the collapse experiment, which is what makes it persuasive: same models, one changed variable, opposite outcome.

Synthetic data isn't poison — when it's curated

There's a stronger point hiding here, too. Far from being toxic, synthetic data is already a proven, useful ingredient in good models — as long as it's curated rather than blindly recycled. Microsoft's Phi models are the standard example: a small model trained partly on "textbook-quality" synthetic data generated by a stronger model, and filtered for quality, that outperformed much larger models trained on more, messier data.

The distinction matters, and it's easy to blur. Phi's synthetic data comes from a more capable model and is curated — it is not a model recursively learning from its own raw output. So it doesn't disprove model collapse; it addresses a different situation. The honest summary of both results together is this: recursive self-training with no fresh real data is a known failure mode, and curated synthetic data from a capable source is a proven asset. Those aren't in tension — they're answers to two different questions.

Is it actually happening in the wild?

This is where honesty matters most. What's demonstrated is model collapse in controlled recursive-training experiments — deliberately closed loops in a lab. What's documented about the real web is that AI-generated content is proliferating: content farms, fabricated blog posts, a surge of AI answers on forums. That's real, and it's the input condition the scare worries about. But it is not the same thing as a model collapsing.

What's speculated — and it is speculation, not measurement — is that this contamination will inevitably degrade future frontier models. As of now there is no documented case of a deployed frontier model actually collapsing. The gap between "demonstrated in the lab under replacement" and "inevitable at web scale" is exactly where the scare lives, and it's a gap the confident headlines tend to leap without noticing.

That doesn't make the concern imaginary. How much unfiltered AI text ends up in future scrapes, and in what ratio, is a real open question, and labs spend real effort on data filtering and provenance because of it. But "worth managing" and "inevitable doom" are very different claims.

Four things that aren't the same

Because "collapse" gets used loosely, it's worth keeping four ideas apart. Model collapse is degradation across successive generations of models trained on prior models' output. Mode collapse is a different, older problem where a single generative model (classically a GAN) only learns to produce a few varieties of output — same word, unrelated failure. Catastrophic forgetting is when a network abruptly loses old knowledge as it learns a new task. And data poisoning is a deliberate attack — someone injecting malicious examples to corrupt a model. Model collapse is none of these: it's an emergent, unintentional degradation of a specific training loop.

The honest bottom line

Model collapse is real, and the research behind it is good. But it's a result about a particular training regime — recursive learning that replaces real data with synthetic — not a prophecy about AI as a whole. Change one variable, accumulate instead of replace, and the collapse goes away in the same study. Curate your synthetic data instead of blindly recycling it, and it becomes an asset. The interesting truth isn't "AI is doomed to rot." It's that "AI eating itself" is a known, well-understood, and largely avoidable failure mode — and the people building these systems have known which word matters for a while now.

For the other side of this coin — how AI-made data is deliberately, and usefully, put to work — see our explainer on synthetic data.