The Curse of Recursion: Training on Generated Data Makes Models Forget

Until recently almost all text on the web was written by people. If future models are trained on web scrapes that include text written by earlier models, each generation learns partly from its predecessor's output. The paper asks what that does over many generations, and names the result model collapse: a degenerative process in which models forget the true data distribution.

The authors identify two sources of error that compound. Statistical approximation error: each generation sees only a finite sample, so events that are rare in the true distribution may not appear at all and cannot be relearned. Functional approximation error: models cannot represent the distribution exactly, for example a single Gaussian fitted to a mixture. Over generations, tails disappear first ("early collapse"), then the distribution narrows toward a point with little variance ("late collapse").

The theory works through a discrete distribution, where the only absorbing states are single points, and a one-dimensional Gaussian refitted from samples each generation, giving the variance of samples from generation n. Experiments with Gaussian mixture models and variational autoencoders show fitted distributions narrowing and tails vanishing. For language, OPT-125m is fine-tuned on wikitext2, each generation then writes a full-size training set for the next by continuing 64-token blocks, and perplexity on the real test set worsens over generations; generated text drifts into repetitive, improbable sequences by generation 9.

The conclusion is about data: access to genuine human-generated data, and records of which data came from models, will become more valuable.

Watch for closed loops in customer pipelines. Retraining a classifier on its own past predictions, fine-tuning an assistant on conversations it wrote, or generating synthetic training data from the deployed model all feed a model its own output. The paper's mechanism applies to each.

Keep real data in every round. In our run of the paper's Gaussian example, 10% real data per generation keeps the fitted distribution essentially intact over 1,000 generations, while pure self-training collapses in most chains. In a customer retraining loop, always anchor each round with fresh human-labeled or observed data.

The rare cases go first. Collapse removes low-probability events before anything else. Those are often the cases that matter: fraud, rare diseases, unusual contract clauses. Monitor recall on rare classes across retraining rounds, not just average accuracy.

Track provenance. Label which training examples are model-generated, so you can cap their share, filter them, or remove them later.

Average metrics can look fine during collapse. Our simulation shows the mean fitted variance near 1 at generation 100 while the median has dropped by 40%. Report distributions across runs and quantiles of outputs.