Extracting Training Data from Large Language Models

Language models trained on web text were widely assumed not to memorize individual documents. They train for about one epoch on deduplicated data, and their training loss and test loss are almost equal, which is the usual sign that a model has not overfit. This paper shows that the average hides the worst case: a small number of training examples are stored verbatim and can be recovered by anyone who can query the model.

The authors studied GPT-2, whose weights and training source were public, and checked their findings with OpenAI against the original training set. They generated 200,000 samples of 256 tokens with each of three sampling setups, then ranked the samples by six scores. The best scores compare GPT-2's confidence on a sample with a reference that should not have memorized it: a smaller GPT-2, the zlib compressor, or GPT-2 itself on the lowercased text. Text that the large model finds much more predictable than the reference does is a likely memory. For each of the 18 combinations they sent 100 candidates for manual checking.

Of the 1,800 candidates, 604 were verbatim training text. Most were common material such as news, license text and log files. Some were not: 46 contained named individuals, 32 contained contact details, 35 contained random high-entropy strings such as UUIDs, and one sample reproduced a person's name, address, email, phone and fax number. Several came from a single training document.

A controlled test on Reddit URLs that appeared in exactly one document measured how much repetition memorization needs. GPT-2 XL (1.5 billion parameters) emitted every URL repeated 33 times or more, the 345-million-parameter model fewer, and the 117-million-parameter model only one, with help. Larger models memorized more. The paper closes with mitigations: differentially private training, careful deduplication and data sourcing, filtering outputs, and auditing models with methods like these.

Whatever you train on can come back out. Customers fine-tune models on support tickets, contracts, chat logs and code. This paper is the standard evidence that a model can reproduce a training record verbatim, to anyone who can query it, without any overfitting showing up in the loss curve. If the training set holds personal data or secrets, treat the model weights as holding them too.

Repetition is the variable you control. The striking examples were repeated many times inside one document: a forum dump that listed the same links dozens of times, a page with a UUID 10 to 311 times. Document-level deduplication keeps all of those repeats because they sit in one document. Before a fine-tune, count repeated spans inside documents, not only duplicate documents.

The audit is cheap and runs before release. Plant a few fake records (canaries) in the training data, train, and measure how highly the model ranks each one among all strings of the same format. Our toy run shows the pattern: a fake five-digit pin went from a random rank to the model's first guess after appearing four times. That is an audit you can show a security reviewer.

Clean the data first; it is the control that works. The paper is direct that sanitizing data is imperfect and that differential privacy is hard to apply to scraped text. For an FDE, the practical stack is: remove or tokenize personal fields before training, deduplicate within and across documents, keep secrets out of corpora entirely, and add an output filter for known sensitive patterns as a second line.

Larger models raise the risk. The 1.5-billion-parameter model needed 33 repeats where the smaller ones failed entirely. Moving a customer to a bigger base model for quality also moves them toward more memorization of the same data.