Deduplicating Training Data Makes Language Models Better

Large language model datasets are scraped from the web, and the web repeats itself. Product pages differ only in place names and dates, the same news story is syndicated across sites with different headers, and boilerplate appears on thousands of pages. Datasets like C4 had removed exact duplicates of paragraphs or document prefixes, which misses most of this.

The paper introduces two tools. ExactSubstr concatenates the whole dataset and builds a suffix array, a sorted list of every suffix's starting position, in which repeated strings end up next to each other. Scanning adjacent entries finds every span of at least 50 tokens that occurs more than once, and all but one copy is removed. NearDup represents each document by its set of 5-grams, uses MinHash to find pairs likely to have high Jaccard similarity, confirms them with token edit similarity above 0.8, and groups matches into connected components, keeping one document per cluster.

Applied to C4, RealNews, LM1B and Wiki-40B, NearDup found near-duplicates in 3.04% of C4 training examples and 13.63% of RealNews, and ExactSubstr removed 7.18% of C4's tokens. Validation sets overlapped training: 4.6% of C4's validation examples and 14.35% of RealNews's had a near-duplicate in training. One C4 cluster held 250,933 documents.

The authors trained 1.5B-parameter models on original, NearDup and ExactSubstr versions of C4 for about two epochs. Deduplicated models matched or beat the original on held-out perplexity, scored worse on validation examples that had duplicates in training (evidence the original model had memorized them), and in 100,000 unprompted samples emitted about a tenth as much text copied verbatim from training.

Deduplicate before fine-tuning or indexing. Customer corpora are full of repeats: email threads quoting each other, templated tickets, policy documents in several versions. Duplicates overweight some content in training and crowd retrieval results with copies of the same passage.

Decontaminate evaluation sets. If 4.6% of a public validation set overlaps training, a customer's evaluation built from the same document pool will too. Run near-duplicate detection between training data and test data before trusting a score.

Less duplication means less memorized output. Text repeated many times in training is what models reproduce verbatim. When customer data contains personal or confidential text, deduplication lowers the chance that a fine-tuned model echoes it, alongside proper PII removal.

Use both kinds of detector. Exact span matching catches repeated boilerplate inside otherwise different documents; near-duplicate matching catches documents that differ by a few words. They remove overlapping but different content: 77% of what NearDup removed from C4 also had a 50-token exact match.

Tune thresholds on your data. MinHash's candidate probability is a steep S-curve around a similarity threshold. Our calculation shows the paper's own printed C4 example pair is too dissimilar to be a direct candidate at its settings; it can only be joined through other pages in the same cluster.