IN THIS BREAKDOWN · 10 SECTIONS
You can read 24% of this breakdown.
The paper in one minute
By 2020, language models were mostly trained on Common Crawl, the large public web scrape, and evidence was growing that more diverse data improved generalization. The best curated datasets were private. EleutherAI built the Pile as an open alternative and as a benchmark for how well models know many domains.
The Pile has 22 components. Some are existing datasets (Books3, Project Gutenberg, OpenSubtitles, English Wikipedia, DM Mathematics, EuroParl, Enron Emails). Fourteen were built for it, including PubMed Central papers, ArXiv papers, GitHub code, FreeLaw court opinions, Stack Exchange, USPTO patent backgrounds, PubMed abstracts, Ubuntu IRC logs, Hacker News, YouTube subtitles, PhilPapers, NIH grant abstracts, OpenWebText2 and BookCorpus2. Pile-CC is a new Common Crawl extraction using jusText on raw web archives for cleaner text. Higher-quality components are repeated within an epoch (Wikipedia three times, Gutenberg 2.5, many others twice), so the 825 GiB of raw text becomes 1,254 GiB of training data per epoch.
As a benchmark, the Pile reports bits per UTF-8 byte (BPB), which is comparable across tokenizers. GPT-2 and GPT-3, evaluated zero-shot, struggled on components such as academic writing and mathematics. GPT-2-sized models trained on the Pile beat models trained on CC-100 and raw Common Crawl on every component, with large gains on academic, code, legal and math text, while matching them on WikiText and LAMBADA.
A large part of the paper is documentation: document lengths and tokenization, languages, topic models, toxicity, and sentiment toward gender, religion and race, plus a discussion of legality, consent and authors' rights, with data sheets for each component.
Research notes: the problem, approach and results
component datasets; 14 newly constructed for the Pile (Section 1, Table 1)
raw size; 1,254 GiB per epoch after up-weighting high-quality components (Table 1)
GPT-2 tokens per byte across the Pile, used to convert loss to bits per byte (Section 3.1)
Large language models were trained mostly on Common Crawl, and the best datasets, such as GPT-3's, were private. Researchers lacked a large, open, diverse corpus, with documentation of what is in it, for training and for testing cross-domain knowledge.
Assemble 22 high-quality subsets, 14 of them newly built (PubMed Central, ArXiv, GitHub, FreeLaw, Stack Exchange, USPTO and others), plus a better-extracted Common Crawl subset, weight them by quality, define held-out splits, and measure models in bits per byte.
An 825 GiB corpus (1,254 GiB per weighted epoch). GPT-2 and GPT-3 did poorly on several components such as academic text; GPT-2-size models trained on the Pile beat models trained on raw Common Crawl or CC-100 on every component, with similar results on standard benchmarks. The paper documents topics, bias, legal status and data concerns.
Why it matters to an FDE
Know what the model read. Open models such as GPT-Neo and GPT-J were trained on the Pile. Its mix (a quarter academic and legal text, 8% GitHub, email and chat logs) shapes what those models know and how they write. When a customer picks an open model, the training corpus documentation tells you where it will be strong and what it might reproduce.
Weight data sources on purpose. The Pile repeats high-quality sources up to three times per epoch and samples Common Crawl once. Customer fine-tuning or continued pretraining should make the same decision explicitly: how much of each source, and why.
Compare models in bits per byte. Per-token perplexity depends on the tokenizer, so two models with different tokenizers cannot be compared that way. Bits per byte can. Use it when evaluating language models on a customer's own text.
Document the corpus like a product. The paper's datasheets, topic analysis and bias measurements are the template for documenting a customer's training or retrieval corpus: sources, licenses, personal data, known skews.
Check licensing and personal data before training. The Pile includes copyrighted books (Books3) and real people's emails (Enron). The paper discusses fair use and consent; a customer project needs explicit decisions on both.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 290 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
Components and weights, Pile-CC, splits and BPB, results against Common Crawl, concerns
Free account - 04How it works
Building the new components, Pile-CC extraction, weighting, deduplication, splits and the BPB metric
Premium - 05The numbers that matter
Component sizes and weights, BPB by model, the size-controlled comparison, and the scaling line
Premium - 06Try it yourself
Recheck Table 1, convert loss to bits per byte, and find which models the scaling line was fit on
Premium - 07Where it breaks
Licensing and consent, personal data, English focus, residual duplication, and a scaling-line slip
Premium - 08Using it on a deployment
Read a model's data card, build a weighted, documented corpus, and evaluate in bits per byte
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium