Pretraining on the Test Set Is All You Need

In September 2023, shortly after phi-1.5, a small model trained on curated "textbook-quality" data, reported strong benchmark results, Rylan Schaeffer posted a three-page paper in the same style. It introduces phi-CTNL, pronounced "fictional": a 1-million-parameter transformer pretrained on a novel, expert-curated, non-synthetic data mixture of under 100,000 tokens. The mixture is the test sets of the benchmarks it is then evaluated on: ARC, BoolQ, GSM8K, HellaSwag, HumanEval, MBPP, MMLU, OpenbookQA, PIQA, SIQA, SQuAD and WinoGrande.

phi-CTNL scores 100% on all of them, beating much larger models. The paper reports two "emergent" properties in the language of a model release. Its pretraining loss falls faster than a power law as epochs increase, and it shows a grokking-like ability to output the canary strings that benchmark authors embed so their data can be detected in training corpora. Figure 1's caption notes the numbers come from the authors' own pipeline "and we might have made them up". Table 1 estimates data contamination at 100%. A final disclaimer says the paper is satire, aimed at a field that is "generally undermined by boastful claims made without serious investigation of data contamination risks", while expressing respect for the curated-data work it parodies.

The method is one sentence long. The point is that every reported number, chart and phrase looks like a real release, and nothing in a results table tells a reader which kind of paper they are reading.

Ask about contamination before comparing scores. Vendors and model cards quote public benchmark numbers. A customer choosing between models needs to know whether those test sets were in the training data, and most releases say little. The satire is a reminder that the most impressive number in a table is also the one most worth checking.

Prefer evaluations the model cannot have seen. The defense that works is a test set built after the model's training cutoff, or from the customer's own data, kept out of any shared prompt logs. Our toy shows why: a model that memorized reworded copies of the test passes an n-gram overlap check and fails completely once the numbers change.

Re-sample instead of re-using. If test items come from templates, generate fresh instances for each evaluation. Memorization collapses on new instances and skill does not, as GSM-Symbolic showed on a public benchmark.

Treat canaries and overlap checks as partial. Canary strings only work when the copy that leaked kept the canary line. Long n-gram overlap finds verbatim copies and misses paraphrases, translations and solutions posted in discussion threads.

Keep your own golden set private. Once a customer's evaluation set is pasted into prompts, tickets or shared documents, it can end up in some model's training data, and its scores stop meaning what they meant.