Textbooks Are All You Need

Following TinyStories, the same Microsoft group asked whether data quality can stand in for model size on a real task: writing Python functions from docstrings. They argue that public code datasets teach badly. Many files depend on other files, many are boilerplate, and the real logic is buried in long, undocumented functions.

So they built three datasets. Filtered code: from about 35 billion tokens of Python in The Stack and Stack Overflow, GPT-4 rated about 100,000 samples for "educational value for a student learning basic coding concepts", and a random forest on a code model's embeddings learned to predict that rating for the rest, keeping about 6B tokens. Synthetic textbooks: under 1B tokens of Python textbook text written by GPT-3.5, with topics and audiences varied to avoid repetition. CodeExercises: about 180M tokens of GPT-3.5-written function docstrings with solutions, varied by constraining function names.

phi-1-base, a 1.3B-parameter transformer, was trained on the first two (together called CodeTextbook) for about 8 passes, roughly 50B tokens, in under 4 days on 8 A100 GPUs. It scored 29% on HumanEval. Fine-tuning on CodeExercises for 7 hours produced phi-1, at 50.6% on HumanEval and 55.5% on MBPP. A 350M-parameter version reached 45%. The authors also show qualitatively that fine-tuning improved tasks absent from the exercises, such as using Pygame and Tkinter.

Because exercises written by GPT-3.5 could resemble HumanEval, Section 5 checks contamination. Few 13-gram overlaps were found. A stronger check splits HumanEval into problems with and without close matches in the exercises (by embedding and syntax-tree similarity), retrains phi-1 after pruning up to 40% of the exercises, and compares with StarCoder on each subset.

Small specialized models can be trained well on curated data. For a customer that needs a narrow capability cheaply, private or on small hardware, filtering existing data with a model-trained classifier and adding synthetic examples is a realistic recipe.

Fine-tuning data should match the task, and then you must test beyond it. CodeExercises were written in HumanEval's format, function plus docstring. That is good practice for a target task and a warning for evaluation: the benchmark lead concentrated on problems similar to the exercises.

Split the test set by similarity to training data. The paper's Table 3 is the method to copy. For each test item, find whether training data contains something close; report performance on matched and unmatched items separately. A model that only wins on matched items has learned the distribution, which may be fine, but the claim should say so.

Read the quality filter's examples. The code the paper shows as "high educational value" has a bug that our test exposes. A model-labeled filter scores what code looks like, not whether it works. When building a filter for a customer, sample what it keeps and execute or review it.

Expect brittleness outside the training style. The authors note phi-1 degrades with grammatical mistakes in prompts and lacks knowledge of less common packages.