IN THIS BREAKDOWN · 10 SECTIONS
You can read 18% of this breakdown.
The paper in one minute
SmolLM2 is a 1.7B-parameter Llama-style model (24 layers, width 2,048, tied embeddings) trained by Hugging Face on about 11 trillion tokens, far past compute-optimal, to make a small model as strong as possible. The paper is mostly about data.
The authors compared existing datasets with small ablation runs and, where those were weak, built new ones with the same recipe: have Llama-3-70B or Llama-3.1-70B rate a sample of documents for educational value, train a small classifier on those ratings, score the whole corpus and keep documents above a threshold. FineMath (up to 54B tokens) is math web text that shows step-by-step reasoning; Stack-Edu (about 125B tokens) is well-documented code from 15 languages; SmolTalk is an instruction-tuning mix built around a new synthetic conversation set, MagPie-Ultra, generated with Llama-3.1-405B.
Training ran in four stages because one run cost about $250,000. Stage 1 was 90% English web (FineWeb-Edu and DCLM) and 10% code. After each stage the authors looked at scores and changed the mix: more code, then small math datasets, then Stack-Edu in place of raw code, and finally, during the learning-rate decay, the best math (FineMath4+) and code. Math went from 7.27 to 22.07 in that last 1T tokens.
The base model leads Qwen2.5-1.5B and Llama3.2-1B on several knowledge and reasoning benchmarks and trails Qwen2.5 on math and code. After instruction tuning and DPO it leads on IFEval and trails Qwen2.5 on MMLU-Pro and GSM8K. All datasets and models are released.
Research notes: the problem, approach and results
average math score before and after the final 1T-token decay stage, when the best math data was added
MultiPL-E gain on C++ from keeping only the 23% of code an LLM-trained classifier rated educational
approximate GPU cost of the 1.7B training run (about 1e23 FLOPs), the reason the mix was tuned during one run
Small models have little capacity, so what they are trained on matters more than for large ones. Public math, code and instruction datasets were too small or too noisy, and a full training run costs too much to try several data mixes.
Build new datasets by having an LLM rate samples for educational value, training cheap classifiers on those ratings and filtering at scale (FineMath, Stack-Edu), plus a synthetic instruction set (SmolTalk). Train for 11T tokens in four stages, changing the mix by hand after each stage based on its scores, and add the best math and code during the final learning-rate decay.
SmolLM2-1.7B's base model beats Qwen2.5-1.5B on MMLU-Pro (19.4 vs 13.7) and TriviaQA but trails it on GSM8K (31.1 vs 61.7), MATH and HumanEval. Filtering code by educational quality raised MultiPL-E by 4 to 8 points per language while keeping as little as 23% of the tokens. Math scores tripled in the last 1T tokens.
Why it matters to an FDE
Few customers will pre-train a model, but many will build training or fine-tuning sets from their own messy data: support tickets, internal code, documents. This paper is the clearest public account of how to curate such data and check that it worked.
The filter recipe transfers. Have a strong LLM score a few hundred thousand examples against a written rubric, train a cheap classifier on those scores, run it over everything, and pick a threshold by small experiments. On code, keeping the 23% to 43% rated most educational in C++, JavaScript and Python raised MultiPL-E by 4 to 8 points. The same recipe works for ticket quality, document relevance or example difficulty.
Thresholds trade quality against repetition. A strict filter leaves less data; if training needs more tokens than that, you repeat it. The authors aimed for no more than 4-5 epochs per dataset, and one heavily filtered math set stopped improving around 10 epochs.
When you add specialized data matters. Small, high-quality sets were held back until the end, where they moved scores the most.
Base-model wins may not survive tuning. The base model beat Qwen2.5-1.5B on MMLU-Pro by 5.7 points; after instruction tuning it trailed by 4.9. Evaluate the model the customer will actually use.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 149 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
What each dataset decision bought, where SmolLM2 wins and loses, and three misreadings
Free account - 04How it works
Ablation setup, the classifier filters, the four stages, context extension and post-training
Premium - 05The numbers that matter
Web, math and code ablations, stage-by-stage scores, base and instruct comparisons
Premium - 06Try it yourself
Recompute the data figures, simulate an LLM-labeled quality filter, and size the KV cache
Premium - 07Where it breaks
Math and code gaps, hand-tuned schedule, benchmark-adjacent data, missing variance and slips
Premium - 08Using it on a deployment
Applying the filter recipe to a customer's data, and choosing a small model by the right evaluation
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium