TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

Models with around 125 million parameters, such as GPT-2 small, trained on broad web text, struggle to write a coherent paragraph. Eldan and Li ask whether that is about scale or about the data. A web corpus asks a model to learn grammar, facts about everything, and reasoning all at once. What if the corpus kept the core of language and reduced the world it describes?

They had GPT-3.5 and GPT-4 write TinyStories: short stories of a few paragraphs using only words a 3 to 4 year old would understand. To prevent the generators from repeating themselves, each prompt required one random noun, one random verb and one random adjective from a list of about 1,500 basic words, plus a random subset of features such as dialogue, a plot twist, a bad ending or a moral. A second version, TinyStories-Instruct, prefixes each story with instructions (words to include, a sentence to contain, features, or a summary) so models can be tested on following them.

They trained small GPT-Neo-style models, mostly from about 1M to 35M parameters (up to about 80M in one comparison) and from 1 to 8 layers, each in under 30 hours on one GPU. Evaluation used about 50 hand-written story beginnings: each model wrote 10 completions per beginning, and GPT-4 graded each like a teacher on grammar, creativity and consistency, plus plot and instruction-following for the instruct variant.

Even tiny models wrote fluent stories, and models with about 28M parameters tracked characters and plot consistently. Grammar emerged earliest; consistency and creativity needed more size. Depth helped context tracking, width helped factual knowledge. Attention heads and neurons in the smallest models had readable roles. The authors check for memorization with n-gram overlap and conclude the models produce new stories, while noting that copying of plot templates cannot be ruled out.

Narrow the domain to shrink the model. When a customer needs language generation in a tight domain (status messages, simple summaries, form-filling dialogue), a small model trained on a focused corpus can be enough. TinyStories is the clean demonstration that data scope, not only parameter count, decides what a small model can do.

Use random constraints to make synthetic data diverse. Asking a model for "a story" a million times gives near-copies. Forcing random combinations of required words and features spreads the outputs. The same trick works for synthetic tickets, emails or forms: sample required entities, intents and attributes per item.

A model grader gives multidimensional scores cheaply. GPT-4 scored grammar, creativity and consistency separately, which revealed that capabilities emerge at different sizes. For customer evaluations, separate rubric dimensions are more useful than one number, provided you calibrate the grader against people.

Overlap checks catch copying of words, not of structure. The paper's Rouge-2 tests show completions do not repeat training text word for word. Our test shows a training story with 14 words swapped scores as low as the paper's evidence of novelty. For customer models trained on templates, test for template reuse directly.

Watch the grader's leniency. In the paper's own table, a 1-layer model's completion in which a girl climbs a ladder "and rescued Lucy", herself, received 7 of 10 for consistency.