Tokenization & Tokens
A language model does not read characters or words. It reads tokens: sub-word chunks produced by a tokenizer, each mapped to an integer the model embeds. Tokens are the unit of the context window and of billing, and the way text splits into them explains a surprising number of model quirks, which is why almost every loop opens here.
Transcript
Two things go wrong for people who think of a prompt as a pile of words. The bill comes in higher than the word count predicted, by about a third on plain English and by multiples on code. And the model, asked to count the letters in a word, gets it wrong in a way no calculator would. Both have the same cause. The model never saw the words. It saw tokens.
A neural network operates on numbers, so text has to become a sequence of integers first. The naive options are bad at the extremes. One token per character? Enormous sequences. One token per word gives you a vocabulary of millions, and it falls apart on names, typos and code. Sub-word tokenization is the compromise that won: frequent words become one token, rare ones break into a few pieces, and nothing is ever out of vocabulary, because the fallback is bytes.
The dominant method is byte-pair encoding. Watch it on a toy corpus of five words with counts. Start with every word split into characters plus an end-of-word marker, then count adjacent pairs. L plus O appears in low, lower and lowest, nine times in all, so it becomes the first merged token. Recount. Now lo plus W is also nine, so low becomes a token. Low followed by the end marker appears five times, so that merges next. After three merges, low is a single token, while lowest is still low, E, S, T.
A real tokenizer runs this loop tens of thousands of times over billions of words. That's why "the" ends up whole and "unbelievable" doesn't. Once learned, the merge list is frozen. Replaying those merges in order is the whole of tokenization at run time, and what comes out is a list of integers.
Here's an illustrative split of "Tokenizing isn't obvious." Six tokens. Notice the leading space is part of the token, so spacing and punctuation change the split. And one word, "isn't", becomes two tokens. Counting tokens isn't counting words, and it isn't counting characters.
Now the money. APIs bill per token, and the context window is measured in tokens. For English, a rough rule is about four characters, or 0.75 words, per token. Code and JSON run far more than prose of the same length. Other scripts can run several times English for the same meaning. One study found lengths differing by up to fifteen times in the worst pairs. And a UUID or a number splits into arbitrary fragments.
The same fact explains the quirks. Why can't the model count the letters in a word, or reliably reverse a string? Because it never saw letters. Why did a trailing space change the output? Because it changed the tokenization. And tokenizers differ by model family, so the same text can be a different number of tokens, and a different cost, on each provider.
The answer that loses is "a token is basically a word, so this prompt is about 500 tokens." Before you quote a budget or a bill, run the real text through the model's own tokenizer. A word count isn't an estimate. It's a guess. The token strip and the merge diagram are on fdeinterviews.com.
TL;DR: A tokenizer chops text into sub-word units (tokens), each mapped to an integer the model actually consumes. Tokens, not words or characters, are the unit of the context window and of API pricing. Most modern tokenizers use byte-pair encoding, which merges frequent character sequences into single tokens, so common words are one token and rare words split into several.
Why models read tokens, not text
Two things go wrong for people who think of a prompt as a pile of words. The bill comes in higher than the word count predicted, by about a third on plain English and by multiples on code. And the model, asked to count the letters in a word, gets it wrong in a way no calculator would. Both have the same cause: the model never saw the words. It saw tokens.
A neural network operates on numbers, so text has to become a sequence of integers before the model sees it. The naive options are bad at the extremes: one token per character makes sequences enormous and forces the model to relearn spelling everywhere, while one token per word produces a vocabulary of millions and falls apart on anything unseen (names, typos, code, other languages). Sub-word tokenization is the compromise that won. Frequent words become a single token, while rare ones break into a few meaningful pieces, so the vocabulary stays a manageable size (tens of thousands of tokens) and nothing is ever out of vocabulary, because the fallback is bytes.
The dominant method is byte-pair encoding (BPE). It starts from individual bytes and repeatedly merges the most frequent adjacent pair into a new token, learned once over a big corpus. The result: "the" is one token, but "unbelievable" might split into "un", "bel", "iev", "able". A rough rule of thumb for English is about 4 characters or 0.75 words per token.
The merge step is easier to trust once you have watched it run. Take a toy corpus, labeled as such, of five words with counts: low 5, lower 2, lowest 2, new 3, newer 1. Start with every word split into characters plus an end-of-word marker. Count adjacent pairs across the corpus, weighting each by its word's count. The pair l+o appears in low, lower and lowest, 9 times in all, so it becomes the first merged token, lo. Recount. lo+w is now also 9, so low becomes a token. Recount again. low followed by the end marker appears 5 times, once per whole-word "low", so that merges next. After three merges "low" is a single token while "lowest" is still low + e + s + t. A real tokenizer runs this loop tens of thousands of times over billions of words, which is why "the" ends up whole and "unbelievable" does not, and why the merge list, once learned, is frozen: replaying those merges in order is the whole of tokenization at run time.
The two phases in the diagram are why the same sentence costs different amounts on different providers. The merge list is learned once, from one corpus, and every call after that is just replaying those merges in the order they were learned.
A worked example
text: "Tokenizing isn't obvious."
tokens: ["Token", "izing", " isn", "'t", " obvious", "."]
ids: [ 9126, 4954, ... ] # each token -> an integer
The split above is illustrative; different tokenizers draw the boundaries differently, which is the last misconception below. Two things to notice. The leading space is usually part of the token (" isn" not "isn"), which is why spacing and punctuation change the split. And a single "word" like "isn't" becomes multiple tokens. Counting tokens is not counting words, and it is not counting characters.
In the token strip above, count the pieces in "antidisestablishmentarianism": four tokens for one word in this illustration (the exact count depends on the tokenizer), while " A" and " rare" are one token each. That ratio, common words kept whole and rare words shattered, is the entire cost story in one row.
Why interviewers probe this
The answer that loses is "a token is basically a word, so this prompt is about 500 tokens." It is wrong by roughly a third on ordinary English and by multiples on code or non-English text, and every cost estimate built on it inherits the error. This is the foundational screen, and a clear explanation signals you actually understand the stack. The practical consequences are what they want:
- Cost and context. APIs bill per token and the context window is measured in tokens, so "how long is this prompt" is always a token question, never a word one. A document that looks short can be token-heavy if it is full of rare words, code, or non-English text.
- Languages and code are not equal. English is dense in the tokenizer; many other languages and lots of code tokenize into far more tokens per character, so the same content costs more and eats more context. Petrov et al. (2023, 'Language Model Tokenizers Introduce Unfairness Between Languages') tokenized the same text across languages and found lengths differing by up to 15 times in the worst pairs, and the gap survived even in tokenizers trained deliberately for multilingual support.
- The quirks it explains. "Why can't the model count the letters in a word or reliably reverse a string?" Because it never saw letters, it saw tokens. "Why did a trailing space change the output?" Because it changed the tokenization.
The strong answer connects tokenization to a real decision: estimating cost, budgeting a prompt against the context window, or explaining a character-level failure to a teammate.
The same meaning, priced four ways:
| Text | Tokens, roughly | What it does to you |
|---|---|---|
| Common English prose | 1 per 4 characters, about 1.33 per word (0.75 words per token) | The baseline every rule of thumb assumes |
| Code, JSON, indentation | Far more than prose of the same length | Prompts with code cost more than they look |
| Non-Latin scripts | Several times English for the same meaning, up to 15x in the worst pairs | The "why is this locale expensive" ticket |
| A UUID, a hash, a number | Split into arbitrary fragments | Expensive to carry, and why arithmetic on digits is unreliable |
Common misconceptions
- "A token is a word." Sometimes, for common words. Often a token is a sub-word piece, and punctuation and leading spaces are tokens too.
- "Token count is roughly the word count." It is closer to characters / 4 for English, and much higher for code and many non-English scripts. Never assume; measure with the model's own tokenizer.
- "The model sees letters." It sees token IDs. Character-level tasks (counting, spelling, reversing) are hard precisely because the unit of perception is the token.
- "All models tokenize the same." Tokenizers differ by model family, so the same text can be a different number of tokens (and a different cost) across providers.
Key takeaways
- Tokens are sub-word units mapped to integers; they are the unit the model, the context window, and billing all operate on.
- Byte-pair encoding merges frequent character sequences, so common words are one token and rare ones split into several; English averages roughly 4 characters per token.
- Token count is not word count and not character count, and it is higher for code and non-English text.
- Many "dumb" model failures (letter counting, spacing sensitivity) are tokenization artifacts, not reasoning failures.
- Before quoting a budget or a bill, run the real text through the model's own tokenizer. A word count is not an estimate; it is a guess.