Tokenization & Tokens

TL;DR: A tokenizer chops text into sub-word units (tokens), each mapped to an integer the model actually consumes. Tokens, not words or characters, are the unit of the context window and of API pricing. Most modern tokenizers use byte-pair encoding, which merges frequent character sequences into single tokens, so common words are one token and rare words split into several.

Why models read tokens, not text

A neural network operates on numbers, so text has to become a sequence of integers before the model sees it. The naive options are bad at the extremes: one token per character makes sequences enormous and forces the model to relearn spelling everywhere, while one token per word produces a vocabulary of millions and falls apart on anything unseen (names, typos, code, other languages). Sub-word tokenization is the compromise that won. Frequent words become a single token, while rare ones break into a few meaningful pieces, so the vocabulary stays a manageable size (tens of thousands of tokens) and nothing is ever out of vocabulary, because the fallback is bytes.

The dominant method is byte-pair encoding (BPE). It starts from individual bytes and repeatedly merges the most frequent adjacent pair into a new token, learned once over a big corpus. The result: "the" is one token, but "unbelievable" might split into "un", "bel", "iev", "able". A rough rule of thumb for English is about 4 characters or 0.75 words per token.

The two phases in the diagram are why the same sentence costs different amounts on different providers. The merge list is learned once, from one corpus, and every call after that is just replaying those merges in the order they were learned.

A worked example

text:    "Tokenizing isn't obvious."
tokens:  ["Token", "izing", " isn", "'t", " obvious", "."]
ids:     [ 9126,    4954,    ...                          ]   # each token -> an integer

Two things to notice. The leading space is usually part of the token (" isn" not "isn"), which is why spacing and punctuation change the split. And a single "word" like "isn't" becomes multiple tokens. Counting tokens is not counting words, and it is not counting characters.

In the token strip above, count the pieces in "antidisestablishmentarianism": four tokens for one word, while " A" and " rare" are one token each. That ratio, common words kept whole and rare words shattered, is the entire cost story in one row.

Why interviewers probe this

The answer that loses is "a token is basically a word, so this prompt is about 500 tokens." It is wrong by roughly a third on ordinary English and by multiples on code or non-English text, and every cost estimate built on it inherits the error. This is the foundational screen, and a clear explanation signals you actually understand the stack. The practical consequences are what they want:

• Cost and context. APIs bill per token and the context window is measured in tokens, so "how long is this prompt" is always a token question, never a word one. A document that looks short can be token-heavy if it is full of rare words, code, or non-English text.
• Languages and code are not equal. English is dense in the tokenizer; many other languages and lots of code tokenize into far more tokens per character, so the same content costs more and eats more context. Petrov et al. (2023, 'Language Model Tokenizers Introduce Unfairness Between Languages') tokenized the same text across languages and found lengths differing by up to 15 times in the worst pairs, and the gap survived even in tokenizers trained deliberately for multilingual support.
• The quirks it explains. "Why can't the model count the letters in a word or reliably reverse a string?" Because it never saw letters, it saw tokens. "Why did a trailing space change the output?" Because it changed the tokenization.

The strong answer connects tokenization to a real decision: estimating cost, budgeting a prompt against the context window, or explaining a character-level failure to a teammate.

The same meaning, priced four ways:

Common misconceptions

• "A token is a word." Sometimes, for common words. Often a token is a sub-word piece, and punctuation and leading spaces are tokens too.
• "Token count is roughly the word count." It is closer to characters / 4 for English, and much higher for code and many non-English scripts. Never assume; measure with the model's own tokenizer.
• "The model sees letters." It sees token IDs. Character-level tasks (counting, spelling, reversing) are hard precisely because the unit of perception is the token.
• "All models tokenize the same." Tokenizers differ by model family, so the same text can be a different number of tokens (and a different cost) across providers.

Key takeaways

• Tokens are sub-word units mapped to integers; they are the unit the model, the context window, and billing all operate on.
• Byte-pair encoding merges frequent character sequences, so common words are one token and rare ones split into several; English averages roughly 4 characters per token.
• Token count is not word count and not character count, and it is higher for code and non-English text.
• Many "dumb" model failures (letter counting, spacing sensitivity) are tokenization artifacts, not reasoning failures.
• Before quoting a budget or a bill, run the real text through the model's own tokenizer. A word count is not an estimate; it is a guess.