TL;DR: A token is a subword chunk (the unit the model reads and bills on). For English, 1 token is roughly 4 characters or 0.75 words, and the FDE consequences are cost, latency, context budget, and arithmetic/spelling failures.
How to approach it
The answer that reads junior is "a token is basically a word." It is nearly right, and it produces bills that are wrong by a third and cost plans that trim the cheap side of the request. Define it in one sentence, then go straight to consequences: billing, latency, context budgeting, and characteristic model failures. Interviewers at AI labs ask this to see whether you connect a basic concept to the operational questions customers actually raise.
A strong answer
A token is the unit a model reads and writes: a subword chunk produced by a tokenizer (typically byte-pair encoding). Common words are one token ("the"); rarer words split into pieces ("tokenization" becomes 2 to 3 tokens). Rule of thumb for English: 1 token is about 4 characters or 0.75 words, so 1,000 tokens is roughly 750 words.
Seeing one split makes it stick. A BPE-style tokenizer handles a sentence something like this (boundaries vary by model, so treat this as the shape, not gospel):
"Unbelievably, the API returned NULL for user_4821."
Un|believ|ably|,| the| API| returned| N|ULL| for| user|_|48|21|.
Everything the model "sees" is that sequence of chunks. Frequent words survive whole, rare words shatter, punctuation and whitespace ride along with neighbors, and the ID at the end becomes arbitrary fragments. Every consequence below falls out of this picture.
Follow the spine and the bill explains itself. The same token is counted twice, once against the context ceiling and once against the input rate, and then every token the model writes is counted a third time at several times the price.
Why an FDE cares, in four buckets:
-
Cost. APIs bill per token, input and output priced separately (output usually 3 to 5x input). If a customer's RAG app stuffs 20k tokens of retrieved context into every call at, say, $3 per million input tokens, that is $0.06 per request before any output; at 100k requests/day that is $6k/day on context alone. Being able to do this arithmetic live is the difference between a demo engineer and an FDE.
-
Latency. Input tokens drive time-to-first-token; output tokens are generated one at a time, so verbose answers are slow answers. "Make responses shorter" is a legitimate latency fix.
-
Context budgeting. The context window is measured in tokens, not pages. A 100-page contract is around 50k tokens; that either fits or it doesn't, and tokenization is how you know.
-
Non-English and code inflation. Tokenizers are trained mostly on English; Japanese, Hindi, or niche code formats can take 2 to 4x more tokens for the same content. A customer's "same app, why is the German deployment 2x the cost?" question is a tokenization question. The 2 to 4x figure is the everyday range; the tail is worse. Petrov et al. (2023, 'Language Model Tokenizers Introduce Unfairness Between Languages') tokenized the same text across languages and found lengths differing by up to 15 times in the worst pairs, and the disparity survived even in tokenizers trained deliberately for multilingual support.
Tokenization also explains famous failure modes: models botch arithmetic and string reversal because "12345" may be split arbitrarily ("123", "45") and the model never sees individual characters or digits as clean units. Counting letters in "strawberry" fails for the same reason. The right production fix is usually a tool call (calculator, code execution), not a better prompt.
| Text | Roughly | Why it matters to you |
|---|---|---|
| Common English word | 1 token | The baseline intuition |
| 100 words of English prose | ~130 tokens | The rule of thumb: words times 1.3 |
| A UUID or a hash | 15-25 tokens | IDs are expensive and people paste them constantly |
| Code with symbols and indentation | Far more than prose | Code prompts cost more than they look |
| Non-English, especially non-Latin script | 2-4x English for the same meaning | A real fairness and cost issue for global deployments |
| Numbers | Split unpredictably | Part of why arithmetic is unreliable |
The last two rows are the FDE-relevant ones. A customer serving a non-English market pays several times more per unit of meaning, and that shows up in their bill and in their context budget rather than in any benchmark.
What interviewers probe next
- "How would you estimate the monthly token bill for a use case?" Requests/day times (avg input + output tokens) times price; have the roughly $1 to $15 per million tokens range for current frontier models in your head. Then do one out loud, because this is exactly the live exercise: a support assistant at 50k requests/day, 2,000 input tokens (system prompt plus retrieved context), 300 output tokens, at illustrative rates of $3 and $15 per million. Input: 50,000 x 2,000 x $3/M = $300/day. Output: 50,000 x 300 x $15/M = $225/day. Call it $525/day, roughly $16k/month, and note out loud that the two lines are the same order of magnitude despite output being 5x the price, because there are far fewer output tokens. That last observation is what tells the interviewer you understand the structure and not just the multiplication.
- "Why does the same prompt cost differently on two providers?" Different tokenizers (token counts differ by roughly 10 to 30%) and different prices; compare cost per task, not per token.
- "How do you count tokens before sending?" Provider tokenizer libraries (for example tiktoken) or count endpoints; never estimate by characters for budget-critical paths.
The four reflexes this question is built to catch:
| What people reach for | Why it fails | What to say instead |
|---|---|---|
| "A token is a word" | Rare words split, IDs and punctuation shatter, and counts drift a third or more from a word count | "A subword chunk; about 0.75 English words on average" |
| "Trim the prompt to cut cost" | Output tokens cost several times input and are generated one at a time | "Trim the answer first; it is the expensive, slow side" |
| "Same app, same cost in every market" | Non-English text tokenizes into far more pieces for the same meaning | "Budget per locale, measured with the real tokenizer" |
| "Ask the model to count letters or add numbers" | It never sees characters or digits as units; "12345" may arrive as "123", "45" | "Route it to a tool: a calculator or code execution" |
Common mistakes
- Saying "a token is a word" without the subword nuance. Minor alone, but it cascades into wrong cost estimates.
- Having no numbers at all. This question exists to test fluency; "tokens cost money" without a rough price or the 4-chars heuristic reads junior.
- Blaming arithmetic failures on "the model being dumb" rather than the tokenization plus next-token-prediction explanation, and not knowing the tool-use fix.
- Forgetting output tokens cost more than input. It flips many optimization decisions.
Key takeaways
- 1 token is roughly 4 English characters or 0.75 words; reach for tiktoken when budgets depend on it.
- Output tokens cost several times input, so trim the answer before the prompt.
- Non-English text and code inflate token counts 2 to 4x, which explains "why is this locale more expensive" tickets.
- When a customer asks why one market costs more, answer with a tokenizer and not a guess: tokenize the same content in both languages in front of them.