Neural Machine Translation of Rare Words with Subword Units

Translation is an open-vocabulary problem: new names, numbers, compounds and inflections keep appearing. Neural translation models of 2015 read and wrote from a fixed list of 30,000 to 50,000 words. Anything else became an unknown token, and systems copied or looked up those words in a dictionary afterward. That fails for words that need to be built from parts, such as the German compound for "sewage water treatment plant", or names that must be transliterated into another alphabet.

Sennrich, Haddow and Birch represent words as sequences of subword units instead. Their main method adapts byte pair encoding (BPE), a 1994 compression algorithm. Start with every word in the training data split into characters plus an end-of-word marker. Count every adjacent pair of symbols, weighted by how often each word occurs, and replace the most frequent pair everywhere with a new merged symbol. Repeat. Each merge adds one symbol, so the vocabulary size is the number of characters plus the number of merges, the method's only setting. Frequent words end up as single symbols and rare words as a few known pieces, so no word is ever unknown.

At translation time, a new word is split into characters and the learned merges are applied in order. The model translates sequences of subword units, and pieces are joined back into words afterward. The authors also learn merges jointly on the source and target languages, so that names are split the same way on both sides, which helps transliteration.

On the WMT 2015 English to German and English to Russian tasks, models with subword vocabularies of 60,000 to 90,000 symbols beat a word-level model with a 500,000-word target vocabulary and a dictionary back-off, by up to 1.1 and 1.3 BLEU. The gains were concentrated in rare and unseen words, which make up 9% to 11% of the test words. BPE, in byte-level and other variants, became the standard way language models turn text into tokens.

Tokens are the unit you pay for and the unit that fills the context window. Every LLM API prices and limits by tokens, and those tokens come from a BPE-style vocabulary learned on the provider's data. When you estimate cost or check whether a contract fits in context, count tokens with the model's own tokenizer, not words.

The token count depends on the text. Common English words are one token; rare words, names, code identifiers, numbers and many non-English languages break into several. In our run, a vocabulary learned on nine papers needed 1.52 symbols per word on a tenth, and a German compound became 15 pieces. Budget per language and per document type.

Odd model behavior often traces to tokenization. Counting letters, comparing numbers, matching product codes or handling rare names can fail because the model never sees the characters as units. When a customer reports a strange error on identifiers or numbers, inspect the tokens.

Domain vocabularies can matter for your own models. If you train or fine-tune a small model on a specialized corpus, the tokenizer's merges reflect the text it was learned on. Heavy fragmentation of domain terms makes sequences longer and learning harder.

No unknown words is the design goal. Byte-level and character fallback mean modern models can read any input. They may still read it badly, but they will not drop it, which matters for names and IDs in customer data.