BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

By 2018, pre-training a language model on unlabeled text and then adapting it to a task had proven itself. OpenAI's GPT trained a Transformer left to right and fine-tuned it; ELMo trained separate left-to-right and right-to-left models and fed their features into task-specific networks. Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova at Google argued both were limited by unidirectionality: a left-to-right model's representation of a word cannot use the words that follow it.

You cannot train an ordinary language model in both directions at once, because each word would indirectly see itself. BERT (Bidirectional Encoder Representations from Transformers) avoids this with a masked language model: pick 15% of the tokens in a sequence, hide them, and train the model to predict the originals from everything else. To reduce the mismatch with fine-tuning, where no tokens are hidden, a chosen token is replaced with [MASK] 80% of the time, a random token 10% of the time, and left unchanged 10% of the time. A second task, next sentence prediction (NSP), gives the model two text spans and asks whether the second really followed the first (half the time) or was drawn at random.

Inputs are WordPiece tokens with a [CLS] token at the start and [SEP] between and after the two spans; each position adds a token embedding, a segment embedding (A or B) and a position embedding. BERT-Base has 12 layers and 110M parameters, matched to GPT's size; BERT-Large has 24 layers and 340M. Both were pre-trained on BooksCorpus and English Wikipedia (3.3 billion words).

For a task, add one output layer and fine-tune every weight: a classifier on the [CLS] vector for sentence or pair classification, or start and end vectors over token outputs for question answering. With that recipe BERT set new best results on eleven tasks.

Generative models get the attention, but BERT-style encoders still do much of the quiet work in production: embedding models for search, rerankers, classifiers for routing and moderation, named-entity taggers.

Encoders are cheap and fast. A 110M-parameter encoder classifies or embeds text in milliseconds on a CPU. For high-volume routing, tagging or deduplication, it often beats calling a large model on cost and latency.

Fine-tuning needs little data. The paper's results include tasks with a few thousand labeled examples, fine-tuned in under an hour on one accelerator. A customer with 3,000 labeled tickets can train a solid classifier.

Know what the [CLS] vector is not. The authors note the vector is not a meaningful sentence representation without fine-tuning. Using raw BERT outputs as search embeddings works poorly; use a model trained for embeddings.

Small-data fine-tuning is unstable. The authors ran several random restarts for BERT-Large on small datasets and kept the best on the dev set. Budget for multiple runs and a held-out test set.