Efficient Estimation of Word Representations in Vector Space

In 2013, the standard way to learn word vectors was a neural network language model: look up vectors for the previous few words, pass them through a hidden layer, and predict the next word over the whole vocabulary. The vectors it learned were useful, but the hidden layer and the output over a large vocabulary made training expensive, so these models saw at most a few hundred million words with 50 to 100 dimensions.

Mikolov and colleagues at Google measured training cost as the number of parameters touched per training word and found that the hidden layer dominated. They proposed two models without one. CBOW (continuous bag of words) averages the vectors of four words before and four after a position and predicts the word in the middle. Skip-gram goes the other way: it takes the middle word and predicts each word within a randomly chosen range of up to ten positions on either side. Both use a hierarchical softmax over a Huffman tree of the vocabulary, so predicting a word costs about the logarithm of the vocabulary size instead of the full vocabulary.

To measure quality, they built a test of 19,544 analogy questions, such as "Athens is to Greece as Oslo is to ?", in five semantic types (capitals, currencies, city-in-state, family relations) and nine syntactic types (plurals, tenses, comparatives). A question is answered by vector arithmetic: take Greece minus Athens plus Oslo and find the nearest word vector. Only an exact match counts.

Skip-gram learned the semantic relations far better than earlier models, and CBOW was strong on syntax and faster. With 300 dimensions on 783 million words, skip-gram scored 53.3% against 24.6% for the best previously published vectors. Trained on 6 billion Google News words with 1,000 dimensions on a distributed system, skip-gram reached 65.6%. More data and more dimensions helped, but only together. The authors also released their code; it became known as word2vec.

Every embedding service descends from this idea. Turning text into vectors where nearby means related is what semantic search, clustering, deduplication and retrieval for RAG rely on. Modern embedding models use transformers and train on sentence pairs, but the contract is the one this paper popularized: similar use, similar vector, compared by cosine similarity.

Cheap and simple beat clever and slow. The paper's main move was to remove the expensive part of the model and spend the savings on far more data. That trade still decides many deployments: a smaller embedding model on the customer's own text can beat a larger general one on their task.

Analogy scores are a narrow test. The benchmark counts only exact top-1 matches. In our small run, the right answer was in the top 3 for 98% to 100% of questions, yet the two examples we printed were both misses at top 1. Evaluate embeddings on the customer's retrieval or classification task, not on analogies.

Static vectors have one meaning per word. "Bank" gets a single vector whether it means a river or a lender, and words outside the vocabulary get none. That is why contextual models replaced word2vec for most uses, and why subword tokenization (the next paper on this site) matters.

Word vectors still have niche value. For a customer with a large domain corpus and tight latency or cost limits, training word or item vectors on their own data (product IDs in baskets, codes in claims) is quick and often good enough.