Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

In 2019 BERT was the best model for judging whether two sentences mean the same thing, but it did that by reading both sentences together in one pass. That is fine for one pair. For search it is a disaster: to find the most similar pair among 10,000 sentences you need about 50 million passes, which the authors measured at roughly 65 hours on a V100 GPU. Finding the closest of Quora's 40 million questions to a new one would take over 50 hours per query.

The obvious shortcut was to run BERT on each sentence alone and use its output as a sentence vector. The authors tested that and found it was bad. Averaging BERT's outputs scored 54.8 on a set of similarity benchmarks, and the popular [CLS] token vector scored 29.2, both below simply averaging old GloVe word vectors at 61.3.

Their fix was Sentence-BERT. Two copies of BERT with shared weights each read one sentence, their outputs are averaged into one vector per sentence, and the network is fine-tuned on sentence pairs labeled as entailing, contradicting or unrelated so that the vectors line up with meaning. After that, each sentence is encoded once, and comparing any two is a cosine similarity, a few multiplications.

The 65-hour search dropped to about 5 seconds, and the vectors beat every earlier sentence embedding method. Every embedding model behind a vector database today follows this design.

Every retrieval system you build rests on the trade-off this paper measured.

Embeddings are a compromise, and the paper says how big. When both were trained on the same data, the BERT cross-encoder still beat SBERT on the similarity benchmark (88.8 against 86.1 for the large models). On a harder task, judging whether two arguments make the same claim with the same reasoning on a topic never seen in training, SBERT fell about 7 points behind. A vector built without seeing the other text cannot weigh every word against it.

So production search has two stages. Retrieve candidates with embeddings, because only vectors make search over millions of documents cheap, then rerank the top few dozen with a cross-encoder, because only it reads the query and document together. That pipeline is the direct consequence of this paper's numbers, and recommending it is one of the most common pieces of advice you will give.

Not every model's output is a good embedding. Raw BERT vectors were worse than 2014-era word averages for similarity. A model that is excellent at one task can produce vectors that are poor for search. Use a model trained for embeddings, and test it on the customer's own queries before trusting a public leaderboard.