TF-IDF and BM25

TL;DR: BM25 ranks documents by counting query-term matches (term frequency), weighting rarer terms more (inverse document frequency), then applying two corrections that make it work in practice: it saturates repeated terms so the 20th occurrence barely helps, and it normalizes for document length so long docs do not win automatically. It needs no training, runs in milliseconds, and is the strong lexical baseline that hybrid search pairs with vectors.

Start with TF-IDF

The core intuition is two competing forces.

Term frequency (TF): a document that mentions "idempotency" five times is probably more about idempotency than one that mentions it once. More matches, higher score.

Inverse document frequency (IDF): but if a word appears in almost every document, matching it tells you nothing. "The" is in everything; "idempotency" is in a handful. IDF weights each term by how rare it is across the corpus, so matching a rare term counts far more than matching a common one. Roughly, idf(t) = log(N / df(t)), where N is the number of documents and df(t) is how many contain term t.

Classic TF-IDF multiplies these: tf(t, d) * idf(t), summed over query terms. It works, but it has two flaws that BM25 fixes.

What BM25 adds

The spine above scores a single document end to end, with saturation and length normalization drawn as the two steps that separate BM25 from raw counting.

BM25 (the "Okapi" weighting scheme) keeps TF and IDF and corrects raw TF in two ways.

Saturation. Raw TF is linear: a term appearing 50 times scores 50x a single occurrence, which is absurd. A document that says "GPU" 50 times is not 50 times more relevant than one that says it twice. BM25 runs TF through a saturating function controlled by k1 (typically 1.2 to 2.0), so the score rises fast for the first few occurrences then flattens.

Computed with the worked example's constants below (k1 = 1.5, b = 0.75, a document of average length, the term "idempotency" with idf = 6.91), the contribution of that one term as it repeats:

The second occurrence adds about 3 points, the tenth about 0.4, and the fiftieth almost nothing. The ceiling is idf * (k1 + 1), here 17.3, and no amount of repetition passes it. Raw TF-IDF at fifty occurrences scores twenty times higher than BM25, which is the "GPU fifty times" document winning on volume.

Length normalization. Long documents contain more words, so they accidentally accumulate more matches. BM25 penalizes documents longer than the corpus average, controlled by b (typically 0.75, where b=0 disables it and b=1 applies it fully).

The same term appearing three times, computed at four document lengths against an average of 300 words:

Three mentions in a document four times the average length count about as much as one mention in an average one. The effect runs both ways: a document shorter than average gets a boost, which is why a two-line FAQ entry that names the term can outrank a long chapter that also names it. That is usually what you want, and b is the dial when it is not.

The full score for a document d against query Q:

score(d, Q) = sum over t in Q of:
    idf(t) * ( f(t,d) * (k1 + 1) ) / ( f(t,d) + k1 * (1 - b + b * |d| / avgdl) )

where f(t,d) is the term frequency in d, |d| is the document length in words, and avgdl is the average document length in the corpus.

The idf(t) used on this page is the plain TF-IDF form, log(N / df(t)), which is enough to see the mechanism. Lucene and Elasticsearch, where most people meet BM25, use log(1 + (N - df + 0.5) / (df + 0.5)), a smoothed version that stays positive even for a term in nearly every document. The rankings it produces follow the same intuition; the numbers differ slightly.

A short worked example

Corpus of N = 1,000,000 documents, avgdl = 300 words. Query: idempotency key. Constants k1 = 1.5, b = 0.75.

• "idempotency" appears in df = 1,000 docs, so idf ≈ log(1e6 / 1e3) = log(1000) ≈ 6.9.
• "key" appears in df = 200,000 docs, so idf ≈ log(1e6 / 2e5) = log(5) ≈ 1.6.

A candidate document of length |d| = 300 (exactly average, so the length factor is 1) with "idempotency" appearing 3 times and "key" twice:

import math

def bm25_term(f, df, N, dl, avgdl, k1=1.5, b=0.75):
    idf = math.log(N / df)
    norm = f * (k1 + 1) / (f + k1 * (1 - b + b * dl / avgdl))
    return idf * norm

N, avgdl, dl = 1_000_000, 300, 300
score  = bm25_term(f=3, df=1_000,   N=N, dl=dl, avgdl=avgdl)   # "idempotency"
score += bm25_term(f=2, df=200_000, N=N, dl=dl, avgdl=avgdl)   # "key"
print(round(score, 2))   # ~13.81

The rare term "idempotency" carries almost all the weight (its IDF is ~4x higher), and its third occurrence adds far less than its first thanks to saturation. That is BM25 doing exactly what you want: reward rare, specific matches, ignore filler, and refuse to be gamed by repetition.

Why it is still a strong baseline

BM25 needs no training, no embeddings, no GPU, and no corpus-specific tuning beyond two constants. It runs in milliseconds over millions of documents via an inverted index, and it nails the cases dense vectors fumble: exact part numbers, function names, error codes, SKUs, legal clause numbers, and rare jargon. When someone searches Error 0x80070643, BM25 returns the doc containing that exact string; a vector model maps it into a fuzzy neighborhood and may return a different error that "feels" similar.

Its blind spot is the mirror image: it scores on shared words, so it misses paraphrase. "How do I cancel my plan" will not match a passage titled "ending your subscription" if they share no terms. That is exactly why modern retrieval runs BM25 and dense vector search side by side and fuses the results: each catches what the other drops. See hybrid-search and retrieval-augmented-generation.

Why interviewers probe this

Reaching for an embedding model first on every retrieval problem is a tell that someone has read one tutorial and shipped nothing. The strong move is to start with BM25 as a baseline, measure recall, and only add a vector-databases layer where paraphrase actually hurts. The follow-up they hold back: "your vector search underperforms on internal docs, why?" The answer names BM25's strength on the codes and acronyms that dense models blur, and proposes hybrid rather than swapping one for the other.

Common misconceptions

• "BM25 is obsolete now that we have embeddings." It is the lexical half of hybrid search and often beats vectors alone on identifier-heavy corpora.
• "More matches always means a higher score." Saturation flattens repeated terms, and IDF can make one rare match outweigh many common ones.
• "It ignores document length." The b parameter normalizes for length so long documents do not win by volume.
• "TF-IDF and BM25 are the same." BM25 adds frequency saturation and length normalization, which is why it works in production where raw TF-IDF struggles.

Key takeaways

• BM25 = term frequency, weighted by inverse document frequency, with saturation (k1) and length normalization (b).
• Rare terms dominate the score via IDF; repeated terms saturate so repetition cannot game the ranking.
• No training, no GPU, milliseconds over millions of docs, and unbeatable on exact codes, IDs, and rare jargon.
• Its weakness is paraphrase, which is why it pairs with dense vectors in hybrid search rather than being replaced by them.