TL;DR: Run BM25 and dense retrieval in parallel to maximize recall, fuse the two ranked lists with Reciprocal Rank Fusion, then rerank the fused shortlist with a cross-encoder for precision. Defend every stage with an ablation table and a named latency budget.
How to approach it
Structure as a funnel with a job per stage: recall stages (BM25 + dense, run in parallel, cast a wide net) then a precision stage (reranker, reorder the net's contents). For each stage be ready with: what it rescues, what it costs, and how you'd measure its contribution.
A strong answer
Stage 1a, BM25 over an inverted index. Rescues: exact identifiers (SKU-88210), error codes, names, acronyms, rare in-domain jargon the embedding model has never seen. Cost: nearly free (<10ms on Elasticsearch/OpenSearch at enterprise scale). Failure: vocabulary mismatch, where paraphrased questions miss.
Stage 1b, dense retrieval (ANN over embeddings, HNSW index). Rescues: paraphrase, synonyms, conceptual queries, multilingual matching. Cost: ~10–50ms ANN lookup plus one embedding call. Failure: "related but wrong," semantically adjacent chunks that differ in the detail that matters (wrong year, wrong customer, wrong version).
Fusion: take top 50–100 from each leg and merge with Reciprocal Rank Fusion, scoring each doc by Σ 1/(60 + rank). RRF needs no score calibration (BM25 and cosine scores live on incomparable scales), which is why it's the production default over weighted-sum schemes that need tuning per corpus.
Step 2 is one box because the two legs run at the same time, not one after the other. The stages are ordered by what they optimize: recall up to step 4, precision after it.
Working RRF once, with the constant at its standard k=60, shows why it behaves the way it does. A document ranked #1 by BM25 but absent from the dense leg scores 1/61 ≈ 0.0164. A document ranked #3 by BM25 and #5 by dense scores 1/63 + 1/65 ≈ 0.0313, nearly double, so consensus between legs beats a single leg's top hit. That is usually the behavior you want: a chunk both retrieval philosophies agree on is rarely garbage, while a single-leg #1 might be a BM25 keyword fluke or an embedding near-miss. The 60 is a damping constant, not magic: it flattens the difference between adjacent ranks (rank 1 scores 0.0164, rank 10 scores 0.0143, only ~15% apart) so one leg's confident-but-wrong head cannot steamroll the merge. All of this needs only ranks, never the raw scores, which is exactly why no per-corpus calibration exists to go stale.
Stage 2, cross-encoder reranker (Cohere Rerank, BGE-reranker, or similar) scores each (query, chunk) pair jointly with full attention, instead of comparing pre-computed vectors. Rescues: fine-grained relevance the bi-encoder bottleneck destroys. It can see that the query's "termination for convenience" doesn't match the chunk's "termination for cause." Cost: this is the expensive stage at ~50–300ms for 50–100 candidates, so it only ever sees the fused shortlist, never the corpus. Typical payoff: +5–15 points NDCG/recall-in-top-5 over fusion alone, routinely the best quality-per-engineering-hour buy in the stack.
Then top 3–8 reranked chunks go to the generator. Total retrieval budget: ~100–400ms, dominated by the reranker. That is fine for chat, worth trimming (smaller rerank candidate set) for stricter latency SLAs.
Measure each stage's contribution by ablation on a golden set: recall@10 for BM25 alone, dense alone, hybrid, hybrid+rerank, sliced by query type (ID-lookups vs natural language). That table is also your customer-facing justification for every component's existence.
The shape of that ablation:
| Configuration | Recall@10 | Latency |
|---|---|---|
| BM25 alone | Low (misses paraphrase) | <10ms |
| Dense alone | Medium (misses exact IDs) | ~10–50ms |
| Hybrid (RRF) | High | ~60ms |
| Hybrid + reranker | Highest (+5–15 NDCG/recall@5) | +50–300ms |
What interviewers probe next
"Where do metadata filters go?" Pre-filter both legs (date, source, ACL) inside the index, not post-filter, or your top-k drains away. "When would you skip the reranker?" Hard latency budgets or tiny corpora where k=5 recall is already high. "Query rewriting?" A stage 0: expand acronyms, decompose multi-part questions, cheap and complementary. "Can a long-context model replace all this?" Stuffing 100 chunks costs ~10–50x in tokens and reintroduces lost-in-the-middle, so retrieval precision still pays.
Common mistakes
Pure-vector answers, the giveaway of tutorial experience. Hand-waving fusion ("combine the scores") without knowing why RRF exists. Putting the reranker over the whole corpus (it's O(candidates), not an index). And presenting the stack without an ablation plan: at retrieval-centric companies like Glean, every stage must earn its latency with a measured recall delta.
Key takeaways
- Each stage has a distinct job: BM25 rescues exact tokens, dense rescues paraphrase, the reranker rescues fine-grained precision.
- RRF fuses incomparable score scales without per-corpus tuning, which is why it beats weighted-sum in production.
- Retrieve wide, rerank narrow, and defend top-k and latency with an ablation table sliced by query type.