TL;DR: Climb a cost-ordered ladder rather than picking a favorite: confirm BM25 is in the stack, then query rewriting with a glossary, then a reranker, and only fine-tune embeddings as a last resort when the eval shows the jargon slice still lags.
How to approach it
Don't pick a favorite; present a cost-ordered ladder and a diagnosis that tells you which rung the problem lives on. First clarify the failure: is jargon failing at the query side (users type "TPS report," index says "transaction processing summary"), the document side (embedding model maps internal acronyms to noise), or both? Sample 20 failing queries and look; the answer usually falls out.
A strong answer
Rung 0, make sure BM25 is in the stack. Jargon and acronyms are exactly what lexical search handles natively; if the system is pure-vector, hybrid retrieval may fix most of this in a day. State this first; it's the senior move.
Rung 1, query rewriting / expansion. Cheapest semantic fix: an LLM pre-processing step that expands acronyms from a curated glossary ("TPS to transaction processing summary"), generates 2–3 paraphrases, or decomposes compound questions, then retrieves with all variants and fuses with RRF. Costs one extra LLM call (~100–300ms, fractions of a cent), needs no training data, ships in days, and is trivially reversible. A 200-term glossary built with the customer's SMEs is often the highest-ROI artifact of the whole engagement. Document-side twin: contextual enrichment, prepending chunk headers/summaries that spell out acronyms at indexing time.
Since the glossary is the rung that does the most work, it is worth seeing at actual size, because "a 200-term glossary" sounds grander than it is. Five real-shaped rows:
TPS -> transaction processing summary
P45 -> employee offboarding process (not the UK tax form)
red route -> priority-1 escalation path (ops slang, not in any doc title)
Falcon -> internal name for the vendor payment system
comp check -> competitive rate comparison (sales) | compensation review (HR)
Each row is one SME conversation, and the last two rows are why an LLM cannot generate this file for you: "Falcon" means nothing outside this company, and "comp check" needs the department context that a rewrite step can apply ("user is in the sales workspace, expand accordingly"). At query time the rewrite is mechanical: "why is my Falcon batch stuck?" becomes "why is my Falcon (vendor payment system) batch stuck?", retrieval runs on both forms, RRF merges. The ambiguous-row problem also explains the maintenance warning below: when sales renames a process, a stale row silently rewrites queries wrong, which is why the glossary needs a named owner and a review cadence like any other config that shapes production behavior.
Rung 2, reranker. A cross-encoder reranks the candidate set with full query-document attention, rescuing fine-grained relevance. Note the dependency honestly: a reranker can only reorder what stage 1 retrieved, so if jargon means the right chunk never reaches the candidate set, reranking fixes nothing. It helps when recall@50 is decent but precision@5 is bad. Off-the-shelf (Cohere Rerank, BGE) ships in days; fine-tuning the reranker on a few thousand labeled pairs is still cheaper than embedding fine-tuning.
Rung 3, fine-tune the embedding model. The heavy fix: contrastive training on (query, relevant-chunk) pairs, typically 1k–10k pairs mined from click logs, eval labels, or synthetic generation. It teaches the model that "P45 process" means the customer's offboarding flow. Costs: weeks not days, and you now own a model artifact (re-embedding the whole corpus on every update, regression risk on general queries, MLOps burden inside the customer's stack). Choose it only when rungs 0–2 plateau and the eval shows persistent semantic-gap failures, and keep a held-out general-query slice to catch regressions.
Decide with data: a golden set sliced into jargon-heavy vs normal queries, measure recall@10 per rung, and stop climbing when the jargon slice converges with the normal slice.
What interviewers probe next
"Where do the fine-tuning pairs come from?" (Mined hard negatives matter more than positives; synthetic queries per chunk via LLM work surprisingly well.) "Glossary maintenance?" (Assign an owner at the customer; stale glossaries quietly rot retrieval.) "What if latency budget forbids query rewriting?" (Cache expansions and do document-side enrichment instead, which is free at query time.)
Common mistakes
Jumping to fine-tuning first, which signals you've never carried the operational cost. Not knowing the reranker's blind spot (it can't rescue what wasn't retrieved). Skipping the BM25 check. And giving no measurement plan per rung, which turns a clean cost ladder back into guesswork.
Key takeaways
- Order the fixes by cost and reversibility; the cheap glossary/query-rewrite rung often closes most of the gap in an afternoon.
- A reranker reorders only what was retrieved, so it cannot rescue jargon that never reached the candidate set.
- Fine-tuning embeddings is last because it costs weeks and a re-embedding migration every time the corpus or model changes.