TL;DR: The dot product blends direction and magnitude (
a · b = |a| |b| cos θ); cosine keeps only direction. Match the metric to how the embedding model was trained: normalized-vector models make dot product and cosine identical, and raw dot product on un-normalized vectors lets magnitude (often length) corrupt ranking.
How to approach it
Start from what the dot product actually computes, then connect it to the two places it shows up: attention scores inside the model, and similarity ranking in a vector database. The interviewer is checking whether you treat the similarity metric as a setting you chose deliberately, or as a default you copied from a tutorial. The strong move is to tie the choice back to how your embedding model was trained, because that, not preference, is what decides dot-product versus cosine.
A strong answer
The dot product of two vectors equals the product of their magnitudes times the cosine of the angle between them: a · b = |a| |b| cos θ. So it blends two things: direction (are these vectors pointing the same way, meaning semantically aligned) and magnitude (how long each vector is). Cosine similarity strips the magnitude out and keeps only direction, by dividing through by both norms. That single difference is the whole question.
Inside the model, attention scores tokens with scaled dot products of queries and keys, QKᵀ / √d_k. The scaling matters: without dividing by the square root of the head dimension, dot products of high-dimensional vectors grow large, push softmax into a near one-hot regime, and starve the gradients. The dot product is used here because magnitude is meaningful; the model learns to make some signals "louder."
In retrieval, the right metric depends on your embedding model. Many modern embedding models (OpenAI's, most sentence-transformers, Cohere Embed) are trained to produce normalized vectors, so every vector has length one. When magnitudes are all one, the dot product equals cosine similarity: they give identical rankings, and the dot product (maximum inner product search, MIPS) is just the cheaper way to compute it. That is why most vector databases default to inner product.
Two dimensions are enough to watch the metrics disagree (numbers computed, not sketched). Query q = [1, 0]. Document d1 = [0.9, 0.4] points nearly the same way and has norm 0.985. Document d2 = [2.0, 2.0] sits at 45 degrees but is long, norm 2.83, the kind of vector a verbose generic chunk produces. Raw dot product: q · d1 = 0.9, q · d2 = 2.0, so the misaligned document wins on length alone. Cosine: 0.914 versus 0.707, and the aligned document wins. Same three vectors, opposite rankings, and the only thing that changed is whether magnitude was allowed to vote. Scale this to 1,536 dimensions and a few million chunks and you have the production bug in the trap below.
The trap is using a raw dot product on un-normalized embeddings. There, magnitude leaks into the score, and magnitude often tracks token count or term frequency, so longer or more "generic" documents score higher regardless of relevance. Symptom: your RAG keeps surfacing the same few verbose chunks. Fix: normalize the embeddings (then dot product and cosine agree), or use cosine directly. Conversely, if your model was trained with a raw inner-product objective, normalizing throws away signal it intended you to keep.
The one-line rule worth saying: match the metric to the training objective. Cosine (or normalized dot product) for the cosine-trained models that dominate today, raw inner product only when the model was built for it.
What interviewers probe next
- "Why the √d_k scaling in attention?" It keeps dot-product variance stable as dimension grows, so softmax doesn't saturate and kill gradients.
- "Cosine and normalized dot product give the same ranking, so why prefer one?" They rank identically on unit vectors; inner product is faster and is what MIPS-based indexes (FAISS, ScaNN) optimize, so you store normalized vectors and query with dot product.
- "Your retrieval favors long documents. Why?" Un-normalized embeddings letting magnitude dominate; normalize, or switch to cosine.
- "When is Euclidean distance the right call instead?" When absolute position matters, not just angle; for normalized vectors L2 distance is monotonic with cosine anyway, so it rarely changes the ranking.
Common mistakes
- "Always use cosine." It is the safe default, but on already-normalized vectors it is identical to the dot product and just slower, and on inner-product-trained models it can be wrong.
- Mixing metrics between indexing and querying, or normalizing at one stage but not the other; that silently corrupts every score.
- Forgetting the attention scaling factor and hand-waving softmax; interviewers at the labs notice.
- Treating magnitude as noise. In attention it is signal the model learned on purpose; in retrieval it is usually noise you want to remove. Knowing which is which is the point.
Key takeaways
- Dot product = direction times magnitude; cosine drops magnitude and keeps direction only.
- On normalized vectors the two rank identically, so store normalized and query with the cheaper dot product.
- Magnitude is learned signal in attention but usually noise in retrieval; the long-chunk bias is the classic un-normalized symptom.