Embeddings & Vector Representations
An embedding turns a piece of text into a list of numbers positioned so that similar meanings land near each other in space, which lets you search by meaning instead of by keyword. Embeddings are the engine under RAG, semantic search, clustering, and deduplication, so FDE loops expect you to explain cosine similarity and the pitfalls that quietly break a vector index.
Transcript
You search your documents for car. The page that answers you said automobile, and because keyword search matches strings and nothing else, it never shows up.
Embeddings fix this by mapping text to a point in a space with hundreds to a couple thousand dimensions, where position encodes meaning, so car and automobile land near each other and car and banana land far apart. Once your text is a vector, find similar things becomes find nearby vectors: a math problem with fast, well-understood solutions.
The model is trained on pairs of texts known to belong together, like a question and its answer, and adjusted so paired texts end up close and unrelated ones end up far apart. Repeat that over an enormous number of pairs and distance stands in for meaning. That shape belongs to one model.
To rank by meaning, you take the cosine of the angle between two vectors. Take vector A equal to one, two, and vector B equal to two, three. The dot product is one times two plus two times three. That's eight. Divide by both lengths and the cosine comes out at 0.992: nearly the same direction. Now take vector C, equal to minus two, one. Its dot product with A is zero, so the cosine is zero, even though C is exactly as long as vector A.
It runs from minus one to one. One means the same direction, zero means unrelated, and negative means opposed. Length never entered the answer. That's the point.
That's why most pipelines normalize vectors to unit length first. After that, the dot product equals the cosine, and for two unit vectors the distance between them, squared, is two minus twice their cosine. As cosine goes up, distance goes down, so sorting by either gives the same list. A vector index can use whichever is cheaper.
Now the rule that breaks most first attempts. Your documents go through an embedding model once, when you build the index, and after that every query must go through the exact same model. Index with one model and query with another, and the vectors sit in incompatible spaces. The numbers are still numbers, so nothing errors, and the results just turn into garbage.
Here's the answer that loses in the interview: upgrade to the better embedding model and re-index new documents as they come in. Half the corpus is now in one space and half in another. Nothing errors. Similarity scores across the two halves are meaningless, and search returns noise to every user who asks. Re-embed everything in one job, and cut over at once.
One last trap. Cosine measures how close two meanings are. Two sentences can be near identical in meaning, and neither one answers the user. That's why rerankers sit on top of vector search: a second model reads the query and each candidate together, and because that's slower, it only runs on the short list.
So when they ask what happens after you re-index half your documents with a new model, say it plainly: the two halves are in different spaces, and you re-embed everything with one model. The worked example is on fdeinterviews.com.
TL;DR: An embedding is a vector of numbers that captures the meaning of text, placed so that semantically similar items sit close together and you can rank relevance by the angle between vectors (cosine similarity). The rule that breaks most first attempts: you must index and query with the exact same embedding model, or the geometry no longer lines up and your search returns noise.
The intuition
Keyword search matches strings. Search "car" and you miss every document that said "automobile." Embeddings fix this by mapping text to a point in a high-dimensional space (often hundreds to a couple thousand dimensions) where position encodes meaning. "Car" and "automobile" land near each other; "car" and "banana" land far apart. The model learned this geometry from enormous amounts of text, so directions in the space roughly correspond to semantic relationships.
How it learned that is worth one paragraph, because it explains the pitfalls later on. An embedding model is trained on pairs of texts known to belong together, such as a question and its answer or two sentences from the same passage, and adjusted so that paired texts end up close and unrelated texts end up far apart. Repeat that over an enormous number of pairs and the space takes on a shape where distance stands in for meaning. Nothing about that shape is shared with a model trained separately, which is why two models' vectors cannot be compared.
Once your text is a vector, "find similar things" becomes "find nearby vectors," a math problem with fast, well-understood solutions.
Measuring similarity
The standard measure is cosine similarity: the cosine of the angle between two vectors. It ranges from -1 to 1, where 1 means the same direction (very similar), 0 means unrelated, and negative means opposed. A two-dimensional example makes the formula concrete; real vectors have hundreds of dimensions, and the arithmetic is the same. Take a = (1, 2) and b = (2, 3). The dot product is 1×2 + 2×3 = 8. The lengths are √5 ≈ 2.236 and √13 ≈ 3.606. Cosine is 8 ÷ (2.236 × 3.606) ≈ 0.992: nearly the same direction. Now c = (-2, 1): the dot product with a is -2 + 2 = 0, so the cosine is 0 and the two are unrelated, even though c is exactly as long as a. Length never entered the answer, which is the point of the measure. Cosine looks at direction, not length, which is why most pipelines normalize vectors to unit length first. After normalization the dot product equals the cosine directly, and ranking by Euclidean distance gives the same order (the nearest by distance is the most similar by cosine), so a vector index can use whichever is cheaper. The reason is a one-line identity: for two unit vectors the squared distance between them equals 2 minus twice their cosine, so as cosine goes up, distance goes down, and sorting by either gives the same list.
| Measure | What it compares | When the ranking is safe |
|---|---|---|
| Cosine similarity | The angle between vectors, length ignored | Always, for vectors from one model |
| Dot product | Angle and length together | Only after normalizing to unit length, when it equals cosine |
| Euclidean (L2) distance | Straight-line distance | After normalization, when nearest by distance is nearest by cosine |
In the flowchart all three sentences pass through the same box, and that box is the contract. Swap it for a different model and the "shared space" on the right stops being shared, with no error to tell you so.
Where you use them
- Semantic search and RAG. Embed your documents once, embed the query at request time, return the nearest neighbors. This is the retrieval half of RAG.
- Clustering. Group similar items (support tickets, product reviews) without predefined labels by clustering their vectors.
- Deduplication and near-duplicate detection. Two near-identical paragraphs sit almost on top of each other; a similarity threshold catches them even when the wording differs.
- Classification and recommendation. Nearest-neighbor over labeled examples gives a cheap, retrain-free classifier, and "items near this one" gives recommendations.
Read the diagram as two clocks running at different speeds. The top half runs once, when you build the index; the bottom half runs on every query. Step 7 is where most production failures start, because nothing errors when the two halves disagree.
The pitfalls that bite
A few mistakes recur in production:
- Mismatched models. Indexing with one embedding model and querying with another produces vectors in incompatible spaces. The numbers are still numbers, so nothing errors; results just quietly become garbage. If you upgrade the model, you must re-embed the entire corpus.
- Domain mismatch. A general-purpose model may not separate the fine distinctions in legal, medical, or internal-jargon text. Evaluate on your own data before trusting it. That is the measured conclusion, not caution: Muennighoff et al. (2022, 'MTEB: Massive Text Embedding Benchmark') ran 33 models across 8 task types, 58 datasets and 112 languages and found no single embedding method dominated across all tasks.
- Forgetting normalization. Mixing normalized and unnormalized vectors, or assuming the database uses cosine when it defaults to L2, gives subtly wrong rankings.
- Treating similarity as relevance. Cosine measures semantic closeness, not correctness for the task. Two sentences can be near-identical in meaning yet neither answers the user. This is why rerankers exist on top of vector search. A reranker is a second model that reads the query and each candidate together and scores whether the candidate answers it, which is slower per pair and therefore run only on the short list that vector search returns.
Why interviewers probe this
The answer that loses is "we'd upgrade to the better embedding model and re-index new documents as they come in." Half the corpus is now in one space and half in another, nothing errors, and search quietly returns noise. It tests whether you understand the layer beneath RAG rather than just calling an API. The screening follow-up is usually "you re-indexed half your documents with a new embedding model, what happens?" The answer they want: the two halves are in different spaces, similarity scores across them are meaningless, and you must re-embed everything with one model. A candidate who knows that has actually run an index in anger.
Common misconceptions
- "Embeddings understand text the way a person does." They capture statistical co-occurrence of meaning learned from training data, not grounded understanding. Out-of-domain text degrades them.
- "Any two embeddings can be compared." Only if they came from the same model. Cross-model comparisons are not meaningful.
- "Higher cosine similarity always means a better answer." It means semantically closer, which is not the same as relevant or correct.
- "More dimensions are always better." Larger vectors cost more storage and compute per query; the right size is a measured trade-off, not a maximum.
Key takeaways
- An embedding places text in a vector space where proximity means semantic similarity, enabling search by meaning.
- Rank with cosine similarity; normalize to unit vectors so direction is what counts.
- Always index and query with the same model, and re-embed the whole corpus if you change models.
- Semantic closeness is not the same as relevance or correctness, which is why reranking sits on top of raw vector search.
- When you change embedding models, re-embed everything in one job and cut over at once. A half-migrated index has no error message.