TL;DR: Treat this as open-set embedding retrieval, not classification: train an image encoder with a metric-learning loss, index reference-photo embeddings in an ANN store, and recognize a query by nearest-neighbor match with a calibrated similarity threshold that returns "unknown" rather than guessing. A flat softmax over millions of long-tailed, constantly-growing classes is the wrong shape for the problem.
How to approach it
The first job is to reject the obvious framing. "Recognize landmarks" sounds like classification, but the catalog is in the millions, grows daily, is brutally long-tailed (the Eiffel Tower has a million photos, a village church has three), and the user can point the camera at a tree, a person, or a landmark you have never indexed. A fixed softmax cannot represent that. So state the shape: this is retrieval over learned embeddings with open-set abstention. Then clarify scale and surface: how many landmarks, is this on-device (Apple-style, privacy-first) or server-side (Google Lens), and is "I don't know" an acceptable answer (it must be). From there: encoder and loss, index, threshold, labeling, evaluation.
A strong answer
Train an image encoder (a vision transformer or a strong CNN backbone) to map a photo to a unit-norm embedding, using a metric-learning / contrastive loss (ArcFace, triplet, or a SimCLR-style objective) so that two photos of the same landmark land close together and different landmarks land far apart. Build a reference index of embeddings for known landmarks (multiple canonical photos per landmark, across angles, seasons, day/night). At query time, embed the photo, do an approximate nearest-neighbor (ANN) search (HNSW or IVF-PQ, via FAISS or ScaNN) against the index, and decide by the top match's similarity: above the threshold, return the landmark; below it, return unknown. This is why retrieval beats classification here: adding a new landmark is inserting vectors into the index, no retraining, and the long tail just needs a few reference photos, not thousands of training images per class.
The open-set threshold is the part most candidates skip and the part that decides whether the product is trustworthy. A confident wrong answer ("that's the Taj Mahal" pointed at a strip mall) destroys more trust than an honest "I'm not sure." Calibrate the threshold on a held-out set that includes negatives: photos of non-landmarks and of landmarks deliberately left out of the index, so the model learns where the unknown region sits. Use top-1 similarity, ideally with a margin check (top-1 minus top-2) so a query that is equally close to ten landmarks is flagged ambiguous rather than forced. Tune the threshold by the cost of the surface: a tourist-info feature can afford to guess; a "tag this in the user's library" feature should abstain readily.
Labels and data are the unglamorous core. Sources: curated reference sets (Wikimedia/Wikipedia geotagged images), GPS metadata on user photos (a strong but noisy label, since a photo near the Colosseum is probably of it), and human verification for the head. The long tail is the hard part: most landmarks will never get hand labels, so lean on GPS-clustered photos plus weak supervision, and accept that tail precision will trail the head. Be explicit about bias: training data skews toward Western, photogenic, heavily-touristed sites, so the model is systematically worse on landmarks in under-photographed regions, and you should measure recall by region and by landmark popularity, not just an aggregate.
| Dimension | Closed-set classifier (softmax over N) | Embedding + ANN retrieval |
|---|---|---|
| New landmark | Retrain / add a class | Insert vectors into the index, no retrain |
| Long tail (few photos) | Class undertrained, poor | Works from a few reference photos |
| Unknown / non-landmark | Forced to pick a class | Threshold returns "unknown" |
| Scale to millions | Output layer explodes | ANN search stays sublinear |
On-device vs server is a genuine tradeoff, not a default. On-device (Apple's instinct) gives low latency and keeps photos private, but a phone can only hold a compressed encoder and a small index, so it handles popular landmarks and punts the long tail. Server-side (Google Lens) holds the full multi-million index and a bigger model but costs a network round trip and ships the user's photo off the device. A sound design is hybrid: a small on-device model and index for the head and for an instant "is this even a landmark" gate, falling back to the server for the long tail and ambiguous cases. Latency budget: interactive recognition wants sub-300ms end-to-end, so on-device inference is tens of milliseconds and the server path is dominated by the round trip plus ANN lookup (single-digit to low-tens of ms on a well-built index).
Evaluation. Offline, this is a retrieval problem: precision@1 and recall@k on a labeled query set, reported separately for head vs tail and by region, plus the open-set metric that matters most, the false-accept rate on the negative set (how often it confidently names a non-landmark) traded against coverage (fraction of true landmarks it is willing to name). Plot the precision/coverage curve and pick the threshold from it. Online, A/B on tap-through and correction rate (users who edit or dismiss the result), with hard guardrails: a rise in confident-wrong reports, a drop in tail recall, or a latency regression should block the launch. The feedback loop: user confirmations and corrections become new labels and new reference photos, especially valuable for the tail, and thumbs-down cases seed the next training round.
What interviewers probe next
- "How do you say 'I don't know'?" Calibrate a similarity threshold on a held-out set that includes non-landmarks and held-out landmarks, add a top1-minus-top2 margin check for ambiguity, and tune the operating point by the surface's tolerance for a wrong guess.
- "Millions of classes, you cannot retrain for each new landmark." That is the argument for retrieval: new landmarks are index insertions, and the encoder is trained for general visual similarity, not a fixed class list.
- "On-device or server?" Hybrid: small on-device model and head index for latency and privacy, server fallback for the long tail and ambiguous queries. State the index-size and round-trip numbers behind the call.
- "Where is it biased?" Toward Western, touristy, well-photographed sites; measure recall by region and popularity and supplement tail data deliberately, because an aggregate score hides the gap.
Common mistakes
Framing it as "train a CNN to classify N landmarks" with no retrieval, which cannot scale, cannot add landmarks without retraining, and starves the long tail. No open-set handling, so the system confidently names a wrong landmark for every photo including photos of non-landmarks. Reporting one aggregate accuracy that hides catastrophic tail and regional gaps. Ignoring the on-device constraint and assuming a full server index is always available. Treating GPS labels as ground truth without acknowledging the noise. And no feedback loop, so user corrections, the cheapest tail labels you will ever get, are thrown away.
Key takeaways
- This is open-set embedding retrieval, not closed-set classification: metric-learning encoder plus ANN index, so new and long-tail landmarks need index inserts and a few reference photos, not retraining.
- A calibrated similarity threshold (with a top1/top2 margin) that returns "unknown" is the trust-defining feature; calibrate it on negatives and held-out landmarks.
- Choose on-device vs server deliberately, usually hybrid: head on-device for latency and privacy, long tail on the server.
- Evaluate retrieval (precision@1, recall@k) and open-set false-accept by head/tail and region, and recycle user corrections as tail labels.