deduplication

At-least-once delivery means duplicates are guaranteed, not hypothetical. Anyone can use a set; the round is whether your set grows forever, and what you do with an event that arrives after its window closed.

Every pipeline eventually double-loads data, so every FDE screen eventually asks this. There's one canonical idiom, plus a determinism detail that decides whether your dedupe is rerun-safe.

Palantir's home turf: entity resolution disguised as a dedupe task. The naive O(n²) comparison is the trap, blocking, match scoring, transitive closure, and the golden-record question are where the interview actually happens.

The Databricks Data Engineer self-join classic. One inequality predicate does all the work, get it wrong and you double-count every pair and match products with themselves. Here's the clean version and the fan-out trap.