TL;DR: Exactly-once delivery is impossible across a network, but exactly-once processing is achievable: assume at-least-once delivery everywhere and make every side-effecting consumer idempotent with a stable key recorded in the same transaction as the effect.
How to approach it
This is a vocabulary check with a production twist. Define the three delivery semantics crisply, then immediately ground them in something an FDE actually deploys: a webhook, a billing event, a Kafka consumer. The candidates who pass this question don't stop at definitions; they explain why exactly-once is mostly a myth and what you do about it.
A strong answer
There are three nominal guarantees. At-most-once is fire and forget: if the network hiccups, the message is gone. Fine for metrics samples, fatal for payments. At-least-once means the sender retries until it gets an acknowledgment, so the receiver will eventually see duplicates (a retry after a timed-out ack delivers the same message twice). This is what SQS, Kafka consumers, and virtually every webhook system actually give you. Exactly-once means every message processed once and only once, which, across two independent systems over an unreliable network, is impossible to guarantee at the transport layer. It's the Two Generals problem in disguise.
The senior move is the reframe: "exactly-once delivery doesn't exist, but exactly-once processing is achievable, and you build it yourself with at-least-once delivery plus idempotent consumers." Concretely, every message carries a stable idempotency key (event ID, not timestamp); the consumer records processed keys in the same transaction as its side effect; duplicates become no-ops. Kafka's "exactly-once semantics" is exactly this pattern (transactional producers plus offset commits in one transaction), not magic at the network layer.
The duplicate is worth watching arrive, because it is not exotic; it is one lost packet:
| Time | Producer | Consumer | Customer's card |
|---|---|---|---|
| t0 | Sends evt_9f2c, starts ack timer | ||
| t1 | Receives, charges $49.99, sends ack | Charged once | |
| t2 | Ack lost in transit; timer fires | ||
| t3 | Retries evt_9f2c (correct behavior!) | ||
| t4 | Receives again. Without idempotency: charges again | Charged twice |
Nobody misbehaved. The producer must retry (the alternative is at-most-once and silently dropped payments), so the duplicate is a certainty at scale, and the consumer is the only place it can be absorbed. The pattern that absorbs it is small enough to write out, and interviewers respond well to candidates who can:
BEGIN IMMEDIATE;
INSERT INTO processed_events(event_id) VALUES ('evt_9f2c')
ON CONFLICT DO NOTHING;
-- rowcount 0 -> duplicate: COMMIT and ack, touch nothing else
-- rowcount 1 -> first delivery: perform the side effect, same transaction
INSERT INTO ledger(event_id, amount_cents) VALUES ('evt_9f2c', 4999);
COMMIT;
Run twice with the same event ID, this produces exactly one ledger row (verified: second delivery hits the conflict, rowcount 0, no-op). The load-bearing detail is that the key claim and the charge commit or roll back together. The tempting version, check a Redis set for the key and then write to Postgres, only shrinks the window: two redeliveries in flight at once both pass the check, and a crash between "effect written" and "key recorded" replays the effect on restart.
Why an FDE cares: you deploy systems into customer environments where networks flap, pods get OOM-killed mid-batch, and someone replays a day of events after an outage. If your pipeline charges a customer's customer twice or sends an LLM-generated email twice, that's your incident review. So the default posture is to assume at-least-once everywhere, design every side-effecting handler to be idempotent, and treat "we'll just be careful not to retry" as a bug waiting for load.
What interviewers probe next
- "Where do you store the idempotency keys, and for how long?" Same datastore as the side effect (so dedup and effect commit atomically), with a TTL longer than your maximum retry horizon.
- "What if the side effect is external, an email or a Stripe charge?" Push the key downstream: Stripe accepts an
Idempotency-Keyheader; for email, record intent before sending and accept a small double-send window. - "When is at-most-once actually the right choice?" High-volume telemetry where a lost sample is cheaper than dedup infrastructure.
Common mistakes
Claiming your favorite queue "supports exactly-once" without explaining the consumer-side contract: that's the tell the interviewer is fishing for. Defining the terms correctly but having no answer for "so what do you do about duplicates?" Using a timestamp or hash-of-payload as the idempotency key (two legitimate identical events collide). And hand-waving "we'd dedupe" without putting the dedup check and the side effect in the same transaction, which just shrinks the race window instead of closing it.
Key takeaways
- At-least-once delivery plus idempotent consumers is the only exactly-once anyone actually ships.
- Idempotency key must be a producer-assigned event ID, never a timestamp or payload hash.
- Record the key in the same transaction as the side effect, or you have only shrunk the race, not closed it.
- For external effects (Stripe, email), propagate the key downstream so the third party dedupes too.