TL;DR: Under load, processing time exceeds the visibility timeout, the broker assumes the worker died and redelivers, so two workers run the same message. Fix it with idempotent processing: claim the key with INSERT ON CONFLICT DO NOTHING in the same transaction as the side effect, not check-then-act.
How to approach it
Name the root-cause pattern quickly (this is the at-least-once-delivery bug as it appears in production) then earn the points on why load triggers it and on a fix that's actually race-free. Interviewers use this to find people who've debugged real queue consumers, not just diagrammed them.
A strong answer
The mechanism. The queue (SQS, RabbitMQ, Kafka, Celery+Redis) is at-least-once: a message is redelivered until acknowledged. Under load, processing time stretches (bigger batches, slower downstream calls, CPU contention) until it exceeds the visibility timeout or ack deadline. The broker assumes the consumer died, redelivers, and now two workers process the same message concurrently. Nothing "changed" in the code; the latency distribution shifted until its tail crossed a configured threshold. Variants of the same family: a worker that crashes (or is OOM-killed, or hits a deploy-time SIGKILL) after the side effect but before the ack; consumer-group rebalances reassigning in-flight partitions; aggressive client retries on a slow producer enqueueing duplicates at the front door.
Diagnose to confirm. Log message IDs with attempt counts (SQS exposes ApproximateReceiveCount); correlate duplicates with processing-duration percentiles versus the visibility timeout; check broker metrics for redeliveries and rebalances. You should be able to show "p95 processing time hit 31s; visibility timeout is 30s."
Fix in two layers. The shallow fix is tuning: raise the visibility timeout above worst-case processing time (or heartbeat-extend it for long tasks), slow consumption, scale workers. Necessary, not sufficient, because crashes and rebalances still duplicate.
The real fix is idempotent processing. Every message carries a stable idempotency key (event ID from the producer, not a timestamp, not a payload hash). The consumer claims the key and performs the side effect atomically: insert the key into a processed-keys table with a unique constraint in the same database transaction as the work; a duplicate hits the constraint and no-ops. The subtle race candidates miss is check-then-act (SELECT key, then process, then INSERT), which leaves a window where two concurrent workers both pass the check; under load, exactly when duplicates occur, both succeed. The unique constraint (or INSERT ... ON CONFLICT DO NOTHING claimed first, or a Redis SET NX lease with TTL if the side effect is external) closes it. For external side effects like Stripe charges or emails, propagate the key downstream (Idempotency-Key header) so the third party dedupes too.
The check-then-act race is easy to nod along with and better to have watched, so we reproduced it with two database connections playing the two workers (executed; the interleaving is exactly the one load produces). Both workers SELECT for the key: both see nothing, because neither has written yet. Both then process and insert: the ledger ends with two rows for one message. Same fixture with a primary-key constraint and claim-first ordering: worker A's INSERT OR IGNORE claims the key and processes; worker B's insert affects zero rows and B no-ops; the ledger ends with one row. The lesson compressed: the SELECT told each worker about a past that was already stale by the time it acted, while the constraint makes the datastore adjudicate the present. Any dedup whose read and write are separable is a race under exactly the concurrency that causes duplicates in the first place, which is why the fix must be a single atomic claim and not a smarter check.
The FDE close: ship the fix with a detection metric, duplicate-key hits per hour, so you can tell the customer "duplicates are now caught and counted, here's the graph," and write up why it only bit under load.
What interviewers probe next
- "Why a unique constraint and not a distributed lock?" A lock serializes and adds a failure mode (lock holder dies); the constraint makes the datastore arbitrate, which it already does well. Locks are for when the side effect can't be transactional.
- "The side effect is calling another team's non-idempotent API." Wrap it: record intent first, pass your key if they accept one, otherwise accept a small duplicate window and reconcile.
- "How long do you keep processed keys?" Longer than the maximum redelivery horizon (including DLQ replay); TTL or partition-drop old keys.
Common mistakes
Blaming "a bug in the queue": the broker is doing exactly what at-least-once promises. Fixing only the timeout and declaring victory. Proposing check-then-act dedup that races under the very load that causes the problem. Hash-of-payload keys (legitimate identical messages collide). And no story for external side effects, which is where double-processing actually costs the customer money.
Key takeaways
- Root cause is processing time crossing the visibility timeout, not a queue bug; the broker is honoring at-least-once.
- Tuning the timeout is necessary but not sufficient; crashes and rebalances still duplicate.
- Claim the key first with INSERT ON CONFLICT DO NOTHING in the side-effect transaction; check-then-act races under load.
- Prefer a unique constraint over a distributed lock, and propagate the key to external APIs that accept one.