FDEInterviews logo
System Design & Production Engineering / 04
easyRetoolMicrosoftDatabricks

What is a dead-letter queue, and how do you handle a poison message in production?

One malformed message can silently wedge an entire pipeline, and it happens at every customer eventually. The DLQ answer interviewers want covers what most candidates forget: what happens *after* the message lands there.

Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.

TL;DR: A DLQ catches messages that fail processing past a max-retry count so a poison message can't wedge the pipeline; the answer that scores covers the lifecycle after landing there: alert on depth, preserve diagnostic context, triage transient versus deterministic, and replay idempotently.

How to approach it

Define the mechanism in two sentences, then show you understand the operational lifecycle: detection, alerting, diagnosis, replay. Interviewers ask this easy question to set up harder ones (job queues, webhook systems), and they're checking whether you think of a DLQ as a destination or as a workflow.

A strong answer

A dead-letter queue is where messages go after they've failed processing too many times, instead of retrying forever or being dropped. The scenario that motivates it is the poison message: a message that deterministically crashes its consumer (malformed JSON, an unexpected schema, a payload that triggers a code bug). Under at-least-once delivery with naive retries, a poison message gets redelivered endlessly, burning compute, spamming logs, and in FIFO or limited-concurrency systems, blocking every message behind it. One bad record wedges the pipeline, which is a customer-visible outage caused by a single row.

The mechanism: configure a max receive/retry count (say 5); on the Nth failure, the broker moves the message to the DLQ rather than redelivering. SQS, RabbitMQ, Kafka (via patterns), Azure Service Bus, and Pub/Sub all support this natively.

rendering diagram…

What separates strong answers is the lifecycle after dead-lettering:

  • Alert on DLQ depth > 0: a silent DLQ is just a slower way to lose data. Page on rate spikes; ticket on singletons.
  • Preserve diagnostic context: attach the exception, stack trace, consumer version, and attempt timestamps when dead-lettering, so you can diagnose without reproducing.
  • Triage the cause. A transient failure that exhausted retries (downstream outage) wants bulk replay back to the main queue once the dependency recovers; a deterministic poison message wants a code fix or schema fix first, then replay, otherwise it boomerangs back.
  • Distinguish retryable from non-retryable errors upstream: a 400-class validation failure should go to the DLQ on the first attempt (it will never succeed); only 500-class or transient errors deserve the full retry budget with exponential backoff and jitter.

The last bullet has arithmetic behind it that makes the design point stick. Give each message a standard retry budget with exponential backoff of 1, 2, 4, 8, 16 seconds: a message that can never succeed still burns 31 seconds and five processing attempts before it reaches the DLQ. Now put that in a FIFO message group with a 30-second visibility timeout and concurrency pinned by ordering: the poison message holds the head of its group for 5 x 30 = 150 seconds, two and a half minutes during which every message queued behind it waits. If a bad upstream export drops a few hundred such records across the day, the pipeline spends accumulated hours stalled behind messages whose failure was knowable on byte one, because a JSON parse error at attempt five is the same parse error as attempt one. That is the entire argument for classifying errors at first failure: the retry budget exists to outlast transient weather, and spending it on deterministic failures converts a data-quality problem into a latency outage.

The FDE angle: pipelines you deploy ingest customer data, which is always dirtier than scoped. A DLQ with good tooling turns "your integration broke overnight" into "37 of 2M records failed on a new date format; here they are; here's the fix," a conversation that builds trust instead of burning it.

What interviewers probe next

  • "How do you replay safely?" Idempotent consumers (replay means duplicates by design), rate-limited re-injection so you don't stampede recovery, and ordering caveats if the domain cares.
  • "What if messages contain PII, how long can they sit in a DLQ?" Retention policy aligned to data agreements; possibly encrypt or redact payloads, store a pointer instead.
  • "DLQ is filling at 100/sec. DLQ or circuit breaker?" That's not poison, that's an outage; stop consuming and fix the dependency rather than dead-lettering the world.

Common mistakes

Defining the DLQ and stopping: no alerting, no replay story, which means in practice you've built a data-loss queue with extra steps. Retrying non-retryable errors five times before dead-lettering. Forgetting that replay introduces duplicates and ordering changes. And missing the FIFO head-of-line blocking point, often the exact reason the interviewer reached for this question.

Key takeaways

  • A DLQ you never alert on is just a slower way to lose data; page on depth > 0.
  • Triage 400-class validation failures to the DLQ on attempt one; only transient 500s earn the retry budget.
  • Poison messages cause head-of-line blocking in FIFO or limited-concurrency systems: one bad row stalls everything.
  • Replay safely means idempotent consumers and rate-limited re-injection, since replay duplicates by design.
That one was free — and so are 10 answers per topic without an account. Signing in doubles that to 20, opens the Plus lessons in the courses, and remembers which topics you keep getting wrong.no card · Google sign-in · nothing to cancel
HOW DID IT GO?
0
READING SIGNED OUT

Signing in doubles your free answers, from 10 to 20 per topic, and the site starts remembering you: mastery per topic, bookmarks, and a next-focus recommendation. Free, no card.

Sign in free
UP NEXT ON YOUR JOURNEY
FEDITOR'S NOTE

The reveal most candidates walk into: a DLQ you never alert on is just a slower way to lose data, and interviewers love asking 'so how would you know it's filling?' Have a crisp answer for replay safety too, because re-injecting a DLQ means duplicates by design, which quietly chains this back to the idempotency question. The sharpest framing is to triage by error class at ingestion: 400-style validation failures belong in the DLQ on attempt one, not after burning five retries that can never succeed.

DISCUSSION · 0

No comments yet — be the first to share your approach.