idempotency

The question that separates demo-builders from production engineers. Idempotency keys, the saga pattern, dry-run modes, approval tiers: the distributed-systems toolkit applied to agents, with the trap most answers miss.

The staff-level reliability question for agent platforms. Refunds went out, emails sent, tickets closed, and now you have to triage at fleet scale without double-refunding. The recovery model interviewers actually score.

The pipeline-design question that exposes whether you've ever been paged. One property, idempotency, solves reruns, late data, and backfills at once, and there are exactly three patterns that deliver it.

The aggregate-restatement problem that breaks naive backfills: events arrive days late and out of order, so any partition you already published may be wrong. The staff answer is event-time partitioning plus bounded restatement, not a bigger watermark.

CDC design questions usually stop at the happy-path MERGE. This one stress-tests it: an at-least-once stream that double-delivers, hard deletes, soft deletes for audit, and a GDPR erasure that must physically purge even the bronze change log. The guarantees have to compose.

Every queue, webhook, and retry loop you'll ever deploy at a customer hides this distinction. Interviewers use it to separate people who've shipped from people who've read about shipping.

The bug only appears under load, the code 'hasn't changed,' and the customer just got charged twice. There's one root-cause pattern behind almost every version of this incident, and a fix with a subtle race most candidates miss.

Every request emits a usage event worth real money, clients retry on timeouts, and tenants must be stopped at their quota and never overcharged. The hard parts are idempotent usage events and the succeeded-but-confirmation-lost case where the work happened but the client never heard back.

Every integration lesson assumes a queue and nobody designs one. Ordering per entity rather than globally, a dead letter path somebody actually reads, and what your pipeline does when the consumer is slower than the producer for four hours.

The LLM billing round has three trapdoors: retries double-charging customers, metering latency leaking into inference, and month-end numbers that don't match the provider invoice. The design that survives all three is the answer.