Incident · advanced · 30 min · Premium
Queue backlog at 3am
Calder's action queue has been filling for forty minutes and nothing is draining it. Contain it, find out why the consumers went away, get the backlog moving without shipping anything twice, and tell the morning shift what happened.
Brief
- Customer
- Calder Equipment (synthetic customer from the Production AI Agents course; every queue item, log and metric in this incident is made up for the exercise)
- Situation
- 03:10. The pager:
action-queue depth 180 and climbing; oldest item 41 min. Approved replacement requests are enqueued but none reach the replacement service. The replacement service itself reports healthy. The queue consumers were restarted by a deploy at 02:28 and there is no on-call for the platform team until 07:00. - Your role
- You are the FDE on call for Calder's agent deployment. You own the queue's config and the console. The items in the queue are approved actions with idempotency keys; draining them is safe, draining them twice is not.
- Goal
- Get the depth under ten without losing or duplicating an action, name the root cause, and write the status update the morning shift reads first.
Objectives
- 1.Get the queue depth under ten without losing or duplicating an action
- 2.Commit to the root cause (you commit to an answer)
- 3.Write the status update the morning shift reads first
Mission workspace
This mission is part of premium, with every challenge and incident in the lab. The brief above is the whole problem; the box, the hints and the debrief are what you unlock.
Get full Premium access · ₹2,000 / $25
Every answer, concept and course, plus Premium PDF guides, companion files, the full practice-test bank and work-sample downloads. Referral Premium excludes guide PDFs and their companion files.
6 months · One payment · No auto-renewal
Study alongside free video lessons.