TL;DR: Stabilize and communicate before root-causing: confirm and segment in the first 5 minutes, ask "what changed?" and roll back any correlated deploy in the next 10, then follow the latency through the stack. By minute 30 you want impact reduced, not root cause found.
How to approach it
Narrate it as a timeline, not a topic list, because the question literally asks for your first 30 minutes. The rubric rewards three behaviors: stabilizing and communicating before root-causing, asking "what changed?" early, and segmenting the problem (which endpoints, which customers, p50 vs p99) instead of staring at one global graph.
A strong answer
Minutes 0 to 5, confirm and frame. Is it real and is it everywhere? Check the measurement first (a broken metrics pipeline mimics an outage), then segment: all endpoints or a few? All customers or one big one? Is p50 also up (systemic: saturation, dependency) or p99 only (a subset: hot key, GC, one slow shard)? 9am Monday is a clue: weekend-deployed changes meeting weekday traffic, cold caches after a quiet weekend, or a customer's batch job hitting your API at business open. Open an incident channel and post a one-liner: impact, scope, next update in 15 minutes. Communicating early is part of the engineering, not a distraction from it.
Minutes 5 to 15, what changed? Most incidents are caused by a change. Check, in order: deploys (yours) over the weekend, config and feature-flag changes, dependency status pages (cloud provider, LLM API, database), and traffic shape (RPS, payload sizes, and one-customer concentration). A single tenant suddenly sending 50x volume or 10x payloads produces exactly this signature. If a deploy correlates, roll back now and diagnose later: reverting a suspect deploy is minutes; proving it guilty is hours.
Steps 2 to 4 take five minutes and decide which half of the stack you spend the other twenty-five in. Step 7 is the one that gets skipped by people who want to understand before they act.
Minutes 15 to 30, follow the latency through the stack. Use traces (or timing logs) to find where the time went: queueing at the load balancer (saturation), application time (CPU, GC pauses, lock contention), database (slow queries, lock waits, missing connection-pool capacity, since pool exhaustion is a classic p99-only killer), or downstream calls (an LLM provider's own degraded latency passes straight through to you). Check resource saturation: CPU, memory, connection counts, queue depths. Mitigate with the cheapest reversible lever: scale out if it's saturation, shed or rate-limit the offending traffic if it's one tenant, fail over or degrade gracefully if it's a dependency. By minute 30 you want impact reduced and a confident statement of which layer owns the problem; root cause can come after lunch.
Connection-pool exhaustion deserves its mechanism spelled out, because it is the canonical p99-only killer and the arithmetic makes the signature obvious. A pool of 50 connections at 100ms per query serves at most 500 requests/sec. Traffic averaging 300 rps sails through, and the dashboards look healthy. Now a 2-second burst arrives at 800 rps: 1,600 requests show up, the pool serves 1,000, and 600 queue behind it; the last one queued waits 600/500 = 1.2 seconds for a connection before its 100ms of actual work (arithmetic verified). Between bursts the queue drains and the median request never touches it, so p50 sits at 100ms all morning while p99 reads over a second. That is the whole signature: the damage is confined to whoever stands in the transient queue, which is definitionally the tail. It also explains why the fix is not "the database is slow" (it is not; every query still takes 100ms) but pool sizing, burst smoothing, or shedding, and why staring at database dashboards during a p99-only incident wastes the first fifteen minutes.
What interviewers probe next
- "p50 is normal, only p99 is up. What's your shortlist?" Connection-pool exhaustion under burst, GC pauses, a hot shard or key, cache misses for a subset, retries amplifying a flaky dependency.
- "It's the LLM provider being slow and you can't fix them. Now what?" Timeouts plus fallback model, queue and degrade non-critical features, stream so perceived latency drops, and communicate the dependency to the customer with data.
- "Nothing changed on your side." Something changed somewhere: traffic, data shape, a TLS cert, a quota tier, a noisy neighbor. "Nothing changed" means you haven't found it yet.
Common mistakes
Diving into one pet theory ("it's the database") without segmenting first. Listing twenty checks with no order or time-boxing, when the question is prioritization under pressure. Forgetting to communicate (silent heroics fail FDE rubrics; a customer is watching this incident). Debugging the root cause while users burn when a rollback was available in minute six. And never asking "what changed?", the single highest-yield question in operations.
Key takeaways
- Stabilize and communicate first; root cause can wait until after lunch.
- "What changed?" is the highest-yield question; if a deploy correlates, roll back before you prove guilt.
- Segment immediately: p50 up means systemic, p99-only means a subset like pool exhaustion or a hot key.
- "Nothing changed" always means you haven't found the change yet, not that there wasn't one.