Challenge · intermediate · 25 min · Free account
429 at nine o'clock
Every Monday at nine Meridian's claims assistant slows to a crawl and a third of runs fail. The provider is rate limiting, and the router's retries are making it worse. Read the storm, name the mechanism, and tame it with one config change.
Brief
- Customer
- Meridian Health (synthetic customer; every metric, trace and log on this box is made up for the exercise)
- Situation
- Monday 09:04. The weekend's claims backlog lands at once and the claims assistant's error rate climbs past 30%. Meridian's platform lead has already asked the model provider for a higher limit and been told it will take a week. The router's retry settings have never been touched since the pilot.
- Your role
- You are the FDE on the Meridian deployment with the operator console and the router's config file. You cannot change the provider's limit. You can change how the router behaves when it hits it.
- Goal
- Name the error class the provider is returning and the mechanism that turns a limit into a storm, change the router so retries spread out instead of synchronising, and show the router's 429 rate and p95 falling on the live system.
Objectives
- 1.Name the HTTP status the provider is answering with (you commit to an answer)
- 2.Name the mechanism that turns a limit into a storm (you commit to an answer)
- 3.Make retries spread out inside the backoff window, with a bounded count
- 4.Show the router's 429 rate under 3% and p95 under 8 seconds on the live system
Mission workspace
This mission opens with a free account. The brief above is the whole problem; the box, the hints and the debrief are behind sign-in.
a free account unlocks the core curriculum tier · no card