Retries, Exponential Backoff and Jitter

TL;DR: Retry transient failures, but back off exponentially (1s, 2s, 4s, 8s) so you stop hammering a service that is already in trouble, and add random jitter so a thousand clients do not all retry at the same instant. Cap the attempts and the total time, and only retry operations that are idempotent.

Why naive retries make things worse

A downstream service hiccups and returns 503. The obvious fix is to retry. The naive version retries immediately, in a tight loop. Now multiply that by every client in the fleet. The service that was briefly overloaded is now hit with the original traffic plus a flood of instant retries, which keeps it overloaded, which causes more failures, which triggers more retries. That feedback loop is a retry storm, and it routinely turns a five-second blip into a half-hour outage.

There is a second, subtler problem. If all your clients fail at the same moment (a deploy, a brief network partition) and they all wait the same fixed interval before retrying, they retry in perfect unison. Each retry wave is as synchronized as the failure that caused it. This is the thundering herd: the load never smooths out, it just arrives in synchronized spikes.

Exponential backoff and jitter

Backoff fixes the first problem by increasing the wait after each failure: 1s, 2s, 4s, 8s. This gives a struggling service room to recover instead of being kept on the floor. Jitter fixes the second by randomizing each client's wait so the herd spreads out across the window instead of stacking on the same tick.

Steps 2 and 3 are gates rather than stages: a retry that skips either of them is how a timeout becomes a double charge, or how a permanent error burns a retry budget it was never going to use.

import random, time

def call_with_retry(op, max_attempts=5, base=1.0, cap=30.0, deadline=60.0):
    start = time.monotonic()
    for attempt in range(max_attempts):
        try:
            return op()  # op MUST be idempotent: it may run more than once
        except Transient as e:
            if attempt == max_attempts - 1:
                raise
            backoff = min(cap, base * (2 ** attempt))   # 1, 2, 4, 8, capped
            sleep = random.uniform(0, backoff)          # full jitter
            if time.monotonic() - start + sleep > deadline:
                raise                                   # respect total timeout
            time.sleep(sleep)
    raise RuntimeError("retry loop exited without a result")  # defensive guard

The random.uniform(0, backoff) is "full jitter": each client picks a random point in [0, backoff], which spreads the herd most aggressively. Two guards stop the loop from misbehaving: a max attempt count and a total deadline, so you fail fast instead of retrying for ten minutes against a service that is plainly down. Note also that you only retry on transient errors. A 400 (bad request) or 401 (auth) will fail identically every time, so retrying it just wastes the budget.

Why retried operations must be idempotent

A retry means the operation might run more than once. If the first attempt actually succeeded but the response was lost on the way back, your retry runs it again. For a read that is harmless. For "charge the card" or "send the email" it is a double-charge. Anything you retry must be safe to run twice, which means an idempotency key or an upsert. Backoff and idempotency are a pair: backoff makes retries safe for the server's load, idempotency makes them safe for the data.

Why interviewers probe this

This separates people who have run distributed systems from those who have only called APIs. The reserved follow-up is almost always "what stops the retries from making the outage worse?", and the strong answer names backoff, jitter, a cap, and a circuit breaker that stops retrying entirely once a service is clearly down. A great candidate connects it back to idempotency unprompted, because that link is where the real production bugs live.

Common misconceptions

• "Retry until it works." Unbounded retries against a down service are a self-inflicted outage. Cap attempts and total time.
• "Backoff alone is enough." Without jitter, synchronized clients just stampede on a slower clock. You need both.
• "Retry everything that fails." Only transient errors (timeouts, 503, 429) are worth retrying. A 4xx will fail the same way forever.
• "Jitter is a micro-optimization." At fleet scale it is the difference between load that smooths out and load that arrives in cliffs.

Key takeaways

• Immediate retries cause retry storms; fixed-interval retries cause thundering herds. Exponential backoff plus jitter fixes both.
• Always bound retries with a max attempt count and a total deadline, and only retry transient errors.
• Any retried operation must be idempotent, or a lost response turns one logical action into two real ones.
• Pair backoff with a circuit breaker so you stop retrying entirely once a dependency is clearly down.