Production at Scale · Simulator 05

Retry amplification storm

When a service starts failing, clients retry — and those retries generate more load on an already struggling service, causing more failures, which cause more retries. Drag fail rate and max retries together and watch the traffic multiplier explode. This is the mechanism behind the "thundering herd on failure" that takes down services during incidents.

Math companion to Retries & backoff and Circuit breaker — interactive amplification only; full contract and failure paths live on those lessons.

Bars: baseline vs amplified QPS. Dashed line = system capacity. Curve = multiplier across all fail rates for the chosen retry count, with the current point marked (●).

By the numbers: retry storm math

If every request has an independent failure probability f, and a client retries up to R more times on failure, the expected number of attempts per original request is the truncated geometric series:

# Probability of still failing after k attempts = f^k
# Expected attempts = sum_{k=0}^{R} f^k  (geometric series)

attempts     = (1 − f^(R+1)) / (1 − f)      # when f ≠ 1
             = R + 1                          # when f = 1 (all attempts fail)

amplifiedQPS = baseQPS × attempts
extraLoad    = amplifiedQPS − baseQPS

# The vicious cycle: amplified load → higher f → more retries → …
# You need: amplifiedQPS ≤ capacity at ALL points in the cycle

At low fail rates the series converges quickly — 5% failure with 3 retries gives ≈1.05× amplification. But as f → 1, amplification approaches R + 1. The danger zone is the middle: a 50% outage with 3 retries gives 1.875× ((1 − 0.5⁴) / 0.5 = 1.875), potentially pushing an overloaded system past capacity and making the failure rate worse.

Retry Amplification worked trace

Under the hood: how retry storms actually work

A retry storm is a positive feedback loop of demand amplification. When a downstream dependency experiences transient latency or packet loss, the client-side execution path amplifies the incident:

1. Initial Timeout: The client triggers a request. If the server does not respond within the client's timeout window (e.g. 500ms), the client aborts the connection.
2. Re-issue Request: The client immediately issues another request to retry:

   for (int i = 0; i <= MAX_RETRIES; i++) {
     try {
       response = client.send(request);
       break;
     } catch (TimeoutException e) {
       sleep(backoff(i)); // if backoff is 0, retries fire instantly!
     }
   }

3. Retry Amplification: As failures increase, the volume of retries quickly consumes the thread pools, memory, and database connections of the downstream service, turning a minor issue into a major outage.

How to debug & inspect it

To identify retry storms, monitor the ratio of incoming requests to unique customer sessions or trace duplicate request IDs in system logs.

Symptom → Cause → Fix:

Math companion — this page trains intuition with numbers. Full design contracts live in rel-05-retries-backoff.html.

When not to stop at the simulator

• Do not ship a production design from the sim alone — you still need multi-instance state, failure modes, and client contracts.
• Do not treat steady-state math as peak; model 3–5× peaks and retry amplification separately.

Why the numbers matter

If you cannot re-derive the formulas on this page from first principles (arrivals, service time, capacity, amplification), you do not own the control. Recompute by hand once, then change one input and predict the chart.

Failure translation

Map each sim control to an ops action: raise limit → client 429; add cache hit ratio → DB QPS falls; add jitter → retry peak flattens; add workers → lag falls until lock or DB saturates.

Sources & further reading

• Retries, backoff & circuit breakers (the lesson this models)
• AWS — Avoiding overload with timeouts
• Google SRE — Handling overload · Netflix — Hystrix & retry budgets