Production at Scale · Simulator 01

Rate limiter at scale

A multi-tenant token-bucket limiter, one bucket per API key. Drag the total request rate up toward 100M req/s, change how many tenants share the traffic, and push a single "hot" key — and watch the busiest key's bucket drain and the 429s appear. This is the algorithm from the rate-limiting lesson, made live.

Math companion to Rate limiting algorithms — interactive fleet/hot-key math only; full algorithms, Redis Lua, and HTTP 429 contract live on that lesson.

InteractiveMath companionModels rel-03

💡 How to Read the Simulator

The Chart (Top): Shows a real-time graph of the busiest API key's token bucket. The green area represents available tokens (it fills up to capacity B and empties as requests come in).
Throttling / Drops: When the green area hits the bottom (empty bucket), you will see vertical red lines in the chart. These indicate 429 Too Many Requests errors where requests were blocked.
The Controls (Middle): Adjust the total traffic rate, number of API keys, limits, and how much traffic one "hot" key takes.
The Readouts (Bottom): Display actual counts of allowed vs. throttled requests per second, the overall percentage of 429 errors, and the current token count.

Green area = tokens in the busiest key's bucket (full at top, empty at bottom). Red lines = requests dropped (429) because that key's bucket was empty.

By the numbers: simulator math

Each API key gets its own bucket: capacity B (the burst it can spend at once), refilling at r tokens/second (its sustained allowed rate). The fleet-level numbers come straight from that, per key:

# Traffic split across K keys; one "hot" key takes hot% of it
hot_qps      = QPS × hot%
per_key_qps  = (QPS − hot_qps) / (K − 1)        # the other keys share the rest

# A key can sustain at most r req/s; the excess is throttled
allowed      = min(hot_qps, r) + min(per_key_qps, r) × (K − 1)
throttled    = QPS − allowed
rate_429     = throttled / QPS

The key insight you can see: raising total QPS doesn't throttle anyone until a single key's share crosses its own limit r. Spreading the same load over more keys (raise K) keeps everyone under the limit; concentrating it on one hot key (raise Hot key share) throttles that key while the others sail through — per-key isolation in action.

✅ Try this

1. Set QPS to 100M with K = 1,000,000 keys → per-key rate is ~100/s, under a 500/s limit → ~0% 429s even at 100M. 2. Now drag Hot key share to 50% → that one key tries ~50M/s against a 500/s limit → its bucket pins empty (solid red) and the global 429 rate jumps, even though 999,999 keys are fine. 3. Raise r (give keys a bigger allowance) or B (bigger burst) and watch the red thin out.

⚠️ Modeled, not measured

This is a first-principles model of the token-bucket algorithm, not a capture of any company's production traffic. It shows the behaviour (how throttling responds to load, key count, and hot keys) — the real limits, key counts, and traffic shapes at Stripe/AWS/etc. are not public. Treat the numbers as illustrative.

Under the hood: how it actually works

Behind the simulator slider is an active token-bucket evaluation loop. In a production environment, this state is maintained inside a distributed store like Redis to ensure consistency across multiple application servers. Every API call triggers a read-compute-write cycle:

Identify tenant: The API Gateway parses the request headers (e.g., Authorization: Bearer sk_live_xyz) to resolve the tenant key.
Fetch bucket state: The gateway issues a command to fetch the current token count and the timestamp of the last refill.
Compute refill: It calculates elapsed time since the last request and determines how many tokens have accumulated: tokens = min(B, tokens + elapsed_seconds * r).
Evaluate and save: If tokens >= 1, it decrements the token count by 1, saves the new state, and returns HTTP 200. Otherwise, it denies the request with HTTP 429.

To avoid race conditions under high concurrency, these steps are executed as a single, atomic Redis Lua script, ensuring no two app servers read the same intermediate state.

How to debug & inspect it

Debugging a distributed rate limiter requires checking key existence, verifying TTLs, and reproducing the 429 status code with a high-throughput script.

Inspect state in Redis:

$ redis-cli HGETALL "rl:acct_42" 1) "tokens" 2) "84.32" 3) "last_refill" 4) "1718900023.150" $ redis-cli TTL "rl:acct_42" (integer) 45

Symptom → Cause → Fix:

Symptom	Likely Cause	Fix
One key experiences high 429s while others are idle	The hot key exceeded its configured refill rate `r`	Configure client-side exponential backoff with jitter or upgrade the tenant's tier
All tenants hit 429 limits simultaneously	The limiter uses a global key (e.g., `rl:global`) instead of per-tenant keys	Modify the gateway key generation schema to isolate keys: `rl:{tenant_id}`
Limiter stops blocking traffic completely	Redis is unreachable and the code fails open	Ensure Redis is highly available (clustering/sentinel) and monitor connection health

Math companion — this page trains intuition with numbers. Full design contracts live in rel-03-rate-limiting.html.

When not to stop at the simulator

Do not ship a production design from the sim alone — you still need multi-instance state, failure modes, and client contracts.
Do not treat steady-state math as peak; model 3–5× peaks and retry amplification separately.

Why the numbers matter

If you cannot re-derive the formulas on this page from first principles (arrivals, service time, capacity, amplification), you do not own the control. Recompute by hand once, then change one input and predict the chart.

Failure translation

Map each sim control to an ops action: raise limit → client 429; add cache hit ratio → DB QPS falls; add jitter → retry peak flattens; add workers → lag falls until lock or DB saturates.

Sources & further reading

Rate limiting algorithms (the lesson this models) · Inside Stripe's API
Stripe — Scaling your API with rate limiters · AWS API Gateway throttling