Production at Scale · Simulator 02
Cache hit ratio at scale
A cache in front of your origin is the highest-leverage latency lever you have. Move the hit-ratio slider from 0 → 99% and watch effective latency collapse — even a slow cache beats going to origin most of the time. Then push QPS into the millions and see how the origin is shielded. This is the model behind the caching lesson, made live.
- The Latency Comparison Bars (Top): Compare "No Cache" (always hitting the slow origin database) vs "With Cache" (which hits the fast CDN cache for a fraction of requests). The length of the bar shows effective request latency.
- The Traffic Breakdown Bar (Bottom): Shows what portion of the traffic is served instantly from the cache (green segment) vs what portion misses the cache and must go to the database (red segment).
- The Goal: Drag the Cache hit ratio slider from 0% to 99%. Observe how even a minor cache hit ratio (e.g., 50%) dramatically cuts latency, and how 99% hit ratio reduces origin load to near zero.
Top bars: "no cache" vs "with cache" latency, scaled to origin latency. Bottom bar: how much of total QPS hits the origin (red) vs is served from cache (green).
By the numbers: cache math
Every request is served either from cache (probability h) or falls through to origin (probability 1 − h). The effective latency is the weighted average:
eff_latency = h × L_cache + (1 − h) × L_origin
origin_QPS = QPS × (1 − h) # only misses reach origin
origin_load = 1 − h # fraction of total traffic
speedup = L_origin / eff_latency # how many times faster vs no cache
# Worked example: h = 0.9, L_cache = 1 ms, L_origin = 50 ms
eff_latency = 0.9 × 1 + 0.1 × 50 = 0.9 + 5.0 = 5.9 ms
origin_load = 1 − 0.9 = 10% # origin sees only 1 in 10 requests
speedup = 50 / 5.9 ≈ 8.5×The nonlinearity is the key insight: going from 90% → 99% hit rate halves origin load again (10% → 1%) and cuts effective latency another ~5×. The last 1% of misses still dominates tail latency when L_origin is large — keep origin fast even behind a good cache.
Hit Rate Sensitivity Table
| Hit Rate (h) | Effective Latency (ms) | Origin Load | Speedup Factor | Cache Reads → Write Sequence |
|---|---|---|---|---|
| 0% (Cache Cold) | 50.0 ms | 100% | 1.0× | GET cache (miss) → GET origin (50ms) → SET cache |
| 50% | 25.5 ms | 50% | 2.0× | GET cache (50% hit/miss) → GET origin (if miss) |
| 90% | 5.9 ms | 10% | 8.5× | GET cache (90% hit) → Return 1ms |
| 99% | 1.5 ms | 1% | 33.3× | GET cache (99% hit) → Return 1ms |
| 99.9% | 1.05 ms | 0.1% | 47.6× | GET cache (99.9% hit) → Return 1ms |
1. Set QPS to 10M, hit rate 90%, L_origin 200 ms → origin sees 1M req/s, eff latency ~21 ms. 2. Raise hit rate to 99% → origin drops to 100k req/s, eff latency ~3 ms — same origin capacity now handles 10× more total QPS. 3. Now drag L_cache up to 10 ms → latency creeps back up; a slow cache kills the benefit. Keep your cache fast and your hit rate high.
This is a first-principles model of cache latency, not a capture of any company's production traffic. It assumes steady-state hit ratio, no cold-start, no cache stampedes, and no replication lag. Real systems see lower effective hit rates during deploys, TTL expirations, and traffic spikes. Treat the numbers as illustrative.
Under the hood: how caching actually works
Caching operates at the HTTP protocol layer using headers and gateways (like Redis or CDNs) to intercept requests before they hit the origin server. When a client requests a resource:
- Reverse Proxy Check: The CDN or load balancer checks its internal storage (usually key-value hashes indexed by URL and query params).
- HTTP Header Handshake: If cached, the proxy evaluates standard headers such as
Cache-ControlandETag:HTTP/1.1 200 OK Cache-Control: public, max-age=3600 ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4" - Conditional Validation: If the local copy is stale, the client sends a conditional request with
If-None-Match. If unchanged, the server returns304 Not Modified(saving body bandwidth).
How to debug & inspect it
To inspect and troubleshoot cache behaviors, use the browser's Network tab or command-line HTTP clients to observe the latency and cache headers.
Symptom → Cause → Fix:
| Symptom | Likely Cause | Fix |
|---|---|---|
| Cache hit rate is 0% | Cache-Control is set to no-store or contains unique session cookies |
Remove dynamic cookie values from responses or set Cache-Control: public, max-age=300 |
| Private user details are cached publicly | Cache-Control: public applied to authenticated endpoints |
Modify endpoint response to return Cache-Control: private, no-store |
| Massive backend latency spikes on cache TTL expiration | Cache stampede where all concurrent clients attempt to regenerate the cache at once | Use locking/single-flight patterns on the origin or serve stale cache during background refresh |
Math companion — this page trains intuition with numbers. Full design contracts live in rel-07-caching.html.
When not to stop at the simulator
- Do not ship a production design from the sim alone — you still need multi-instance state, failure modes, and client contracts.
- Do not treat steady-state math as peak; model 3–5× peaks and retry amplification separately.
Why the numbers matter
If you cannot re-derive the formulas on this page from first principles (arrivals, service time, capacity, amplification), you do not own the control. Recompute by hand once, then change one input and predict the chart.
Failure translation
Map each sim control to an ops action: raise limit → client 429; add cache hit ratio → DB QPS falls; add jitter → retry peak flattens; add workers → lag falls until lock or DB saturates.
Sources & further reading
- Caching strategies (the lesson this models)
- AWS — Caching best practices · Redis client-side caching
- Facebook — Scaling Memcache (NSDI 2013)