Production at Scale · Simulator 06
Capacity & the latency hockey-stick
Why do engineers keep servers at 60–70% utilization and refuse to "just run them hotter"? The M/M/1 queueing model answers it: latency rises hyperbolically as utilization approaches 100%, producing the famous hockey-stick curve. Drag QPS up, watch the dot climb the curve — and watch the cost.
- The Hockey-Stick Curve: Shows how average request latency (Y-axis) responds to server CPU utilization (X-axis). The dot (●) shows where your system operates.
- Target Line: The vertical dashed line shows your safe target utilization (e.g. 70%).
- The Goal: Drag the QPS slider up or decrease the **Server count**. Notice that as server utilization crosses 80%, the dot climbs the steep vertical wall of the curve. Request response times shoot through the roof, demonstrating why 100% utilization is a disaster.
Curve: response time vs utilization (M/M/1). Current operating point shown as a dot (●). The vertical dashed line marks ρ = target utilization. Notice how small increases in ρ near 1.0 produce massive latency increases.
By the numbers: capacity and queueing math
Given a target utilization u, you need enough servers so no single server is overloaded. The M/M/1 model (single-queue, exponential service) gives the mean response time:
# Minimum servers to keep utilization ≤ target
servers = ceil(QPS / (perServer × targetUtil / 100))
# Actual utilization with that fleet size
ρ = QPS / (servers × perServer)
# M/M/1 mean response time (service time + queueing delay)
responseTime = serviceMs / (1 − ρ) # diverges as ρ → 1
# Illustrative monthly cost ($0.10/hr per server, 730 hr/month)
monthlyCost = servers × $0.10 × 730The key insight: responseTime = serviceMs / (1 − ρ) has a singularity at ρ = 1. At 50% utilization you pay 2× the service time; at 90% you pay 10×; at 99% you pay 100×. That is why keeping utilization at ≤ 70% is not waste — it is the headroom that absorbs traffic spikes without blowing latency SLOs.
Utilization vs Latency worked trace
| Utilization (ρ) | Service Time (ms) | Queueing Delay (ms) | Total Latency (ms) | Slo Budget Consumed (of 50ms) |
|---|---|---|---|---|
| 10% | 20.0 ms | 2.2 ms | 22.2 ms | 44.4% |
| 50% | 20.0 ms | 20.0 ms | 40.0 ms | 80.0% |
| 80% | 20.0 ms | 80.0 ms | 100.0 ms | 200.0% (BREACHED) |
| 95% | 20.0 ms | 380.0 ms | 400.0 ms | 800.0% (CRITICAL) |
1. Set target utilization to 95% — notice how the response time blows up near that operating point and how little headroom you have for spikes. 2. Drop it back to 65% — you need more servers and it costs more, but the operating point sits in the flat part of the curve, well away from the knee. 3. Raise QPS to 100M with a small per-server capacity — watch servers and cost scale up. 4. Increase per-server capacity (vertical scaling) vs increasing server count (horizontal) — compare the cost change.
This uses the M/M/1 queueing approximation (Poisson arrivals, exponential service, single queue). Real systems have multiple queues, non-exponential service times, connection limits, and GC pauses that change the shape of the curve — but the qualitative behaviour (latency diverges as ρ → 1) holds universally. The $0.10/hr cost is purely illustrative. Treat numbers as directional, not operational.
Under the hood: how CPU saturation & response time curves actually work
Under heavy request volumes, CPU core starvation introduces non-linear scheduling delays inside the operating system kernel. The thread scheduling mechanism explains the latency knee:
- Request Arrival: The network card places incoming packets into a kernel-space TCP backlog ring buffer.
- Process Scheduling: The kernel's completely fair scheduler (CFS) allocates time slices to application worker threads. When CPU utilization crosses 85-90%, threads spend more time waiting in the run queue (runnable status) than actually executing:
# OS Scheduler Task State: Runnable → Running Thread-1: [ RUNNING ][ WAIT ][ RUNNING ] Thread-2: [ WAIT ][ RUNNING ][ WAIT ][ RUN ] - Backlog Overflow: As threads stall in the scheduling queue, they pull requests slower from the TCP socket buffer. The buffer fills, resulting in packet drops and client-side connection timeouts.
How to debug & inspect it
To audit and troubleshoot capacity constraints, inspect OS scheduler statistics and check network socket backlog depths on the server nodes.
Symptom → Cause → Fix:
| Symptom | Likely Cause | Fix |
|---|---|---|
| Request latency rises exponentially while CPU utilization is near 90%+ | CPU core starvation causing scheduler queue delays (M/M/1 knee) | Autoscale the server fleet horizontally or configure load shedding to fail fast |
| Connection timeouts (504 Gateway Timeout) while CPU is low (<15%) | TCP listen backlog is full because thread-pool sizes are set too low | Increase application thread-pool limits or tune kernel parameter net.core.somaxconn |
| Servers run out of memory (OOM crash) during sudden load spikes | Unbounded in-memory request buffering consumes all virtual memory | Enforce queue size limits and return HTTP 503 Service Unavailable when full |
Math companion — this page trains intuition with numbers. Full design contracts live in rel-13-capacity-estimation.html.
When not to stop at the simulator
- Do not ship a production design from the sim alone — you still need multi-instance state, failure modes, and client contracts.
- Do not treat steady-state math as peak; model 3–5× peaks and retry amplification separately.
Why the numbers matter
If you cannot re-derive the formulas on this page from first principles (arrivals, service time, capacity, amplification), you do not own the control. Recompute by hand once, then change one input and predict the chart.
Failure translation
Map each sim control to an ops action: raise limit → client 429; add cache hit ratio → DB QPS falls; add jitter → retry peak flattens; add workers → lag falls until lock or DB saturates.