Production at Scale · Simulator 04
Queue & backpressure
A bounded queue between producers and a worker pool. Drag the producer rate above workers × per-worker throughput and watch the queue fill, consumer lag balloon, and backpressure begin dropping messages. This is the model behind Kafka consumer lag, SQS queue depth, and any async pipeline that can be overwhelmed.
- The Chart (Top): Shows the queue depth over time (how full the queue is). The line goes up when message ingress is faster than workers can process, and down when workers catch up.
- Congestion (Red Fill): When the queue depth reaches 100% (the top of the chart), the chart turns red, indicating that the queue is full and any new incoming messages are being dropped (backpressure).
- The Goal: Drag the producers rate above the combined worker capacity (workers $\times$ per-worker speed). Watch the queue fill up, consumer lag spike, and red drops appear. Then, add more workers or increase their speed to drain the queue (green line).
Line = queue depth over time, normalized to queue max. Red = queue at capacity (messages being dropped). Green = draining.
By the numbers: queue math
The queue grows whenever producers outpace consumers, and shrinks whenever consumers catch up. The key relationships:
# Consumer throughput (total msgs/s the worker pool can handle)
throughput = workers × perWorker
# Net fill rate: positive means queue is growing
net = producer − throughput
# Queue depth evolves each tick (clamped to [0, qmax])
depth(t+dt) = clamp(depth(t) + net × dt, 0, qmax)
# Consumer lag: how far behind in time the consumers are
lag = depth / max(1, throughput) # seconds
# You need this many workers to keep up
min_workers = ceil(producer / perWorker)The fundamental rule: you need workers ≥ producer / perWorker to prevent unbounded queue growth. Below that threshold, the queue fills to capacity and backpressure (dropping or blocking) becomes unavoidable. Consumer lag is the leading indicator — it rises before you start dropping.
Worker Scale Sensitivity Table
| Producers | Per-Worker Rate | Workers Needed | Under-provisioned Lag Growth Rate | Queue Operation Sequence |
|---|---|---|---|---|
| 10,000/s | 1,000/s | 10 | +1,000 msgs/s (with 9 workers) | ENQUEUE → Queue Full → DROP / BLOCK |
| 50,000/s | 2,000/s | 25 | +10,000 msgs/s (with 20 workers) | ENQUEUE → Queue Full → DROP / BLOCK |
| 100,000/s | 5,000/s | 20 | +25,000 msgs/s (with 15 workers) | ENQUEUE → Queue Full → DROP / BLOCK |
1. Set producer to 10 k/s, 10 workers at 2 k/s each → throughput = 20 k/s → queue drains. 2. Raise producer to 30 k/s → net = +10 k/s → watch the queue fill and lag climb. 3. Add workers until throughput exceeds producer again — the queue drains. 4. Set qmax very small (e.g. 1 k) with a fast producer → backpressure starts almost immediately.
This is a first-principles model of a bounded queue, not a capture of any real message broker's behaviour. Real systems (Kafka, SQS, RabbitMQ) have rebalancing, partition assignment, fetch batching, and flow-control mechanisms that affect observed lag. The model shows the structural dynamics — treat numbers as illustrative.
Under the hood: how queues & backpressure work
A message queue acts as a structural shock absorber. Internally, brokers write incoming payloads to disk-backed ring buffers (such as commit logs in Kafka) or memory queues:
- Producer Enqueue: A client app sends a batch write request. If the broker's buffer is within limits, it appends the message and returns HTTP 202.
- Consumer Pull: Worker instances poll the queue in a loop:
while (true) { const batch = queue.poll(size: 100, timeout: 50); process(batch); queue.commit(); } - Backpressure Enforcement: When consumer processing lag exceeds limits or when the queue hits capacity, the broker stops accepting new writes, returning status codes like
RESOURCE_EXHAUSTEDor blocking TCP socket reads.
How to debug & inspect it
Use command-line CLI tools provided by message brokers to check active consumer positions and calculate total consumer lag.
Symptom → Cause → Fix:
| Symptom | Likely Cause | Fix |
|---|---|---|
| Lag climbs steadily during traffic spikes | Consuming workers are under-provisioned or processing is too slow | Autoscale the worker fleet horizontally or optimize consumer thread performance |
| Unbalanced lag (one partition has massive lag, others have zero) | Skewed partition key hashing sending most requests to one worker | Modify routing key generation to spread traffic evenly across partitions |
| Workers repeatedly crash and rebalance, losing throughput | Processing a batch takes longer than the broker's maximum poll interval | Decrease the fetch batch size or increase the maximum processing timeout setting |
Math companion — this page trains intuition with numbers. Full design contracts live in rel-10-event-driven-pubsub.html.
When not to stop at the simulator
- Do not ship a production design from the sim alone — you still need multi-instance state, failure modes, and client contracts.
- Do not treat steady-state math as peak; model 3–5× peaks and retry amplification separately.
Why the numbers matter
If you cannot re-derive the formulas on this page from first principles (arrivals, service time, capacity, amplification), you do not own the control. Recompute by hand once, then change one input and predict the chart.
Failure translation
Map each sim control to an ops action: raise limit → client 429; add cache hit ratio → DB QPS falls; add jitter → retry peak flattens; add workers → lag falls until lock or DB saturates.