Failure Case Studies · Lesson 03

The 2017 AWS S3 Outage

One mistyped parameter on a routine maintenance command removed far more servers than intended. What should have been a brief intervention turned into a four-hour outage that rippled across the internet — and left the AWS status dashboard too broken to report on itself.

What happened

The morning of February 28, 2017 started with a routine debugging session. An AWS engineer was investigating a slowdown in a billing system and needed to remove a small number of servers from one of its supporting subsystems to resolve the issue. The removal was done with an internal maintenance tool — the kind of tool used regularly by operations teams. But something went wrong with the parameters.

1. Feb 28, 2017, morning: An AWS engineer begins debugging a slowness issue in the billing system. A maintenance command is prepared to remove a small set of servers from a subsystem.
2. Incorrect parameter entered: Due to a typo or input error, the command receives a larger value than intended. Rather than removing a small subset of servers, it targets a much larger group.
3. S3 index servers removed: Among the servers taken offline is a substantial portion of the S3 index subsystem — the internal component responsible for tracking where every stored object lives across the fleet.
4. Index and placement lose capacity: Per AWS PES, the index subsystem manages metadata and location of objects (needed for GET/LIST/PUT/DELETE). The placement subsystem allocates new storage for PUTs and depends on the index. Removing more capacity than intended forced full restarts of both.
5. Recovery begins — and stalls: Restarts and safety checks take far longer than expected. These subsystems had not been fully restarted at current scale for years (PES). Recovery is multi-hour, not a quick flip.
6. ~4 hours of S3 unavailability in US-EAST-1: From 9:37 AM to about 1:54 PM PST (PES milestones: index partial capacity ~12:26, index full ~1:18, PUT/placement ~1:54), S3 in US East (N. Virginia) cannot serve normal API traffic.
7. Blast radius extends across the internet: Many major sites and AWS services in US-EAST-1 degraded (contemporaneous reports named Slack, Trello, Quora, and others). Industry estimates (e.g. Cyence) put large corporate losses on the order of ~$150M — third-party estimates, not AWS figures. EC2/EBS/Lambda paths depending on S3 also failed. The Service Health Dashboard could not update correctly while its assets depended on S3.

Root cause

No single decision caused the outage. Three separate conditions had to be true simultaneously, and when they aligned, the blast radius was catastrophic:

The design lessons

The S3 outage illustrates several interconnected principles that belong in every reliability-focused system design conversation:

Blast radius reduction. The goal is to ensure that a failure in one component affects the smallest possible set of users and services. The S3 outage was so damaging in part because there was essentially no blast radius boundary: a single subsystem's failure could reach every service that had ever stored something in US-EAST-1. Good system design places hard boundaries — network, data, or logical — between failure domains.

Dependency isolation. The Service Health Dashboard depending on S3 is the canonical cautionary example of circular dependency between a service and its monitoring system. When the thing you monitor is also the thing your monitoring system runs on, you lose observability precisely when you need it most. Critical operational infrastructure — status pages, alerting pipelines, runbook repositories — must run on different infrastructure from the services they monitor.

Cell-based architecture. If the S3 index subsystem had been partitioned into smaller, independently deployable and restartable cells, the bad command would have taken down one cell rather than the entire subsystem. Cell architecture trades some resource efficiency for resilience: the blast radius of any single failure is bounded by the cell boundary. It also keeps restart times predictable because each cell is small enough to restart quickly.

Runbook and tooling safety guards. Runbooks are instructions written for humans; humans make errors. Safety guards are constraints encoded into the tools themselves. A minimum-remaining-capacity check on the drain command — "refuse to remove more than 5% of a tier without a secondary confirmation" — would have stopped this incident at the source. The lesson is that operational tools handling destructive actions must be designed like APIs: with validation, type constraints, and explicit confirmation steps for high-impact operations.

Graceful degradation. Services that depend on a storage layer should have a fallback posture: serve stale cached data, disable non-critical features, or return a graceful error rather than crashing entirely. Every layer of graceful degradation between a dependency failure and the end user is a reduction in blast radius.

How to avoid it

Five concrete mitigations directly address the failure modes this incident revealed:

(a) Safety guards on destructive operational commands. Any command that removes capacity from a live system should enforce: a maximum percentage cap (e.g., refuse to remove more than 5% of a tier in one invocation), a dry-run mode that shows exactly which nodes would be affected before execution, and an explicit --confirm flag required to actually execute the destructive path. The guard must live in the tool, not just the procedure.

(b) Regular practice-restarts of critical subsystems. Subsystems should be restarted on a scheduled cadence — even when there is no operational reason to do so — to keep restart procedures exercised, benchmarked, and fast. If a restart takes 20 minutes today and you never practice it, it may take 4 hours when you actually need it. Treat restart time as a metric to be monitored and optimized, just like latency or error rate.

(c) Cell-based architecture. Partition large subsystems into independently deployable, independently restartable cells. Each cell owns a fraction of the keyspace or capacity. A bad command or a bad restart in cell-03 does not affect cells 01, 02, or 04. The blast radius of any single failure becomes bounded by the cell boundary. Cells also make capacity scaling and gradual rollouts far safer.

(d) Out-of-band status communication. The status page, alerting infrastructure, incident chat channels, and runbook hosting must not depend on the service they monitor. Host the status page on a separate CDN, in a different region, with no runtime dependency on the affected service. During an outage is exactly the wrong time to discover that your communications infrastructure is down too.

(e) Design for graceful degradation. Services that use S3 (or any durable storage) as a dependency should have a degraded mode: return the last cached version of data, disable write paths while reads continue, or return a structured error that allows the caller to handle the situation gracefully rather than crashing. The goal is that a storage outage causes a feature to work at reduced capacity, not to fail completely.

# Example: a drain-nodes admin command with safety guards

# Bad — no guardrails, accepts arbitrary count
drain-nodes --tier s3-index --count 1000  # ILLUSTRATIVE only — not a published AWS figure   # removes excessive capacity, no confirmation

# Good — dry-run + percentage cap + confirmation required
drain-nodes --tier s3-index --count 5 --dry-run
# → Would remove: node-042, node-107, node-219, node-304, node-418
# → That is 0.4% of tier (5 / tier_size). Limit: 5%. Safe to proceed.
# → Re-run with --confirm to execute.

drain-nodes --tier s3-index --count 1000  # ILLUSTRATIVE only — not a published AWS figure --dry-run
# → ERROR: count N is 100% of tier (limit: 5%).
# →        Reduce --count or request an override with --override-reason.

Under the hood: the precise failure mechanism

The outage unfolded through a deterministic chain of dependency failures. Each step below is traceable to the AWS post-event summary and the known internal architecture of S3.

1. The maintenance tool had no minimum-remaining-capacity check. The internal drain tool accepted a --count parameter specifying how many servers to remove from a subsystem tier. No lower-bound guard existed: the tool would accept any integer value up to the full size of the tier without warning, without a preview, and without a confirmation prompt. The interface gave no indication of what percentage of the tier a given count represented.
2. A typo sent the wrong count. The engineer intended to remove a small number of servers for an S3 subsystem used by the billing process. AWS's post-event summary states that an incorrect input removed a larger set of servers than intended — AWS did not publish the exact counts. Because the tool executed immediately — no dry-run, no percentage-of-tier confirmation — the command completed before the error was recognized.
3. Far more capacity than intended was removed from critical S3 subsystems. AWS PES: the command removed a larger set of servers than intended, including substantial capacity from the index subsystem (metadata + location of objects; required for GET/LIST/PUT/DELETE) and the placement subsystem (allocates storage for new PUTs; depends on index). Exact server counts were not published — do not invent them. What matters is that remaining capacity was insufficient, forcing full subsystem restarts and multi-hour recovery.
4. Index loss cascades to Metadata and PUT via hard dependency. Per AWS PES, two subsystems matter for this incident: the index (metadata and location information for all objects — necessary to serve GET, LIST, PUT, and DELETE) and the placement subsystem (allocation of new storage during PUTs; needs the index). When index capacity was removed and restarts began, those request types failed. Placement recovery lagged, so PUT health returned later than GET/LIST (PES milestones: index partial ~12:26 PM PST, index full ~1:18 PM, PUT/placement ~1:54 PM PST). Do not invent a third peer "metadata" service separate from index.
5. Read operations also failed. S3 GET requests resolve object location through Index before fetching bytes from the storage fleet. With Index non-functional, even reads returned errors. The result: reads fail, writes fail, metadata operations fail. S3 in US-EAST-1 was effectively non-operational across all object operations.
6. No cell partitioning meant all index capacity was one logical unit. A cell-based architecture would partition the Index subsystem into independently-operated shards — say, 20 cells each holding 1/20th of the object namespace. A drain command affecting one cell would degrade 5% of traffic while the other 19 cells continued serving requests normally. S3's Index was not partitioned this way in 2017. All index capacity was a single logical unit, so removing a large fraction of servers from the tier removed a large fraction of total index capacity — a blast radius proportional to the entire region's S3 traffic.
7. Recovery stalled on in-memory state rebuild at massive scale. Engineers began restarting the Index subsystem to restore service. The restart process requires the subsystem to rebuild its in-memory index — the object → location table — from persistent storage before it can begin serving requests. This rebuild time is roughly linear in the number of entries: at S3's 2017 scale (trillions of objects), the table had grown enormous. The subsystem had been running continuously for years without a restart; nobody had benchmarked restart time at current scale, and there were no runbooks with realistic time estimates. What might have taken minutes at the scale when the procedure was last tested now took hours.
8. Sequencing errors during restart extended the outage further. The correct restart sequence is: Index first, then its dependents (Metadata, then PUT). Attempting to restart Metadata or PUT before Index has fully rebuilt its state causes those subsystems to fail again immediately, requiring another restart cycle. Sequencing errors during the recovery window added additional delay on top of the Index rebuild time.
9. The AWS Service Health Dashboard became a victim of the outage it was meant to report. The dashboard serving status.aws.amazon.com delivered its static assets — JavaScript, CSS, images — from Amazon S3 in us-east-1. When S3 in that region failed, the dashboard's assets became unretrievable. Customers attempting to check the AWS status page encountered a partially or fully degraded dashboard: the monitoring and communication tool had a circular dependency on the failing service.

Monitoring timeline

Failure chain and the guardrail that would have contained each stage

How to debug & inspect it

To debug administrative operations and prevent catastrophic command typos, run CLI tools in dry-run mode and inspect system partition metrics. Out-of-band monitoring tools must be verified to have no dependency on the system they monitor.

Diagnose cluster operations and dependency failures using the table below:

Debug checklist:

1. Confirm all destructive commands default to --dry-run and require --confirm to execute.
2. Audit internal tool inputs for validation limits that prevent draining too much capacity at once.
3. Map and document the boot sequence dependencies to ensure correct ordering during incident recovery.
4. Verify that status monitoring systems do not share any infrastructure or storage with production endpoints.

When you are the client of a hard dependency outage

• Do not spin SDK retries at full RPS for hours when the dependency is 100% down — open a circuit, durable-queue, degrade noncritical work.
• Do not host status/ops tooling only on the system it monitors.
• Do not run unguarded capacity-removal tools without max-removal caps, dry-run, and audit.

# New command: drain-nodes
# Removes the specified nodes from a storage tier.
# Usage: drain-nodes --tier <name> --nodes <comma-separated node IDs>
#
# Example:
drain-nodes --tier s3-index --nodes node-001,node-002,node-003

Sources & further reading

These links point to the primary source material for this incident and the AWS reliability guidance that followed it. All explanatory prose above is original.

• AWS Post-Event Summary — the primary public record of the February 28, 2017 S3 disruption
• AWS Builders' Library — Static Stability Using Availability Zones