The System That Tried to Serve Everyone
It’s 8:00 PM on Black Friday. The campaign email goes out and traffic jumps 6x in under a minute. Your service does what most services are built to do: it accepts every single request.
Queues fill up. Latency climbs past the client timeout. Clients retry. Within minutes, nobody can buy anything — not the first user, not the millionth. The system failed because it was too polite to say no.
Now imagine a different version. When the spike hits, the service recognizes it’s at its limit and starts turning away a slice of traffic with a fast, friendly “we’re busy, try again in a moment.” Most users complete their purchases. Nobody writes a postmortem titled “Total Outage.”
That difference has a name: Load Shedding. 💥
Why “Accept Everything” Is a Trap
Every system has a safe throughput limit. Below it, more traffic means more useful work. Above it, something counterintuitive happens: more traffic means less useful work.
When requests pass that limit, latency rises until it crosses the client timeout. Now your servers spend CPU on requests whose clients already gave up. Effective availability can drop to around 50%. Then clients retry at the same time — the retry storm — the load doubles, and effective availability goes to 0%. A partial overload becomes a full outage.
You might think “Auto Scaling will handle it.” It helps, but it takes minutes to warm up new capacity, your database and third-party dependencies don’t scale with your web tier, and cloud capacity is finite. Scaling is the long-term answer. Load Shedding is what keeps you alive in the minutes before scaling arrives — and protects you when scaling can’t help at all.
What Load Shedding Actually Is
Load Shedding is a defensive pattern where you deliberately reject part of the traffic so the rest can be served well.
Instead of accepting every connection until everything dies, you refuse a slice of requests early — with a waiting page, a queue screen, or a fast HTTP 429 Too Many Requests / 503 Service Unavailable — so the remaining capacity stays healthy.
The core insight: a fast “no” is far cheaper than a slow “yes” that times out. A rejected request costs microseconds. A request that sits in a queue for 30 seconds and then times out costs CPU, memory, a connection, and usually a retry. Shedding protects goodput — the requests that actually succeed — instead of raw throughput.
Deciding What to Shed
Not all traffic is equal, and good shedding reflects that:
That last one is often forgotten. If the client gave up after 10 seconds, processing that request at second 15 is pure waste.
Netflix is a great reference here. Its Zuul gateway uses priority throttling, classifying traffic by device (mobile, TV, browser) and by criticality:
- Non-critical — logs and metrics collection. Shed first.
- Degraded experience — secondary UI features. Shed next.
- Critical — the playback core. Protected until the very end.
Users might lose a row of recommendations, but the video keeps playing. The lesson: classify your traffic before the incident, not during it.
Deciding When to Shed
The trigger matters as much as the policy. Shed too late and you collapse anyway; shed too early and you reject customers for nothing.
- Concurrency (in-flight requests) — the most reliable signal. It reacts instantly and reflects real saturation.
- Queue time — how long requests wait before work starts. Rising queue time is the earliest warning of overload.
- Latency against your SLO — if p99 is heading past your target, you’re about to burn error budget.
- CPU alone — a weak signal. Spiky, lagging, and blind to I/O-bound bottlenecks.
Here’s a simple priority-aware shedder driven by in-flight requests:
Non-critical work starts shedding at 50% of the limit, while critical traffic keeps the full capacity. Simple, cheap, and easy to reason about at 3 AM.
Deciding Where to Shed
Shed as early — and as cheaply — as possible, in layers:
- Edge — API Gateways, Load Balancers, Nginx, Envoy. Rejecting here costs almost nothing.
- Application — code that understands priorities, tenants, and critical paths.
- Queues and workers — drop expired messages before processing them.
At the edge, Nginx gives you a starting point in a few lines:
On AWS, you can automate it end to end: CloudWatch detects a latency spike, EventBridge triggers a Lambda, and the Lambda adjusts weighted target groups on the Application Load Balancer, sending around 20% of traffic to a static page on S3. No human in the loop — pure incident response automation. 🔧
Shedding Well (and Not Hurting Yourself)
Load Shedding is a sharp tool. Badly calibrated, it rejects valid requests for no reason. A few rules I always follow:
- Track shed rate as its own SLI. If you’re shedding every day, you have a capacity problem, not a spike problem.
- Send
Retry-Afterand use client backoff with jitter. Otherwise your 429s become fuel for the next retry storm. - Test thresholds before you need them. Load tests, stress tests, game days. An untested shedding rule is just a hope.
- Pair it with Circuit Breakers. Shedding protects you from your callers; circuit breakers protect you from your dependencies.
Saying No Is a Reliability Feature
Back to Black Friday. The difference between the two versions of that night wasn’t more servers — it was a decision made in advance about what matters most and what can wait.
Load Shedding isn’t a system failure — it’s a conscious defense strategy. It’s always better to turn away a fraction of your users than to lose all of them. Classify your traffic, pick your signals, shed early and cheaply, and let your system say no so it can keep saying yes. 🚀
Want to go deeper on resilience patterns like Load Shedding, Circuit Breakers, and SLO-driven alerting? That’s exactly what the trainings at mugnos-it.com are built for.
Cheers,
Douglas Mugnos
MUGNOS-IT 🚀
