gw-06 — Resilience: Load Balancing, Retries, Circuit Breaking, Adaptive Concurrency

The JD lists "uplift the resilience posture for traffic traversing the cloud" as a primary responsibility. Resilience is the set of data-plane behaviors that keep the gateway (and the services behind it) serving during the inevitable: a slow origin, a dead instance, a traffic spike, a dependency brown-out. Done well, the gateway absorbs trouble. Done poorly, the gateway amplifies it — a retry storm turns a blip into an outage, a missing concurrency limit turns a slow dependency into a full collapse.

This lab covers the resilience toolkit a Cloud Gateway engineer must own: client-side load balancing (round-robin, power-of-two-choices, least-loaded, zone-aware), outlier ejection, timeouts, retries with budgets and hedging, circuit breakers, adaptive concurrency limits, and load shedding. The unifying theme is the one Netflix and Google learned the hard way: the failure modes that take you down are usually self-inflicted amplification.

You will implement power-of-two-choices load balancing and an adaptive concurrency limiter, and reproduce a retry storm to see amplification (and a budget stop it).

1. What is it?

Resilience patterns split into choosing where to send a request (load balancing + outlier ejection), deciding whether and how to send it again (timeouts, retries, hedging), and protecting yourself and your dependencies from overload (circuit breakers, concurrency limits, load shedding).

request ─▶ [pick endpoint: LB over healthy subset]
        ─▶ [send with a timeout]
        ─▶ on failure: retry? (budget + idempotency) / hedge?
        ─▶ if endpoint is bad: eject it (outlier detection)
        ─▶ if dependency is failing: open circuit (fail fast)
        ─▶ if WE are overloaded: shed/limit at admission (adaptive concurrency)

The deepest of these — and the most interview-worthy — is adaptive concurrency: instead of a fixed thread/connection limit, the gateway measures latency and infers the right in-flight concurrency from Little's Law, shedding load at admission when the system is at its knee. It's a control loop, conceptually like TCP congestion control.

2. Why does it matter?

• Amplification is how outages happen. A 1% origin error rate with a 3× retry policy and no budget becomes a 3% extra load exactly when the origin is struggling — pushing it further over, causing more errors, more retries: a metastable failure that doesn't recover even after the trigger clears. The single most important resilience insight is "don't amplify," and the JD's "resilience posture" is largely about not building amplifiers.

• The gateway is the natural place for cross-cutting resilience. Putting retries/circuit-breaking/limits at the edge means every service gets them without re-implementing, and the gateway can make fleet-wide decisions (shed the least important traffic first).

• Tail latency dominates user experience. p99/p99.9, not p50, is what users feel, and it's where hedging, outlier ejection, and good LB pay off. A single slow instance in a naive round-robin set drags the tail; P2C and outlier ejection route around it.

• It's where data-driven root-cause lives. The JD's emphasis on "identify root causes using data" maps directly here: distinguishing "the dependency is slow" from "we're amplifying with retries" from "we're concurrency-limited" requires the right metrics and the discipline to read them (gw-11).

3. How does it work?

Load balancing: round-robin → P2C → least-loaded

• Round-robin spreads evenly but is blind to load — it keeps sending to a slow/overloaded instance.
• Least-connections / least-loaded picks the instance with the fewest in-flight requests — load-aware, but requires global state (which instance has how many) that's expensive/stale at scale.
• Power of Two Choices (P2C) is the sweet spot: pick two instances at random, send to the less loaded of the two. No global coordination, yet provably avoids the worst hot spots ("the power of two random choices" result — exponential reduction in max load vs one random choice). This is what modern proxies (Envoy, Finagle) default to.

P2C:  a, b = two random endpoints
      pick whichever has fewer in-flight (or lower EWMA latency)

• Zone/locality-aware LB prefers same-zone endpoints (lower latency, no cross-AZ data cost) while keeping a fallback to other zones — the edge cares about this for both latency and cloud-egress cost.

Outlier ejection

Continuously watch per-endpoint error/latency. When one crosses a threshold (e.g. consecutive 5xx, or latency >> the set's median), eject it from rotation for a cool-down, then probe it back. This is passive health checking from real traffic — it reacts faster than periodic active health checks and catches "the instance is up but broken."

Timeouts, retries, budgets, hedging

• Timeout — the foundational primitive (gw-03). Every call is bounded. A retry without a tight timeout is useless (you wait forever, then wait again).
• Retry — re-send on a retryable failure (connection refused, 503, timeout) — but only for idempotent requests, and never retry a request that the origin may have already processed unless it's safe.
• Retry budget — cap retries to a fraction (e.g. 10%) of total requests, fleet-wide, so retries can never become a large multiple of base load. This is the anti-amplification mechanism; prefer it to fixed per-request retry counts.
• Hedging — for latency (not errors): if a request hasn't responded by the p95, send a second copy to another endpoint and take the first to answer. Cuts the tail dramatically; costs extra load, so it's also budgeted.

Circuit breaker

When a dependency is failing, stop calling it for a while — fail fast instead of piling up doomed requests (which exhausts your concurrency and slows everything). The classic state machine:

CLOSED ── error rate exceeds threshold ──▶ OPEN ── cool-down elapses ──▶ HALF-OPEN
   ▲                                                                        │
   └──────────────── probe succeeds ──────────────────────────────────────┘
          (probe fails in HALF-OPEN ──▶ back to OPEN)

OPEN = reject immediately (or serve a fallback). HALF-OPEN = let a trickle through to test recovery. This bounds the damage a sick dependency can do to the gateway itself.

Adaptive concurrency (the deep one)

A fixed concurrency limit is always wrong: too low wastes capacity, too high lets a slow dependency build an unbounded queue (latency explodes). Adaptive concurrency measures and adjusts. The theory is Little's Law:

L = λ × W        (in-flight = arrival_rate × latency)

When the system is healthy, increasing concurrency raises throughput with flat latency. Past the knee, latency rises but throughput doesn't — you're queueing. An AIMD-style controller (like TCP congestion control, or the Gradient algorithm Netflix open-sourced in concurrency-limits) tracks the minimum observed latency (RTT_noload) and the current latency, and shrinks the limit when current >> minimum (a queue is forming), grows it when they're close. Requests over the limit are shed at admission (fast 503) rather than queued — bounded latency under overload.

Load shedding & prioritization

When you must drop, drop the least important traffic first (criticality tiers): shed retries before originals, background before interactive, non-paying before paying. The gateway is the right place to encode this because it sees all traffic.

4. Core terminology

5. Mental models

• Retries without a budget are a payday loan. They feel like help in the moment and bury you exactly when you're weakest. A budget caps the interest: retries can add at most X% load, never a multiple.

• A circuit breaker is a fuse, not a switch you flip. It trips automatically to protect the house when current spikes, and resets cautiously (half-open). Without it, a shorted appliance (sick dependency) burns down the house (exhausts the gateway).

• Adaptive concurrency is cruise control on a hill. A fixed throttle (fixed limit) either crawls on the flat or redlines on the climb. Cruise control measures speed (latency) and adjusts the throttle (limit) to hold the target — backing off the instant the engine strains (latency rises above the no-load minimum).

• P2C is hiring with two résumés instead of one or all. Reading every résumé (global least-loaded) is too slow at scale; picking one blindly (random/round-robin) sometimes hires the worst. Comparing just two random candidates avoids the worst hire almost as well as reading them all — for almost no cost.

• Load shedding is triage, not failure. Under overload you will drop something; choosing to drop the least important traffic on purpose is the system staying in control, versus dropping random (often the most important) traffic by collapsing.

6. Common misconceptions

• "More retries = more reliable." The opposite past a point: retries amplify load during failures and cause metastable collapse. Reliability comes from budgeted retries + timeouts + circuit breaking + shedding, not from a bigger retry count.

• "Round-robin is fine, instances are identical." They aren't, even when identical — GC pauses, noisy neighbors, a slow disk, an unlucky cache miss make instances momentarily slow. Load-aware LB (P2C) and outlier ejection route around transient slowness; round-robin walks right into it.

• "A fixed thread/connection limit protects me." A fixed limit set for the healthy case becomes a queue-builder when the dependency slows (in-flight piles up to the limit, latency explodes). Adaptive limits shrink as latency rises; fixed limits don't.

• "Hedging fixes errors." Hedging is for latency (tail), not errors — sending a second copy of a request that fails for a deterministic reason just doubles the failures. Use retries (budgeted) for errors, hedging for tail latency.

• "The circuit breaker should open on the first error." Too sensitive and it flaps, cutting healthy traffic. Tune thresholds (error rate over a window, minimum request volume) so it trips on sustained failure, not noise.

7. Interview talking points

• "How do you keep a slow dependency from taking down the gateway?" Timeout every call → circuit-break the dependency when it's sustainedly failing (fail fast, serve fallback) → adaptive concurrency limit so in-flight to it is bounded and excess is shed at admission → shed the least-important traffic first. The thread: bound the blast, don't queue the doomed.

• "Walk me through a retry storm and how to prevent it." A blip → everyone retries → retries are the new load → origin stays over its knee → metastable. Prevent with: tight timeouts, retry budgets (cap retries to ~10% of traffic), exponential backoff + jitter, circuit breaking, and not retrying non-idempotent requests. Mention you'd verify with retries / requests ratio in metrics (gw-11).

• "Why power-of-two-choices over round-robin or least-loaded?" RR is load-blind; global least-loaded needs expensive/stale coordination; P2C picks the better of two random endpoints and gets near-least-loaded behavior with O(1) local info — the "power of two random choices" result. Pair it with EWMA latency and outlier ejection.

• "Explain adaptive concurrency limiting." Little's Law: in-flight = rate × latency. Track minimum (no-load) latency; when current latency climbs above it, a queue is forming, so shrink the limit (AIMD / gradient); when they're close, grow it. Shed over-limit requests at admission. It's congestion control for your service, and it needs no magic-number tuning. Cite Netflix's concurrency-limits.

• "When is it safe to retry?" Idempotent requests, retryable failures (connection refused, 503, timeout where you know the origin didn't act), within budget, with backoff+jitter. Never blindly retry a POST that may have committed. gRPC: retry only on specific grpc-status codes (gw-02).

• "You must drop traffic — what goes first?" Criticality tiers: shed retries before originals, background/prefetch before interactive, and protect the highest-value flows. The gateway encodes this because it sees everything. Tie to Netflix's "the show must go on" — degrade gracefully, never hard-fail the core experience.

8. Connections to other labs

• gw-03 (API gateway) — all of this lives in the endpoint phase; the per-request timeout is the foundation everything else builds on.
• gw-04 (connection management) — LB and outlier ejection operate within the subset; pool acquisition is itself a concurrency limit.
• gw-05 (Pushy) — reconnect backoff+jitter and connect-path load shedding are this lab applied to persistent connections.
• gw-08 (Envoy/xDS) — Envoy implements all of these (P2C, outlier detection, circuit breakers, adaptive concurrency) configured via xDS; the control plane tunes them per route/cluster.
• gw-11 (observability) — you can't run any of this blind; the retry ratio, circuit state, ejection events, and concurrency limit are all metrics.
• db-17 (Raft) — quorum/majority intuition shows up in "how big must a subset be to survive f failures" and in not amplifying during partitions.