gw-06 — The Hitchhiker's Guide to Data-Plane Resilience

Companion to CONCEPTS.md, with the runnable toolkit in src/go/resil/. The unifying lesson: the failures that take you down are usually self-inflicted amplification — and every primitive here is a way to refuse to amplify.

bash scripts/verify.sh runs the tests and the retry-storm simulator, which prints the whole thesis of this lab:

offered load per tick (capacity = 120; spike at ticks 10-15):
  tick:        0    4    8   12   16   20   24   28   32   36
  naive:     100  100  100  680 5000 5000 5000 5000 5000 5000
  budgeted:  100  100  100  220  120  100  100  100  100  100
after the spike clears:
  naive offered load    = 5000  (STILL OVER CAPACITY — metastable collapse)
  budgeted offered load = 100  (recovered to baseline)

A transient spike (6 ticks) becomes a permanent outage under naive retries, because the retries become the new load and feed back on themselves. A retry budget caps the feedback and the system self-heals the instant the spike ends. Internalize that picture; it's the single most important resilience insight, and it's a guaranteed interview story.

1. Power of Two Choices (balancer.go)

Round-robin is load-blind: it keeps feeding a slow instance. Global least-loaded needs expensive, stale coordination. P2C is the sweet spot — pick two endpoints at random, send to the lower-score one — and the "power of two random choices" result says this gets you near-optimal max-load with O(1) local info.

score = (inflight+1) × (ewmaNs+1) blends live load and an EWMA of latency. TestP2CRoutesAroundSlowEndpoint makes one of five endpoints 100× slower and shows it's picked < 1% of the time: a high score loses every pairing, so traffic flows around a "gray failure" (up but slow) automatically. Round-robin would send it a flat 20%.

Outlier ejection — and the panic threshold

Observe(endpoint, latency, ok) updates the EWMA and, on consecutive failures, ejects the endpoint for a growing cool-down. TestOutlierEjectionRespectedByPick confirms Pick never returns an ejected endpoint; TestEjectionExpires confirms it comes back after the cool-down. The maintainer detail is TestPanicModeWhenTooManyEjected: if more than half the fleet would be ejected (a fleet-wide brown-out, not one bad instance), the balancer stops ejecting and uses everyone. Without that panic threshold, a correlated failure ejects the whole fleet and you black-hole all traffic — turning a brown-out into a total outage. Envoy has this exact safeguard for the exact same reason.

2. Adaptive concurrency (limiter.go)

A fixed concurrency limit is always wrong: set for the healthy case, it becomes a queue-builder the moment a dependency slows (in-flight piles up to the limit, latency explodes). Adaptive infers the right limit from latency via Little's Law (L = λ × W): track the minimum (no-load) latency; when current latency climbs above it a queue is forming, so shrink the limit; when they're close, grow it. Excess requests are shed at admission — bounded latency under overload.

The tests pin the control loop:

  • TestAdaptiveGrowsWhenHealthy: at current == minRTT the gradient is 1, so the limit grows (toward max) to find the throughput knee.
  • TestAdaptiveShrinksUnderLatency: a 100× latency jump collapses the limit toward its fixed point (L = 0.5L + √L ⇒ L ≈ 4), proving it backs off hard when a queue forms.
  • TestAdaptiveShedsAtLimit: once in-flight hits the limit, Acquire returns false — the request is shed, not queued.

Why shed instead of queue? A request the client already timed out on is doomed work; queueing it wastes capacity that live requests need. This is congestion control for your service, and — like TCP's — it needs no magic-number tuning. Cite Netflix's concurrency-limits.

3. Retry budgets & circuit breakers (budget.go)

Budget is a token bucket: each served request accrues ratio tokens, each retry spends one, an empty bucket means no retry. TestRetryBudgetCapsAmplification shows 20 requests at ratio 0.5 permit exactly 10 retries, then no more. Why a budget and not a per-request retry count? Under correlated failure (every request failing at once), a per-request "retry 3×" multiplies the entire load by 4 — precisely when the dependency can least afford it. A budget bounds retries to a fraction of total traffic, so amplification is capped no matter how many requests fail. That's the difference between the naive and budgeted lines in the simulator.

CircuitBreaker is the CLOSED → OPEN → HALF-OPEN state machine. TestCircuitBreaker walks the full cycle: it opens after sustained failures (fail fast — stop calling a sick dependency and piling up doomed requests), rejects during the cool-down, lets a single probe through when half-open, reopens if the probe fails, and closes if it succeeds. Tune Threshold so it trips on sustained failure, not noise — too-sensitive a breaker flaps and cuts healthy traffic.

4. How they compose (the call path)

A resilient origin call layers all four:

budget.OnRequest()
if !breaker.Allow() { return fallback }            // fail fast on sick dep
rel, ok := limiter.Acquire(); if !ok { return 503 } // shed at overload
ep := p2c.Pick(); p2c.Acquire(ep)                   // load-aware endpoint
resp, err := callWithTimeout(ep)                    // bounded latency (gw-03)
p2c.Release(ep); p2c.Observe(ep, latency, err==nil) // feed LB + ejection
breaker.Record(err == nil)
rel(latency, err != nil)                            // feed the limiter
if retryable(err) && idempotent && budget.TryRetry() { ... }

Each guard refuses to amplify in a different way. Together they keep a brown-out contained instead of cascading.

5. Hands-on

cd src/go
bash ../scripts/verify.sh           # tests + the metastability demo

# Play with the budget ratio and watch the recovery threshold move:
# (edit resilsim's ratio, or parameterize it as an exercise)

6. Exercises

  1. Wire it into gw-03: build a resilient http.RoundTripper that wraps ProxyEndpoint.Transport with budget + breaker + limiter + P2C over a gw-04 subset. Drive wrk through a flaky origin and watch the retries/requests ratio stay capped.
  2. Add hedging: after p95, send a second request to another endpoint and take the first response; budget the hedges. Show it cuts the tail when the set is healthy and hurts under broad slowness — so gate it on the set being mostly healthy.
  3. Make the breaker rate-based: open on error-rate-over-a-window with a minimum request volume (not consecutive failures), so a low-traffic endpoint doesn't trip on two unlucky errors.
  4. Criticality-aware shedding: when the limiter sheds, drop retries before originals and background before interactive. Implement a priority and prove the core experience survives overload.
  5. Reproduce metastability with real services: stand up a flaky origin and a load generator; show the naive-retry storm and the budget fix on actual retries/requests metrics (gw-11).