gw-06 — Analysis

The design-review treatment of the LB + adaptive limiter you build, with the trade-offs to defend.

Required behaviors

Load-aware balancing. The balancer must route around a momentarily-slow endpoint, not walk into it (round-robin fails this).
Bounded amplification. Retries can never exceed a configured fraction of base traffic; verified by a retries/requests metric.
Fail fast on sick dependencies. A circuit breaker opens on sustained failure (rate over a window with a min-volume floor), not on noise, and probes recovery in half-open.
Bounded in-flight under overload. Adaptive concurrency keeps latency bounded by shedding at admission past the knee, with no magic-number tuning.
Graceful degradation. When shedding, the least-important traffic goes first; the core experience is protected.

P2C with EWMA latency. The balancer picks two random endpoints and takes the one with the lower EWMA in-flight/latency score. EWMA (not instantaneous) damps noise; two random choices avoid global state. The step measures max-load vs round-robin under a skewed endpoint.
Outlier ejection from live traffic. Passive: count consecutive failures / compare latency to the set median; eject for a cool-down that grows with repeated ejections; probe back. Reacts faster than active health checks and catches "up but broken."
Gradient-style adaptive limiter. Track RTT_noload (a long-window minimum) and current RTT; limit ← limit × (RTT_noload / RTT_current) bounded by AIMD-style growth/backoff, à la Netflix concurrency-limits. Over-limit requests get a fast 503. No fixed thread pool to mis-size.
Retry budget as a token bucket. Retries draw from a bucket refilled at budget × request_rate; empty bucket = no retry. This caps amplification globally, unlike per-request retry counts which multiply under correlated failure.

P2C needs in-flight counts per endpoint. Cheap locally, but with per-event-loop state (gw-04) each loop has a partial view. Decide whether load counters are per-loop or shared; per-loop is lock-free but slightly less globally accurate — usually fine.
Outlier ejection can cascade. Eject too aggressively during a fleet-wide brown-out and you eject everyone, leaving no capacity. Cap the fraction of a set you'll eject (e.g. ≤50%) — a panic threshold, exactly as Envoy does.
Adaptive concurrency vs bursty traffic. A limiter tuned for steady traffic can shed legitimate bursts; too sloppy and it lets a queue build. The min-RTT window length is the key knob; document it and how you'd tune it from data.
Retry budgets need a shared denominator. "10% of traffic" is fleet-wide; a purely per-instance budget can still amplify if failures correlate. Decide the scope and accept the approximation if you keep it per-instance.
Hedging vs load. Hedging cuts the tail but adds duplicate load (budgeted, e.g. only hedge 5% of requests at p95). Under broad slowness, hedging makes things worse — gate it on the set being mostly healthy.

Per-route/per-cluster policy from the control plane (gw-08): different timeouts, retry budgets, circuit thresholds, and limits per backend.
Request criticality/priority propagated end-to-end so shedding is semantically correct, not just "drop the newest."
Coordinated bulkheads: isolate resource pools per dependency so one sick dependency can't starve others.
Distributed rate limiting (a shared token store) when per-instance limits aren't enough (gw-03's "stateless gateway holds state" point).
Full observability: retries/requests, circuit state transitions, ejection events, limiter value vs in-flight, shed counts by criticality (gw-11).