gw-04 — The Hitchhiker's Guide to Curbing Connection Churn

Companion to CONCEPTS.md, with the runnable code in src/go/connpool/. This lab reproduces the headline result of Netflix's "Curbing Connection Churn in Zuul" — in code you can run, measure, and break.

Connection churn is the rate at which you open/close connections to your backends. Every new connection costs a TCP handshake plus (for TLS) an expensive asymmetric-crypto handshake. At 1M+ rps a gateway that opens a fresh origin connection per request burns enormous CPU on handshakes and floods origins' accept queues (gw-01). The fix is three techniques, each a file in this package: pooling, per-event-loop pools, and subsetting.

Run bash scripts/verify.sh and you'll see the real numbers:

connection math:
  everyone-to-everyone : 500000 connections
  subset (size 20)      : 10000 connections  (50.0x fewer)
coverage per origin (ideal 10.00):
  min=5  p50=10  max=15
stability when one origin leaves (lower is better):
  Van der Corput ring : 11 / 500 gateways changed
  hash-mod subsetting : 259 / 500 gateways changed

Those three blocks are the lab. Let's earn each one.

1. Pooling: stop re-handshaking (pool.go)

Pool.Get(origin) returns a warm connection from the idle set if one exists (incrementing Reused), else dials a new one (incrementing Created — the churn counter). Put returns it for reuse, or closes it if the idle set is full.

The tests quantify the win:

  • TestNoPoolIsAllChurn: with maxIdle=0, 100 get/put cycles produce Created == 100 — pure churn, a handshake every request.
  • TestPoolingReducesChurn: with maxIdle=8, the same 100 cycles produce Created == 1, Reused == 99 — one warm connection, reused.

That's the entire pooling thesis in two tests: churn collapses from "every request" to "≈zero."

The eviction trap (over-tuning re-creates churn)

TestIdleEvictionCausesChurn uses an injectable clock (p.nowFn) to prove the failure mode: with a too-short idleTTL, a connection put back and retrieved after the TTL is evicted and re-dialed — Created climbs again. This is the bug juniors introduce when they "clean up idle connections aggressively": they reintroduce the very churn pooling removed. The right idleTTL is tuned to the traffic's inter-request gap, backed by keepalive (gw-01) to detect genuinely dead peers — not set to "a few hundred ms to be safe."

2. Per-event-loop pools (looppools.go)

A single shared pool means every checkout/return contends on one lock, and a connection accepted on loop A might be handed to a request running on loop B — a cross-thread handoff that hurts cache locality and ordering. LoopPools gives each event loop its own pool; a request on loop i only ever touches pool[i]. Lock-free, no handoff — the specific change the Netflix talk highlights.

TestPerLoopIndependence shows the trade-off honestly: loop 0 and loop 1 each dial their own connection to the same origin (created == 2), then loop 0 reuses its own (reused == 1). So per-loop pools raise the minimum connection count ~K×. That's not a regression — it's why subsetting exists, and the two are designed to be used together. The contention you remove (at 1M+ rps, one lock is a real bottleneck) is worth far more than a modest rise in idle connections, and subsetting claws that rise back.

3. Subsetting (ring.go) — the N×M → N×subset fix

Without subsetting, N gateways × M origins = N×M connections. TestSubsetConnectionMath does the arithmetic: 500 gateways × 1000 origins = 500,000 connections; with a subset of 20, 10,000 — a 50× reduction. Both numbers grow with the fleet, so the product is the thing that explodes; subsetting turns multiplication into addition.

But naive subsetting fails two ways, and the ring fixes both:

Balance — Van der Corput, not random

VanDerCorput(i) bit-reverses i and reads it as a fraction in [0,1). TestVanDerCorputKnownValues pins the sequence: 0, .5, .25, .75, .125, .625, .375, .875. Notice it never clumps — each new point lands in the largest remaining gap. We place gateways on the ring with this sequence (gateways have stable indices 0..G-1), so their subset "windows" evenly cover the origins. TestSubsetBalance asserts every origin is covered (no cold origins) and the max coverage stays within a sane band of the ideal — the demo shows min=5, p50=10, max=15 against an ideal of 10. Random placement would clump (hot and cold origins); Van der Corput spreads.

Stability — hash for origin identity, not slice index

The subtle, maintainer-level point: origins are positioned by a stable hash of their identity (originPos = FNV-1a / 2³²), not by their index in a slice. Why it matters: if you positioned origins by slice index and one origin left, every later origin's index — and thus ring position — would shift, reshuffling the whole fleet's subsets and causing a re-subsetting churn storm (the fix causing the problem). With stable hashed positions, removing one origin only perturbs the gateways whose window touched it.

TestSubsetStabilityVsHashMod proves it, and the demo quantifies it: when one origin leaves, the Van der Corput ring changes 11/500 gateways' subsets; hash-mod changes 259/500. Hash-mod uses (seed+j) mod len, so changing len moves nearly everything — exactly the instability that makes re-subsetting dangerous. Stability under membership change is a first-class requirement, because EDS/Endpoint- Slice churn (gw-08/gw-09) means membership changes constantly.

This is the detail that separates "I read the blog" from "I could have written it": the low-discrepancy sequence gives balance, and stable per-origin positioning gives stability under churn. You need both, and they come from different mechanisms.

4. How it composes with the rest of the phase

  • The pool is the http.RoundTripper/Transport behind gw-03's endpoint filter — swap it in and the gateway stops churning origin connections.
  • The subset is computed from the EDS membership an xDS control plane pushes (gw-08), which itself derives from Kubernetes EndpointSlices (gw-09). Rapid pod churn is why stability matters; you'd debounce re-subsetting so the fix doesn't thrash.
  • Load balancing (gw-06: P2C, outlier ejection) operates within the subset, not over the whole fleet.
  • HTTP/2 origin multiplexing (gw-02) is the other churn lever: a handful of h2 connections per origin carry hundreds of concurrent requests — at the cost of the concentration/HOL trade-off.

5. Hands-on

cd src/go
bash ../scripts/verify.sh                 # tests + the demo above

# Play with the parameters and watch the trade-offs:
go run ./cmd/churnsim -origins 2000 -gateways 1000 -subset 10   # tiny subset: fewer conns, less resilience headroom
go run ./cmd/churnsim -origins 100  -gateways 50   -subset 50   # subset≈origins: approaches full mesh

Questions the demo answers experimentally:

  • How does shrinking the subset trade connection count against the number of origins each gateway can lose before it's under-provisioned?
  • How does coverage balance change with subset size and fleet ratio?
  • How does ring stability compare to hash-mod as you change origins?

6. Exercises

  1. Wire the pool into gw-03: implement an http.RoundTripper backed by Pool (one per origin), drop it into ProxyEndpoint.Transport, and watch Created flatten under wrk keep-alive load vs Connection: close load.
  2. Subset-aware pooling: make each per-loop pool only dial origins in this gateway's subset. Re-measure IdleTotal() — confirm the per-loop K× rise (§2) is bounded by the subset size.
  3. Debounce re-subsetting: feed a rapidly-flapping membership list and add a debounce so the ring rebuilds at most every N ms. Show that without it, re-subsetting itself causes churn (the fix becoming the bug).
  4. Size the subset for resilience: derive how large the subset must be so that losing f origins still leaves enough capacity — a quorum-style argument (connect it to db-17's majority reasoning).
  5. Compare to consistent hashing: implement a consistent-hash ring and compare its stability and balance to Van der Corput placement. When would you pick each?