gw-04 — Connection Management: Curbing Connection Churn

This lab is built directly on the Netflix talk named in the JD — "Curbing Connection Churn in Zuul." It's the most Netflix-specific topic in the phase and a perfect interview centerpiece because it ties together everything from gw-01 (sockets, TLS handshake cost) through gw-03 (the endpoint phase) into one concrete, measured win: Zuul went from opening thousands of origin connections per second to about 60, an ≈8× reduction in TCP opens, by fixing how the gateway manages connections to its backends.

The two ideas that did it: per-event-loop connection pools (so a request and its origin connection live on the same thread, eliminating cross-thread handoff and contention) and subsetting (so a large gateway fleet doesn't fan out a connection to every origin instance — it talks to a balanced subset). The subset is chosen with a low-discrepancy sequence (a binary Van der Corput sequence) so the load lands evenly even as the fleet and the origin set change.

You will build a connection pool, add per-event-loop partitioning, implement deterministic subsetting, and measure the churn drop with a load generator — reproducing the shape of the Netflix result.

1. What is it?

Connection churn is the rate at which you open and close TCP (and, worse, TLS) connections to your backends. Every new connection costs a TCP handshake (1 RTT) plus, for TLS, an expensive asymmetric-crypto handshake (1–2 RTT + CPU). A gateway that opens a fresh origin connection per request pays that tax constantly: latency on the request path and CPU burned on handshakes, plus pressure on the origin's accept queue (gw-01).

Connection management is the set of techniques that drive churn toward zero:

• Keep-alive / connection reuse — don't close the origin connection after a response; reuse it for the next request to that origin.
• Connection pooling — keep a pool of warm, reusable connections per origin; check one out per request, return it after.
• Per-event-loop pools — partition the pool by event-loop thread so a request never has to acquire a connection owned by another thread (no lock contention, no cross-thread handoff, better cache locality).
• Subsetting — each gateway instance connects to a subset of origin instances, not all of them, so total connections = gateways × subset_size instead of gateways × origins, while load stays balanced.
• HTTP/2 origin multiplexing — one h2 connection carries many concurrent requests, so a handful of connections per origin suffices.

2. Why does it matter?

• It's a top-line efficiency metric at Netflix. The talk's headline: ~8× fewer TCP opens, churn from thousands/s to ~60/s. At 1M+ rps across 80+ Zuul clusters, that's an enormous reduction in handshake CPU and tail latency. This is the kind of "tangible impact on the backbone" the JD is hiring for.

• Churn is a hidden CPU sink. New engineers see high gateway CPU and look at filters; veterans look at connections.created.rate and the TLS handshake count. Recognizing churn as a CPU problem (not just a latency one) is a senior signal.

• It's a fleet-scale combinatorics problem. Without subsetting, a fleet of N gateways × M origins = N×M connections; both grow, so the product explodes. A 500-instance gateway fleet to a 1000-instance origin = 500,000 connections, most idle, each a memory and keepalive cost on the origin. Subsetting turns N×M into N×subset. This is exactly the math the interviewer wants you to do out loud.

• Balance is the hard part. Naive subsetting (random, or hash mod) creates hot origins and cold origins, especially as instances come and go. The low-discrepancy (Van der Corput) approach keeps the subset assignment evenly spread and stable under churn of the membership itself — change one gateway or origin and only a small, balanced part of the assignment moves.

3. How does it work?

The churn problem, drawn

NO POOLING (churn): every request handshakes a new origin connection
  req1 ─ TCP+TLS handshake ─▶ origin ─ response ─ CLOSE
  req2 ─ TCP+TLS handshake ─▶ origin ─ response ─ CLOSE      ← pay every time
  req3 ─ TCP+TLS handshake ─▶ origin ─ response ─ CLOSE

POOLING (no churn): handshake once, reuse
  req1 ─ checkout ─▶ [warm conn] ─▶ origin ─ response ─ return
  req2 ─ checkout ─▶ [warm conn] ─▶ origin ─ response ─ return   ← no handshake
  req3 ─ checkout ─▶ [warm conn] ─▶ origin ─ response ─ return

Per-event-loop pools (the Zuul insight)

A gateway runs K event-loop threads (gw-01/gw-03). If there's one shared pool, every checkout/return contends on a lock, and a connection accepted on loop A might be handed to a request running on loop B — a cross-thread handoff that hurts cache locality and ordering. The fix: one pool per event loop. A request running on loop A only ever uses connections owned by loop A. No lock, no handoff, the whole request/response cycle stays on one thread.

        ┌── event loop 0 ──┐   pool0: [conn→originX][conn→originY]
 reqs ─▶│   request runs    │── checkout/return within the loop, lock-free
        └───────────────────┘
        ┌── event loop 1 ──┐   pool1: [conn→originX][conn→originZ]
 reqs ─▶│   request runs    │
        └───────────────────┘

The trade-off (be ready to state it): per-loop pools mean each loop needs its own warm connections, so the minimum connection count rises with loop count. Subsetting is what keeps that in check.

Subsetting — N×M → N×subset

Each gateway picks a fixed-size subset of the origin instances to talk to. The requirements:

1. Balanced load on origins. Every origin should be in roughly the same number of gateways' subsets (so it gets ~equal traffic).
2. Stable under membership churn. When one gateway or one origin is added/removed, only a small, balanced portion of subset assignments should change (minimize connection re-establishment — otherwise re-subsetting causes churn).
3. No coordination. Each gateway computes its subset locally from the membership list (which the control plane provides, gw-08).

Low-discrepancy subsetting (Van der Corput)

The Netflix approach builds a balanced distribution ring using a low-discrepancy sequence. A low-discrepancy sequence fills the [0,1) interval as evenly as possible for any prefix length — unlike random points, which clump. The binary Van der Corput sequence is the canonical one: take the integer i, write it in binary, reverse the bits, and read it back as a fraction.

i   binary   reversed   value
0   .0        .0        0.0
1   .1        .1        0.5
2   .10       .01       0.25
3   .11       .11       0.75
4   .100      .001      0.125
5   .101      .101      0.625
6   .110      .011      0.375
7   .111      .111      0.875

Notice the values never clump: each new point lands in the largest remaining gap. Map gateways and origins onto this ring and each gateway deterministically picks the subset_size origins nearest its position; because the sequence is balanced, every origin is covered ~equally, and adding/removing one member shifts only a small balanced slice.

HTTP/2 origin multiplexing

If the origin speaks h2, one connection multiplexes hundreds of concurrent requests (gw-02), so you need only a few connections per origin per loop. This is the other big churn lever: h1 needs pool_size ≈ peak concurrency; h2 needs a handful. The trade-off is the h2 concentration/HOL issue from gw-02.

Connection lifecycle and health

A pooled connection isn't free to keep forever:

• Idle eviction — close connections idle past a TTL so you don't hold thousands of cold sockets (and so a silently-dead origin gets cleaned up). But evict too aggressively and you re-create churn.
• Max lifetime — recycle connections after a max age so DNS/endpoint changes and rebalancing eventually take effect (the gRPC MAX_CONNECTION_AGE tension from gw-02).
• Health/validation — validate a connection on checkout (or rely on keepalive + TCP_USER_TIMEOUT, gw-01) so you don't hand out a half-dead connection and turn one origin failure into a latency cliff.

4. Core terminology

5. Mental models

• Pooling is reusing a taxi; churn is calling a new taxi for every block. The handshake is the taxi pulling up and you buckling in. Keep the same taxi (keep-alive) and the per-trip overhead vanishes. TLS makes the "buckling in" a full safety briefing — expensive enough that reuse is a CPU win, not just latency.

• Subsetting is potluck seating. 500 guests (gateways) and 1000 dishes (origins): nobody can sample every dish. Assign each guest a balanced subset so every dish gets eaten by about the same number of guests. Random seating leaves some dishes mobbed and others untouched; the Van der Corput seating chart spreads everyone evenly and barely reshuffles when one guest or dish is added.

• Low-discrepancy = "always fill the biggest gap next." Random darts clump and leave holes; the Van der Corput sequence places each dart in the largest empty space. That's exactly the property you want when assigning a changing set of gateways to a changing set of origins without coordination.

• A pool is a cache of expensive objects. All the cache problems apply: sizing (too small → churn, too big → idle waste), eviction (too eager → churn, too lazy → stale/dead connections), and the thundering herd when the cache is cold (a deploy empties every pool at once → a churn spike exactly when you can least afford it).

6. Common misconceptions

• "Just make the pool huge." A huge pool holds thousands of idle connections — memory on both ends, keepalive traffic, and at fleet scale it's the N×M explosion subsetting exists to prevent. Size to peak per-loop concurrency, then subset.

• "Subsetting hurts balance." Naive subsetting does. The whole point of the low-discrepancy approach is balanced coverage and stability under membership change. Done right, subsetting improves balance versus everyone-talks-to-everyone with random LB.

• "HTTP/2 to the origin eliminates the need for pooling." It reduces connection count, but you still pool the (few) h2 connections, still manage their lifecycle, and now you have the concentration/HOL trade-off and the MAX_CONNECTION_AGE-vs-churn tension to manage.

• "Idle eviction is purely good." Evicting idle connections fights churn reduction: evict too soon and you re-handshake on the next request. The right answer is a TTL tuned to traffic shape plus keepalive to detect dead peers — not aggressive eviction.

• "Re-subsetting is cheap." Recomputing subsets on every membership change can cause a churn storm (drop and re-establish many connections at once). Stability-under-change is a first-class requirement, which is why the sequence choice matters.

7. Interview talking points

• "How would you reduce connection churn at a gateway?" This is a near-certain question given the JD. Answer in layers: (1) keep-alive + pooling to stop per-request handshakes; (2) per-event-loop pools to kill lock contention and cross-thread handoff; (3) subsetting to stop the N×M fan-out; (4) low-discrepancy subset selection for balance + stability; (5) h2 origin multiplexing where applicable; (6) careful idle/lifetime TTLs. Quote the result shape: thousands/s → ~60/s, ≈8×.

• "Do the connection math." 500 gateways, 1000 origins, everyone-to-everyone = 500k connections. With a subset of 20: 500×20 = 10k. State both numbers and the balance requirement.

• "What's a low-discrepancy sequence and why use it here?" It fills the interval evenly for every prefix, so subset assignment is balanced and changes minimally when membership changes. The binary Van der Corput sequence is "reverse the bits of i." Contrast with random (clumps → hot/cold origins) and hash-mod (rebalances everything when the modulus/membership changes).

• "Why per-event-loop pools specifically?" A request handled on loop A acquiring a connection from a shared pool means lock contention and possibly a connection owned by loop B (cross-thread handoff, cache misses, ordering hazards). Per-loop pools keep the entire request/response on one thread — the same "never leave the event loop" discipline from gw-03.

• "What's the cost/downside of your scheme?" Per-loop pools raise the minimum connection count (each loop warms its own); subsetting trades a little resilience headroom (fewer origins per gateway) for efficiency, so the subset must be large enough to survive losing a few origins; idle TTLs trade memory for churn. Senior answers name the trade-offs unprompted.

• "A deploy just caused a connection-churn spike. Why?" Cold pools: every restarted gateway re-establishes its connections at once (thundering herd). Mitigations: stagger the deploy, pre-warm pools, jitter reconnects, ramp traffic. Ties to gw-12.

8. Connections to other labs

• gw-01 (L4) is where the handshake cost and TIME_WAIT live; this lab is the optimization layer over those sockets.
• gw-02 (L7) — h2 origin multiplexing is a churn lever, with the concentration/HOL trade-off.
• gw-03 (API gateway) — the pool is the Transport behind the endpoint filter; per-event-loop pools mirror the event-loop model.
• gw-06 (resilience) — subsetting interacts with load balancing (P2C over the subset) and outlier ejection (eject within the subset).
• gw-08 (Envoy/xDS) — the control plane supplies the membership list (EDS) that subsetting is computed from; Envoy has built-in subset LB.
• gw-09 (Kubernetes networking) — EndpointSlices are the membership source in K8s; pod churn drives the stability-under-change requirement.