gw-09 — The Hitchhiker's Guide to Kubernetes Networking Internals

Companion to CONCEPTS.md, with runnable simulations in src/go/k8snet/. The gateway fleet runs on Kubernetes (Netflix's Titus→K8s journey); these are the two networking behaviors that actually page you.

You can't spin up a real cluster in a unit test, and reading the Kubernetes docs doesn't build the reflex you need on call. So this lab simulates the two highest-impact behaviors deterministically: the drain / EndpointSlice propagation race and conntrack exhaustion. Run bash scripts/verify.sh:

drain / EndpointSlice race (3 kube-proxies; 10 req/proxy/tick):
  WRONG (exit first):           100 requests dropped
  RIGHT (fail readiness first):   0 requests dropped
  RIGHT but grace too short:    400 requests dropped (SIGKILL mid-propagation)
conntrack exhaustion (table size 100, 1000 requests):
  churn (new connection each):  900 packets dropped
  keep-alive (reuse 1 flow):      0 packets dropped

Those two blocks are the most common self-inflicted gateway outages on Kubernetes. Internalize them and you can debug "every deploy drops a few requests" and "the node randomly drops connections under load" — two incidents that otherwise eat a day each.

1. The drain race (drain.go)

A Service is fronted by many kube-proxies, each with its own local view of the endpoints, updated from EndpointSlices. Removal is eventually consistent: when a pod leaves, each proxy stops routing to it only after it observes the update — and they don't all observe it at the same instant.

SimulateDrain models three kube-proxies with propagation delays [2,3,5] ticks:

  • WRONG (readinessFirst=false) — the pod exits immediately, but the proxies keep routing to it for their propagation delays. Result: (2+3+5)×10 = 100 requests dropped (TestDrainRaceDropsWithoutReadiness). This is the "every deploy drops a few requests" bug, and it's why a naive sleep 30 && exit doesn't fix anything — sleeping doesn't make the LB stop sending you traffic.

  • RIGHT (readinessFirst=true) — readiness fails first, so removal starts propagating while the pod keeps serving; it only exits after every proxy has stopped routing to it (within the grace period). Result: 0 dropped (TestDrainNoDropWithReadinessFirst). The ordering is always: fail readiness → drain → exit. This is the Kubernetes form of the gw-01 drain ordering, made precise.

  • RIGHT but grace too short — TestDrainGraceTooShort adds a proxy that takes 50 ticks to update with only a 10-tick grace: the pod is SIGKILLed mid-propagation and the slow proxy's traffic drops. This is the gw-05 problem: a high-density WebSocket node needs a grace period (terminationGracePeriodSeconds) of minutes; the default 30s would SIGKILL 200k connections mid-migration. Same model, different timeout.

In a real pod, "fail readiness first" is a preStop hook that flips the readiness probe (and triggers your app drain — gw-01/gw-05), with terminationGracePeriodSeconds sized to cover propagation + in-flight-request completion.

EndpointSlice/Shard model the other scalability detail: endpoints are sharded (~100 per slice) so a 1000-pod Service doesn't ship one giant object on every change (TestEndpointSliceSharding). Those slices are the membership the control plane turns into EDS (gw-08) and the gw-04 subset ring consumes.

2. conntrack exhaustion (conntrack.go)

On nodes that do netfilter/NAT (most Kubernetes setups), every connection consumes a slot in the kernel's nf_conntrack table. Conntrack models it: Track(flow) reuses an existing flow for free but consumes a slot for a new flow, dropping when full.

SimulateConntrack(100, 1000, ...):

  • churn (a new connection per request) → 1000 new flows into a 100-slot table → 900 drops (TestConntrackExhaustionUnderChurn). The symptom is "the node randomly drops connections under load" with a cryptic nf_conntrack: table full in dmesg — easy to misdiagnose as a network problem.
  • keep-alive (reuse one flow) → 0 drops (TestConntrackOkWithKeepAlive).

The punchline ties the phase together: connection reuse (gw-04) is not just a CPU/latency optimization — it's also what keeps you under the conntrack ceiling. The durable fix for conntrack exhaustion is fewer connections (pool + keep-alive + h2 multiplexing); raising nf_conntrack_max is the stopgap.

3. The rest of the model (read CONCEPTS for depth)

The simulations cover the two incident-grade behaviors; CONCEPTS.md covers the full picture you must be able to narrate:

  • The packet path: Service DNS → ClusterIP → kube-proxy DNAT (iptables / IPVS / eBPF) → pod IP → CNI (veth, overlay vs native routing) → container.
  • kube-proxy is L4 only — connection-level DNAT, not request/latency aware; this is why h2/gRPC behind a bare ClusterIP hits the multiplexing trap (gw-02) and why you still need real LB (gw-06).
  • CNI wiring (veth pairs, IPAM, overlay vs native), NetworkPolicy, externalTrafficPolicy (Cluster vs Local — even spread + SNAT vs client-IP preservation), and NRI/OCI hooks for per-workload runtime customization (the Netflix talk).

docs/analysis.md lists the hands-on cluster exercises (trace a packet with nsenter, watch EndpointSlices during a rollout, reproduce the drain race and conntrack exhaustion on a kind cluster) to do once you have the simulated reflexes.

4. Exercises

  1. Reproduce both on a real kind cluster: run a wrk loop against a Service while deleting a pod with vs without a preStop that fails readiness; then lower nf_conntrack_max and blast short connections. Confirm the simulated numbers match reality qualitatively.
  2. Trace a packet: nsenter into a pod's netns, find its veth peer, dump the iptables/ipvsadm rules for the ClusterIP, follow the DNAT to a pod IP.
  3. Size a grace period: given EndpointSlice propagation P and in-flight request duration D, derive terminationGracePeriodSeconds for an L7 node and for a 200k-connection WebSocket node (gw-05).
  4. Model externalTrafficPolicy: extend the drain sim to compare Cluster (even spread, SNAT, loses client IP) vs Local (client-IP preserved, node-local only) and the load-imbalance trade-off.
  5. Wire to gw-08: turn the sharded EndpointSlices into EDS resources and feed the gw-04 subset ring; debounce re-subsetting under pod flapping so the fix doesn't cause churn.