gw-09 — Analysis

This lab is operational rather than build-from-scratch: you can't re-implement the kernel network stack in a step, so the work is reproducing and reading the real machinery on a local cluster (kind / minikube / k3d) and connecting it to the drain, membership, and resilience concerns of the rest of the phase.

What to actually do (in lieu of steps/ code)

  1. Trace a packet. On a kind cluster, deploy a Service + 3 pods. nsenter into a pod's netns, find its veth peer on the node, dump the iptables/ipvs rules for the ClusterIP, and follow the DNAT to a pod IP. Write the path down hop by hop.
  2. Watch EndpointSlices during a rollout. kubectl get endpointslices -w while you kubectl rollout restart a Deployment. Observe pods leaving/joining and the propagation delay. This is the gw-04 membership churn and the gw-08 EDS source.
  3. Prove the drain race. Run a wrk loop against a Service while you delete a pod with (a) no preStop/short grace and (b) a preStop that fails readiness + sleeps. Show (a) drops requests and (b) does not. This is gw-01/gw-05 drain on real Kubernetes.
  4. Exhaust conntrack (carefully). On a test node, lower nf_conntrack_max, blast many short connections, watch nf_conntrack_count hit the cap and connections drop. Then enable keep-alive (gw-04) and watch the flow count fall.
  5. See the gRPC/h2 ClusterIP trap. Send many gRPC calls over one channel to a ClusterIP-fronted Service; observe the skew (gw-02); switch to a headless Service + client-side LB and watch it even out.

Required understanding (the design-review bar)

  • Drain ordering on K8s: preStop (fail readiness, start app drain) → EndpointSlice removal (eventually consistent) → SIGTERM → grace → SIGKILL. Keep serving until grace expires; never exit on first SIGTERM. Grace = seconds for L7, minutes for high-density WebSocket nodes (gw-05).
  • Membership source: EndpointSlices → (control plane) → EDS (gw-08) → subset ring (gw-04). Pod churn drives stability-under-change.
  • kube-proxy is L4 only: connection-level DNAT, not request/latency aware; explains why real LB (gw-06) and L7 (gw-02) still live in the gateway, and why h2/gRPC need more than a ClusterIP.
  • Hidden ceilings: conntrack table, ephemeral ports, iptables rule bloat / sync latency, MTU on overlays. These are the "mysterious latency/drops" root causes the JD's data-driven debugging targets.

Tradeoffs worth flagging

  • Overlay vs native routing. Overlays (VXLAN) work anywhere but add encapsulation overhead and MTU pitfalls; native routing / real VPC IPs (AWS VPC CNI, Calico BGP) are faster and simpler to debug but cloud/topology-specific.
  • iptables vs IPVS vs eBPF. Simplicity/ubiquity vs scale vs performance+observability. The right answer depends on Service count and the value of L7/eBPF visibility.
  • externalTrafficPolicy Cluster vs Local. Even spread + SNAT (loses client IP) vs client-IP preservation + node-local routing (possible imbalance). For an edge LB that needs the client IP/identity (gw-01, gw-07), Local (or PROXY protocol upstream) is often required.
  • NRI/OCI runtime customization vs portability. Customizing the runtime per workload (Netflix's NRI talk) unlocks specialized networking/sidecar behavior but adds a node-level component to operate; the win is doing it without forking Kubernetes.

What production adds beyond a local cluster

  • A production CNI at scale (IPAM exhaustion, BGP/route-table limits, NetworkPolicy enforcement, multi-cluster).
  • eBPF observability (Hubble) to actually see the dataplane.
  • Node-level runtime customization (NRI/OCI hooks) for the gateway/origin workloads.
  • PodDisruptionBudgets + long grace periods + surge control so fleet rollouts don't drain too many nodes at once (gw-12).
  • Tuned kernel limits (conntrack, ephemeral ports, somaxconn) baked into the node image for connection-heavy gateway workloads.