gw-09 — Kubernetes Networking Internals: CNI, kube-proxy, Services

The JD requires a "solid understanding of Kubernetes internals (Networking, CNI, CRDs, and Operator patterns)." This lab covers the networking half; gw-10 covers CRDs/operators. The gateway fleet runs on Kubernetes (Netflix's "Managing Netflix's Compute with Kubernetes" talk is the Titus→K8s story), so a Cloud Gateway engineer must understand how a packet actually reaches a pod, how Services and EndpointSlices work (they're the membership source for gw-04/gw-08), and how readiness gates the graceful drain you built in gw-01/gw-05.

This is the layer where the abstractions (Service, Pod, Endpoint) meet the kernel (veth pairs, iptables/IPVS/eBPF, conntrack). When a gateway has a mysterious latency or connection problem, the answer is often here — in kube-proxy rules, conntrack table limits, or a readiness race during a deploy.

1. What is it?

Kubernetes networking rests on a few hard rules and a stack of components that implement them. The Kubernetes network model:

1. Every pod gets its own IP (no NAT between pods).
2. Pods can reach every other pod directly by IP, across nodes, without NAT.
3. Agents on a node (kubelet, etc.) can reach pods on that node.

How those rules are realized:

 Service (stable virtual IP + DNS name)              ← the abstraction
    │  selects pods by label → EndpointSlices (the real pod IPs)
    ▼
 kube-proxy (or Cilium/eBPF) programs the node       ← the dataplane glue
    │  iptables / IPVS / eBPF maps ServiceIP → a pod IP
    ▼
 CNI plugin wires each pod's network namespace        ← the pod plumbing
    │  veth pair: pod netns ↔ node bridge/overlay
    ▼
 the packet reaches the pod (your gateway / origin)

• CNI (Container Network Interface) is the spec + plugins that give a pod its network: create the pod's network namespace, a veth pair (one end in the pod, one on the node), assign an IP (IPAM), and set up routing/overlay so cross-node traffic works (Calico, Cilium, AWS VPC CNI, flannel...).
• Service is a stable virtual IP (ClusterIP) + DNS name fronting a set of pods. EndpointSlices (the scalable successor to Endpoints) list the actual ready pod IPs behind a Service.
• kube-proxy programs the node so traffic to a ClusterIP is load-balanced to a backing pod IP — via iptables (rule chains), IPVS (kernel L4 LB, scales better), or it's replaced entirely by eBPF (Cilium) for performance and observability.

2. Why does it matter?

• The gateway runs here, and so do its origins. Drain (gw-01, gw-05) depends on readiness probes + EndpointSlice updates + terminationGracePeriodSeconds + preStop. Get the ordering wrong and every deploy drops connections. This is the operational glue for everything in the phase.

• EndpointSlices are the membership source. The subsetting in gw-04 and the EDS in gw-08 ultimately read pod IPs from EndpointSlices. Their churn (pods coming/going on every deploy/scale) is what makes stability-under-change a requirement.

• "Mysterious" gateway problems live in this layer. conntrack table exhaustion (a busy gateway opens huge numbers of flows), iptables rule bloat at thousands of Services (kube-proxy sync latency), NodePort/ externalTrafficPolicy quirks dropping the client IP, MTU mismatches on overlays causing fragmentation/black-holes. The JD's "root cause using data" maps directly to debugging these.

• Netflix moved compute to Kubernetes. Understanding how custom controllers, container runtime customization (NRI/OCI hooks), and the CNI fit together is the context for "Managing Netflix's Compute" and the NRI talk — and for running a gateway fleet on K8s.

3. How does it work?

A packet's journey to a pod

client → Service DNS (kube-dns/CoreDNS) → ClusterIP (virtual)
   → kube-proxy rule (iptables/IPVS/eBPF) rewrites dst to a pod IP (DNAT)
   → routed to the node hosting that pod (overlay/underlay via CNI)
   → node's veth/bridge → pod's veth → pod netns → container :port
   ← conntrack remembers the mapping so replies are un-DNAT'd correctly

The conntrack table is load-bearing: every connection through a ClusterIP creates a conntrack entry; a busy gateway can exhaust nf_conntrack_max and start dropping connections — a classic outage with a one-line sysctl fix once you know to look.

CNI in detail

When the kubelet starts a pod, it calls the CNI plugin with ADD:

1. create the pod's network namespace
2. create a veth pair: vethXXXX (node side) <-> eth0 (pod side)
3. IPAM: allocate a pod IP from the node's CIDR
4. wire the node side into a bridge / set routes / program eBPF
5. for overlays: encapsulate cross-node traffic (VXLAN/Geneve) or use
   native routing (Calico BGP, AWS VPC CNI = real VPC IPs)
6. return the assigned IP to kubelet

DEL tears it down on pod stop. The CNI spec is deliberately tiny — a binary + JSON config — which is why there are so many implementations.

kube-proxy modes

Services, EndpointSlices, and readiness

• A readiness probe failing removes the pod from its Service's EndpointSlices → kube-proxy stops sending it new traffic. This is the hook that makes graceful drain work (gw-01/gw-05): flip readiness first, then drain.
• EndpointSlices shard endpoints into chunks (default ≤100 each) so a 1000-pod Service doesn't ship one giant object on every change — a scalability fix over the old Endpoints object.
• externalTrafficPolicy: Local keeps the client source IP (no extra hop/SNAT) but only routes to pods on the receiving node — a real trade-off for an edge LB (preserve IP vs even spread).

The drain ordering you must get right

pod deletion:
  1. pod marked Terminating; removed from EndpointSlices (async!)
  2. preStop hook runs (e.g. flip readiness, start app drain — gw-01/gw-05)
  3. SIGTERM sent to the container
  4. grace period (terminationGracePeriodSeconds) counts down
  5. SIGKILL if still alive

The race: EndpointSlice removal (step 1) is eventually consistent across all kube-proxies — some nodes may still send traffic for a moment. So your app must keep serving during preStop/grace, not exit on the first SIGTERM. This is exactly the gw-01 "fail readiness, then drain with a deadline" pattern, and why a high-density WebSocket node (gw-05) needs a long grace period.

4. Core terminology

5. Mental models

• A Service is a phone extension; EndpointSlices are the directory; kube-proxy is the switchboard. You dial the extension (ClusterIP); the switchboard looks up who's currently at that extension (EndpointSlices) and connects you to one of them (DNAT). Readiness is someone marking themselves "do not connect new calls."

• CNI is the patch panel for pods. Each pod gets a virtual cable (veth) from its private room (netns) to the building's network. The CNI plugin is the technician who runs the cable, assigns the jack (IP), and decides whether floors are connected by real wiring (native routing) or tunnels (overlay).

• conntrack is the gateway's hidden quota. Every flow takes a slot; the table is finite. A connection-heavy gateway (especially with churn — gw-04) silently fills it and starts dropping, looking like a random network failure until you check nf_conntrack_count vs _max.

• Readiness-then-drain is leaving a party politely. You tell the host you're heading out (fail readiness → removed from rotation), finish your current conversation (drain in-flight), then leave (exit). Leaving mid-sentence (exit on first SIGTERM) drops whoever you were talking to.

6. Common misconceptions

• "Service load balancing is real load balancing." kube-proxy does L4, connection-level, roughly-even DNAT — not request-aware, latency-aware, or L7. It pins a whole connection (so h2/gRPC hit the multiplexing trap, gw-02) and doesn't do P2C/outlier ejection (gw-06). Real LB is the gateway's job, not kube-proxy's.

• "A pod is removed from rotation the instant it's deleted." EndpointSlice propagation to every kube-proxy is eventually consistent; there's a window where traffic still arrives. Hence preStop + grace + keep-serving-during-drain.

• "All CNIs are equivalent." They differ enormously: overlay vs native routing (latency, MTU), IPAM scale, NetworkPolicy support, eBPF vs iptables dataplane, cloud integration (real VPC IPs vs encapsulation). The choice shapes performance and observability.

• "iptables kube-proxy is fine at any scale." Rule count and sync latency grow with Services×endpoints; at thousands of Services, programming latency and packet-path cost become real. IPVS or eBPF are the scale answers.

• "externalTrafficPolicy doesn't matter." Cluster SNATs and hides the client IP (bad for an edge that needs it — gw-01 PROXY protocol / gw-07 identity); Local preserves it but can imbalance load. It's a deliberate edge decision.

7. Interview talking points

• "Trace a packet from a client to a pod." Service DNS → ClusterIP → kube-proxy DNAT (iptables/IPVS/eBPF) to a pod IP → routed via CNI (overlay or native) to the node → veth into the pod netns → container; conntrack tracks the mapping for the return path. Naming conntrack and the kube-proxy modes signals depth.

• "How do you drain a gateway pod without dropping requests on Kubernetes?" preStop hook flips readiness (and triggers app drain) → pod removed from EndpointSlices (eventually) → keep serving in-flight during the grace period → exit before SIGKILL. Set terminationGracePeriodSeconds long enough (seconds for L7, minutes for a 200k-connection WebSocket node — gw-05). Acknowledge the EndpointSlice propagation race.

• "What's the gotcha with gRPC/HTTP-2 behind a ClusterIP?" kube-proxy is connection-level L4, so one long-lived h2 connection pins all its streams to one pod (gw-02's multiplexing trap). You need L7 LB (a real proxy / mesh / headless Service + client-side LB), not bare ClusterIP.

• "A busy gateway is randomly dropping connections — where do you look?" conntrack exhaustion (nf_conntrack_count near _max), ephemeral-port exhaustion, kube-proxy sync latency / rule bloat, readiness flapping. Reduce flows with connection reuse (gw-04); raise limits as a stopgap. Data-driven, exactly as the JD asks.

• "iptables vs IPVS vs eBPF for kube-proxy?" iptables: simple, ubiquitous, O(n) rule growth → slow at scale. IPVS: kernel hash-table L4 LB, scales to many Services. eBPF (Cilium): fastest, best observability, bypasses iptables, replaces kube-proxy. Pick by scale and observability needs.

• "How does the gateway learn its origins' addresses?" EndpointSlices (watched by the control plane / mesh) → EDS (gw-08) → the gateway's cluster membership → subsetting (gw-04). Pod churn drives the stability requirement.

8. Connections to other labs

• gw-01 / gw-05 (drain) — readiness + preStop + grace period are the Kubernetes machinery that makes graceful drain actually work.
• gw-04 (connection management) — EndpointSlices are the membership the subsetting ring consumes; pod churn is why stability matters; conntrack limits are the L4 ceiling.
• gw-06 (resilience) — kube-proxy is not a substitute for real LB; this lab explains why you still need P2C/outlier ejection.
• gw-08 (Envoy/xDS) — the control plane watches the K8s API (Services/EndpointSlices) to compute EDS.
• gw-10 (Gateway API/operator) — CRDs + the operator pattern, built on the same API machinery; the Gateway API replaces Service-type LoadBalancer/Ingress for L7.
• gw-12 (migration) — running the gateway fleet on K8s, NRI/OCI runtime customization, and the Titus→K8s migration story.