gw-08 — Envoy & the xDS Control Plane

The JD lists "Envoy Gateway, Kubernetes Gateway API, Istio Gateway" and, crucially, splits the work into "data plane and control plane." This lab is where that split becomes concrete. Envoy is the modern canonical data plane — a high-performance C++ L4/L7 proxy. By itself it proxies whatever its static config says; its power comes from being dynamically configured by a control plane over the xDS protocol. The control plane is the source of truth (which listeners, routes, clusters, endpoints, secrets exist); the data plane is a fleet of Envoys that fetch and apply that config in real time.

This is the same data-plane/control-plane architecture as Netflix's move toward a service mesh ("Zero Configuration Service Mesh with On-Demand Cluster Discovery"), and it's the design pattern behind the Gateway API operator in gw-10. Understanding xDS is understanding how a gateway fleet is operated at scale: you don't redeploy 80 clusters to change a route — the control plane pushes it.

You will build a minimal xDS control plane in Go using go-control-plane and drive a real Envoy with it, watching config flow from your server into the live proxy.

1. What is it?

Envoy is a self-contained proxy with a clean internal model:

Listener  — a port Envoy listens on (e.g. :443)
  └─ Filter chain — ordered network/HTTP filters (the gw-03 idea, in C++)
       └─ HTTP Connection Manager — terminates L7, runs HTTP filters
            └─ Route config — match request → route → cluster
                 └─ Cluster — a logical upstream service (a backend)
                      └─ Endpoints — the concrete instances (ip:port) of that cluster

Each layer is configurable dynamically via a corresponding xDS ("x" Discovery Service) API:

xDS is the gRPC (or REST) protocol Envoy uses to subscribe to these resources from a control plane. The control plane computes the desired state and streams updates; Envoy applies them without a restart. ADS (Aggregated Discovery Service) multiplexes all of them over one stream for ordering guarantees; Delta xDS sends only what changed.

2. Why does it matter?

• The JD explicitly wants data-plane + control-plane fluency. This lab is that split. In a design interview, leading with "I'd separate a p99-optimized data plane from a correctness-optimized control plane that pushes config via something xDS-shaped" is the strongest possible opening (gw-00 INTERVIEW.md).

• It's how you operate a fleet without redeploys. Routes, clusters, endpoints, certs, and resilience policy all change via control-plane push. Connection subsetting (gw-04) consumes EDS membership; cert rotation (gw-07) consumes SDS; canary routing (gw-12) is an RDS change. The control plane is the operational nerve center.

• Config propagation is a distributed-consistency problem. Pushing config to a fleet has the same hazards as replicating a log (db-16…20): ordering (CDS must arrive before the EDS that references it, or Envoy drops endpoints — that's why ADS exists), partial rollout (some Envoys updated, some not), and thundering herds (a change that invalidates everything at once). Your consensus intuition transfers directly.

• Envoy is the industry data plane. Istio, Gateway API implementations (Contour, Envoy Gateway), Consul, and many clouds run Envoy under the hood. Reading an Envoy config and reasoning about a filter's buffering/lifecycle is a baseline skill (the JD says you should be able to read C++).

3. How does it work?

The data-plane/control-plane loop

        ┌──────────────── CONTROL PLANE ────────────────┐
        │ source of truth (k8s CRDs / service discovery) │
        │            ↓ reconcile                          │
        │   desired state → SnapshotCache (versioned)     │
        │            ↓ xDS (gRPC stream, per-node)         │
        └───────────────────┬────────────────────────────┘
                            │  LDS/RDS/CDS/EDS/SDS
        ┌───────────────────▼────────────────────────────┐
   ───▶ │  Envoy fleet (DATA PLANE): apply config live     │ ───▶ origins
 client └──────────────────────────────────────────────────┘

1. The control plane watches a source of truth (Kubernetes objects in gw-10, or a service registry).
2. It computes a snapshot of desired resources (listeners, routes, clusters, endpoints) with a version.
3. Each Envoy opens a gRPC stream and subscribes; the control plane sends the resources for that Envoy's node ID.
4. Envoy ACKs the version it applied (or NACKs with an error if the config is invalid — a critical feedback signal).
5. On any change, the control plane bumps the version and pushes the delta; Envoy applies it without dropping connections.

The xDS request/response protocol (state-of-the-world)

Envoy → DiscoveryRequest { type_url, version_info, resource_names, response_nonce, [error_detail] }
CP    → DiscoveryResponse { type_url, version_info, resources[], nonce }
Envoy → DiscoveryRequest (ACK: echoes version_info == applied; same nonce)
         ...or NACK: version_info == LAST GOOD, error_detail set

The ACK/NACK + nonce dance is the heart of xDS correctness: the control plane knows exactly which version each Envoy is running and whether a push was rejected. This is a versioned, acknowledged replication protocol — recognizably the same shape as the log-matching/ commit machinery from db-17.

Why ADS (ordering) matters

If CDS (clusters) and EDS (endpoints) come on separate streams, EDS for cluster X might arrive before CDS defines X — Envoy can't place the endpoints and may drop traffic. ADS puts everything on one ordered stream so the control plane can guarantee "clusters before endpoints, routes before the listeners that reference them." This is literally a causal-ordering problem (db-16's logical clocks intuition).

Delta vs State-of-the-World

• SotW: every update re-sends the full resource set for a type. Simple; expensive when the set is large and changes are small.
• Delta xDS: sends only added/removed resources. Essential at scale (imagine EDS for 10,000 endpoints changing one at a time). The Netflix "on-demand cluster discovery" work is in this spirit — don't push everything to everyone; discover and push what's actually needed.

Reading an Envoy config (you'll be asked)

static_resources:
  listeners:
  - address: { socket_address: { address: 0.0.0.0, port_value: 8443 } }
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          route_config:
            virtual_hosts:
            - domains: ["api.local"]
              routes:
              - match: { prefix: "/v1/play" }
                route: { cluster: playback }
          http_filters:
          - name: envoy.filters.http.router
  clusters:
  - name: playback
    type: EDS              # endpoints come from the control plane
    lb_policy: LEAST_REQUEST   # P2C (gw-06)
    outlier_detection: {...}    # gw-06
    circuit_breakers: {...}     # gw-06

Everything you built by hand in gw-03/gw-04/gw-06 has a declarative Envoy equivalent — and the control plane fills it in dynamically.

4. Core terminology

5. Mental models

• The control plane is the brain; the data plane is the muscle. The brain decides what should be true (routes, clusters, endpoints) and sends instructions; the muscle executes on the request path, fast and dumb. They fail differently: a brain outage means "config stops changing" (the muscle keeps running on last-known-good — a key resilience property); a muscle outage means "requests drop." Optimize each for its failure mode.

• xDS is git pull for proxy config, with acknowledgments. Each Envoy subscribes and pulls the latest version; it reports back exactly which commit (version) it's on, and rejects (NACK) a bad commit without losing the last good one. The control plane always knows the fleet's deployed version distribution.

• ADS ordering is "define the word before you use it." Sending endpoints for a cluster Envoy hasn't been told about is referencing an undefined variable. ADS guarantees declaration-before-use across resource types over one ordered stream.

• Last-known-good is the data plane's seatbelt. If the control plane vanishes, Envoy keeps serving on the config it last applied. So a control-plane outage degrades change velocity, not availability — a deliberate, vital design choice you should call out.

6. Common misconceptions

• "The control plane is on the request path." It must not be. If every request consulted the control plane, its availability would cap the data plane's. Config is pushed ahead of time and cached; the request path never waits on it. (Contrast with the push-registry lookup in gw-05, which is on the delivery path and is therefore a low-latency datastore, not a control plane.)

• "xDS is just config files over the network." The ACK/NACK + versioning + ordering (ADS) make it a consistency protocol. Treating it as dumb file sync leads to the classic bugs: endpoints before clusters, partial fleet rollout, no visibility into who applied what.

• "Push the new config to everyone at once." A simultaneous fleet-wide push is a thundering herd and a blast-radius maximizer. Stage it: validate, canary a few Envoys, watch their ACKs and metrics, then ramp (gw-12). xDS gives you the per-node ACK signal to do this safely.

• "Envoy and the control plane are one product." They're deliberately separate; many control planes (Istio, Contour, Gloo, your own) drive the same Envoy. The interface is xDS. That separation is why you can build a custom control plane (this lab) without touching the data plane.

• "NACKs are rare edge cases." A NACK means an Envoy rejected your config — it's running stale while you think you shipped. Unmonitored NACKs are a silent "my change didn't take" failure. Alert on them.

7. Interview talking points

• "Design the control plane for a gateway fleet." Source of truth → reconcile to desired state → versioned snapshot per node → xDS push with ACK/NACK → last-known-good on control-plane outage → staged/canaried rollout. Emphasize it's off the request path and that config propagation is a consistency problem (cite ADS ordering).

• "Why is the data/control-plane split the right architecture?" Different SLOs and failure modes: data plane optimized for p99 and availability (keeps serving on last-known-good); control plane optimized for correctness and safe propagation. You can scale, deploy, and reason about them independently.

• "Walk me through xDS." Envoy subscribes per resource type (LDS/RDS/CDS/EDS/SDS) on a gRPC stream; control plane streams versioned resources; Envoy ACKs the applied version or NACKs with an error keeping last-known-good; nonce correlates responses; ADS aggregates for ordering; delta xDS for scale. It's a versioned, acknowledged replication protocol.

• "Why does ADS exist?" Ordering across resource types: clusters before their endpoints, routes before the listeners that reference them. Separate streams race and cause transient drops. It's a causal-ordering guarantee — same family as the logical clocks in db-16.

• "What happens to traffic if the control plane goes down?" Nothing immediately — Envoy serves on last-known-good config. You lose the ability to change config (new endpoints, routes, certs). So you make the control plane highly available for change velocity, but its outage isn't a data-plane outage. Design certs/SDS so they don't expire during a plausible control-plane outage.

• "How would you safely roll out a config change to 80 clusters?" Validate → push to a canary set of Envoys (one node group) → watch ACKs (not NACKs) + RED metrics → ramp by node groups with automated rollback on SLO/NACK breach. The per-node ACK is your rollout signal.

8. Connections to other labs

• gw-03 (API gateway) — Envoy's HCM + filter chain is the production twin of your hand-built gateway; routes are RDS.
• gw-04 (connection management) — EDS supplies the endpoint membership your subsetting computes over; Envoy has built-in subset LB and per-worker pools.
• gw-06 (resilience) — P2C (LEAST_REQUEST), outlier detection, circuit breakers, adaptive concurrency are all Envoy config pushed via xDS.
• gw-07 (security) — SDS pushes certs/keys for hot rotation.
• gw-09 (Kubernetes) — EndpointSlices are the EDS source; the control plane watches the K8s API.
• gw-10 (Gateway API/operator) — the operator is a control plane: it reconciles Gateway/HTTPRoute CRDs into xDS (or vendor config).
• db-16…20 (consensus) — config propagation ordering, versioning, and ACK are the same consistency problems you solved in the consensus phase.