gw-08 — The Hitchhiker's Guide to Envoy & the xDS Control Plane

Companion to CONCEPTS.md, with the runnable mini-xDS in src/go/xds/. The JD splits the work into "data plane and control plane" — this lab is that split, in code.

Envoy by itself proxies whatever its static config says; its power is being dynamically configured by a control plane over xDS. The control plane is the source of truth (which listeners/routes/clusters/ endpoints exist); the data plane is a fleet of Envoys that fetch and apply that config in real time. This lab models the protocol mechanics — versioned snapshots, ordering, ACK/NACK, last-known-good, and a reconcile loop — in stdlib, so they're legible. Production is envoyproxy/go-control-plane driving real Envoy; the concepts are identical.

Run bash scripts/verify.sh and watch the whole control loop:

reconcile #1 (initial):                  [envoy] applied v=5bfda895b0aa -> ACK
reconcile #2 (no change -> debounced):   [cp] desired state unchanged; nothing pushed
reconcile #3 (scale up -> new version):  [envoy] applied v=6c41278c9e89 -> ACK
reconcile #4 (inconsistent -> rejected): [cp] rejected: route "r" references undefined cluster "ghost"
final: applied=6c41278c9e89  current=6c41278c9e89  lastNACK=""

1. Resources and the snapshot (resource.go)

The four xDS resource families map to Envoy's model:

A Snapshot bundles them at a Version and is pushed atomically — Envoy never sees a half-applied set.

Consistency = why ADS exists

Snapshot.Consistent() enforces the cross-resource invariants:

• every Route references a defined Cluster,
• every Listener references a defined Route,
• every Endpoints references a defined Cluster.

TestSnapshotConsistency proves it rejects a route to a ghost cluster and a listener to a missing route. This is exactly why ADS (Aggregated Discovery Service) exists: if clusters and endpoints arrive on separate streams, EDS for cluster X can land before CDS defines X, and Envoy drops the endpoints. ADS puts everything on one ordered stream so the control plane can guarantee declaration-before-use — a causal-ordering problem, the same family as db-16's logical clocks.

Content-hash versioning

Snapshot.Fingerprint() hashes the resources order-independently (TestFingerprintStability: same resources in any add order → same hash; different resources → different hash). Versioning by content hash means identical desired state ⇒ identical version ⇒ no spurious push (debounce), and reconciles are idempotent across control-plane restarts. This is the same determinism discipline as the byte-identical dumps in the consensus phase (db-16…20).

2. The cache and the ACK/NACK state machine (cache.go)

SnapshotCache holds the latest snapshot per node and tracks, per node, the version each Envoy ACKed (or its last NACK). That tracking is the rollout signal: the control plane always knows the fleet's deployed version distribution.

TestAckNackFlow walks the protocol:

1. SetSnapshot(node, v1) validates + stores + pushes; Envoy receives it and Ack(node, v1) → AppliedVersion == v1.
2. SetSnapshot(node, v2) pushes v2, but Envoy Nack(node, v2, reason) (it failed local validation). The applied version stays at v1 — Envoy keeps serving last-known-good — and the NACK reason is recorded.
3. A later Ack clears the NACK.

Two maintainer-level truths fall out:

• A control-plane outage is not a data-plane outage. Envoy serves on last-known-good config; you lose the ability to change config, not the ability to serve. TestSetSnapshotRejectsInconsistent shows a rejected push leaves the previous snapshot in force. Design certs/SDS so they don't expire during a plausible control-plane outage.
• Unmonitored NACKs are a silent failure. A NACK means "Envoy rejected your change and is running stale while you think you shipped." LastNack is the metric you alert on (gw-11).

3. The reconcile loop (reconcile.go)

Reconciler is level-triggered: it derives the desired snapshot from a Source (the K8s API in gw-10, or a service registry), versions it by content hash, and pushes only when it changed and is consistent.

TestReconcileDebounceAndChange: first reconcile pushes; a second with identical desired state pushes nothing (debounce — vital under the EDS/ EndpointSlice churn of gw-09); a membership change pushes a new version. TestReconcileKeepsLastGoodOnError: an inconsistent desired state returns an error and pushes nothing — last-known-good remains. This is the exact same reconcile shape as the Kubernetes operator in gw-10; gw-10's operator is a control plane whose Source is the K8s API.

4. How it stitches the phase together

• The Route/Cluster resources are gw-03's route table, pushed dynamically instead of hard-coded.
• Endpoints (EDS) is the membership gw-04's subsetting ring consumes; its churn is why the reconcile loop debounces.
• Resilience policy (gw-06: LB, outlier detection, circuit breakers) is Envoy cluster config the control plane pushes.
• TLS certs (gw-07) are pushed via SDS, the same protocol.
• In gw-10, a Kubernetes operator replaces the Source with watches on Gateway-API CRDs and EndpointSlices — and you have Envoy Gateway / Contour in miniature.

5. Hands-on

cd src/go
bash ../scripts/verify.sh        # tests + the control-loop demo

# Drive a real Envoy with go-control-plane (the production path) — see
# steps/01-minimal-xds-server.md for the bootstrap config and the
# envoyproxy/go-control-plane wiring. The mechanics you'd reason about
# (snapshot/version/ACK/NACK/ADS ordering) are exactly what this lab
# implements.

6. Exercises

1. Per-node canary snapshots: give a subset of node IDs a different snapshot version and verify (via AppliedVersion per node) that you can ramp a config change node-group by node-group (gw-12).
2. Delta xDS: extend the push to send only changed resources rather than the full set; measure the bytes saved when one of 10,000 endpoints changes.
3. On-demand discovery: only push clusters a node has actually referenced (the Netflix "zero-config service mesh" idea) instead of the whole world.
4. Drive real Envoy: wire go-control-plane's SnapshotCache and point an Envoy at it (steps/01); confirm curl localhost:9901/config_dump shows your version and that killing the control plane leaves traffic serving (last-known-good).
5. Alert on NACKs: expose LastNack/applied-version-skew as metrics (gw-11) and write the burn-rate-style alert for "N nodes failed to apply the latest config."