gw-10 — The Hitchhiker's Guide to CRDs, Operators & the Gateway API

Companion to CONCEPTS.md, with the runnable operator in src/go/operator/. It closes the loop with gw-08: an operator is a control plane whose source of truth is the Kubernetes API.

The JD wants "Operator patterns" and the "Kubernetes Gateway API." This lab builds the operator pattern in its smallest correct form — an in-memory apiserver, a level-triggered reconcile loop, finalizers, and status — so you understand exactly what controller-runtime does for you. Run bash scripts/verify.sh:

apply (service missing):       Accepted=false(ServiceNotFound) Programmed=false dpPushes=0
service created (self-heal):   Accepted=true(Valid) Programmed=true(Synced) dpPushes=1
reconcile x2 (idempotent):     Accepted=true Programmed=true dpPushes=1
delete issued (Terminating):   exists=true ...
reconcile (finalizer cleanup): exists=false dpPushes=2 dpRoutes=0

That lifecycle — self-heal, idempotent, finalizer cleanup — is the whole pattern.

1. Level-triggered reconciliation (reconcile.go)

The single most important operator concept: a controller reconciles to desired state, it doesn't react to events. Reconcile(name) reads the current state every call and converges — so missing an event, or restarting the controller, never leaves the world permanently wrong.

TestLevelTriggeredSelfHealing proves it: apply an EdgeRoute whose target Service doesn't exist → status Accepted=false/ServiceNotFound. Then create the Service and reconcile again without touching the CR → status flips to Accepted=true/Programmed=true on its own. The controller re-derived the truth from current state. An edge-triggered "on add do X" controller couldn't do this — it would be stuck on the stale "service missing" it saw at creation. This is the thermostat (not light-switch) mental model made concrete.

Idempotency

TestIdempotentReconcile reconciles identical state five times and asserts the data plane was pushed once (dpPushes == 1). The MemDataPlane versions config by content hash, so reprogramming the same route set is a no-op — exactly the gw-08 content-hash debounce. Reconcile runs on every watched change and on resync, so under churn it's called constantly; it must be idempotent and fast or it backs up the work queue.

Rebuild the whole set, don't mutate incrementally

reprogram() rebuilds the data-plane config from all live EdgeRoutes every time, rather than adding/removing one. That's what makes reconcile idempotent and correct after missed events — and it's why TestDeleteReprogramsRemaining shows that deleting r1 leaves the data plane with exactly r2.

2. Status as the contract (reconcile.go)

A user applies a CR and walks away; status.conditions is how they (and dashboards) learn whether it took effect. The crucial ordering: Programmed=true is set only AFTER the data-plane push succeeds. TestStatusProgrammedOnlyAfterSuccess forces the data plane to fail (FailNext) and asserts status becomes Programmed=false/PushFailed — the CR analog of an xDS NACK (gw-08). A controller that reports success before the side effect actually happened is lying to its users; status must reflect reality. Surface Programmed=false / generation-vs-observed skew as a metric and alert on it (gw-11).

3. Finalizers: clean teardown (store.go, reconcile.go)

When you kubectl delete a CR, Kubernetes sets a deletionTimestamp but keeps the object while any finalizers remain — giving the controller a chance to clean up external state first. TestFinalizerCleanup walks it: the live object carries a finalizer; Delete puts it in Terminating (still Exists); the next Reconcile deprograms the data plane (the route is already excluded from ListLive because it's being deleted), then removes the finalizer, which lets the object actually disappear.

Why deprogram before the object is gone? Otherwise there's a window where the CR is deleted but traffic still flows to a dead backend. Finalizers close that window. The maintainer hazard (CONCEPTS §6): a finalizer whose cleanup can never complete (data plane permanently unreachable) blocks deletion forever — so bound the cleanup and keep a break-glass to remove a stuck finalizer.

4. The controller / work queue (controller.go)

Controller drains a work queue, calling Reconcile per key (TestControllerProcessesQueue). A real controller-runtime controller adds caching informers (watch once, serve many cached reads), a rate-limited requeue (retry with backoff), and leader election (one active replica, HA without double-writes). The level-triggered core — enqueue on change, reconcile to desired state — is exactly what you built. When you scaffold with kubebuilder, you're filling in this Reconcile method; everything else is the framework.

5. The Gateway API and how this becomes Envoy Gateway

The CONCEPTS file covers the Gateway API — the standard, role- separated routing CRDs (GatewayClass/Gateway/HTTPRoute) that replace Ingress. An HTTPRoute is the declarative form of gw-03's route table; this operator's EdgeRoute is a stand-in for it. Swap the MemDataPlane for gw-08's xDS SnapshotCache and point the Source at the Kubernetes API (Gateway-API CRDs + EndpointSlices, gw-09), and you have Envoy Gateway / Contour in miniature: watch CRDs → reconcile into xDS → push to Envoy → write status back. gw-08 + gw-10 together are a complete, K8s-native gateway control plane.

6. Hands-on

cd src/go
bash ../scripts/verify.sh        # tests + the lifecycle demo

# Then build the real thing with kubebuilder (steps/01): define the CRD,
# implement this exact Reconcile against a real apiserver, and watch
# `kubectl get edgeroute -o yaml` show the status conditions you set.

7. Exercises

  1. Wire to gw-08: implement DataPlane over gw-08's SnapshotCache so the operator reconciles CRs into xDS and an Envoy serves them.
  2. Watch the referenced Service: enqueue the EdgeRoute when its target Service's endpoints change (gw-09), so endpoint churn re-triggers reconcile — then your data plane always has current endpoints.
  3. CRD versioning as migration: add a v2 spec field, write a conversion from v1, and keep both clients working — a migration at the API level (gw-12).
  4. Leader election: run two controllers and add a simple lease so only one reconciles; prove no double-writes.
  5. Real kubebuilder operator (steps/01): scaffold, define the CRD, port this Reconcile, deploy to kind, and confirm self-heal + finalizer cleanup on a live cluster.