gw-10 — Gateway API, CRDs, and the Operator Pattern

The JD asks for "Kubernetes internals (… CRDs, and Operator patterns)" and names "Kubernetes Gateway API, Istio Gateway." This lab closes the loop: the Gateway API is the standard, role-oriented, extensible Kubernetes way to express L7 routing (the successor to Ingress), and the operator pattern is how you implement a gateway's control plane as a Kubernetes controller — watch custom resources, reconcile them into data-plane config (the xDS of gw-08, or vendor config), and report status back.

This is where gw-08 (control plane) and gw-09 (K8s internals) combine. An operator is a control plane that speaks Kubernetes. You will build a controller-runtime operator that watches Gateway/HTTPRoute resources and reconciles them into proxy configuration — the exact shape of Envoy Gateway, Contour, and Istio.

1. What is it?

A Custom Resource Definition (CRD) extends the Kubernetes API with your own object types. Once registered, a CRD's resources are first-class: kubectl get, RBAC, watches, status — all work. A controller/operator is a process that watches those resources and drives the real world to match them, via the reconcile loop:

        ┌─────────── reconcile loop (level-triggered) ───────────┐
 watch ▶│  observe desired state (the CRD spec)                   │
        │  observe actual state (the data plane / external system)│
        │  compute the diff                                       │
        │  take actions to converge actual → desired             │
        │  write status back to the CRD                           │
        └──────────────────────▲─────────────────────────────────┘
                               │ re-trigger on any change or resync

The key property is level-triggered, declarative reconciliation: the controller doesn't react to events ("a route was added"); it repeatedly drives toward the desired state ("these routes should exist"), so it's self-healing and idempotent — it converges no matter how many events it missed or how it was restarted.

The Gateway API is a set of standard CRDs for L7 routing, split by role:

It replaces Ingress (which was too limited and annotation-hell) with a typed, role-separated, extensible model — and it's portable across implementations (Envoy Gateway, Istio, Contour, NGINX, cloud LBs).

2. Why does it matter?

• It's the named technology and the future of K8s ingress. "Gateway API, Istio Gateway" is in the JD. Ingress is legacy; the Gateway API is where the industry (and a migration-minded team) is heading — and migrations are the team's bread and butter (gw-12).

• The operator pattern is how gateway control planes are built on K8s. Envoy Gateway, Contour, and Istio are operators: watch Gateway API CRDs → reconcile into Envoy xDS. Building one teaches you exactly how the gateway you'd operate is wired, and the JD's "Operator patterns" requirement is literally this.

• Reconciliation is the safest way to manage fleet config. Because it's level-triggered and idempotent, an operator self-heals: drift, partial failures, and restarts all converge. This is the same desired-state discipline as gw-08's reconcile loop, formalized by Kubernetes.

• Status reporting closes the operability loop. A good operator writes back status (Accepted/Programmed/conditions) so users and dashboards know whether their HTTPRoute actually took effect — the CRD analog of xDS ACK/NACK (gw-08) and a key observability surface (gw-11).

3. How does it work?

The reconcile loop (controller-runtime)

Reconcile(ctx, req):                       # req = namespaced name of a changed object
    obj = get(req)                         # desired state from etcd
    if not found: return                   # deleted; finalizers handle cleanup
    actual = observe_external_state(obj)   # what the data plane currently has
    if actual != desired(obj):
        program_data_plane(desired(obj))   # converge (e.g. push xDS — gw-08)
    set_status(obj, Programmed=true)       # report back
    return (requeue if needed)

It runs whenever a watched object changes and on a periodic resync, so a missed event never leaves you wrong forever. It must be idempotent (reconciling the same state twice = no-op) and fast (work is queued; slow reconciles back up the queue).

How a request flows from CRD to data plane

app dev:  kubectl apply HTTPRoute (prefix /v1/play -> Service playback)
   │ watch
operator: reconcile -> compute routes/clusters/endpoints
   │ (endpoints from EndpointSlices — gw-09)
   ▼
data plane config: Envoy xDS snapshot (gw-08)  OR  reload a proxy
   ▼
Envoy/proxy serves /v1/play -> playback pods
   │
operator: write HTTPRoute.status.conditions = [Accepted, Programmed]

The operator is the bridge from the declarative K8s API (gw-09) to the imperative data-plane config (gw-08). That's the whole job.

CRD machinery you must know

• OpenAPI schema + validation — the CRD declares its spec's shape; the API server validates on write (plus webhooks for richer rules).
• Versioning + conversion webhooks — v1alpha2→v1 etc., with a conversion webhook so old and new clients coexist (vital for migrations).
• Finalizers — block deletion until the controller cleans up external state (e.g. deprogram the data plane before the Gateway object disappears), preventing orphaned config.
• Owner references — child objects (a Deployment the operator creates for a Gateway) get garbage-collected with the parent.
• status subresource — separates user-written spec from controller-written status; conditions communicate Accepted / Programmed / errors.
• Leader election — only one controller replica reconciles at a time (HA without double-writes).

Why level-triggered beats edge-triggered

Edge-triggered ("do X when event Y fires") loses correctness if you miss an event (controller was down, queue dropped it). Level-triggered ("make the world match the spec, repeatedly") is self-correcting: it re-derives the right state from scratch each time. This is the single most important operator concept and a frequent interview probe.

4. Core terminology

5. Mental models

• An operator is a thermostat, not a light switch. A switch is edge-triggered (you flip it, once). A thermostat is level-triggered: it continuously compares actual temperature to the set point and acts to close the gap, forever. Miss a moment and it just corrects on the next read. That's why operators self-heal.

• The Gateway API is org-chart-as-API. Ingress mashed everyone's concerns into one annotated object. Gateway API splits them by role: the infra team owns GatewayClass, the platform team owns Gateway (ports/certs), app teams own HTTPRoutes — each with its own RBAC. The schema encodes the org boundaries, which is why large orgs adopt it.

• CRD + controller = "teach Kubernetes a new noun and a new verb." The CRD adds the noun (Gateway); the controller adds the behavior (what it means for a Gateway to exist). Kubernetes becomes a general-purpose reconciliation engine for your domain.

• Status is the receipt. A user applies an HTTPRoute and walks away; status.conditions is how they learn it was Accepted and Programmed (or why not). Without it, "I applied it but is it live?" is unanswerable — the CRD version of an unmonitored xDS NACK (gw-08).

6. Common misconceptions

• "Operators react to events." They reconcile to desired state. Events just trigger a reconcile; the reconcile re-derives everything from current state, so missing an event is survivable. Building an edge-triggered "on add do X, on delete do Y" controller is the classic beginner mistake that breaks on restart.

• "Gateway API is just Ingress v2." It's a role-separated, extensible, portable, typed model with first-class L4/L7/TLS/gRPC routes, cross-namespace ReferenceGrant, and status conditions. Ingress couldn't express most of this without vendor annotations.

• "CRDs make Kubernetes a database for my config." They're desired state, not just storage — the value is the controller continuously enforcing them. A CRD with no controller is inert YAML.

• "Reconcile should be fast because it's called rarely." It's called on every watched change and on resync; under churn it's called a lot. It must be idempotent and quick; slow reconciles back up the work queue and delay convergence across all objects.

• "One controller replica is a SPOF." Run multiple with leader election: one reconciles, the others stand by, failover is automatic. You get HA without two controllers fighting over the same resources.

7. Interview talking points

• "Explain the operator/reconcile pattern." Level-triggered, declarative, idempotent reconciliation: observe desired (CR spec) + actual (external) state, converge, write status; re-run on change and resync. Contrast level- vs edge-triggered and explain why level-triggered self-heals. This is the answer that proves you understand Kubernetes, not just use it.

• "What is the Gateway API and why does it exist?" Standard, role-oriented, extensible L7 routing CRDs (GatewayClass/Gateway/ HTTPRoute) replacing Ingress; portable across implementations (Envoy Gateway, Istio, Contour); typed L4/L7/gRPC/TLS routes with status conditions. The role separation maps to org structure and RBAC.

• "How would you build a gateway control plane on Kubernetes?" An operator: watch Gateway API CRDs + EndpointSlices (gw-09) → reconcile into Envoy xDS snapshots (gw-08) → write status back. It's gw-08's reconcile loop, but the source of truth is the K8s API and the framework is controller-runtime (work queue, caching informers, leader election).

• "Why finalizers and owner references?" Finalizers block deletion until you deprogram the data plane (no orphaned config / no traffic to a deleted backend); owner references cascade-delete the child objects an operator creates. Both prevent the "deleted the CR but the side effects linger" class of bugs.

• "How do you safely evolve a CRD?" Versioned API (v1alpha2→v1) with a conversion webhook so old/new clients coexist; additive, optional fields; a deprecation window. This is a migration (gw-12) at the API level — exactly the "leading large-scale migrations" the JD prizes.

• "How do users know their route is live?" status.conditions (Accepted, Programmed) written by the controller — the CRD analog of xDS ACK/NACK. Surface it in dashboards/alerts (gw-11). An operator that doesn't report status is unobservable.

8. Connections to other labs

• gw-08 (Envoy/xDS) — the operator reconciles CRDs into the xDS snapshots gw-08 serves; together they are a complete K8s-native gateway control plane.
• gw-09 (K8s networking) — built on the same API machinery; EndpointSlices feed the operator's endpoint computation; finalizers interact with pod/Service lifecycle.
• gw-03 (API gateway) — an HTTPRoute is the declarative form of gw-03's route table; the operator programs the data plane that runs the filter chain.
• gw-07 (security) — Gateway listeners reference TLS secrets; the operator wires certs (and can drive SDS).
• gw-12 (migration) — migrating Ingress → Gateway API, and rolling out CRD changes safely, are textbook large-scale migrations; CRD versioning/conversion is migration tooling.
• db-16…20 (consensus) — Kubernetes is etcd-backed (Raft, db-17); the reconcile loop's "converge to a replicated desired state" is the consensus mindset at the application layer.