gw-10 step 02 — Program the data plane, and clean up with finalizers

Goal

Make the operator a real control plane: reconcile the full set of routes into a data-plane config (the gw-08 xDS snapshot), and use a finalizer to deprogram cleanly on delete so no orphaned config or traffic-to-a-deleted-backend remains.

Code — the DataPlane programmer (bridges to gw-08)

package controller

import (
	"context"
	"sort"

	netv1 "github.com/10xdev/gw10/api/v1"
	"sigs.k8s.io/controller-runtime/pkg/client"
)

type DataPlane interface {
	Program(ctx context.Context, er *netv1.EdgeRoute) error
}

// xdsProgrammer rebuilds the WHOLE snapshot from all EdgeRoutes and
// pushes it (gw-08 SnapshotCache). Rebuilding the full set — rather than
// mutating incrementally — keeps reconcile idempotent and correct.
type xdsProgrammer struct {
	client.Client
	cache SnapshotSink // your gw-08 cache.SetSnapshot wrapper
}

func (p *xdsProgrammer) Program(ctx context.Context, _ *netv1.EdgeRoute) error {
	var list netv1.EdgeRouteList
	if err := p.List(ctx, &list); err != nil {
		return err
	}
	routes := make([]netv1.EdgeRoute, 0, len(list.Items))
	for _, er := range list.Items {
		if er.DeletionTimestamp.IsZero() { // exclude routes being deleted
			routes = append(routes, er)
		}
	}
	// Deterministic order -> stable version -> no spurious pushes (gw-08).
	sort.Slice(routes, func(i, j int) bool { return routes[i].Name < routes[j].Name })

	snap := buildSnapshotFromRoutes(routes) // -> listeners/routes/clusters/EDS
	return p.cache.Set(ctx, "edge-envoy", snap)
}

Code — finalizer for clean teardown

const finalizer = "net.10xdev.io/deprogram"

func (r *EdgeRouteReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	var er netv1.EdgeRoute
	if err := r.Get(ctx, req.NamespacedName, &er); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	// Handle deletion: run cleanup, THEN drop the finalizer to let it go.
	if !er.DeletionTimestamp.IsZero() {
		if controllerutil.ContainsFinalizer(&er, finalizer) {
			// Deprogram: rebuild the snapshot WITHOUT this route so the
			// data plane stops sending it traffic before the object vanishes.
			if err := r.Programmer.Program(ctx, &er); err != nil {
				return ctrl.Result{Requeue: true}, nil // retry cleanup
			}
			controllerutil.RemoveFinalizer(&er, finalizer)
			_ = r.Update(ctx, &er)
		}
		return ctrl.Result{}, nil
	}

	// Ensure the finalizer is present on live objects.
	if !controllerutil.ContainsFinalizer(&er, finalizer) {
		controllerutil.AddFinalizer(&er, finalizer)
		_ = r.Update(ctx, &er)
	}

	// ... normal reconcile (program + status) from step 01 ...
	return ctrl.Result{}, nil
}

Tasks

1. Implement xdsProgrammer reusing your gw-08 SnapshotCache. Apply two EdgeRoutes; confirm Envoy serves both (config came from CRDs → operator → xDS → Envoy: the full stack).
2. Add the finalizer. kubectl delete an EdgeRoute; confirm the object stays in Terminating until the operator deprograms it, then disappears — and Envoy stops routing that prefix before the object is gone (no orphaned config).
3. Break-glass: simulate the data plane being unreachable during delete; show the object is stuck in Terminating (finalizer can't complete), then remove the finalizer by hand and explain why that's the emergency escape hatch.

Acceptance

• End-to-end: kubectl apply EdgeRoute → Envoy serves the route; kubectl delete → route deprogrammed before deletion completes.
• Finalizer prevents orphaned data-plane config; you can articulate the stuck-finalizer hazard and its break-glass.

Discussion prompts

• Why rebuild the entire snapshot from all EdgeRoutes each reconcile instead of incrementally adding/removing one? (Idempotency, correctness after missed events, matches gw-08's content-hash versioning.)
• Why deprogram before the object is deleted (finalizer) rather than after? (Avoid a window where the object is gone but traffic still flows to a dead backend.)
• This operator + gw-08 is exactly Envoy Gateway / Contour in miniature. What does a production version add (Gateway API conformance, webhooks, multi-tenancy, leader election, metrics)?