gw-08 step 02 — A reconcile loop and live endpoint updates (EDS)

Goal

Turn the static snapshot into a living control plane: a reconcile loop watches a source of truth, recomputes the desired snapshot on change, and pushes new endpoints to Envoy with no dropped requests — the operational pattern behind every gateway fleet (and the Kubernetes operator in gw-10).

Code — the reconcile loop

package cp

import (
	"context"
	"crypto/sha256"
	"encoding/hex"
	"sort"
	"time"

	"github.com/envoyproxy/go-control-plane/pkg/cache/v3"
)

// Source is the desired state (in prod: the K8s API or a service
// registry; here: a value you mutate).
type Source interface {
	Endpoints() []string // current healthy endpoints for "playback"
}

// Reconciler watches the source and pushes a new snapshot when desired
// state changes. Version = hash(desired state), so identical state never
// triggers a spurious push (debounce by content).
type Reconciler struct {
	Cache  cache.SnapshotCache
	NodeID string
	Source Source

	lastVersion string
}

func (r *Reconciler) Run(ctx context.Context) {
	t := time.NewTicker(500 * time.Millisecond) // or a real watch/event
	defer t.Stop()
	for {
		select {
		case <-ctx.Done():
			return
		case <-t.C:
			r.reconcileOnce(ctx)
		}
	}
}

func (r *Reconciler) reconcileOnce(ctx context.Context) {
	eps := r.Source.Endpoints()
	sort.Strings(eps) // deterministic -> stable version for same set
	version := hashVersion(eps)
	if version == r.lastVersion {
		return // no change: no push (debounce)
	}
	snap := MakeSnapshot(version, eps)
	if err := snap.Consistent(); err != nil {
		// Never push an inconsistent snapshot — keep last-known-good.
		return
	}
	if err := r.Cache.SetSnapshot(ctx, r.NodeID, snap); err != nil {
		return
	}
	r.lastVersion = version
}

func hashVersion(eps []string) string {
	h := sha256.New()
	for _, e := range eps {
		h.Write([]byte(e))
		h.Write([]byte{0})
	}
	return hex.EncodeToString(h.Sum(nil))[:12]
}

The experiment — add/remove an endpoint live

type mutableSource struct{ mu sync.Mutex; eps []string }
func (s *mutableSource) Endpoints() []string { s.mu.Lock(); defer s.mu.Unlock(); return append([]string{}, s.eps...) }
func (s *mutableSource) set(eps []string)    { s.mu.Lock(); s.eps = eps; s.mu.Unlock() }

func main() {
	src := &mutableSource{eps: []string{"127.0.0.1:9001"}}
	snapCache := cache.NewSnapshotCache(true, cache.IDHash{}, nil)
	r := &cp.Reconciler{Cache: snapCache, NodeID: "edge-envoy-1", Source: src}
	go r.Run(context.Background())
	go cp.Run(context.Background(), snapCache, ":18000")

	// Later: scale up -> a new endpoint appears in the source -> EDS push.
	time.Sleep(10 * time.Second)
	src.set([]string{"127.0.0.1:9001", "127.0.0.1:9002"}) // reconcile pushes v2
}

Drive continuous load through Envoy while you add/remove endpoints:

wrk -t2 -c50 -d60s http://127.0.0.1:10000/ &
# during the run, mutate the source (add :9002, then remove :9001)
watch -n1 'curl -s localhost:9901/clusters | grep playback'  # endpoints change live

Tasks

1. Implement the reconcile loop with content-hash versioning. Add an endpoint to the source and confirm Envoy's /clusters shows it within a reconcile tick — with zero errors in the wrk run.
2. Remove an endpoint; confirm Envoy stops sending it traffic gracefully (existing requests finish; new ones avoid it).
3. Show the debounce: mutate the source to the same set; confirm no new version is pushed (the hash is unchanged).
4. Tie it to gw-04: this EDS membership is exactly what your subsetting ring consumes. Rapidly flap an endpoint and discuss debouncing the re-subset so the fix doesn't cause churn.

Acceptance

• Live endpoint add/remove reflected in Envoy with no dropped requests.
• Content-hash versioning prevents spurious pushes (no-op reconciles don't bump the version).
• A correct statement of how this feeds gw-04 subsetting and gw-10's operator.

Discussion prompts

• Why hash the desired state for the version instead of a counter? (Same state ⇒ same version ⇒ idempotent reconcile; survives control-plane restarts without spurious pushes.)
• Rapid pod churn (gw-09) makes the source flap. How do you debounce so Envoy isn't pushed thousands of EDS updates/sec, without making legitimate scale-ups too slow?
• This loop is exactly a Kubernetes controller's reconcile (gw-10): watch desired state → compute → converge. What does controller-runtime add (work queue, rate limiting, leader election) and why?