gw-03 step 02 — Routing table, hop-by-hop hygiene, and hot-reload

Goal

Add a data-driven routing table that maps requests to origin clusters, strip hop-by-hop headers correctly, and hot-reload the route table without dropping a request — the property that makes a gateway operable at fleet scale (and a preview of the control plane in gw-08).

Code — the route table

package gw

import (
	"strings"
	"sync/atomic"
)

type Route struct {
	Host    string // "" = any
	Prefix  string // path prefix
	Method  string // "" = any
	Header  [2]string // optional [name,value] predicate, e.g. canary steering
	Cluster string
}

type RouteTable struct {
	routes []Route
}

// Match returns the first route whose predicates all match, longest
// prefix first (so /v1/play/start beats /v1).
func (t *RouteTable) Match(host, method, path string, hdr func(string) string) (string, bool) {
	best := -1
	bestLen := -1
	for i, r := range t.routes {
		if r.Host != "" && r.Host != host {
			continue
		}
		if r.Method != "" && r.Method != method {
			continue
		}
		if !strings.HasPrefix(path, r.Prefix) {
			continue
		}
		if r.Header[0] != "" && hdr(r.Header[0]) != r.Header[1] {
			continue
		}
		if len(r.Prefix) > bestLen {
			best, bestLen = i, len(r.Prefix)
		}
	}
	if best < 0 {
		return "", false
	}
	return t.routes[best].Cluster, true
}

// Router holds an atomically-swappable table for lock-free hot-reload.
type Router struct {
	tbl atomic.Pointer[RouteTable]
}

func (rt *Router) Load() *RouteTable { return rt.tbl.Load() }

// Swap atomically replaces the table. In-flight requests keep using the
// snapshot they read; new requests see the new table. No dropped request.
func (rt *Router) Swap(t *RouteTable) { rt.tbl.Store(t) }

Code — the routing filter (inbound)

type RoutingFilter struct{ R *Router }

func (RoutingFilter) Type() Phase { return Inbound }
func (RoutingFilter) Order() int  { return 50 } // after auth(10), before endpoint
func (RoutingFilter) ShouldFilter(c *RequestContext) bool { return true }

func (f RoutingFilter) Apply(c *RequestContext) {
	cluster, ok := f.R.Load().Match(
		c.Req.Host, c.Req.Method, c.Req.URL.Path, c.Req.Header.Get)
	if !ok {
		c.Resp.Status = 404
		c.Resp.Body = []byte("no route")
		c.Stop()
		return
	}
	c.RouteName = cluster
	c.Attributes["cluster"] = cluster
}

Code — hop-by-hop header hygiene

A proxy must consume, not forward, hop-by-hop headers (RFC 9110 §7.6.1) and must normalize forwarding headers:

var hopByHop = []string{
	"Connection", "Proxy-Connection", "Keep-Alive", "Proxy-Authenticate",
	"Proxy-Authorization", "Te", "Trailer", "Transfer-Encoding", "Upgrade",
}

func sanitizeForOrigin(c *RequestContext) {
	h := c.Req.Header
	// Also remove anything named by the Connection header.
	for _, name := range strings.Split(h.Get("Connection"), ",") {
		if n := strings.TrimSpace(name); n != "" {
			h.Del(n)
		}
	}
	for _, n := range hopByHop {
		h.Del(n)
	}
	// Append client IP to X-Forwarded-For (don't trust an inbound one
	// from an untrusted client — gw-07 trust-boundary discussion).
	if ip := clientIP(c.Req); ip != "" {
		prior := h.Get("X-Forwarded-For")
		if prior != "" {
			h.Set("X-Forwarded-For", prior+", "+ip)
		} else {
			h.Set("X-Forwarded-For", ip)
		}
	}
}

Hot-reload demo

// Build an initial table.
r := &Router{}
r.Swap(&RouteTable{routes: []Route{
	{Host: "api.local", Prefix: "/v1/play", Cluster: "playback"},
	{Host: "api.local", Prefix: "/v1/search", Cluster: "search"},
	{Prefix: "/", Cluster: "edge-fallback"},
}})

// Later, e.g. on a control-plane push or SIGHUP, swap in a new table
// that adds a canary route — with ZERO dropped requests:
r.Swap(&RouteTable{routes: append(current,
	Route{Prefix: "/v1/play", Header: [2]string{"x-canary", "true"},
	      Cluster: "playback-canary"})})

Tasks

1. Implement the route table + routing filter; verify /v1/play/start → playback, /v1/search/q → search, /anything → edge-fallback, longest-prefix wins.
2. Add a header-predicate canary route (x-canary: true → playback-canary) and confirm steering works (gw-12 uses exactly this for migrations).
3. Hot-reload: run wrk continuously against the gateway, call Swap on a ticker to add/remove a route, and show zero errors during the swaps (atomic pointer = no lock, no dropped request).
4. Verify hop-by-hop headers are stripped and X-Forwarded-For is appended (inspect what the origin receives).

Acceptance

• Correct longest-prefix + predicate matching.
• Continuous wrk load through repeated Swaps with no 5xx/dropped requests.
• Origin sees a clean header set with a correct X-Forwarded-For.

Discussion prompts

• Why an atomic.Pointer swap instead of locking the table on every request? (Read path is hot — 1M+ rps; writes are rare.)
• How does this hot-reload generalize to a fleet of gateways receiving a control-plane push (gw-08)? What new failure modes appear when the push is partial across the fleet?
• Why must you not trust an inbound X-Forwarded-For from an untrusted client, and how does that interact with the PROXY protocol (gw-01)?