gw-12 — The Hitchhiker's Guide to Leading a Large-Scale Gateway Migration

Companion to CONCEPTS.md, with the runnable rollout engine in src/go/rollout/. This is the capstone: it's how you ship gw-01…gw-11 to a real fleet without anyone noticing. "Evidence of leading large-scale migrations" is the JD's differentiator for a Distributed Systems Engineer 5.

There's no new protocol here — the capstone is the method: taking a risky change to a critical, high-traffic system and shipping it safely, measurably, with instant rollback. Every Netflix talk in the JD is the story of one of these (Zuul 1→2, the churn rollout, Pushy's re-architecture, Titus→Kubernetes). Run bash scripts/verify.sh:

=== healthy rollout ===
  shadow  mirrored traffic; no user impact
  canary    1% ... -> PROMOTE
  ramp      5% ... -> PROMOTE
  ramp     25% ... -> PROMOTE
  ...
  => reached 100%, rolledBack=false
=== rollout that breaches SLO at 25% ===
  ramp     25%  canErr=0.080 -> ROLLBACK
          auto-rollback to 5% (old path still warm)
  => reached 5%, rolledBack=true
=== shadow validation (new path differs on 1% of requests) ===
  mirrored=10000 diffs=100 diffRate=1.00%  (caught before any user was exposed)

1. The migration ladder (rollout.go)

The four pillars of any survivable migration are reversibility, validate-before-commit, gradual-ramp-with-guardrails, and stakeholder alignment. The first three are mechanized in Run, which walks the ladder lowest-risk to highest-commitment:

shadow (0%) → canary (1%) → ramp (5% → 25% → 50%) → full (100%)

TestLadderRampsToFullWhenHealthy: when the canary matches the baseline at every stage, it ramps all the way to 100%. TestLadderRollsBackOnSLOBreach: when the new path breaches the SLO at 25% (a bug only visible under real load — the most dangerous kind), it auto-rolls-back to the last good stage (5%) and stops. You never skip rungs; each one buys evidence that the next is safe.

The crucial property is in the rollback line: "old path still warm." Reached tracks the last safely-promoted percent, and rollback returns there instantly because the old path was never decommissioned. A rollback that requires a redeploy isn't a rollback — it's a hope. Reversibility is the whole game.

2. Automated canary analysis (rollout.go)

Analyze is the SLO gate: it compares the canary's golden signals (gw-11) to the baseline and decides PROMOTE or ROLLBACK. TestAnalyzePromoteWhenHealthy, ...RollbackOnErrors, and ...RollbackOnLatency pin the three outcomes: within tolerance → promote; error rate exceeds the delta → rollback; tail latency exceeds the ratio → rollback.

The key word is automated. A human watching a dashboard at 1% can't gate a fleet-wide ramp safely — they're too slow and not statistically rigorous. Real systems use Kayenta-style statistical comparison (the lab uses simple thresholds to make the mechanics legible). The decision is data-driven, exactly the JD's "identify root causes using data" — you don't "push through" a degraded canary on a hunch.

3. Shadow traffic: zero-risk validation (shadow.go)

Before any user touches the new path, you mirror production traffic to it, discard (or diff) the response, and serve the user from the old path. Shadow runs n requests through both and counts diffs. TestShadowDiff catches a new path that differs on 1% of requests; the important test is TestShadowZeroRiskWhenNewPathBroken: even when the new path always returns garbage, all 50 users are still served by the old path — the shadow only compares, never returns. That's the property that makes shadowing the cheapest, safest pre-canary validation: you fly the new system through real production weather, but a crash hurts no one.

The gateway is the perfect place to mirror (Envoy does it natively, gw-08; your gw-03 endpoint can fork the request). The diffing — does the new path produce acceptable output? — is where real bugs surface before exposure.

4. The org side (where migrations actually fail)

The code mechanizes the technical ladder; the hard part is people, and it's what the JD lists as core duties ("driving alignment with stakeholders," "setting partner expectations," "technical mentorship," "high-quality design/code reviews"). The CONCEPTS file and docs/analysis.md give the playbook:

  • Write the design doc (goals, non-goals, alternatives, rollout + rollback plan, risks accepted) and run the review as the author.
  • Make the new path strictly better so adoption is pulled, not pushed; set a clear deprecation date; provide migration tooling.
  • "Done" means soaked + decommissioned, flags removed, runbooks updated, partners off the old path — not "it's at 100%."

The capstone exercise in docs/analysis.md asks you to take one gw-* change (e.g. gw-01→gw-04 pooling) through this exact ladder with a written design doc — the artifact a "5" is judged on.

5. NRI/OCI hooks: migrating onto Kubernetes without forking it

The Netflix "Container Runtime Customization" talk is a migration lesson: they moved workloads onto standard Kubernetes by using containerd NRI plugins and OCI hooks to inject per-workload networking/storage/sidecar behavior at the runtime layer — bespoke behavior without forking Kubernetes. The general principle for any platform migration: find the extension point that lets you migrate incrementally, instead of a big-bang rewrite or a fork. (gw-09 covers the K8s networking substrate this rides on.)

6. Hands-on

cd src/go
bash ../scripts/verify.sh        # tests + the healthy/breaching/shadow demos
go run ./cmd/rolloutsim

Then do the capstone (docs/analysis.md): drive a real gw-* change through shadow → canary → ramp using the gw-11 metrics as the gate, with a written design doc and a rehearsed rollback.

7. Exercises

  1. Statistical canary: replace threshold Analyze with a Mann- Whitney/CI-based comparison over samples (Kayenta-style); show it tolerates noise but catches a real regression.
  2. Sticky canary: keep the same cohort on the canary across stages (consistent measurement) instead of re-sampling per request.
  3. Blast-radius staging: extend the ladder to roll out region-by- region / cell-by-cell with a PodDisruptionBudget-style cap (gw-09) so you never drain too much at once.
  4. Wire to the phase: gate the ladder on real gw-11 RED metrics from a gw-03 gateway running a gw-04 change; auto-rollback via a gw-08 config flip.
  5. Game day: rehearse the rollback before the ramp — trigger a synthetic breach and confirm the auto-rollback returns to the warm old path with zero errors.