gw-12 — Capstone: Leading a Large-Scale Gateway Migration

The JD calls out "Evidence of leading large-scale migrations is a plus," plus driving alignment with stakeholders, setting partner expectations, technical mentorship, and high-quality design/code reviews. This capstone is where the technical phases (gw-01…gw-11) become a leadership problem. Anyone can write a better gateway; a "5" gets the whole fleet there — across billions of requests, dozens of stakeholder teams, and a zero-downtime requirement — without anyone noticing.

Netflix's history is a catalog of exactly these migrations: Zuul 1 → Zuul 2 (blocking → async, gw-03), the connection-churn rollout (gw-04), Pushy's re-architecture and density jump (gw-05), Titus → Kubernetes ("Managing Netflix's Compute"), and container runtime customization with NRI/OCI hooks to run specialized workloads on standard Kubernetes. This lab distills the playbook common to all of them and turns it into something you can speak to with authority.

There is no new protocol here — the capstone is the method: how to take a risky change to a critical, high-traffic system and ship it safely, measurably, and with the org aligned behind you.

1. What is it?

A large-scale migration is moving a critical system from state A to state B while it keeps serving production traffic, where the blast radius is "everything" and the rollback must be instant. The canonical gateway migrations:

The method that makes any of them survivable rests on four pillars: (1) make it reversible, (2) validate with production traffic before you commit (shadow), (3) ramp gradually with automated guardrails (canary + SLO-gated rollout), and (4) keep humans aligned the whole way (design reviews, stakeholder comms, mentorship).

2. Why does it matter?

• It's the differentiator for a "5". The technical chops in gw-01…gw-11 get you to senior. Leading a migration — scoping it, aligning a dozen teams, shipping it with zero customer impact, and mentoring others through it — is what the role is actually hiring the 5 for. "Evidence of leading large-scale migrations" is the tiebreaker.

• The gateway's blast radius makes migrations uniquely scary. A bug in one service hurts that service; a bug in the edge hurts everything (gw-03). So the edge team has the most refined migration discipline in the company, and you'll be expected to wield it from early on.

• It's mostly a people problem. The hardest part isn't the code — it's getting partner teams to adopt the new path, setting realistic expectations, running the design review that surfaces the objection you didn't think of, and reviewing others' code to the fleet-wide bar. The JD lists all of these as core responsibilities for a reason.

• It's how the talks happen. Every talk in the JD is the story of a migration. Being able to narrate one — what you tried, what broke, what you measured, how you de-risked — in the exact shape of those talks is how you impress the panel and, later, become the person who gives the talk.

3. How does it work?

The migration ladder (lowest risk to highest commitment)

1. DARK / SHADOW     mirror prod traffic to the new path; DON'T serve its
                     responses to users; diff new-vs-old offline.
2. CANARY            serve a tiny % of real traffic on the new path;
                     compare RED metrics + SLOs vs the control group.
3. STICKY CANARY     keep the same users on the canary so you measure a
                     consistent cohort, not request-level noise.
4. RAMP             increase the % in stages (1% → 5% → 25% → 50% → 100%)
                     with automated rollback on SLO/NACK breach (gw-08/11).
5. CUTOVER + SOAK   100% on the new path; keep the old path warm for a
                     soak period so rollback stays instant.
6. DECOMMISSION     only after the soak proves stability: remove the old
                     path, delete the flags, update the runbooks.

You never skip straight to a flag flip. Each rung buys evidence that the next is safe.

Traffic shadowing / mirroring (the de-risking superpower)

The gateway is the perfect place to mirror: copy each request to the new path in addition to the real one, discard (or async-diff) the mirror's response, and serve the user from the old path. You get production-shaped load and inputs against the new system with zero user risk. Envoy supports request mirroring natively (gw-08); your gw-03 gateway can fork the request in the endpoint filter. The diffing (does new produce the same/acceptable output as old?) is where real bugs surface before any user is exposed.

Canary analysis (automated, not eyeballed)

canary group:  N% of traffic (or N% of nodes) on the new path
control group: the rest on the old path
compare: RED metrics, SLO burn, resource USE, business KPIs
         (a statistical comparison, e.g. Netflix's Kayenta/ACA)
decision: auto-promote if canary ≈ control; auto-rollback if worse

The key word is automated: a human watching a dashboard can't gate a fleet-wide ramp safely. SLO-gated, statistically-compared, auto-rollback canaries are the standard.

Reversibility and blast-radius control

• Flag everything. The new path is behind a runtime flag (RTDS/ config push, gw-08), so disabling it is a config change, not a deploy.
• Keep the old path warm. Don't decommission until the soak proves it; rollback must be instant, which means the old path must still be there.
• Stage by blast radius. Roll out region by region, cluster by cluster, node group by node group — never globally at once (the thundering-herd / fleet-wide-blast lesson from gw-04/gw-08).
• Cap the damage. Use the connection-churn and reconnect-storm lessons (gw-04/gw-05): staggered rollout, pre-warming, jittered restarts, PodDisruptionBudgets (gw-09) so you never drain too much at once.

The org side (where migrations actually fail)

• Design review as the author. Write the doc: goals, non-goals, alternatives considered, rollout plan, rollback plan, the risks you're not mitigating and why. The review surfaces the objection you missed; running it well is a core JD duty.
• Stakeholder alignment & partner expectations. Partner teams must adopt the new path (e.g. re-declaring routes for Ingress→Gateway-API). You set expectations on timeline, required work, and the deprecation date — and you make the new path easier than the old so adoption is pulled, not pushed.
• Mentorship. A migration is how juniors level up: pair them on a slice, review their code to the fleet bar, and let them own a rung of the ladder. The JD lists "technical mentorship" and "high-quality code reviews" as responsibilities, not nice-to-haves.

NRI / OCI hooks as a migration enabler (the Netflix angle)

Migrating workloads onto Kubernetes (gw-09) without forking it is itself a migration problem. Netflix uses containerd NRI plugins and OCI hooks to inject specialized networking, storage, and sidecar behavior per workload at the runtime layer — so they get bespoke behavior and stay on standard Kubernetes. The lesson generalizes: find the extension point that lets you migrate incrementally without forking the platform, instead of a big-bang rewrite.

4. Core terminology

5. Mental models

• A migration is replacing a plane's engine in flight. You can't land (take downtime). So you add the new engine alongside, test it without relying on it (shadow), switch one of four to it (canary), then the rest gradually — keeping the old one spinning until you're sure (soak). Nobody on board should feel a thing.

• Shadow traffic is a flight simulator fed by real weather. You fly the new system through actual production conditions, but a crash hurts no one because the real passengers are on the other plane. It's the cheapest, safest evidence you can buy.

• The flag is the seatbelt; the warm old path is the parachute. The flag lets you stop instantly; keeping the old path alive means stopping actually lands you somewhere safe. A rollback plan that requires a deploy isn't a rollback — it's a hope.

• Adoption is pulled, not pushed. Teams migrate when the new path is better for them (simpler, faster, safer), not because you sent a deadline email. The best migration leaders make the new path the path of least resistance, then the deadline is a formality.

• The blast radius is a dial, not a switch. Region-by-region, cluster-by-cluster, 1%→100% — every rung turns the dial up a little and gives you a chance to turn it back. Big-bang is the switch that has no "back."

6. Common misconceptions

• "If it passes staging/tests, just flip it." Staging never matches prod's scale, traffic mix, or weird inputs. Shadow + canary against real traffic is the only trustworthy validation for an edge change.

• "Rollback = redeploy the old version." Too slow when the edge is bleeding. Rollback must be a flag/config flip with the old path already warm — seconds, not a deploy cycle.

• "Faster is safer (rip the band-aid)." For a critical fleet, slow and staged is safer: each rung limits blast radius and gives the guardrails time to catch a regression before it's global.

• "The migration is done at 100%." It's done after the soak and decommission: old path removed, flags deleted, runbooks/dashboards updated, partners off the deprecated path. Half-finished migrations (two code paths forever) are a tax everyone keeps paying.

• "It's a technical project." The technical part is often the easy part. Stakeholder alignment, partner adoption, and the design review are where migrations stall or fail. Treat the org work as first-class.

• "Canary = watch a dashboard at 1%." Manual eyeballing doesn't scale and isn't statistically sound. Use automated canary analysis with SLO gates and auto-rollback.

7. Interview talking points

• "Tell me about a large-scale migration you led." The behavioral centerpiece. Structure: the goal + why it mattered → the constraints (zero downtime, N stakeholders) → the rollout ladder you chose (shadow → canary → ramp → soak) → the guardrails (SLO-gated auto-rollback) → the surprise that broke → how data drove the call → the outcome (measured) → what you'd do differently. Mirror the shape of the Netflix talks. Use gw-04/gw-05 as concrete models if you lack a story of your own scale yet.

• "How would you roll out a risky new gateway behavior to the whole fleet?" The ladder: flag it → shadow/mirror and diff → sticky canary with automated analysis → ramp by region/cluster with SLO+NACK-gated auto-rollback → soak → decommission. Name the blast-radius staging and the warm-old-path rollback.

• "How do you validate a change without risking users?" Traffic mirroring at the gateway: real prod inputs hit the new path, responses discarded/diffed, users served from the old path. Zero user risk, production-fidelity evidence. (Envoy mirrors natively; gw-03 forks in the endpoint filter.)

• "How do you get other teams to migrate?" Make the new path strictly better (easier/faster/safer) so adoption is pulled; set clear expectations and a deprecation date; provide tooling/automation for the migration; and report progress transparently. Push only as a last resort.

• "A canary looks slightly worse — promote or roll back?" Default to the SLO/error-budget math (gw-11): if the canary degrades the SLI beyond noise, roll back and investigate; don't "push through" on a hunch. Decisions are data-driven, which is exactly what the JD asks.

• "How do you move workloads onto Kubernetes without a big-bang rewrite?" Find the extension point (NRI/OCI hooks, gw-09) that lets you migrate incrementally while staying on standard K8s; run old and new substrates side by side; migrate cluster by cluster. The principle: prefer an incremental path through an extension point over a fork.

• "What does 'done' mean for a migration?" Soaked at 100%, old path decommissioned, flags removed, runbooks/dashboards/alerts updated, partners fully off the deprecated path. Not "it's at 100%."

8. Connections to other labs

• Everything. This capstone is how you ship gw-01…gw-11 to a real fleet. Each prior lab supplies a piece:
  • gw-03 — the data plane you're migrating (Zuul 1→2 is the archetype).
  • gw-04 / gw-05 — staged rollout, pre-warming, jitter, and drain are the blast-radius controls; both are real Netflix migrations.
  • gw-06 — resilience guardrails keep a bad canary from cascading.
  • gw-08 / gw-10 — flags/config and CRD versioning are the migration machinery (dynamic push, conversion webhooks); xDS canaries by node.
  • gw-09 — PodDisruptionBudgets, drain ordering, and NRI/OCI hooks make the substrate migration safe.
  • gw-11 — SLOs, error budgets, and canary analysis are what gate every rung of the ladder.
• db-16…20 (consensus) — staged, ordered, reversible state change with quorum-style safety is the same mindset as committing a replicated log; a migration is "change the cluster's state without ever being in an unsafe intermediate state."