gw-03 — Building an API Gateway: The Zuul Model

This is the lab the role is named after. An API gateway is the single front door for a fleet of services: it terminates the client protocol (gw-02), authenticates, routes each request to the right backend, applies policy (rate limits, header rewrites), proxies the request with resilience (gw-06), and emits observability (gw-11). At Netflix this is Zuul — 80+ clusters, fronting ~100 backend service clusters, more than 1M requests per second. The "Evolution of Edge @ Netflix" talk is the story of how that gateway went from a blocking, thread-per-request design (Zuul 1) to an asynchronous, non-blocking, Netty-based design (Zuul 2).

You will build a gateway in the Zuul 2 shape: a non-blocking acceptor (from gw-01) wrapping a filter chain with three phases — inbound, endpoint, outbound — and a routing table. The filter-chain model is the single most important architectural idea in this phase, because it's how you "design, develop, and integrate functionality within the data plane" — the exact words of the JD.

1. What is it?

An API gateway is a reverse proxy with a programmable request lifecycle. Every cross-cutting concern that you'd otherwise duplicate in every service — auth, routing, rate limiting, retries, logging, header hygiene, canary steering — is implemented once, as a filter, and runs on every request at the edge.

Zuul's model (which Envoy, Spring Cloud Gateway, and the Kubernetes Gateway API all echo) decomposes the lifecycle into ordered phases:

            ┌──────────────────────── ZUUL ────────────────────────┐
 client ──▶ │ INBOUND filters ─▶ ENDPOINT (route) ─▶ OUTBOUND filters│ ──▶ client
            │   authn/authz          pick origin        response       │
            │   routing decision     proxy the call      rewrite        │
            │   rate limit           (or serve locally)   logging       │
            └────────────────────────────┬──────────────────────────┘
                                          │ proxy
                                          ▼
                                   origin service cluster

• Inbound filters run on the request before routing: decode, authenticate, decorate, decide where it goes, reject early (rate limit / WAF).
• Endpoint filters are the terminal action: usually "proxy to the chosen origin," but can be "serve a static/health response" or "return an error." This is where the origin call (gw-04, gw-06) happens.
• Outbound filters run on the response: rewrite headers, strip hop-by-hop headers, record metrics, emit the access log.

The other half is the threading model. Zuul 2 runs on Netty: a small pool of event-loop threads, each owning many connections, each never blocking. This is the gw-01 acceptor generalized — and the source of the "you must never block the event loop" discipline.

2. Why does it matter?

• It's the literal job. "Designing, developing, and integrating functionality within our data plane and control plane to uplift the security, observability, and resilience posture" is writing filters and the machinery that runs them. Everything else in this phase plugs into this model.

• The blocking→async transition is the team's defining story. Zuul 1 used one thread per request; under load, threads pile up waiting on slow origins and the gateway falls over (thread-pool exhaustion). Zuul 2 uses an event loop so a slow origin parks a cheap continuation, not an expensive thread. ~25% throughput gain, ~25% less CPU. If you understand why (gw-01's C10K), you can speak to it credibly.

• The filter chain is how you reason about ordering and blast radius. "Where does auth run relative to rate limiting? Should the WAF run before or after routing? If an inbound filter throws, do outbound filters still run?" These are daily design-review questions, and the filter model gives you a precise vocabulary for them.

• It's the integration point for every other lab. Security (gw-07) is an inbound filter; the connection pool (gw-04) and resilience (gw-06) live in the endpoint filter; observability (gw-11) is an outbound filter plus instrumentation throughout.

3. How does it work?

The request context

A single mutable request context flows through all phases, carrying the request, the (eventual) response, the chosen route, attributes set by earlier filters, and timing. In Zuul this is SessionContext.

RequestContext {
    request    : the inbound HTTP request (method, path, headers, body stream)
    route      : chosen origin + metadata (set by a routing filter)
    response   : the response being built (set by endpoint/outbound)
    attributes : map for cross-filter data (e.g. authenticated identity)
    timings    : per-phase latency for the access log
    shouldStop : early-exit flag (a filter can short-circuit the chain)
}

Filter shape

Every filter declares its type (inbound/endpoint/outbound), an order (filters run sorted within a phase), a shouldFilter predicate (does it apply to this request?), and an apply body. This is exactly Zuul's ZuulFilter interface:

Filter {
    type()        -> Inbound | Endpoint | Outbound
    order()       -> int                  # lower runs first
    shouldFilter(ctx) -> bool             # cheap applicability check
    apply(ctx)                            # the work
}

The async discipline (the heart of Zuul 2)

On an event loop you must never block. A filter that calls an origin returns a future/promise, and the framework resumes the chain when it completes — the event-loop thread is freed to service other connections meanwhile.

inbound chain (sync, cheap) ─▶ endpoint filter returns Future<Response>
        event loop is now FREE to serve other connections
        ... origin responds ...
        event loop resumes ─▶ outbound chain ─▶ write response

In Go, goroutines + the netpoller give you this for free (a blocked goroutine doesn't block an OS thread). In Java/Netty you do it explicitly with CompletableFuture/Observable and you must be disciplined: one synchronous DB call or blocking lock on the event-loop thread stalls every connection that thread owns. This is the #1 operational hazard of the model and a guaranteed interview topic.

Routing

The endpoint phase needs a route: map the request to an origin cluster. A routing table maps a matcher (host + path prefix + method + header predicates) to a cluster name; the cluster resolves to endpoints via service discovery (gw-08) and gets balanced + pooled (gw-04, gw-06).

route table (longest-prefix / first-match wins):
  Host=api.x.com  Path=/v1/play/*    -> cluster "playback"
  Host=api.x.com  Path=/v1/search/*  -> cluster "search"
  Header x-canary=true  Path=/v1/*   -> cluster "playback-canary"   (gw-12)
  default                            -> cluster "edge-fallback"

Routing is dynamic: the control plane pushes table updates to the data plane without a redeploy (gw-08). That push/no-redeploy property is what makes a gateway operable at fleet scale.

Where each concern lives

4. Core terminology

5. Mental models

• A gateway is middleware with a deploy boundary. A filter chain is the same idea as web-framework middleware (func(next) handler), but it runs as its own fleet in front of every service, so a change is one deploy instead of N. That centralization is the value and the risk — a bad filter breaks everything at once (gw-12 is about shipping changes safely).

• The event loop is a kitchen pass, not a line of cooks. One expediter (event-loop thread) watches every order (connection) and hands off work the instant it's ready. If the expediter stops to chop an onion (a blocking call), the whole pass halts. The async rule ("never block the loop") is "the expediter never picks up a knife."

• Filters are a pipeline with a kill switch. Each stage can pass the request along, modify it, or stop the line and return a response. Auth that fails stops the line before routing — you never want to open an origin connection for a request you're going to 401.

• Routing is a routing table, not an if/else. Treat it as data pushed from a control plane, matched by precedence, hot-reloadable. The moment routing lives in code you redeploy to change a route, and at fleet scale that's the difference between minutes and a day.

6. Common misconceptions

• "Async is just for performance." It changes correctness under overload. A thread-per-request gateway with slow origins exhausts its thread pool and stops serving fast origins too (head-of-line at the pool). The event-loop model degrades gracefully because a slow origin costs a cheap parked continuation, not a thread.

• "A gateway should do everything." Centralizing too much logic makes the gateway a monolith with fleet-wide blast radius and a bottleneck team. The discipline is: cross-cutting infrastructure concerns at the gateway; business logic in services. (Netflix's history with this is instructive — they pushed dynamic Groovy filters, then pulled back toward typed, reviewed filters.)

• "Filters run in the order I wrote them." They run in declared order within their phase. Auth (inbound, order 10) always precedes routing (inbound, order 50) regardless of source layout. Getting ordering explicit is the whole point.

• "Blocking once is fine." On an event loop, one blocking call on the loop thread stalls every connection that thread owns — possibly thousands. There is no "just this once." Offload blocking work to a separate executor or make it async.

• "The gateway is stateless." Mostly — but it holds connection pools, rate-limiter token buckets, circuit-breaker state, and HPACK tables. That state is why draining and config push need care.

7. Interview talking points

• "Design an API gateway." Lead with the data/control-plane split, then the filter-chain lifecycle (inbound→endpoint→outbound), then the threading model (event loop, never block), then where resilience and observability hook in. Name Zuul's three phases explicitly. This single answer demonstrates the whole phase.

• "Why did Zuul go async/Netty, and what did it cost?" Thread-per- request exhausts under slow origins (head-of-line at the thread pool); event loop parks cheap continuations instead. Gain: ~25% throughput and CPU. Cost: every filter must be non-blocking; debugging async stacks is harder; one blocking call poisons a loop. The honest cost/benefit is the senior answer.

• "Where does auth run, and why before routing?" Inbound, early order, before the routing/endpoint phase — so you never open an origin connection or do work for a request you'll reject. Short-circuit on failure. This shows you think about blast radius and resource use.

• "How do you change a route without a redeploy?" Routing is data pushed from the control plane (gw-08); the data plane hot-reloads it atomically. Discuss propagation safety (validate before apply, version configs, roll out incrementally) — that's the consensus intuition from db-17 applied to config.

• "A filter needs to call an external service (e.g., authz). How, without wrecking latency?" Make it async with a tight timeout and a fallback (fail-open vs fail-closed is a security decision — gw-07); cache decisions; never block the event loop; budget the added latency against the request SLO.

• "How do you safely roll out a risky new filter to the whole fleet?" Flag-gate it, shadow it (run it, don't act on the result, compare), canary a small % of traffic, watch RED metrics, auto-roll- back on SLO breach (gw-11, gw-12). Acknowledge the fleet-wide blast radius explicitly.

8. Connections to other labs

• gw-01 (L4) is the acceptor + event loop this gateway wraps; the threading discipline starts there.
• gw-02 (L7) is the protocol decoding the inbound phase depends on.
• gw-04 (connection management) lives in the endpoint phase — the pooled, subsetted origin call.
• gw-06 (resilience) is endpoint-phase logic: retries, circuit breaking, adaptive concurrency, load shedding.
• gw-07 (security) is the inbound authn/authz filter + TLS termination.
• gw-08 (Envoy/xDS) is the control plane that pushes this gateway's routes and clusters; Envoy's HCM + filter chain is the production twin of this lab.
• gw-11 (observability) is the outbound access-log/metrics/trace filter plus instrumentation throughout.
• gw-12 (migration) is how you ship changes to this fleet safely.