gw-07 — Security at the Edge: mTLS, Identity, and Zero-Trust

The JD names "uplift the security… posture for traffic traversing the cloud" as a primary responsibility, and links the talk "The Show Must Go On: Securing Netflix Studios at Scale." Studio/partner traffic (pre-release content, production assets) is among the most sensitive at Netflix, and the gateway is where its security posture is enforced. The modern answer is zero-trust: never trust the network, authenticate every connection cryptographically, and authorize every request.

This lab covers the security mechanisms a Cloud Gateway engineer owns: TLS 1.3 termination, mutual TLS (mTLS), workload identity (SPIFFE/SVID; Netflix's Metatron), certificate issuance and rotation, SNI-based routing and passthrough, and authn/authz at the gateway (token validation, the fail-open vs fail-closed decision). The throughline: identity is the new perimeter.

You will stand up mutual TLS between a client and the gateway, validate the client's identity from its certificate, and enforce an authorization policy — the concrete core of zero-trust at the edge.

1. What is it?

Zero-trust discards the old "hard shell, soft center" model (trust anything inside the network) for "trust nothing; verify everything." Every connection proves who it is with a cryptographic credential, and every request is authorized against a policy. At the edge this means:

• TLS termination — decrypt inbound TLS so the gateway can inspect and route (gw-02/gw-03). The gateway holds the server certificate and private key for the public hostnames.
• mutual TLS (mTLS) — both sides present certificates. The client proves its identity to the gateway (and gateway→origin mTLS proves the gateway to the origin). This is how service-to-service traffic authenticates without passwords or network-location trust.
• Workload identity — a short-lived certificate whose subject is the workload's identity (a SPIFFE ID like spiffe://netflix/studio/asset-service), issued automatically by an identity platform (SPIRE; Netflix's Metatron, which injects credentials at the continuous-delivery phase).
• Authorization — given an authenticated identity, decide if this identity may perform this request (which route, method, resource).

client/workload ──mTLS (both present certs)──▶ [GATEWAY]
   identity = SPIFFE ID from client cert        verify cert chain + identity
                                                authorize(identity, request)
                                                ──mTLS──▶ origin (gateway proves itself)

2. Why does it matter?

• It's a primary JD responsibility and a flagship talk. "Securing Netflix Studios at Scale" is about exactly this: protecting high-value content traffic with strong identity and policy at scale. Being fluent in mTLS, identity, and rotation is directly hireable.

• The edge is the enforcement point. Authenticating and authorizing at the gateway means every backend service inherits a strong security posture without each one re-implementing TLS, token validation, and policy. Centralization (gw-03) applied to security.

• Certificate operations are where security meets reliability. A cert that expires unrotated takes down a service — a self-inflicted outage that's depressingly common. Automated issuance + rotation (short-lived certs, hot reload without dropping connections) is an operational skill the role needs, and it ties back to draining (gw-01) and connection management (gw-04).

• The fail-open/fail-closed decision is a senior judgment call. If the authz service is down, do you let traffic through (fail-open: available but possibly unauthorized) or block it (fail-closed: secure but unavailable)? The answer depends on the traffic's sensitivity, and articulating that trade-off is exactly the judgment a "5" is hired for.

3. How does it work?

The TLS 1.3 handshake (what you must be able to narrate)

client                                   server (gateway)
  │ ClientHello (SNI, ALPN, key_share, cipher suites)  │
  │ ───────────────────────────────────────────────▶  │
  │ ServerHello (key_share, cert, Finished)            │  selects cert by SNI
  │ ◀───────────────────────────────────────────────  │  (one listener, many hosts)
  │ [for mTLS: CertificateRequest]                     │
  │ Certificate (client) + CertificateVerify + Finished│  client proves identity
  │ ───────────────────────────────────────────────▶  │  verify chain + SPIFFE ID
  │ ════════════ application data (1-RTT) ════════════ │

• SNI (Server Name Indication) tells the server which hostname the client wants before the cert is chosen — so one listener serves many domains, each with its own cert. SNI also enables TLS passthrough: an L4 proxy can route by SNI without decrypting (keeps splice, gw-01).
• ALPN negotiates the protocol (h2, http/1.1, h3) inside the handshake.
• mTLS adds the CertificateRequest + client Certificate + CertificateVerify — the client signs to prove it holds the private key for the cert it presented.

Workload identity: SPIFFE / SVID / Metatron

• A SPIFFE ID is a URI naming a workload: spiffe://trust-domain/path (e.g. spiffe://netflix/studio/asset-service). It lives in the certificate's SAN (Subject Alternative Name, URI type).
• An SVID (SPIFFE Verifiable Identity Document) is the X.509 cert carrying that ID, short-lived (hours), auto-rotated.
• SPIRE is the reference issuer; it attests a workload (proves it's really asset-service via node + workload attestation) before issuing its SVID.
• Metatron is Netflix's equivalent: it solves the credential bootstrap problem by injecting identity into each microservice at the continuous-delivery phase, so a service comes up already holding a rotatable identity — no secret-zero to leak.

Certificate rotation without dropping connections

Short-lived certs mean you rotate constantly. The gateway must:

1. Fetch the new cert/key before the old expires (a rotation agent / SDS — Secret Discovery Service in Envoy, gw-08).
2. Hot-swap it for new handshakes via an atomic pointer in the TLS config's GetCertificate callback — existing connections keep their session; new ones use the new cert.
3. Never block the event loop on a cert fetch (gw-03).

This is the same atomic-swap, no-dropped-request pattern as route hot-reload (gw-03) and pool membership (gw-04).

Authorization at the gateway

Once the identity is known (from the client cert, or a validated JWT/ token for end users), authorize the request:

authorize(identity, request):
    policy = lookup(route, method)
    if identity in policy.allowed_identities: ALLOW
    else: DENY (403)
    # decision may call an external authz service — async, timeout,
    # cached, with a fail-open/fail-closed policy per route sensitivity

For end-user traffic the gateway typically validates a signed token (JWT/PASETO): check signature, expiry, audience, scopes — locally (no per-request call) using the issuer's public keys (JWKS), refreshed periodically.

4. Core terminology

5. Mental models

• mTLS is showing ID at both ends, not just the bouncer showing theirs. Regular TLS: the club proves it's the real club (server cert), you walk in anonymously. mTLS: you also show ID, so the club knows exactly who you are and can apply your membership tier (authz).

• A SPIFFE ID is a passport; an SVID is the visa stamped in it. The identity is durable (spiffe://netflix/studio/asset-service); the document proving it (the cert) is short-lived and reissued constantly — so a stolen one expires fast, and revocation is "wait a few hours" instead of a fragile CRL/OCSP dance.

• Short-lived certs trade a revocation problem for a rotation problem. Long certs need revocation infrastructure (CRL/OCSP, often broken); short certs sidestep it but demand rock-solid automated rotation. You're choosing which hard problem to own — and rotation is the more tractable one.

• Fail-open vs fail-closed is a thermostat with a stuck sensor. If the sensor (authz service) dies, do you leave the heat on (fail-open, comfortable but maybe unsafe) or shut it off (fail-closed, safe but cold)? For a public catalog page you might fail-open; for pre-release studio assets you fail-closed. Same mechanism, opposite default by sensitivity.

6. Common misconceptions

• "TLS means we're secure." TLS authenticates the server and encrypts the channel; it says nothing about who the client is or what they're allowed to do. Zero-trust needs mTLS (client identity)

  • authorization (policy). Encryption ≠ authentication ≠ authorization.

• "mTLS is too expensive at scale." The handshake cost is real (asymmetric crypto), which is exactly why connection reuse and pooling (gw-04) matter — amortize the handshake over many requests. Session resumption and h2 multiplexing further reduce it. Churn is the enemy of cheap mTLS.

• "Long-lived certs are simpler." They move the cost to revocation (which usually doesn't work well in practice) and make a leaked key a long-lived disaster. Short-lived auto-rotated certs are the modern, more reliable default.

• "Validate the JWT on every request by calling the auth service." That couples every request to the auth service's availability and latency. Validate signatures locally against cached JWKS; only the rare key-rotation needs a fetch.

• "Fail-open is always safer for availability." It can be a security incident (serving unauthorized access to sensitive data). The right default depends on what's behind the route; treating it as a blanket rule is the mistake.

7. Interview talking points

• "How would you secure service-to-service traffic at the edge?" mTLS with short-lived, auto-rotated workload certs (SPIFFE/SVID; Metatron-style bootstrap at deploy time), identity from the cert SAN, per-route authorization, all enforced at the gateway so backends inherit it. Name the rotation + hot-reload requirement.

• "Walk me through the TLS handshake, and where mTLS changes it." ClientHello (SNI/ALPN/key_share) → ServerHello + cert → (mTLS: CertificateRequest → client Certificate + CertificateVerify) → Finished → 1-RTT data. The client's CertificateVerify signature is the proof it owns the key. SNI selects the server cert and enables L4 passthrough routing.

• "How do you rotate certs without dropping connections?" Fetch ahead of expiry (SDS/rotation agent), atomic-swap via the GetCertificate callback so new handshakes use the new cert while existing connections keep theirs — same no-dropped-request pattern as route hot-reload (gw-03). Never block the event loop on the fetch.

• "Fail-open or fail-closed if the authz service is down?" It depends on the data's sensitivity: fail-closed for studio/pre-release assets (security > availability), consider fail-open with degraded scope for low-sensitivity public traffic. Mitigate the dependency with local token validation + cached decisions so the question rarely arises. This trade-off articulation is the senior signal.

• "Why short-lived certs?" They make revocation a non-problem (a leaked cert expires in hours) at the cost of demanding automated rotation — the more reliable trade. And they fit zero-trust: identity is continuously re-proven.

• "How does mTLS interact with connection churn?" Every new connection pays a full asymmetric handshake; mTLS doubles the cert work. So mTLS makes gw-04's connection reuse a security-cost optimization, not just latency — a nice cross-lab connection to draw.

8. Connections to other labs

• gw-01 (L4) — SNI-based TLS passthrough keeps you at L4 (and keeps splice); the handshake cost is the socket-level reason churn hurts.
• gw-02 (L7) — ALPN negotiates h2/h3 in the handshake; termination is the prerequisite for L7 inspection.
• gw-03 (API gateway) — authn/authz is the inbound filter; cert hot-reload is the same atomic-swap pattern as route reload.
• gw-04 (connection management) — connection reuse amortizes the (expensive, doubled-for-mTLS) handshake.
• gw-08 (Envoy/xDS) — SDS pushes certs to the data plane; the control plane distributes identity and policy.
• gw-09 (Kubernetes) — pod identity, service-mesh mTLS, and secret distribution all live here; SPIRE runs as a node agent + workload API.