gw-07 — The Hitchhiker's Guide to Edge Security (mTLS & Zero-Trust)

Companion to CONCEPTS.md, with the runnable mTLS gateway in src/go/mtls/. Backed by the "Securing Netflix Studios at Scale" talk: high-value content traffic protected by strong identity and policy at the edge.

Zero-trust replaces "trust the network" with "trust nothing; verify everything." Concretely at the edge: every connection proves who it is with a certificate (mTLS), and every request is authorized against a policy. This lab builds all of it from crypto/tls and crypto/x509, including a tiny CA so the whole thing runs offline.

1. Identity: a CA and SPIFFE SVIDs (certs.go)

In production, identity comes from SPIRE or Netflix's Metatron, which attest a workload and issue it a short-lived certificate (an SVID) whose subject is its identity. We model that with a CA that issues leaves carrying a SPIFFE ID in the certificate's URI SAN:

cert, _ := ca.Issue("asset-service", "spiffe://netflix/studio/asset-service", false, time.Hour)

TestIssueAndExtractSpiffe issues such a cert and recovers the identity with SpiffeIDFromState. The key facts:

  • The identity lives in the URI SAN (spiffe://trust-domain/path), not the CN — that's the SPIFFE convention.
  • The cert is short-lived (ttl is small on purpose). Short certs make revocation a non-problem: a leaked cert expires in hours, so you trade a hard revocation problem (CRL/OCSP, which rarely works well) for a tractable rotation problem (§3). This is the modern, more reliable default — say that in an interview.
  • ExtKeyUsage differentiates server (ServerAuth) from client (ClientAuth) certs; the same CA issues both.

2. Mutual TLS (server.go, identity.go)

ServerTLSConfig is the heart:

&tls.Config{
    MinVersion:     tls.VersionTLS13,
    GetCertificate: holder.get,                  // hot rotation (§3)
    ClientAuth:     tls.RequireAndVerifyClientCert, // mTLS: DEMAND a verified client cert
    ClientCAs:      clientCAs,
    NextProtos:     []string{"h2", "http/1.1"},  // ALPN
}

RequireAndVerifyClientCert is what makes it mutual: the server won't complete the handshake unless the client presents a certificate chaining to a trusted CA. TestMTLSEndToEnd proves all three outcomes against a real TLS listener:

  • authorized identity + allowed path → 200 (and the handler sees the identity in X-Spiffe-Id),
  • valid cert but wrong identity for the route → 403,
  • no client cert → the handshake itself fails (the request never reaches the handler).

That last one is the zero-trust property in one assertion: an unauthenticated client cannot even open a connection, let alone send a request.

Authorization and the trust boundary

Authorize middleware pulls the identity from r.TLS — the cryptographically verified peer certificate — and checks a Policy (longest-prefix path → allowed SPIFFE IDs, default-deny). TestPolicyAllow covers allow/deny/wildcard/default-deny.

The single most important security rule in this whole phase: identity comes from the verified cert, never from a client-controlled header. A handler that trusts an inbound X-User or X-Spiffe-Id from the client is trivially spoofable. We set X-Spiffe-Id after verifying, to pass identity to trusted downstreams — but we'd strip any inbound copy first. This is the exact same trust boundary as the PROXY protocol (gw-01) and X-Forwarded-For (gw-03).

The fail-open vs fail-closed decision (what to do if an external authz dependency is down) is a per-route judgment call by data sensitivity — fail-closed for pre-release studio assets, perhaps fail-open with degraded scope for a public catalog page. Articulating that trade-off is the senior signal; the lab keeps authz local (no external dependency) so the question is yours to design (exercise §6.4).

3. Zero-downtime rotation (server.go)

Short certs mean you rotate constantly — so rotation must never drop a connection. CertHolder stores the active cert in an atomic.Pointer read by the TLS config's GetCertificate callback. A rotation agent calls Set with a fresh cert; new handshakes use the new cert while existing connections are untouched. TestHotRotation confirms the active serial changes on Set; the mtlsgw CLI rotates every 30s live.

This is the same atomic-swap, no-dropped-request pattern as gw-03 route reload and gw-04 membership — once you see it three times, you own it. In Envoy this is SDS (Secret Discovery Service) pushing certs over xDS (gw-08); the mechanism here is the local version of that.

4. Why this ties to connection churn (gw-04)

Every new mTLS connection pays a doubled asymmetric handshake (both sides verify a cert). So mTLS makes gw-04's connection reuse a security-cost optimization, not just a latency one: pooling and h2 multiplexing amortize the expensive handshake over many requests. "mTLS is too expensive at scale" is really "connection churn under mTLS is expensive" — and the fix is gw-04, plus TLS 1.3 session resumption.

5. Hands-on

cd src/go
bash ../scripts/verify.sh        # tests -race

go run ./cmd/mtlsgw              # generates a CA + certs, prints a curl command
# in another shell, use the printed command:
curl --cacert /tmp/gw07-ca.crt --cert /tmp/gw07-client.crt --key /tmp/gw07-client.key \
     https://127.0.0.1:8443/studio/assets        # 200, authorized as the SPIFFE id
curl --cacert /tmp/gw07-ca.crt https://127.0.0.1:8443/studio/assets   # TLS error: no client cert
# watch the log: the server cert rotates every 30s with zero dropped connections.

Inspect a cert to see the SPIFFE URI SAN:

openssl x509 -in /tmp/gw07-client.crt -text -noout | grep -A1 'Subject Alternative'

6. Exercises

  1. mTLS to the origin too: have the gateway present its own SVID when dialing the origin (end-to-end identity), and propagate the client identity downstream in request context.
  2. JWT for end users: add local JWT validation (verify signature against cached JWKS, check exp/aud/scopes) for human traffic, with no per-request call to the auth service. Combine with mTLS for service-to-service.
  3. Fail-open vs fail-closed: add an external authz check with a timeout and make the failure policy per-route; demonstrate both behaviors and justify the default for a sensitive vs a public route.
  4. SNI passthrough: implement an L4 path that routes by SNI without terminating TLS (keeping splice, gw-01) for traffic that must stay end-to-end encrypted; contrast with termination's L7 powers.
  5. Rotation under load: drive wrk/curl in a loop against mtlsgw across several rotations and confirm zero failed requests — proving the atomic swap is truly zero-downtime.