gw-02 — L7 Protocols: HTTP/1.1, HTTP/2, gRPC, HTTP/3

Where gw-01 forwards an opaque byte stream, this lab terminates the application protocol: it parses HTTP, understands requests and responses, and can therefore route, retry, rate-limit, and rewrite per request instead of per connection. That is the entire reason an L7 gateway exists. The JD asks for "deep expertise in L7 (HTTP/S, gRPC, WebSockets)" — this lab covers HTTP/1.1, HTTP/2, gRPC, and HTTP/3; WebSockets get their own lab (gw-05) because the persistent-connection proxy is a distinct beast at Netflix (Pushy).

You will hand-write an HTTP/2 frame parser and demonstrate stream multiplexing, HPACK, and flow control — the three things that make h2 fundamentally different from h1 and that every gateway engineer must be able to reason about.

1. What is it?

Layer 7 is the application layer: the protocol carries semantic messages (an HTTP request with a method, path, headers, body), not just bytes. An L7 proxy decodes those messages, which unlocks:

• per-request routing (path/header/method-based), not per-connection;
• retries and hedging (you know where a request starts and ends);
• header manipulation, auth, rate limiting (you can read/modify the message);
• observability per request (status code, latency, route).

The cost is real: you must parse, you hold message state, you can't zero-copy splice, and you inherit every protocol's quirks. The four protocols you must know:

HTTP/1.1   text, one request per connection at a time (pipelining is dead)
HTTP/2     binary frames, many streams multiplexed over one TCP conn
gRPC       RPC framing on top of HTTP/2 (length-prefixed protobuf messages)
HTTP/3     HTTP semantics over QUIC over UDP — fixes TCP head-of-line blocking

2. Why does it matter?

• The gateway's value lives at L7. Routing 1M+ rps to ~100 backend clusters (the Zuul number) is a per-request decision. Retries, circuit breaking, and header-based canary routing (gw-06, gw-12) are all L7. If you can't parse the protocol you can't do the job.

• HTTP/2 multiplexing breaks naive load balancing. With h1, one connection carries one request at a time, so an L4 LB spreading connections roughly spreads load. With h2, one long-lived connection carries hundreds of concurrent streams — so an L4 LB pins all of them to one backend. This is why gRPC (which is h2) needs L7 / per- request LB, and it's a favorite interview trap.

• gRPC is the internal-RPC default at cloud-native shops. A gateway that proxies gRPC must understand h2 framing, the grpc-status trailer, and the difference between transport errors and gRPC status codes. Many "my gRPC load balancing is broken" incidents are really "an h1-era L4 LB in the path."

• HTTP/3 / QUIC is the edge frontier. It moves the transport into userspace over UDP to kill TCP-level head-of-line blocking and speed up connection setup (0-RTT). The edge team will care because it changes everything about connection management, NAT, and load balancing (UDP flows, connection IDs that survive IP changes).

3. How does it work?

HTTP/1.1 — text, and the framing problem

A request is text headers + optional body. The only hard part is knowing where the body ends:

POST /v1/play HTTP/1.1\r\n
Host: api.netflix.com\r\n
Content-Length: 27\r\n          ← body length known up front
\r\n
{"title":"Stranger Things"}

…or, when length isn't known up front, chunked transfer encoding:

Transfer-Encoding: chunked\r\n
\r\n
1b\r\n                          ← next chunk is 0x1b = 27 bytes
{"title":"Stranger Things"}\r\n
0\r\n                           ← zero-length chunk = end
\r\n

Two ways to delimit a body → the source of request smuggling vulnerabilities when a front proxy and a back proxy disagree about which one wins (CL.TE / TE.CL). A gateway must normalize this. Pipelining (multiple requests in flight on one h1 connection) exists in the spec but is effectively dead because of head-of-line blocking; real h1 reuse is serial keep-alive (one at a time, connection stays open — the thing gw-04 pools).

HTTP/2 — binary frames over one connection

h2 multiplexes many logical streams over one TCP connection using binary frames. Every frame shares a 9-byte header:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                 Length (24 bits)              |   Type (8)    |
+---------------+---------------+---------------+---------------+
|   Flags (8)   |R|         Stream Identifier (31)             |
+---------------+-----------------------------------------------+
|                   Frame Payload (Length bytes)               ...

Frame types you must know: HEADERS (start a request/response, carries HPACK-compressed headers), DATA (body), SETTINGS (connection config exchanged at startup), WINDOW_UPDATE (flow control credit), RST_STREAM (cancel one stream), GOAWAY (graceful connection shutdown — critical for draining, gw-01/gw-04), PING, PRIORITY, PUSH_PROMISE (server push, now deprecated).

Three mechanisms make h2 different and are the interview core:

1. Multiplexing. Each request/response is a stream with an odd (client-initiated) ID. Frames from many streams interleave on the wire, demuxed by Stream Identifier. One connection, hundreds of concurrent requests — no h1 connection-pool churn (the gw-04 win is partly "use h2 to the origin").

2. HPACK header compression (RFC 7541). Headers repeat enormously (:authority, user-agent, cookies). HPACK keeps a dynamic table of previously-seen header fields on each side; a header can be sent as a 1-byte index into that table. Stateful, per-connection, and the source of subtle bugs when a proxy rewrites headers (you must keep the encoder/decoder tables consistent). The static table is 61 predefined common headers.

3. Flow control. Per-stream and per-connection credit windows (WINDOW_UPDATE). A receiver advertises how many bytes it'll accept; the sender must not exceed it. This is application-level backpressure layered on top of TCP's — and a misconfigured window is a classic throughput bug (a slow consumer stalls a stream, or tiny windows cap throughput).

The HOL-blocking subtlety: h2 fixes application-level head-of-line blocking (a slow stream no longer blocks others at the h2 layer) but NOT transport-level HOL — because it's all one TCP connection, a single lost TCP segment stalls every stream until retransmit. That residual problem is exactly what HTTP/3 solves.

gRPC — RPC on top of h2

gRPC is a convention over HTTP/2:

:method = POST
:path   = /package.Service/Method
content-type = application/grpc
DATA frames carry: [1-byte compressed-flag][4-byte length][protobuf message]...
trailers (a second HEADERS frame) carry: grpc-status, grpc-message

Key gateway facts: gRPC uses trailers for status, so an HTTP 200 can still be a gRPC error (grpc-status: 14 = UNAVAILABLE). Streaming RPCs keep the stream open for many messages. Load balancing gRPC requires L7 / per-request distribution (the multiplexing trap above).

HTTP/3 / QUIC — HTTP over UDP

QUIC (RFC 9000) reimplements reliable, ordered, multiplexed, encrypted streams in userspace on top of UDP, with TLS 1.3 built into the handshake. HTTP/3 (RFC 9114) is HTTP semantics over QUIC, with QPACK (RFC 9204) as the HOL-blocking-safe replacement for HPACK. What changes for a gateway:

• No TCP HOL blocking — each QUIC stream has independent loss recovery, so one lost packet stalls only its own stream.
• Faster setup — 1-RTT, or 0-RTT for resumed connections.
• Connection migration — a QUIC connection is identified by a Connection ID, not the 4-tuple, so it survives a client changing networks (Wi-Fi→cellular) without a new handshake. Great for mobile (Netflix devices); painful for L4 LBs that hash the 4-tuple.
• UDP everywhere — load balancing, NAT traversal, and DDoS posture all change; many middleboxes still mishandle UDP.

4. Core terminology

5. Mental models

• h1 is one phone line; h2 is a switchboard. With h1 keep-alive you reuse the line but must finish one call before the next. h2 runs many calls on one line at once, tagged by stream ID. h3 gives each call its own line so a dropped packet on one call doesn't mute the others.

• HPACK is a shared shorthand the two parties build as they talk. The first time you say a long header you spell it out and both sides write it down; after that you just say "row 7." Lose sync on the notebook (a proxy that rewrites headers carelessly) and you start decoding garbage.

• Flow control is a debit card, not unlimited credit. The receiver hands the sender a balance (the window); every DATA byte is a charge; WINDOW_UPDATE is a top-up. Forget to top up and the sender stops — a stall that looks like a network problem but is your bookkeeping.

• gRPC over an L4 LB is a party line. All the multiplexed calls pin to whichever backend the single connection landed on, so one backend gets hammered while others idle. The fix is to balance the calls, not the connection — i.e., L7.

6. Common misconceptions

• "HTTP/2 is faster because it's binary." Binary framing helps, but the real wins are multiplexing (no h1 connection churn / no HOL at the h1 layer) and header compression. And h2 adds a problem (TCP HOL across streams) that h3 had to fix.

• "HTTP/2 needs fewer connections, so any LB is fine." The opposite: because h2 uses one long-lived connection with many streams, an L4 LB pins all that load to one backend. h2/gRPC make L7 LB more necessary, not less.

• "gRPC errors show up as HTTP 5xx." No — gRPC status rides in trailers; you can get HTTP 200 with grpc-status: 14. A gateway measuring only HTTP status will report a broken backend as healthy.

• "HTTP/3 is just HTTP/2 over UDP." It re-implements the whole reliable-transport layer (QUIC), changes header compression (QPACK), adds connection migration, and folds TLS into the handshake. It's a new transport, not a swap of L4.

• "Chunked encoding is legacy." It's how you stream a response of unknown length (Server-Sent Events, streaming JSON, gRPC-web). And its interaction with Content-Length is a live security surface (smuggling) every gateway must normalize.

7. Interview talking points

• "What actually changed from HTTP/1.1 to HTTP/2 to HTTP/3?" Give the three-line answer: h2 = multiplexed binary streams + HPACK + flow control over one TCP conn (fixes h1 HOL and connection churn); h3 = the same over QUIC/UDP (fixes the residual TCP HOL, adds 0-RTT and connection migration). Name the cost each introduced.

• "Why does gRPC load balancing break behind a classic LB?" h2 multiplexes all calls over one connection; an L4 (connection-level) LB pins them to one backend. You need L7/request-level LB, or a proxy that re-balances per request, or client-side LB with periodic reconnects (MAX_CONNECTION_AGE).

• "Walk me through HTTP/2 flow control." Two levels (stream + connection), credit advertised via SETTINGS_INITIAL_WINDOW_SIZE and topped up via WINDOW_UPDATE; sender blocks at zero. It's application backpressure on top of TCP's. A too-small window caps throughput on high-BDP links; a stuck consumer stalls a stream.

• "How do you drain an HTTP/2 connection?" GOAWAY with the last-processed stream ID: the peer finishes in-flight streams, opens no new ones, then closes. Tie it back to gw-01 draining — for h2 the drain signal is in-band, which is better than h1 (where you rely on Connection: close per response).

• "What's request smuggling and how does a gateway prevent it?" Front and back proxies disagreeing on body length (Content-Length vs Transfer-Encoding). Defense: reject ambiguous framing, prefer one encoding, normalize before forwarding, and run identical parsing semantics front-to-back.

• "Why is HPACK stateful, and why does that matter for a proxy?" The dynamic table is per-connection shared state between encoder and decoder. A proxy that mutates headers must update its own encoder/decoder consistently, and must bound the table size (SETTINGS_HEADER_TABLE_SIZE) to avoid an HPACK-bomb DoS.

• "When would you push HTTP/3 to the edge?" Mobile/lossy networks (Netflix devices on cellular) where TCP HOL and reconnect cost hurt; weigh against UDP middlebox issues, higher CPU for userspace transport, and the operational cost of new LB/observability tooling.

8. Connections to other labs

• gw-01 (L4) forwards the bytes this lab parses; the moment you terminate L7 you lose splice zero-copy.
• gw-03 (API gateway) runs filters on the parsed request/response this lab produces — routing, auth, rewrite all assume L7 decoding.
• gw-04 (connection management) uses h2 multiplexing to the origin as one way to curb connection churn; GOAWAY is the drain primitive.
• gw-05 (WebSockets) is the other L7 upgrade path (Upgrade: websocket from h1, or CONNECT over h2) — a persistent connection, not a request/response.
• gw-06 (resilience) depends on per-request boundaries (retries, hedging) that only L7 gives you, and must understand gRPC trailers to classify failures correctly.
• gw-08 (Envoy/xDS) — Envoy's HTTP connection manager is a production version of this lab's parser + filter chain.