gw-02 — The Hitchhiker's Guide to L7 Protocols

Companion to CONCEPTS.md, with the runnable HTTP/2 toolkit in src/go/h2/. You will decode the same bytes nghttp -v shows — but from first principles, so you can debug an h2 problem nobody else on the team can.

L7 is where a proxy stops being a dumb pipe (gw-01) and starts understanding traffic: routing per request, retrying, rewriting, observing. The price is that you must parse the protocol, and HTTP/2 is where most engineers' mental model is fuzziest. This guide makes it concrete by building the three mechanisms that make h2 different — multiplexed binary framing, HPACK header compression, and flow control — and then connecting them to the real-world traps: the gRPC load-balancing pitfall and HTTP/3.

1. The frame layer (frame.go)

Everything in h2 is a frame: a 9-byte header + payload.

+-----------------------------------------------+
| Length (24)                                   |
+---------------+---------------+---------------+
| Type (8)      | Flags (8)     |
+-+-------------+---------------+---------------+
|R| Stream Identifier (31)                      |
+=+=============================================+
| Frame Payload (Length bytes)                ...

ReadFrame is a faithful decode: read 9 bytes, pull the 24-bit length, the type, the flags, and the 31-bit stream id (masking the reserved high bit — forgetting that mask is a classic bug that yields astronomical stream IDs). WriteFrame is the inverse, so the package round-trips (see TestFrameRoundTrip).

The teachable property is multiplexing: frames for many streams interleave on one connection, demultiplexed by StreamID. TestReadFramesMultiplex writes HEADERS for stream 1, HEADERS for stream 3, then DATA for each — interleaved — and reads them back in order. One TCP connection, two concurrent requests. That is the entire reason h2 killed the h1 connection-pool churn problem (and why gw-04's churn work leans on h2 to the origin).

Frame types to know by heart: HEADERS (start a request/response), DATA (body), SETTINGS (startup config), WINDOW_UPDATE (flow-control credit), RST_STREAM (cancel one stream), GOAWAY (graceful connection drain — the h2 equivalent of gw-01's connection drain, but in-band), and CONTINUATION (header block spillover — and the vector for the "Rapid Reset"/CONTINUATION-flood DoS class).

MaxFrameLen guards the 24-bit length so a hostile peer can't make you allocate gigabytes — the kind of defensive detail a maintainer adds after the first fuzzing report.

2. HPACK (hpack.go) — the stateful one

Headers repeat enormously (:authority, user-agent, cookies on every request). HPACK compresses them with two tables:

  • a static table of 61 common header fields (RFC 7541 Appendix A), fully reproduced in staticTable;
  • a per-connection dynamic table that both sides build as they talk: the first time you send a header it's spelled out and remembered; after that it's a one-byte index.

Integer coding (the part everyone gets subtly wrong)

HPACK integers use an N-bit prefix then continuation bytes (§5.1). EncodeInt/decodeInt implement it; TestEncodeDecodeInt checks the RFC §C.1 vectors exactly: 10 in a 5-bit prefix is 0x0a; 1337 is 0x1f 0x9a 0x0a; 42 in an 8-bit prefix is 0x2a. The continuation- byte loop has an overflow guard — an unbounded integer is a DoS.

The dynamic table, proven against the RFC

TestHpackRFCRequestExamples decodes the actual RFC 7541 §C.3.1 and §C.3.2 request blocks with one Decoder instance. The second request uses byte 0xbe — an index into the dynamic table — to refer to the :authority: www.example.com that the first request added. If the decoder didn't persist its table across HEADERS frames, this would decode to garbage. That is the single most important HPACK fact for a proxy engineer:

The dynamic table is per-connection shared state between the encoder and decoder. If your gateway rewrites a header (adds x-forwarded-for, strips a hop-by-hop header), it must re-encode with its OWN encoder whose table tracks what it actually sent to the origin — never blindly forward the client's HPACK block. Desync the table and every subsequent header on that connection corrupts.

TestHpackDynamicEviction proves the size accounting and eviction (each entry costs len(name)+len(value)+32 bytes; over maxSize the oldest entries are evicted). Bounding maxSize is your defense against an HPACK-table-bloat DoS.

Huffman, honestly

Real captures Huffman-code header strings. This lab decoder handles integers, literals, and the dynamic table — everything that teaches the mechanics — and surfaces Huffman explicitly (ErrHuffman, TestHpackHuffmanSurfaced) rather than guessing. For production header parsing you use golang.org/x/net/http2/hpack, which includes the 257-symbol Huffman table. The lesson here is the table mechanics; the Huffman codec is a (large, mechanical) lookup you should know exists, not re-implement under interview pressure.

3. Flow control (flow.go) — backpressure, again

h2 adds application-level flow control on top of TCP's: two credit windows, one per stream and one per connection. A sender may transmit DATA only while both are positive; each byte debits both; a WINDOW_UPDATE credits one.

TestFlowControl walks the subtle part: drain a stream to zero and a WINDOW_UPDATE on the stream alone is not enough if the connection window is also zero — you must credit both. That two-level model is why a single greedy stream can't starve the others, and why a misconfigured SETTINGS_INITIAL_WINDOW_SIZE is a throughput bug on high-BDP (bandwidth × delay) links: the window caps in-flight bytes ≈ throughput × RTT, so a too-small window throttles a fast, far link.

This is the same backpressure principle as gw-01's coupled copy, but expressed as explicit credits in the protocol instead of a blocking Write. Recognizing that "flow control" at L4 (TCP receive window), L7 (h2 WINDOW_UPDATE), and in your proxy's buffers (gw-01) are the same idea at three layers is a senior insight.

4. The demux (demux.go) — multiplexing made visible

Demux reconstructs per-stream state from an interleaved frame stream: stream phase (idle → open → half-closed → closed), header count, data bytes, and flow-control debits. TestDemuxMultiplexing feeds HEADERS+ DATA for streams 1 and 3 and asserts ActiveStreams() == [1, 3] — two requests concurrently in flight on one connection — and that an END_STREAM flag moves a stream to half-closed.

The CLI h2inspect wraps this: pipe it a raw h2 byte stream and it prints a frame trace + decoded headers + the multiplexed stream set, the same view as nghttp -v but from your own parser.

5. The gRPC load-balancing trap (the interview favorite)

gRPC is just HTTP/2 with conventions: :path = /package.Service/Method, content-type: application/grpc, length-prefixed protobuf in DATA frames, and — critically — status in trailers (a second HEADERS frame after the body carrying grpc-status). So HTTP 200 + grpc-status 14 is a failed call. A gateway that classifies success by :status alone reports a broken backend as healthy (this matters for gw-06 retries and gw-11 SLOs).

The trap: h2 multiplexes all calls over one long-lived connection, so a connection-level (L4, gw-01) load balancer pins every call to one backend. You need L7/per-request balancing (gw-06), or client-side LB with one subchannel per backend, or MAX_CONNECTION_AGE to force periodic reconnects — which reintroduces exactly the connection churn gw-04 fights. steps/03 reproduces the skew. Be ready to draw this on a whiteboard.

6. HTTP/3 in one breath

h2 fixed h1's application-level head-of-line blocking but not the transport-level kind: it's all one TCP connection, so one lost segment stalls every stream until retransmit. HTTP/3 moves the transport into userspace over UDP (QUIC): independent per-stream loss recovery (no TCP HOL), TLS 1.3 folded into the handshake (1-RTT, 0-RTT resume), and connection migration (a QUIC Connection ID, not the 4-tuple, identifies the connection — so a phone switching Wi-Fi→cellular keeps its connection). For the edge this is huge for mobile (Netflix devices) and painful for L4 LBs that hash the 4-tuple. Header compression becomes QPACK (HOL-blocking-safe). You won't implement QUIC here, but you must be able to say why it exists: to kill the residual TCP HOL that h2's single connection couldn't.

7. Hands-on

cd src/go
bash ../scripts/verify.sh   # or: go test -race ./...

# Build a header block in code and watch it decode (a tiny program, or
# adapt TestHpackEncodeDecodeRoundTrip). Then inspect a real stream:
go build -o /tmp/h2inspect ./cmd/h2inspect

# Generate a cleartext h2c capture with curl/nghttp and pipe it in:
#   nghttp -v --no-tls http://127.0.0.1:9000/ -m 4   # multiplex 4 reqs
# (real headers are Huffman-coded; h2inspect prints frames + marks
#  [huffman] for header blocks — the framing/demux still works.)

8. Exercises

  1. Add Huffman decode for the static-table-only case using golang.org/x/net/http2/hpack behind a build tag, and feed h2inspect a real curl --http2-prior-knowledge capture. Confirm your frame trace matches nghttp -v.
  2. Reproduce the gRPC LB skew (steps/03): one channel, 1000 calls, count per-backend hits through an L4 LB vs an L7 path. Quantify it.
  3. Make a too-small window throttle throughput: lower the initial window in FlowController, simulate a high-RTT link by delaying WINDOW_UPDATEs, and show throughput ≈ window/RTT.
  4. Detect a trailers-only failure: extend Demux to capture the second (trailer) HEADERS frame and surface grpc-status; prove a 200-but-failed call is detectable. This is what gw-11 needs.
  5. Implement GOAWAY draining: track the last-processed stream id, reject new streams after a GOAWAY, and let in-flight finish — the h2 analog of gw-01's connection drain.