gw-02 step 03 — gRPC, trailers, and the load-balancing trap

Goal

Understand gRPC's wire format (it's just HTTP/2 with conventions), see why grpc-status lives in trailers, and reproduce the classic failure: gRPC behind an L4 load balancer pins all calls to one backend.

Background — gRPC on the wire

HEADERS  :method=POST  :path=/echo.Echo/Unary  content-type=application/grpc
DATA     [0][00 00 00 05][protobuf bytes...]    ← 1 flag byte + 4 len + msg
...                                              (more DATA for streaming)
HEADERS  grpc-status: 0  grpc-message:          ← TRAILERS (END_STREAM here)
  • The response status you care about is grpc-status (0 = OK), carried in trailers — a second HEADERS frame after the body.
  • Therefore HTTP 200 + grpc-status 14 is a failed call. A gateway that logs only :status reports it as success. (gw-11 must read trailers.)
  • Streaming RPCs keep the stream open for many DATA frames — the connection is long-lived and multiplexed.

Reproduce the L4 LB trap

Spin up 3 identical gRPC backends, put an L4 balancer in front (round-robin on connections), and a client that opens one channel (one h2 connection). Then send many calls.

# 3 backends on :9001..:9003 (use grpc's helloworld or any echo server)
for p in 9001 9002 9003; do grpc_echo_server --port $p & done

# L4 round-robin in front (your gw-01 proxy, or nginx stream, or envoy
# tcp_proxy) on :9000 -> {9001,9002,9003}
/tmp/l4proxy -listen :9000 -origin 127.0.0.1:9001   # connection-level pin

# One client channel, 1000 unary calls:
ghz --insecure --proto echo.proto --call echo.Echo/Unary \
    -c 1 -n 1000 127.0.0.1:9000

Observe: nearly all 1000 calls land on a single backend, because the one h2 connection was balanced once, at connect time, and all 1000 multiplexed streams ride it. Now compare:

  • L7 balancer (Envoy with http_connection_manager + a gRPC route, or any per-request proxy): the same 1000 calls spread ~evenly across the 3 backends, because each request is balanced.
  • Client-side LB (gRPC round_robin policy with multiple resolved addresses): also spreads, by opening one subchannel per backend.

Tasks

  1. Stand up 3 backends and reproduce the skew through an L4 LB; capture per-backend request counts.
  2. Swap in an L7 path (Envoy gRPC route is the realistic one — see gw-08) and show the distribution evens out.
  3. Make a backend return grpc-status: 14 while keeping HTTP 200. Confirm an L4/HTTP-only view calls it healthy, and that reading the trailer reveals the failure. Note where this matters for retries (gw-06) and SLOs (gw-11).

Acceptance

  • A reproduced, quantified skew (e.g. "987/1000 on backend A") through the L4 LB, and an even split through the L7/client-side path.
  • A demonstration that grpc-status in trailers can disagree with the HTTP status, and a one-paragraph explanation of the consequences for monitoring and retries.

Discussion prompts

  • Three fixes for "gRPC behind a connection-level LB": (a) L7/per-request proxy, (b) client-side LB with one subchannel per backend, (c) force periodic reconnects with MAX_CONNECTION_AGE so connections re-balance. When would you pick each?
  • Why does MAX_CONNECTION_AGE partially fix it but also reintroduce exactly the connection churn that gw-04 is trying to eliminate? (The tension between rebalancing and connection reuse is a real design trade-off — be ready to discuss it.)
  • A streaming RPC pins to one backend for its whole life. How do you drain a backend that holds long-lived gRPC streams? (Tie to GOAWAY and gw-01/gw-05 draining.)