gw-01 — The Hitchhiker's Guide to the L4 Data Plane

A long-form, hands-on companion to CONCEPTS.md. Read with the code in src/go/ open. Everything here is runnable and tested; nothing requires another book.

This guide takes you from "a socket is a file descriptor" to "I can operate a Netty-style L4 proxy at a million connections and explain every kernel counter that moves." It is written the way a maintainer of a proxy would explain it to a new teammate: the why behind each line, the failure that motivated it, and the experiment that proves it.

0. The 60-second map

A Layer-4 proxy is four ideas stacked:

  1. Accept connections off a kernel queue without falling behind.
  2. Copy bytes both ways, coupling the two directions so a slow peer throttles the fast one (backpressure).
  3. Half-close correctly so one direction ending doesn't kill the other.
  4. Drain in-flight connections on shutdown without dropping anyone.

The code in l4/proxy.go is exactly those four ideas and nothing else. By the end you'll understand each at the syscall level and have changed/broken/fixed each one yourself.

1. The kernel model you must carry in your head

When you call net.Listen("tcp", ":8080"), Go performs socket(), bind(), listen(). That listen(fd, backlog) creates two queues the kernel manages for you:

   incoming SYN ───▶ [ SYN queue ]  half-open, awaiting final ACK
                          │  3-way handshake completes
                          ▼
                     [ accept queue ] established, awaiting accept()
                          │  your code calls Accept()
                          ▼
                     a connected socket fd
  • SYN queue size ≈ net.ipv4.tcp_max_syn_backlog. SYN floods fill it; SYN cookies (net.ipv4.tcp_syncookies) let the kernel keep serving without storing half-open state.
  • Accept queue size ≈ min(backlog, net.core.somaxconn). This is the one that bites you. If your Accept() loop is slower than the arrival rate, this queue fills, and the kernel either drops the SYN (client retransmits, looks like latency) or sends RST (client sees "connection reset"). The counter is ListenOverflows in nstat/netstat -s.

Carry this picture everywhere. 80% of "mysterious gateway latency" stories end at "the accept queue was overflowing because something blocked the accept loop."

Why the accept loop must never block

In proxy.go, Serve does the minimum per accepted connection — increment a counter, spawn a goroutine, loop:

conn, err := ln.Accept()
...
p.wg.Add(1)
go func() { defer p.wg.Done(); p.handle(conn) }()

If handle ran inline (no goroutine), the accept loop would stall for the entire lifetime of each connection and the accept queue would overflow instantly under load. The goroutine is not a performance nicety; it's a correctness requirement. In Java/Netty the equivalent is "don't do blocking work on the boss/acceptor event loop."

Go's secret: "two goroutines per connection" looks like the thread-per-connection model that caused the C10K problem. It isn't. Go's netpoller multiplexes all those goroutines over a handful of OS threads using epoll/kqueue; a goroutine blocked in Read parks for ~few KB of stack, not a 1 MB OS-thread stack. You write blocking code; the runtime runs an event loop. That's why this lab is readable and scalable.

2. Backpressure: the bug everyone writes first

The naive proxy spawns a reader goroutine that pushes into an unbounded queue and a writer goroutine that drains it:

// THE CLASSIC BUG — do not ship this
ch := make(chan []byte, 1<<30) // "big enough"
go func() { for { b := read(src); ch <- b } }()
go func() { for b := range ch { write(dst, b) } }()

Point a fast producer (yes | nc) at a slow consumer (an origin you kill -STOP) and watch RSS climb until the OOM killer fires. The producer reads as fast as the kernel delivers; the consumer can't keep up; the queue absorbs the difference without bound. A proxy is a bucket brigade, not a warehouse.

The fix is in proxy.go's pipe:

buf := make([]byte, 32*1024) // ONE bounded buffer
for {
    nr, er := src.Read(buf)
    if nr > 0 {
        nw, ew := dst.Write(buf[:nr]) // BLOCKS until the slow side accepts it
        ...
    }
    if er != nil { break }
}

dst.Write blocks when the destination's socket send buffer is full (the kernel won't accept more until the slow peer ACKs and drains its receive window). While Write blocks, this goroutine doesn't call Read again — so we stop pulling from the fast side. Memory is bounded by one 32 KiB buffer per direction, forever, at any throughput. That is backpressure: TCP flow control on the slow side propagates, through our blocking write, into a paused read on the fast side.

io.Copy would give you the same property for free (it uses a 32 KiB buffer internally) — we hand-rolled the loop only so the byte counter updates during the connection, not just at close. Run TestProxyForwards: it asserts BytesUp/BytesDown move while the connection is still open, which is only true because we count per write.

3. Half-close: the difference between a toy and a proxy

TCP connections are two independent half-duplex streams. Either side can send FIN (no more data this way) while still receiving. Protocols rely on this: an HTTP client sends its request, half-closes, and still reads the response.

A toy proxy does defer client.Close(); defer origin.Close() and copies both ways; when the client half-closes, the toy tears down everything, killing the response. Watch pipe do it right:

func pipe(dst, src net.Conn, counter *atomic.Int64) {
    ... copy src -> dst ...
    if cw, ok := dst.(interface{ CloseWrite() error }); ok {
        cw.CloseWrite() // forward the FIN on THIS direction only
    }
}

When src (say the client) sends FIN, src.Read returns io.EOF, the loop ends, and we call dst.CloseWrite() — propagating the FIN to the origin's read side. The other pipe goroutine (origin→client) is untouched and keeps delivering the response until the origin closes its write side. TestHalfClose proves it: the origin drains to EOF, then writes "DONE", and the client still receives it.

*net.TCPConn satisfies CloseWrite(); the type assertion is how you reach the half-close without giving up the net.Conn interface.

4. Draining: every deploy is a drain

You will redeploy this proxy thousands of times. Each redeploy must not drop in-flight connections. The drain logic in Serve/drain:

ctx cancelled
  └─ ln.Close()           // stop accepting NEW connections
  └─ wait for p.wg         // in-flight handlers finish...
     └─ ...or DrainTimeout fires -> forceCloseAll() the stragglers

TestGracefulDrain holds a connection open, cancels, and asserts (a) Serve returns promptly because the deadline force-closes the straggler, and (b) new dials are refused after shutdown.

The ordering subtlety the test can't show (it needs Kubernetes, gw-09): in production you must fail the readiness probe first, so the load balancer stops sending new connections, then cancel/stop accepting. If you stop accepting before the LB notices, the LB keeps sending connections to a closed port → dropped requests on every deploy. The sequence is always: fail readiness → drain → exit.

The DrainTimeout is a policy: long enough that normal requests finish, short enough that a deploy isn't held hostage by one stuck connection. For an L7 HTTP node, seconds. For a 200k-WebSocket node (gw-05), minutes — and a 30 s default terminationGracePeriodSeconds would SIGKILL 200k live connections. Same code, wildly different timeout.

5. Hands-on lab

Build and run:

cd src/go
go build -o /tmp/l4proxy ./cmd/l4proxy

# An HTTP origin so we can drive real load:
python3 -m http.server 9000 &

# The proxy, printing churn stats every second:
/tmp/l4proxy -listen :8080 -origin 127.0.0.1:9000 -stats 1s &

Experiment A — see throughput and the accept path

# keep-alive load (one connection reused): high rps, ~0 accepts/s
wrk -t4 -c100 -d10s http://127.0.0.1:8080/

# connection-per-request load: every request is a new accept + dial.
# Watch the proxy's "accepted/s" stat explode — THIS is what curbing
# connection churn (gw-04) eliminates.
wrk -t4 -c100 -d10s -H 'Connection: close' http://127.0.0.1:8080/

Experiment B — read the kernel

# accept-queue depth on the listening socket (Recv-Q=current, Send-Q=max):
ss -lnt 'sport = :8080'

# listen-queue overflows + SYN drops (should be ~0 when healthy):
nstat -az 2>/dev/null | grep -Ei 'ListenOverflow|ListenDrop|TCPReqQFull' || \
  netstat -s | grep -Ei 'overflow|listen'

# outbound (origin-side) connections opened — the churn counter:
nstat -az 2>/dev/null | grep -i ActiveOpens

Experiment C — prove the event loop is real

# Go's netpoller IS epoll/kqueue. On Linux:
strace -f -e trace=network,epoll_pwait -p "$(pgrep l4proxy)" 2>&1 | head -40
# On macOS, use dtruss or just trust kqueue. You'll see accept4 + the
# poller syscalls; you will NOT see one thread per connection.

Experiment D — force an accept-queue overflow (the war story)

# Make handlers pile up: STOP the origin so dials hang, lower backlog,
# blast many short connections, and watch ListenOverflows climb.
kill -STOP %1                                  # freeze the python origin
for i in $(seq 1 3000); do (echo | nc -w1 127.0.0.1 8080 &) ; done
nstat -az | grep -i overflow                   # the count rises
kill -CONT %1

The client-side symptom of this is connection timeouts and resets — which look like a network problem but are really "the accept queue overflowed because dials to the origin were hanging." Knowing to look at ListenOverflows is the difference between a 10-minute and a 10-hour incident.

6. Deep dives (the maintainer details)

6.1 TCP_NODELAY and the 40 ms stall

Nagle's algorithm holds small writes to coalesce them into full segments; delayed ACK holds ACKs up to ~40 ms hoping to piggyback on return data. Together, on a small request/response, they deadlock for ~40 ms: sender waits for an ACK (Nagle), receiver waits for data to piggyback the ACK on (delayed ACK). setNoDelay in proxy.go sets TCP_NODELAY on both sides to disable Nagle. For a request/response proxy this is almost always right; for a bulk-transfer path you might leave Nagle on to reduce packet count. It's a latency/throughput knob, not a "go faster" button.

6.2 Zero-copy: where splice(2) would go

Our pipe copies bytes through a userspace buf: read() (kernel→user) then write() (user→kernel) — two copies, two syscalls per buffer. An L4 proxy that never inspects the payload doesn't need to see the bytes. Linux splice(2) moves them between two fds through a kernel pipe with zero userspace copies; high-end L4 proxies hit line rate this way. We don't use it because (a) it's Linux-only and not in Go's stdlib for arbitrary conn→conn, and (b) the moment you add TLS termination or inspection (gw-02/gw-07) you can't splice anyway — you must decrypt and parse in userspace. The L4/L7 cost difference is, quite literally, those two memory copies.

6.3 SO_REUSEPORT: one acceptor per core

A single accept loop is a single point of contention; at very high connect rates the kernel's accept-queue lock and your one goroutine become the bottleneck. SO_REUSEPORT lets N sockets bind the same port; the kernel hashes incoming connections across them, so you run one acceptor per core with no shared lock. To add it here, set the socket option in a net.ListenConfig{Control: ...} (see steps/02). Caveat: it spreads new connections, not load — with long-lived connections (gw-05) one acceptor can still end up hot.

6.4 TIME_WAIT and who pays for close()

The side that actively closes a connection holds it in TIME_WAIT for 2×MSL (~60 s on Linux) so a delayed duplicate segment from the old connection can't corrupt a new one reusing the same 4-tuple. It's correct, but it means the closer pays. A proxy that opens and closes origin connections per request accumulates TIME_WAIT on the origin side and can exhaust ephemeral ports or conntrack. The fixes, in order of preference: don't close (keep-alive/pooling — gw-04), let the client close, widen ip_local_port_range, and tcp_tw_reuse for outbound. Never reach for the long-removed tcp_tw_recycle.

6.5 conntrack: the hidden quota

If your nodes run netfilter/NAT (most Kubernetes setups do — gw-09), every connection consumes a nf_conntrack slot. A connection-heavy proxy can hit nf_conntrack_max and start dropping with cryptic nf_conntrack: table full dmesg lines. Watch net.netfilter.nf_conntrack_count vs _max. The durable fix is fewer connections (gw-04 again); the stopgap is raising the limit.

7. PROXY protocol: keeping the client IP

Put a proxy in front of an origin and the origin's accept() sees the proxy's IP. For L7 you'd add X-Forwarded-For; for L4 (no HTTP to add a header to) the standard is the PROXY protocol — a header prepended to the byte stream before the payload. proxyproto.go implements v1 (text):

PROXY TCP4 203.0.113.7 10.0.0.5 56324 8080\r\n

WriteProxyV1 emits it toward the origin; ParseProxyV1 reads and strips it on the receiving side, leaving the bufio.Reader positioned at the real payload. TestProxyProtocolEndToEnd runs the full loop: a client connects through the proxy, the origin parses the header, and the test asserts the origin recovered the client's actual source port.

The trust boundary (say this in an interview): only parse a PROXY header from peers you trust. If an untrusted client can send one, they spoof their source IP — and now your rate limits, geo rules, and audit logs are lies. Production allowlists the set of IPs permitted to speak PROXY protocol. The exact same caution applies to trusting an inbound X-Forwarded-For at L7 (gw-03/gw-07).

8. Code tour (read in this order)

  1. l4/proxy.go — Listen/Serve/handle/pipe/drain. The whole proxy is here; ~180 lines.
  2. l4/proxyproto.go — the PROXY v1 codec.
  3. l4/proxy_test.go — six tests that boot real in-process proxies; read these to see the behaviors asserted.
  4. cmd/l4proxy/main.go — the CLI + signal-driven drain + the churn-stats sampler.

Run it all: bash scripts/verify.sh (vet + build + -race test).

9. Exercises (do these — they're the interview)

  1. Add SO_REUSEPORT (steps/02) and start two instances on :8080. Prove with ss -tnp that both accept connections. Then explain why long-lived connections can still make one hot.
  2. Add an idle timeout: set a read deadline that resets on activity; close connections idle past N seconds. Watch it reap leaked connections. Why is too-aggressive a timeout its own bug (gw-04)?
  3. Break backpressure on purpose: replace pipe with the unbounded- channel version and reproduce unbounded RSS growth under a kill -STOP'd origin. Revert. Now you've felt the bug.
  4. Make drain force-close visible: add a log line in forceCloseAll and a metric for "stragglers force-closed at deadline." When would a nonzero value be alarming?
  5. Pick a backend per connection: extend OriginAddr to a list and choose one per accepted connection (round-robin). Notice you can only balance connections, not requests — the fundamental L4 limit that motivates L7 (gw-02) and P2C (gw-06).

When you can do all five and narrate the kernel counter each one moves, you can hold the L4 portion of a Cloud Gateway interview.