gw-01 step 03 — Graceful drain, load testing, and reading the kernel

Goal

Make the proxy drain on SIGTERM without dropping in-flight connections, then load-test it and read the kernel counters that prove it's healthy. This is the operational core of the role: every deploy is a drain.

Code — drain with a deadline

import (
	"context"
	"log"
	"time"
)

// RunWithDrain runs the proxy and, on ctx cancellation (SIGTERM),
// stops accepting, then waits up to drainTimeout for in-flight
// connections to finish before returning.
func (p *Proxy) RunWithDrain(ctx context.Context, drainTimeout time.Duration) error {
	runErr := make(chan error, 1)
	go func() { runErr <- p.Run(ctx) }() // Run closes the listener on ctx.Done

	select {
	case err := <-runErr: // listener returned (ctx cancelled or fatal)
		// Now drain: wait for in-flight handlers tracked by p.wg.
		done := make(chan struct{})
		go func() { p.wg.Wait(); close(done) }()

		select {
		case <-done:
			log.Printf("drain complete: all connections closed")
		case <-time.After(drainTimeout):
			log.Printf("drain timeout after %s: forcing exit with stragglers", drainTimeout)
		}
		return err
	}
}

Wire the ordering correctly in main:

	// 1. SIGTERM cancels ctx -> Run stops accepting + closes listener.
	// 2. (Kubernetes preStop already flipped readiness; the LB has
	//    stopped sending NEW connections — see gw-09.)
	// 3. RunWithDrain waits for in-flight, bounded by drainTimeout.
	_ = p.RunWithDrain(ctx, 25*time.Second)

Ordering is everything. Readiness must fail before you stop accepting, so the load balancer removes you from rotation while you can still serve the connections you already have. In Kubernetes the preStop hook + terminationGracePeriodSeconds give you that window (gw-09). Exit too early and every deploy drops requests.

Load test

# Build, run an HTTP origin so wrk can talk through the L4 proxy:
python3 -m http.server 9000 &           # origin
/tmp/l4proxy -listen :8080 -origin 127.0.0.1:9000 &

# Throughput + latency:
wrk -t4 -c200 -d30s http://127.0.0.1:8080/

# Constant-rate (wrk2) to see real tail latency under a fixed load:
wrk2 -t4 -c200 -d30s -R20000 http://127.0.0.1:8080/

Read the kernel — is it healthy?

# Accept-queue depth on the LISTEN socket: Recv-Q = current, Send-Q = max.
# If Recv-Q rides near Send-Q, your accept loop can't keep up.
ss -lnt 'sport = :8080'

# Listen-queue overflows + SYN drops (these should stay ~0):
nstat -az | grep -Ei 'ListenOverflows|ListenDrops|TCPReqQFullDrop'

# Per-socket TCP info (rtt, cwnd, retrans) for established conns:
ss -tni 'sport = :8080' | head

# Are we accumulating TIME_WAIT on the origin (outbound) side?
ss -tan state time-wait | wc -l

# Watch what the proxy is actually doing at the syscall level — you'll
# see epoll_pwait + accept4 + read/write; Go's netpoller IS an event loop:
strace -f -e trace=network,epoll_pwait -p "$(pgrep l4proxy)" 2>&1 | head -40

Tasks

  1. Implement RunWithDrain. Start a long-lived connection (nc), then kill -TERM the proxy. Confirm the existing connection keeps working until it closes (or the deadline), while a new nc connection is refused immediately.
  2. Drive wrk2 at a fixed rate and record p50/p99/p99.9. Then add a 2nd SO_REUSEPORT instance and show the tail improves (less accept-loop contention).
  3. Force an accept-queue overflow: set the origin to kill -STOP so handlers pile up, lower the listen backlog, blast wrk -c2000, and watch ListenOverflows climb in nstat. Explain the client-side symptom (connection timeouts / resets).

Acceptance

  • On SIGTERM, in-flight connections complete (or hit the deadline); new connections are refused. No mid-stream drops in the wrk run that straddles the signal.
  • You can point at ss/nstat output and say whether the proxy is accept-queue-bound, origin-bound, or healthy.
  • You can reproduce a ListenOverflows spike on demand and explain it.

Discussion prompts

  • Why must readiness fail before you stop accepting, not after? Draw the timeline of LB endpoint removal vs your shutdown.
  • A 25s drain deadline force-closes stragglers. For an L7 HTTP node that's usually fine; for a WebSocket node holding 200k connections (gw-05) it's catastrophic. What changes about drain at that scale?
  • wrk opens a fixed connection pool, so it never stresses your accept path after warmup. What load shape does stress accept, and how would you generate it? (Hint: many short connections — the very thing gw-04 eliminates with keep-alive.)