gw-05 — The Hitchhiker's Guide to Pushy (WebSockets at Scale)

Companion to CONCEPTS.md, with the runnable miniature Pushy in src/go/pushy/. Two of the JD's seven talks are about this system; being able to whiteboard and build it is close to a guaranteed interview win.

Everywhere else in this phase, a client makes a request and you proxy it (short-lived, request/response). Pushy inverts that: a device opens one long-lived WebSocket and keeps it open for hours, and the server pushes messages to it at any time. That inversion changes everything — connection density becomes the dominant cost, routing becomes "find the node holding this device," and draining a node becomes the hardest operational problem in the phase. This lab builds all of it.

1. The WebSocket protocol (ws.go)

A WebSocket starts as an HTTP/1.1 request with Upgrade: websocket; the server proves it understood by hashing the client's key with a magic GUID. AcceptKey does exactly that, and TestAcceptKey pins it to the RFC 6455 §1.3 worked example (dGhlIHNhbXBsZSBub25jZQ== → s3pPLMBiTxaQ9kYGzzhZRbK+xOo=). After the 101 Switching Protocols response (ServerHandshake), the connection speaks frames.

WriteFrame/ReadFrame implement the framing (RFC 6455 §5): a small header (FIN bit, opcode, mask bit, 7/16/64-bit length), then payload. Two details matter and are tested (TestFrameRoundTripMasked):

  • Client→server frames MUST be masked (XOR with a 4-byte key) to defeat cache-poisoning of intermediaries; server→client frames are not. Getting the mask direction wrong is the #1 hand-rolled-WebSocket bug.
  • Control frames (Close, Ping, Pong) are handled inline. Conn.ReadMessage auto-answers a ping with a pong (TestPingAutoPong) — that's your application-layer keepalive, holding NAT mappings open and detecting dead peers (the gw-01 keepalive idea at L7).

A net.Pipe lesson baked into the tests: the ping/pong test uses real loopback TCP, not net.Pipe, because net.Pipe is fully unbuffered — a server writing a pong while the client is still writing the next frame deadlocks. That's not a toy concern: any duplex protocol over an unbuffered transport can deadlock if you don't separate read and write paths. (In production each connection has a dedicated write pump for exactly this reason — see §2.)

2. Holding connections (node.go)

Node holds deviceID -> Sink. A Sink is a per-device outbound channel; Send returns false if the device's bounded queue is full rather than blocking. TestSlowConsumerIsolation proves the property that keeps a node alive: a slow device whose queue fills gets its message dropped, while every other device keeps receiving. One slow consumer must never head-of-line the whole node — at 200k connections per node, a single blocking write would stall thousands of devices.

In the real server (cmd/pushyd), each connection gets exactly one writer goroutine (the "write pump") behind a mutex, because concurrent writes to a WebSocket corrupt framing. The bounded queue + single writer

  • drop-on-full is the canonical high-density connection model.

Connection density is the whole cost model. Node.Count() is the metric that matters; the talk's 60k → 200k → 400k journey was a profiling exercise in shrinking per-connection memory (buffers, TLS state), fds, and heartbeat cost. Do the arithmetic: 200k conns × ~10 KB state ≈ 2 GB of connection state per node before any payload.

3. The push registry (registry.go)

To deliver to device D you don't search every node — you look up registry[D] = node. MemRegistry is the in-process stand-in for Netflix's KeyValue store; the interface is what matters.

The load-bearing detail is TTLs. TestRegistryTTL (with an injectable clock) shows an entry expiring: a node that crashes never runs its clean Unregister, so without a TTL its entries would point at a dead node forever, and pushes would silently vanish. Expired entries are treated as misses — that's your crashed-node cleanup. Unregister also only removes an entry if this node still owns it, so it can't clobber a device that already reconnected elsewhere (a real race during drain). Registry staleness is a delivery-reliability property, not a caching nicety.

4. Async delivery (node.go)

A backend doesn't call Pushy directly — it Publishes a PushEvent to a topic (Kafka in production, a channel here) and returns immediately. The MessageProcessor consumes the topic, looks up the registry, and forwards to the owning node. TestDeliveryRouting runs the whole path: connect device-42 to node n2, publish to device-42, and assert it arrives on n2's sink with Delivered == 1 — routed purely by registry lookup, with zero coupling between the publisher and the holding node.

TestRegistryMissPolicy covers the other branch: a push to an unconnected device increments Misses and is dropped (or, per policy, queued for redelivery on reconnect). The processor also tracks ForwardFails for the race where a node drains between the lookup and the forward — the device will reconnect elsewhere and the message is redelivered (§5).

Why decouple with a queue? A redeploying or slow Pushy node must not block publishers; the queue is a mailbox, not a phone call. The cost is delivery latency and per-device ordering (partition Kafka by deviceId if order matters).

5. The hardest part: graceful drain (drain.go)

You cannot SIGKILL a node holding 200k live connections. Drain means: stop accepting new connections, tell connected devices to reconnect, and let them migrate over minutes — without a stampede.

The stampede is the trap. If every device reconnects the instant it's told, they all hit the next node simultaneously, overflow its accept queue (gw-01), fail, retry, and you've turned a deploy into a self-inflicted DDoS. The fix is full jitter: each device waits a uniformly random delay over the reconnect window. Jitter.ReconnectDelays assigns those delays; TestReconnectJitterSpread proves that across 1000 devices and a 30 s window, every time-bucket is non-empty — a smooth ramp, not a spike. SpreadBuckets is the histogram that makes it visible.

Backoff adds exponential backoff with full jitter for unrequested disconnects (TestBackoffWithinCap). Note the distinction the CONCEPTS file draws: backoff de-correlates one client's retries; jitter de-correlates many clients from each other. You need both.

Finally, drain forces at-least-once + idempotent delivery: a message in flight when a device reconnects may be redelivered. Dedup (TestAtLeastOnceDedup) tracks recently-seen MsgIDs with bounded memory so the client suppresses duplicates — which is how you reach "five nines" honestly (at-least-once delivery + idempotent client = effectively-once for the user; true exactly-once is a myth).

6. Hands-on

cd src/go
bash ../scripts/verify.sh        # tests -race

go build -o /tmp/pushyd ./cmd/pushyd
/tmp/pushyd -ws :8090 -admin :8091 &

# Connect a device (first text message = deviceId). With websocat:
#   echo device-42 | websocat ws://127.0.0.1:8090/ws -
curl localhost:8091/stats                                  # connected devices
curl -X POST 'localhost:8091/deliver?device=device-42' -d 'new episode!'

If you don't have websocat, write a 20-line Go client using ClientHandshake + WriteMessage — the package exports both sides.

7. Exercises

  1. Write the drain loop end to end: on SIGTERM, Node sends each device a JSON {"type":"reconnect","afterMs":N} using ReconnectDelays, waits for Count() to fall (bounded by a deadline), then exits. Simulate a fleet and plot reconnect arrivals/s with and without jitter (the spike vs the flat ramp).
  2. Add a write pump: give each connection a bounded outbound channel and a single writer goroutine; show that a kill -STOP'd client gets dropped without affecting others (the §2 property, over real sockets).
  3. Make the registry distributed: back Registry with a real KV (Redis) and add a reconciliation job that sweeps orphaned entries from crashed nodes. Measure lookup latency on the delivery path.
  4. Partition for ordering: route PushEvents through N topic partitions keyed by deviceId and prove per-device ordering survives concurrent processors.
  5. Density profiling: connect 50k fake devices to one Node and measure RSS per connection; find the biggest per-connection cost and shrink it (the 60k→200k journey in miniature).