gw-05 — WebSockets & Persistent-Connection Proxies (Pushy)

Two of the talks in the JD — "Pushy to the Limit" and "Scaling Push Messaging for Millions of Devices" — are about one system: Pushy, Netflix's WebSocket proxy. It holds hundreds of millions of concurrent, persistent WebSocket connections from devices, and lets backend services push a message to any specific device in real time. It sustains a steady 99.999% message-delivery reliability, and the team pushed per-node density from ~60k to ~200k connections (headroom to ~400k) by switching instance types and reworking the architecture.

This lab is where the gateway model inverts. Everywhere else, the client makes a request and you proxy it (request/response, short-lived). Here the client opens one long-lived connection and keeps it open for hours or days, and the server initiates messages. That inversion changes everything: connection density becomes the dominant cost, routing becomes "find the node holding this device's connection," and draining a node for a deploy becomes the hardest operational problem in the phase.

You will build a WebSocket server, a push registry (device → node), and an async delivery path (a backend publishes; the right node pushes to the right device) — Pushy in miniature.

1. What is it?

A persistent-connection proxy maintains a standing connection to each client so messages can flow server → client at any time, not just in response to a client request. WebSocket (RFC 6455) is the canonical transport: it starts as an HTTP request with an Upgrade: websocket header, then the same TCP connection switches to a bidirectional, framed, full-duplex message channel.

client ── HTTP GET /ws  Upgrade: websocket ──▶ Pushy node
client ◀── 101 Switching Protocols ───────────  (handshake done)
        ══════ full-duplex WebSocket frames ══════   (stays open for hours)
        ◀── server can push a message at ANY time ──

Pushy's job is twofold:

1. Hold the connections. Terminate hundreds of millions of WebSockets across a fleet, keep them alive (ping/pong), authenticate them, and survive deploys without dropping them carelessly.
2. Route messages to them. When a backend wants to reach device D, look up which Pushy node holds D's connection in the push registry, deliver the message there, and that node writes it down the socket.

The supporting cast (from the talks):

• Push registry — a low-latency key-value store mapping deviceId → {node, connection metadata}. Netflix moved it from Dynomite (a Redis wrapper) to KeyValue, an internal KV service.
• Message processor — a standalone Spring Boot service consuming from Kafka; backends publish "send X to device D" events, the processor looks up the registry and forwards to the owning node.
• Async, decoupled delivery — the push path is event-driven through Kafka so a slow or redeploying Pushy node doesn't block publishers.

2. Why does it matter?

• It's a flagship system of the team. Two of seven linked talks are about it. Being able to whiteboard Pushy — connection density, push registry, async delivery, graceful drain — is close to a guaranteed win in the loop.

• Persistent connections break every assumption from the rest of the phase. No connection pooling to clients (the connection is the session). Load balancing is "connections," not "requests," and it's sticky for the connection's whole life, so a node can stay hot long after the LB stopped sending it new connections (the gw-01 SO_REUSEPORT caveat, at its most extreme). State per connection is the dominant memory cost.

• Graceful drain is genuinely hard here. You cannot drop 200,000 live connections to deploy. You must coordinate reconnect: signal clients to reconnect (with backoff + jitter so they don't thunder onto the next node), drain over minutes not seconds, and keep the push registry consistent as devices migrate between nodes.

• It's a real-time, at-least-once delivery system. 99.999% reliability over an unreliable mobile network means reconnect logic, message acknowledgment, idempotency, and dealing with a device that's briefly offline — distributed-systems delivery semantics applied to the edge.

3. How does it work?

The WebSocket handshake and framing

The upgrade is an HTTP/1.1 request with specific headers; the server proves it understood by hashing the client key with a magic GUID:

GET /ws HTTP/1.1
Host: push.netflix.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13

→ 101 Switching Protocols
  Sec-WebSocket-Accept: base64(sha1(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"))

After 101, the connection speaks WebSocket frames: a small header (FIN bit, opcode — text/binary/close/ping/pong, mask bit, payload length 7/16/64-bit), then payload. Client→server frames are masked (XOR with a 4-byte key) to defeat cache-poisoning of intermediaries; server→client frames are not. Ping/pong frames keep the connection alive through NATs and detect dead peers (gw-01 keepalive at the app layer).

The push architecture

                         ┌──────────────── Pushy fleet ───────────────┐
 device D ═══WS════════▶ │ node N2  (holds D's live connection)         │
                         │   on connect: registry[D] = N2              │
                         └───────────────▲─────────────────────────────┘
                                         │ deliver(D, msg) → write down socket
   backend ──"send msg to D"──▶ Kafka ──▶ Message Processor ──registry.lookup(D)=N2
                                         │
                              Push Registry (KeyValue): deviceId → node

1. Device connects to some node N2; N2 authenticates it and writes registry[D] = N2 (+ connection metadata, with a TTL).
2. A backend publishes "send msg to D" to Kafka (fire-and-forget, fast).
3. The Message Processor consumes it, looks up registry[D] = N2, forwards the message to N2.
4. N2 writes the message down D's socket. If D isn't connected anywhere (registry miss / stale), the message is dropped or queued per policy.

Connection density — the dominant constraint

At 200k connections/node, the per-connection cost is everything:

• Memory: read/write buffers, TLS state, app session state — keep it tiny. At 200k × 10KB = 2GB just for connection state.
• File descriptors: 200k fds/node → tune ulimit/fs.file-max.
• Event-loop threads: Netty event loops multiplexing all of them; never block (gw-03), or you stall thousands of connections.
• Heartbeat cost: ping/pong to 200k connections is real traffic and CPU; tune the interval against NAT timeout and detection latency.

The talk's density jump (60k→200k→400k) came from a better instance type and trimming per-connection overhead — a profiling-and-budgeting exercise.

Graceful drain (the hard part)

deploy node N2:
  1. stop accepting NEW connections (LB readiness off — gw-09)
  2. tell connected devices to reconnect (a "go away" app message)
  3. devices reconnect to OTHER nodes with backoff + JITTER
     (no jitter ⇒ a reconnect thundering herd onto the next node)
  4. as each device migrates, registry[D] updates to its new node
  5. when connections drain below a threshold (or a deadline), exit

Draining 200k connections takes minutes; the deploy system must tolerate that (long terminationGracePeriodSeconds, gw-09). Botching the jitter turns a deploy into a self-inflicted DDoS on the rest of the fleet — a classic war story to have ready.

4. Core terminology

5. Mental models

• A request/response gateway is a restaurant; Pushy is a switchboard. Diners come, order, eat, leave (short-lived). A switchboard keeps a line open to every subscriber for hours and rings them when there's a call. The cost model flips from "throughput of orders" to "how many lines can one operator hold open."

• The push registry is a hotel front desk. To deliver a message to a guest you don't search every room — you ask the desk which room (node) they're in. Keep the desk's records fresh (TTL, update on check-in/out) or you'll knock on empty rooms (stale registry → lost messages).

• Reconnects without jitter are a stampede through one door. When a node drains, every device bolts for the exit at once; if they all run to the same next node simultaneously you've moved the fire, not put it out. Jitter spreads the stampede over time and across nodes.

• Decoupling via Kafka is a mailbox, not a phone call. Publishers drop a letter and move on; the message processor delivers when it can. A redeploying node doesn't block the publisher — the letter waits. The cost is the delivery delay and the need for the registry to be right when the letter is processed.

6. Common misconceptions

• "WebSockets are just a faster HTTP." They invert the model: server-initiated, long-lived, stateful. The hard problems (density, drain, routing-to-a-connection) don't exist in request/response and are the whole job here.

• "Load balancing persistent connections is the same as requests." It's connection-sticky for the connection's entire life. New connections balance; existing ones don't move. A node can be hot for hours after the LB stops sending it new connections — you rebalance by forcing reconnects, which is exactly the drain problem.

• "Just scale out to add capacity." New nodes only get new connections; existing connections stay put. To actually shift load you must cycle connections (drain/reconnect) — adding nodes alone doesn't cool a hot one.

• "The push registry can be eventually consistent and it's fine." Staleness directly causes lost or misdelivered messages (you push to a node that no longer holds the device). It needs tight TTLs, updates on connect/disconnect, and a miss policy — consistency here is a delivery- reliability property.

• "Heartbeats are free." Ping/pong to hundreds of millions of connections is significant traffic and CPU, and the interval trades liveness-detection latency against cost and NAT-timeout safety. It's a real tuning decision.

7. Interview talking points

• "Design a push-notification system for 300M devices." The Pushy question. Spine: WebSocket fleet holding the connections → push registry (device→node) → Kafka + message processor for async fan-out → reconnect strategy with backoff+jitter → graceful drain. Quantify: conns/node (~200k), memory/conn, registry QPS, fan-out rate.

• "How do you deploy a node holding 200k live connections?" Stop new connections (readiness off) → signal clients to reconnect → backoff + jitter so they don't stampede onto one node → drain over minutes → update the registry as devices migrate → exit on threshold/deadline. Name the reconnect-storm failure mode and the jitter fix.

• "Where does the device→node mapping live and why does consistency matter?" A low-latency KV store (Netflix: KeyValue) with TTLs, updated on connect/disconnect. Staleness = lost/misdelivered messages, so it's a delivery-reliability property, not just a cache.

• "How do you get 99.999% delivery over flaky mobile networks?" At-least-once with acks + idempotency; client reconnect with resume; a brief offline queue or drop policy; the async Kafka path so publishers aren't coupled to delivery. Be explicit about the exactly-once myth (you get at-least-once + idempotency).

• "How did per-node density go from 60k to 200k?" Better instance type + trimming per-connection memory/CPU (buffers, TLS state, heartbeat cost) found by profiling. fds, event-loop threads, and GC pressure are the limits to push on. Shows you think in per-connection budgets.

• "WebSocket vs SSE vs long-poll vs HTTP/2 streams vs WebTransport?" SSE = server→client only, simpler, auto-reconnect, no masking. Long- poll = fallback, high overhead. WebSocket = full-duplex, the workhorse. h2/h3 streams and WebTransport (over QUIC) are the modern alternatives; know the trade-offs (h3/WebTransport survives network changes via connection migration, gw-02).

8. Connections to other labs

• gw-01 (L4) — draining at its hardest; keepalive (ping/pong is the app-layer version); SO_REUSEPORT stickiness taken to the extreme.
• gw-02 (L7) — the WebSocket upgrade is the other HTTP upgrade path; h3/WebTransport is the future transport for this.
• gw-03 (API gateway) — Pushy is a specialized gateway; the event-loop "never block" discipline is life-or-death at 200k conns/node.
• gw-06 (resilience) — reconnect backoff+jitter, load shedding on the connect path, and not letting a reconnect storm cascade.
• gw-08 / gw-09 — the registry is a distributed datastore; drain is gated by Kubernetes readiness + a long termination grace period.
• db-20 (distributed KV) — the push registry is exactly a low-latency distributed key-value store with TTLs; you built one.