gw-05 — Analysis

The design review for the miniature Pushy you build in steps/.

Required behaviors

Hold and authenticate connections. Each WebSocket is authenticated on connect (gw-07), registered, and kept alive with ping/pong; a dead peer is detected and cleaned up.
Route by registry. A message addressed to device D is delivered to the node currently holding D, looked up in the registry; a registry miss follows an explicit policy (drop / queue / error).
Async, decoupled delivery. Publishers enqueue (Kafka in prod; a channel/topic in the lab) and return immediately; delivery happens off that queue so a slow node never blocks a publisher.
Graceful drain. On shutdown: stop new connections, signal reconnect, drain with backoff+jitter over a long window, keep the registry consistent as devices migrate.
Bounded per-connection cost. Small buffers, bounded write queues per connection, idle/keepalive timeouts — density is the constraint.

Registry as an interface. Register/Lookup/Unregister/TTL behind an interface, backed by an in-memory map in the lab but shaped like a KV service (KeyValue/Redis) so the production swap is obvious. TTLs are first-class: a crashed node's entries must expire.
Per-connection bounded write queue. Each connection has a small buffered channel for outbound messages; if it fills (slow client), you drop-oldest or disconnect per policy — never block the delivery path on one slow consumer (head-of-line across devices).
Reconnect with backoff + full jitter. The drain path sends a "reconnect" control message; clients wait random(0, base·2^attempt) capped — full jitter, the AWS-builders'-library recommendation — precisely to avoid the reconnect storm.
Delivery is at-least-once + idempotent. Messages carry an id; clients dedupe. The lab demonstrates a redelivery on reconnect and the client suppressing the duplicate, which is how you reach the "five nines" claim honestly (not exactly-once).

Density vs resilience headroom. Packing 200k–400k connections per node maximizes efficiency but raises blast radius: losing one node forces 200k reconnects. Size nodes and drain windows so a node loss is absorbable by the rest of the fleet without a cascading storm.
Registry consistency vs latency. A strongly consistent registry costs latency on every connect/disconnect (high-churn operations); an eventually consistent one risks misdelivery. The pragmatic answer is a fast KV with short TTLs + update-on-event + a miss policy, accepting a small misdelivery rate covered by retries.
Sticky load that scale-out can't fix. New nodes only attract new connections. The only way to cool a hot node is to cycle its connections, which costs a controlled reconnect wave. Capacity planning must account for connection lifetime, not just connection rate.
Kafka decoupling adds delivery latency and an ordering question. The mailbox model means a small delay and per-key ordering concerns (does device D need messages in order? then partition by D). State the ordering guarantee you're providing.

TLS termination + device identity (gw-07) on every connection at fleet scale, with cert rotation.
A real distributed registry (KeyValue/Redis) with replication, TTLs, and a reconciliation job for orphaned entries (crashed nodes).
Kafka partitioning by device for ordered delivery, consumer-group scaling, and backpressure when delivery falls behind.
Connection-density tuning: instance selection, fd limits, event-loop sizing, GC/allocation profiling, heartbeat-interval tuning — the work behind the 60k→200k→400k jump.
Drain orchestration integrated with Kubernetes (long termination grace, preStop reconnect signal, PodDisruptionBudgets) — gw-09, gw-12.
Delivery-reliability observability: per-message acks, delivery latency, registry hit/miss, reconnect-storm detection (gw-11).