gw-05 step 03 — Graceful drain and avoiding the reconnect storm

Goal

Deploy a node holding many live connections without dropping messages and without a reconnect stampede onto the rest of the fleet. This is the hardest operational problem in the phase and a guaranteed interview topic.

Code — drain with a reconnect signal

package pushy

import (
	"context"
	"encoding/json"
	"time"
)

// Drain stops new connections, asks every connected device to reconnect
// (elsewhere) after a JITTERED delay, and waits for connections to bleed
// off — up to a long deadline.
func (n *Node) Drain(ctx context.Context, deadline time.Duration) {
	// 1. (Caller already flipped readiness so the LB sends no NEW conns.)

	// 2. Tell every device to reconnect after a random delay so they
	//    don't all hit the next node at the same instant.
	n.mu.RLock()
	conns := make([]*deviceConn, 0, len(n.conns))
	for _, dc := range n.conns {
		conns = append(conns, dc)
	}
	n.mu.RUnlock()

	for _, dc := range conns {
		// Full jitter: each client waits random(0, reconnectWindow).
		delayMs := jitterMillis(30_000) // spread reconnects over 30s
		msg, _ := json.Marshal(map[string]any{
			"type": "reconnect", "afterMs": delayMs,
		})
		dc.enqueue(msg) // client schedules reconnect after delayMs + its own backoff
	}

	// 3. Wait for connections to drain, bounded by the deadline.
	end := time.Now().Add(deadline)
	for time.Now().Before(end) {
		n.mu.RLock()
		remaining := len(n.conns)
		n.mu.RUnlock()
		if remaining == 0 {
			return
		}
		time.Sleep(500 * time.Millisecond)
	}
	// 4. Deadline hit: force-close stragglers (they reconnect via backoff).
	n.closeAll()
}

Code — full jitter (the storm-prevention primitive)

import "math/rand"

// jitterMillis returns a uniformly random delay in [0, max) — "full
// jitter" from the AWS builders' library. This is what turns a
// synchronized stampede into a smooth ramp.
func jitterMillis(max int) int { return rand.Intn(max) }

// Client-side reconnect backoff (for reference): exponential with full
// jitter, capped. Used after an UNREQUESTED disconnect too.
func reconnectDelay(attempt int) time.Duration {
	const base, cap = 500 * time.Millisecond, 30 * time.Second
	d := base << attempt
	if d > cap {
		d = cap
	}
	return time.Duration(rand.Int63n(int64(d))) // full jitter
}

The experiment — see the storm, then prevent it

Simulate a fleet (a few nodes, N simulated devices) and a metric of "reconnects per second arriving at the rest of the fleet" during a drain.

A) Drain with NO jitter (all devices reconnect immediately):
   reconnect arrivals spike to ~N in one tick → the next node's accept
   queue overflows (gw-01) → some reconnects fail → clients retry →
   the spike echoes. A self-inflicted DDoS.

B) Drain WITH full jitter over a 30s window:
   reconnect arrivals are a flat ~N/30 per second → every node absorbs
   its share → no accept-queue overflow → smooth migration.

Plot both. The difference is the entire lesson.

At-least-once + idempotency on reconnect

When a device reconnects mid-drain, an in-flight message may be redelivered (the registry pointed at the old node when it was published). Make delivery safe:

- every PushEvent carries MsgID
- the client tracks recently-seen MsgIDs and suppresses duplicates
- the server may redeliver freely (at-least-once) — dedupe makes it
  effectively-once from the user's view

Tasks

1. Implement Drain with full jitter and a long deadline; a simulated client that honors the reconnect message (waits afterMs, then reconnects to another node with its own backoff+jitter).
2. Run experiment A vs B; plot reconnect arrivals/sec at the rest of the fleet. Capture the spike vs the flat ramp.
3. Show a redelivered message during drain and the client suppressing the duplicate via MsgID — demonstrating at-least-once + idempotency.
4. Tie drain to Kubernetes: set a long terminationGracePeriodSeconds and a preStop hook that triggers Drain; explain why a default 30s grace period would force-close 200k connections (gw-09).

Acceptance

• A drain that migrates all devices off a node with no message loss and no reconnect spike (flat arrivals with jitter; a clear spike without it).
• A demonstrated duplicate suppressed by MsgID.
• A correct statement of the Kubernetes grace-period requirement for a high-density node.

Discussion prompts

• Why "full jitter" rather than "exponential backoff" alone? (Backoff spreads retries of one client; jitter de-synchronizes many clients. You need both.)
• Draining 200k connections over 30s = ~6.7k reconnects/sec leaving this node. Is that absorbable by a fleet of M nodes? Do the math and decide the right reconnect window.
• Scale-out adds nodes but existing connections don't move. So how do you actually cool a hot node — and why is that the same machinery as drain?