gw-06 step 03 — Reproduce a retry storm, then stop it with a budget

Goal

See retry amplification turn a small blip into a metastable collapse, then add a retry budget + backoff/jitter + circuit breaker and watch the system stay up. This is the most important resilience lesson and a guaranteed interview story.

Code — retry budget (token bucket) + circuit breaker

package resil

import (
	"sync"
	"time"
)

// Budget caps retries to a fraction of total request volume. Each
// request adds `ratio` tokens; each retry costs 1. Empty bucket => no
// retry. This bounds amplification GLOBALLY, unlike per-request counts.
type Budget struct {
	mu     sync.Mutex
	tokens float64
	ratio  float64 // e.g. 0.1 = retries may be at most ~10% of traffic
	max    float64
}

func NewBudget(ratio float64) *Budget {
	return &Budget{ratio: ratio, max: 100, tokens: 100}
}

func (b *Budget) OnRequest() {
	b.mu.Lock()
	b.tokens = min(b.max, b.tokens+b.ratio)
	b.mu.Unlock()
}

func (b *Budget) TryRetry() bool {
	b.mu.Lock()
	defer b.mu.Unlock()
	if b.tokens >= 1 {
		b.tokens--
		return true
	}
	return false // budget exhausted: DO NOT retry (amplification guard)
}

// CircuitBreaker: CLOSED -> OPEN on sustained failure -> HALF-OPEN probe.
type CircuitBreaker struct {
	mu          sync.Mutex
	state       int // 0=closed 1=open 2=half-open
	failures    int
	threshold   int
	openedAt    time.Time
	cooldown    time.Duration
}

func (c *CircuitBreaker) Allow() bool {
	c.mu.Lock()
	defer c.mu.Unlock()
	switch c.state {
	case 1: // open
		if time.Since(c.openedAt) > c.cooldown {
			c.state = 2 // half-open: allow a probe
			return true
		}
		return false // fail fast — don't call a known-sick dependency
	default:
		return true
	}
}

func (c *CircuitBreaker) Record(ok bool) {
	c.mu.Lock()
	defer c.mu.Unlock()
	if ok {
		c.failures = 0
		c.state = 0 // closed
		return
	}
	c.failures++
	if c.state == 2 || c.failures >= c.threshold {
		c.state = 1
		c.openedAt = time.Now()
	}
}

Code — the call path with all guards

func (g *Gateway) call(ep *Endpoint) (resp, error) {
	g.budget.OnRequest()
	if !g.breaker.Allow() {
		return fallback(), nil // circuit open: fast fallback, no doomed call
	}
	for attempt := 0; ; attempt++ {
		r, err := doWithTimeout(ep, g.timeout)
		g.breaker.Record(err == nil)
		if err == nil || !retryable(err) {
			return r, err
		}
		if attempt >= 1 || !g.idempotent || !g.budget.TryRetry() {
			return r, err // no retry: not idempotent, or budget empty
		}
		time.Sleep(backoffWithJitter(attempt)) // exp backoff + full jitter
	}
}

The experiment — blip → storm → recovery

A) NAIVE (3 retries, no budget, no breaker, no jitter):
   inject a 30s, 20%-error blip on the origin.
   -> each failing request retries 3x -> offered load ~1.6x during the
      blip -> origin pushed further past its knee -> error rate climbs ->
      MORE retries -> load stays high AFTER the blip ends -> metastable:
      the system does NOT recover on its own.

B) GUARDED (budget=10%, breaker, backoff+jitter, idempotent-only):
   same blip.
   -> retries capped at ~10% extra -> breaker opens when errors sustain,
      shedding doomed calls to a fast fallback -> offered load stays near
      baseline -> origin recovers as soon as the blip clears -> system
      self-heals.

Tasks

1. Implement the budget, breaker, and guarded call path.
2. Reproduce scenario A: plot offered load and error rate; show it stays elevated after the trigger clears (metastable).
3. Run scenario B: show offered load stays bounded and the system recovers immediately when the blip ends.
4. Add the metric retries / requests and confirm it's capped at the budget ratio in B and spikes uncontrolled in A (this is the gw-11 signal you'd alert on).

Acceptance

• A reproduced metastable collapse with naive retries, and self-healing with budget + breaker + jitter.
• A retries/requests metric you can point to as the early-warning signal, capped by the budget.

Discussion prompts

• Why does a budget succeed where a smaller per-request retry count fails under correlated failure? (Per-request counts still multiply when every request fails at once; a budget bounds the aggregate.)
• Why retry only idempotent requests, and how do you make a non-idempotent operation safely retryable? (Idempotency keys.)
• Hedging vs retries: when does adding a second request help (tail latency) and when does it hurt (broad slowness)? Where's the budget?