pa-09 — The Hitchhiker's Guide to SLOs & Reliability

Companion to CONCEPTS.md, with the runnable engine in src/go/reliability/. The reliability- engineering discipline that governs how an architect's platform is operated — and how fast it's allowed to change.

bash scripts/verify.sh runs the demo, and the alerting table is the whole lesson:

multi-window burn-rate alerting (page=14x, ticket=1x):
  sustained outage   longBurn=15.0x shortBurn=15.0x -> PAGE
  transient blip     longBurn= 1.0x shortBurn=20.0x -> OK      <- doesn't page!
  slow burn          longBurn= 2.0x shortBurn= 2.0x -> TICKET
  healthy            longBurn= 0.5x shortBurn= 0.5x -> OK
bulkhead isolation: depA saturated; depB unaffected

SLO{Target} → ErrorBudget = 1 − Target. BurnRate(total, bad) = errorRate / errorBudget: TestBurnRate shows a 1% error rate against a 1% budget is burn rate 1 (sustainable — exactly exhausts the window), and 10% is 10× (exhausts in 1/10 the time). That single number converts "are we okay?" into arithmetic. The architect-level use: a healthy budget licenses risk (ship features, run migrations — gw-12); a burning budget mandates stabilization. SLOs end the dev-vs-ops argument by making it math.

Float note: the tests compare burn rates with a tolerance because 1 − 0.99 isn't exact in floating point — a small but real reminder that you never == floats.

Alert(long, short, pageThreshold, ticketThreshold) pages only when both a long window (sustained, not a blip) and a short window (still happening, not already resolved) exceed the page threshold. TestMultiWindowDoesNotPageOnBlip is the key test: a short window at 20× burn but a long window at 1× → no page. That conjunction is the Google-SRE state of the art, and it's the difference between a pager people trust and one they mute. TestMultiWindowTicketsOnSlowBurn shows a slow sustained burn opening a ticket instead.

The design rule that falls out: page on symptoms (SLO burn = users hurting), not causes (CPU, pool exhaustion). Causes go to dashboards for diagnosis; only user impact wakes someone up.

3. Bulkheads (bulkhead.go)

Bulkhead{max} is a counting semaphore: TryAcquire takes a slot or rejects (fail fast, no queue). TestBulkheadsAreIndependent is the point: dependency A is saturated, yet B acquires fine — A's failure can't starve the whole service. This is how you contain the blast radius pa-01 told you to measure. Layer it with circuit breakers (stop calling a failing dependency) and adaptive concurrency (infer the right limit) from gw-06, plus timeouts, for Nygard's full stability toolkit.

4. Where this sits in the architect's toolkit

  • The SLIs come from gw-11 (RED metrics, histograms, traces); this is the alerting + governance layer on top.
  • The containment primitives (breakers, adaptive concurrency) are gw-06; bulkheads here complete them.
  • The error budget gates progressive delivery (gw-12) and GitOps promotion (pa-08) — a burning budget freezes risky rollouts.
  • Reliability is an architecture decision: the -ilities trade off (CAP/PACELC, pa-06) and are designed in, per service, with SLOs making the choice explicit and owned.

5. Hands-on

cd src/go
bash ../scripts/verify.sh
go run ./cmd/relsim

6. Exercises

  1. Real burn-rate windows: feed a time series of (total, bad) per minute and compute rolling 5m/1h/6h burn rates; reproduce the SRE multi-burn-rate alert matrix.
  2. Budget-gated deploys: block a (pa-08/gw-12) promotion when the remaining error budget is below a threshold.
  3. Bulkhead + breaker + timeout: wrap a flaky dependency call in all three (reuse gw-06's breaker/limiter) and show graceful degradation under load.
  4. Priority shedding: when a bulkhead is full, reject low-priority work first (criticality tiers) so the core path survives.
  5. SLO dashboard: compute remaining budget over a 28-day window and project the exhaustion date from the current burn rate.