pa-09 — Reliability Engineering: SLOs, Error Budgets & Bulkheads

The Apple JD asks for knowledge of "observability and reliability engineering, including SLOs, distributed tracing, and circuit breakers." Tracing internals you built in gw-11; circuit breakers and adaptive concurrency in gw-06. This lab adds the reliability-engineering discipline an architect uses to govern the platform: SLOs and error budgets, multi-window burn-rate alerting (page on a real, sustained problem — never on a blip), and bulkheads (concurrency isolation).

You build the SLO + burn-rate engine and a bulkhead, and prove the alerting fires correctly.

1. What is it?

• SLI (indicator) — a measured quality signal (success ratio, p99 latency).
• SLO (objective) — a target for an SLI over a window (99.9% of requests succeed over 28 days).
• Error budget — 1 − SLO: the allowed unreliability. It's a budget you spend on risk (shipping, migrations) and that pauses feature work when exhausted. Reliability becomes arithmetic, not argument.
• Burn rate — how fast you're consuming the budget: observedErrorRate / errorBudget. Burn rate 1 = on track to exactly exhaust the budget over the window; >1 = too fast.
• Multi-window alerting — page only when a long window (it's sustained) and a short window (it's still happening) both show a high burn rate. This is the Google-SRE trick that stops a transient spike from paging you and stops a stale alert from firing after recovery.
• Bulkhead — a fixed concurrency budget per dependency, so a saturated/slow dependency can only exhaust its own slots, not the whole service's.

2. Why does it matter?

• SLOs turn "how reliable?" into a number and a budget. Without them, reliability is a vibe and an argument between dev (ship!) and ops (stop!). With them, a healthy budget licenses risky changes (migrations, gw-12) and a burning budget mandates stabilization — objectively. An architect sets the SLO framework that governs release velocity across teams.

• Alerting quality is a system property you design. Page on every error and you train people to ignore pages (alert fatigue → missed real incidents). Page only on user-visible SLO burn, and rarely. The multi-window/multi-burn-rate scheme is the state of the art, and it's a design decision, not a dashboard afterthought.

• Bulkheads (+ circuit breakers, gw-06) bound blast radius. The pa-01 blast-radius analysis tells you what a failure can reach; bulkheads, breakers, and timeouts are how you contain it so one sick dependency doesn't take the whole service down (cascading failure).

• Reliability is a first-class architecture concern, not an afterthought. The "-ilities" (availability, latency) trade off against each other and against velocity (CAP/PACELC, pa-06). An architect makes those trade-offs explicit, per service, via SLOs.

3. How does it work?

Error budget and burn rate

SLO{Target: 0.999} → ErrorBudget() = 0.001. BurnRate(total, bad) = errorRate / errorBudget. A 1% error rate against a 1% budget is burn rate 1 (sustainable for exactly the window); a 10% error rate is burn rate 10 (exhausts in 1/10 the window). The famous thresholds: ~14.4× burns 2% of a 30-day budget in 1 hour → page; ~6× over 6 hours → page or ticket.

Multi-window, multi-burn-rate alerting

PAGE   if long-window burn >= pageThreshold  AND short-window burn >= pageThreshold
TICKET if long-window burn >= ticketThreshold AND short-window burn >= ticketThreshold
else OK

SLO.Alert(long, short, pageThreshold, ticketThreshold) implements it. The long window proves the burn is sustained (not a blip); the short window proves it's still happening (so the alert clears quickly after recovery). Both must be hot to page — that conjunction is the whole trick. (Production uses several window pairs at different thresholds; the principle is identical.)

Bulkheads

Bulkhead{max} is a counting semaphore: TryAcquire takes a slot or rejects (fail fast, no queueing); Release returns it. Give each dependency its own bulkhead and a saturated dependency exhausts only its slots — the rest of the service keeps serving. Combine with circuit breakers (open when a dependency is failing) and timeouts (bound each call) and adaptive concurrency (gw-06) for the full stability toolkit (Nygard's Release It!).

Graceful degradation

When budget burns or a dependency is bulkheaded/broken, degrade, don't hard-fail: serve stale/cached data, drop optional features, shed low-priority traffic (gw-06). The architect decides, per feature, what "degraded but up" looks like — the Netflix "the show must go on" ethos.

4. Core terminology

5. Mental models

• The error budget is a spending account for risk. Reliability you don't spend is velocity left on the table; overspend (budget burned) and you must stop shipping and stabilize. It turns the dev-vs-ops tug-of-war into arithmetic both sides accept.

• Multi-window alerting is "is it raining AND still raining?" The long window asks "has it been raining a while?" (not a single drop); the short window asks "is it still raining right now?" (so you stop the alarm once it clears). You only sound the flood siren when both are true.

• Bulkheads are a ship's watertight compartments. A hull breach (saturated dependency) floods one compartment, not the whole ship. Without them, water (load) from one leak spreads everywhere and you sink (cascading failure).

• Page on symptoms, alert-elsewhere on causes. The pager is for "users are hurting" (SLO burn). Causes (pool exhausted, breaker open, high CPU) go to dashboards and tickets for diagnosis. Confusing the two is how you get alert fatigue.

6. Common misconceptions

• "Aim for 100% / five nines everywhere." 100% is the wrong target (impossible, and it forbids any risk/velocity). Pick the SLO users actually need; the error budget is the point, not a failure. Five nines is extremely expensive — justify it per service.

• "Alert on every error." That's alert fatigue: people mute the pager and miss the real incident. Alert on SLO burn rate (user impact), multi-window, rarely. Cause-metrics are for dashboards.

• "More retries/timeouts = more reliable." Past a point they amplify load (retry storms, gw-06) and turn a blip into an outage. Reliability comes from budgets, breakers, bulkheads, and shedding — bounding work, not adding it.

• "Reliability is an ops problem." It's an architecture decision: the -ilities trade off and are designed in (boundaries, async, bulkheads, consistency choices). SLOs make those trade-offs explicit and owned.

• "A bulkhead is the same as a circuit breaker." Bulkhead = bound concurrency (isolate saturation); circuit breaker = stop calling a failing dependency (fail fast on errors). They compose; you want both (plus timeouts + adaptive concurrency, gw-06).

7. Interview talking points

• "How do you set and use SLOs?" Define SLIs users feel (success ratio, p99 latency), set an SLO target (the reliability users need, not 100%), derive the error budget (1−SLO), and use the budget to govern velocity: healthy → ship/migrate; burning → freeze and stabilize. The architect provides this framework org-wide.

• "How do you alert on SLOs without alert fatigue?" Multi-window, multi-burn-rate: page only when a long window (sustained) AND a short window (ongoing) both exceed a high burn rate; ticket on slower burns; page on symptoms not causes. This avoids blip-paging and stale alerts.

• "How do you prevent cascading failure?" Bound the blast radius: timeouts on every call, circuit breakers (fail fast on a sick dependency, gw-06), bulkheads (per-dependency concurrency isolation), load shedding, and graceful degradation. Tie to pa-01's blast-radius analysis: contain what a failure can reach.

• "Bulkhead vs circuit breaker vs adaptive concurrency?" Bulkhead caps concurrency per dependency (isolation); breaker stops calls to a failing dependency (fail fast); adaptive concurrency (gw-06) infers the right in-flight limit from latency. Layer all three.

• "100% uptime — good goal?" No: impossible, and it eliminates the error budget that lets you ship and take risks. Pick the SLO users need; treat the budget as something to spend. Over-reliability is as much a failure as under-reliability (wasted velocity/cost).

8. Connections to other labs

• gw-06 — circuit breakers and adaptive concurrency; bulkheads here complete the stability toolkit.
• gw-11 — the SLIs (RED metrics, histograms, traces) these SLOs are computed from; burn-rate is the alerting layer over those signals.
• gw-12 / pa-08 — error budgets gate progressive delivery and promotion; a burning budget freezes risky rollouts.
• pa-01 — blast-radius analysis says what a failure reaches; bulkheads/breakers say how you contain it.
• pa-06 — consistency/availability choices (CAP/PACELC) are SLO decisions.