gw-11 — The Hitchhiker's Guide to Data-Plane Observability

Companion to CONCEPTS.md, with the runnable primitives in src/go/obs/. The JD demands "identify root causes using data" twice — this lab builds the data, correctly.

A gateway sits on the critical path for billions of requests; it sees problems first and is best placed to attribute them. But observability is full of subtle ways to lie to yourself. This lab builds three primitives that get it right: a mergeable histogram (so fleet percentiles are correct), W3C trace propagation (so traces don't break at the proxy), and RED metrics (the signals you alert on).

Run bash scripts/verify.sh:

percentiles can't be averaged across instances:
  instance A p99 = 0.001s
  instance B p99 = 0.100s
  AVG of p99s    = 0.051s   <- WRONG
  MERGED p99     = 0.100s   <- RIGHT (sum buckets, then quantile)
trace propagation through the gateway:
  incoming traceparent: 00-aaaa...-bbbb...-01
  forwarded downstream: 00-aaaa...-cccc...-01
  same trace-id? true   new span-id? true

1. You cannot average percentiles (histogram.go)

This is the single most important — and most violated — rule of latency monitoring. A percentile is a property of a distribution, not a number you can add. TestPercentilesMergeNotAverage makes it undeniable: two instances, one all-1ms and one all-100ms. Averaging their p99s gives 51ms — a number that describes neither instance and isn't the fleet's p99. The fleet p99 is 100ms (half the fleet is slow), and the only correct way to get it is to merge the histogram buckets, then compute the quantile (merged := a.Clone(); merged.Merge(b); merged.Quantile(0.99)).

That's why latency must be a histogram (bucketed counts), not a gauge or a pre-computed summary: histograms with shared bounds are additive, so you sum per-instance buckets fleet-wide and compute the percentile from the merged distribution. TestHistogramQuantile pins the bucket-based quantile (p99 of "99% fast, 1% slow" is the fast bucket; p99.9 is the slow tail). Get this wrong and your dashboards say "p99 is fine" while users suffer — the classic "observability of the observer" failure.

In PromQL this is histogram_quantile(0.99, sum by (le) (rate(..._bucket[5m]))) — sum the buckets, then quantile. The lab's Merge is exactly that sum by (le).

2. Trace propagation: don't sever the trace (trace.go)

A trace is a tree of spans stitched by a propagated context. A proxy has a unique, high-impact responsibility: extract the incoming traceparent, create its own span for the proxy hop (same trace-id, new span-id), and inject it into the request it forwards — so the origin's spans join the same trace.

ParseTraceparent parses the W3C format (00-<32hex trace>-<16hex span>-<2hex flags>) and rejects malformed or all-zero ids (TestParseTraceparent). NewChild is the propagation rule: keep the trace-id and sampling decision, mint a new span-id (TestTracePropagationKeepsTrace). If the gateway drops the traceparent (or — worse — starts a fresh trace), the origin begins a disconnected trace and you lose the ability to see "the gateway added 3ms, the origin added 200ms." That severed-trace bug is unique to proxies and is exactly what the demo's same trace-id? true line guards against.

Extract/Inject work against any header carrier (http.Header.Get/ Set), and TestExtractMissingIsNotOK ensures an absent header isn't silently treated as a valid context.

3. RED metrics (red.go)

RED tracks Rate, Errors, Duration per route — the request-driven service's golden signals. TestRED records 100 OK + 5 5xx and checks Rate=105, ErrorRatio≈0.0476, P99=5ms. Two design choices matter:

  • Bounded labels only (route, status class). Never label by trace-id, user-id, or full path — that's a cardinality explosion that bankrupts your metrics store. High-cardinality detail goes in traces/logs, linked by an exemplar (the trace-id on a log line / metric sample).
  • The retry ratio is a first-class signal. RetryRatio = retries/requests is the early warning for a retry storm (gw-06): when it climbs toward your budget, you're amplifying. Alert on it.

4. Putting it on the pager (the operating discipline)

The CONCEPTS file covers SLOs, error budgets, and burn-rate alerting in depth. The one-line rules:

  • Alert on symptoms, not causes. Page on SLO burn rate (user-facing latency/errors), not on every 5xx. Cause metrics (pool exhaustion, circuit state, NACKs) live on dashboards for diagnosis, not on the pager.
  • The "p99 doubled" drill (gw-00 INTERVIEW.md): scope → golden signals → correlate with deploys/config/origin → walk the path → falsifiable hypothesis → confirm with one metric → mitigate then fix. Every primitive here feeds a step of that drill.

5. Hands-on

cd src/go
bash ../scripts/verify.sh        # tests + the two demos

# Wire RED + trace propagation into the gw-03 gateway's outbound filter
# and endpoint, then drive load and watch a stitched trace + a correct
# fleet p99 (exercise §6.1).

6. Exercises

  1. Instrument gw-03: add an outbound RED filter and trace propagation in the proxy endpoint; expose /metrics; confirm a request produces a single trace spanning gateway+origin and a correctly-merged p99.
  2. Burn-rate alert: implement an SLO (e.g. 99.9% success) and a multi-window burn-rate alert; show it pages on fast budget burn but not on a single blip.
  3. Tail-based sampling: buffer spans for a trace and keep it only if it errored or exceeded a latency threshold; show you catch the rare bad trace while sampling the rest.
  4. gRPC status from trailers (gw-02): make Record read grpc-status from trailers so a 200 + grpc-status 14 is counted as an error — a gateway that only reads :status undercounts failures.
  5. Coordinated omission: drive load with a fixed-rate generator (wrk2) vs a closed-loop one (wrk) and show how the latter hides the tail — then explain why you measure at admission, not just at response write.