gw-11 — Observability for the Data Plane

The JD lists "uplift the … observability … posture for traffic traversing the cloud" as a primary responsibility, and twice demands the ability to "identify root causes using data." A gateway sits on the critical path for billions of requests; it is both the place where problems are seen first and the place best positioned to attribute them. Observability is not dashboards-after-the-fact — it's the designed-in ability to ask new questions of a running system and get answers fast enough to act.

This lab covers the three pillars (metrics, logs, traces) as they apply to a proxy, the RED and USE method, the discipline of golden signals and SLOs/error budgets, distributed tracing across a proxy (context propagation — the subtle part), tail-latency measurement, and eBPF-based visibility. You will propagate W3C trace context through a proxy and build RED metrics you can alert on.

1. What is it?

Observability is the property that lets you understand a system's internal state from its outputs. Three pillars, each answering a different question:

The methods that turn raw signals into judgment:

• RED (for request-driven services like a gateway): Rate, Errors, Duration — per route, per cluster, per status class.
• USE (for resources): Utilization, Saturation, Errors — CPU, memory, fds, connection pools, event-loop lag.
• Golden signals (Google SRE): latency, traffic, errors, saturation — the four to put on every dashboard and page on.

And the discipline that makes them actionable:

• SLI/SLO/error budget: pick a Service Level Indicator (e.g. p99 latency, success rate), set a target (SLO), and the allowed failure (error budget) governs how aggressively you ship vs stabilize (gw-12).

2. Why does it matter?

• The JD makes it a core deliverable and a core skill. "Uplift the observability posture" is the build side; "root cause using data" is the operate side. Every lab's debugging section in this phase assumes the signals this lab defines.

• A gateway is the system's best vantage point. It sees every request, every route, every backend, every status. RED metrics at the gateway attribute a problem to a route/cluster faster than any single service can — which is why edge observability is disproportionately valuable.

• Tail latency is invisible without the right metrics. Averages lie; p99/p99.9 is what users feel and where retries/hedging/outliers (gw-06) show up. You must measure distributions (histograms), not means — and know why you can't average percentiles across instances.

• Tracing across a proxy is where context-propagation bugs live. If the gateway doesn't propagate (and create) trace context correctly, every downstream trace is broken — you lose the ability to see where latency goes. This is a specific, common, high-impact gateway responsibility.

3. How does it work?

RED metrics with the right instruments

rate     = counter: requests_total{route,cluster,method}        (rate() in queries)
errors   = counter: requests_total{...,status_class="5xx"}      (errors/rate ratio)
duration = HISTOGRAM: request_duration_seconds_bucket{route,...} (p50/p99 via quantiles)

Duration must be a histogram (bucketed), not a gauge or summary, so you can compute percentiles and aggregate across instances correctly. A counter for retries (retries_total / requests_total — the gw-06 amplification signal) and gauges for connections, pool utilization, and event-loop lag round it out.

Why you can't average percentiles

p99 of instance A and p99 of instance B do not average to the fleet p99 — percentiles aren't linear. The correct approach: export histogram buckets per instance, sum the buckets fleet-wide, then compute the percentile from the merged histogram. Getting this wrong is a classic "our dashboards say p99 is fine but users are unhappy" bug.

Distributed tracing across the proxy

A trace is a tree of spans (one unit of work each), stitched by a propagated context. The gateway must:

1. Extract incoming context from headers (W3C traceparent, or b3 for Zipkin lineage).
2. Create its own span for the proxy hop (so gateway latency is visible as its own segment).
3. Inject the (possibly new/child) context into the request it forwards to the origin — so the origin's spans join the same trace.

traceparent: 00-<32-hex trace-id>-<16-hex span-id>-01
             │   │                   │               └ flags (01 = sampled)
             │   │                   └ this hop's span id
             │   └ the trace id (same across the whole request tree)
             └ version

If the gateway drops or fails to forward traceparent, the origin starts a new trace and the chain is severed — you can no longer see "the gateway added 3ms, the origin added 200ms." Propagation is the whole game.

Sampling

Tracing every request is expensive; sampling keeps cost bounded. Two models: head-based (decide at the start, propagate the decision in the flags bit — simple, may miss rare errors) and tail-based (buffer spans, decide after seeing the whole trace — can keep all errors/slow traces, costs a collector that buffers). The gateway typically honors the incoming sampling decision and can force-sample errors.

Access logs: structured and sampled

Per-request structured logs (JSON) with route, status, duration, bytes, upstream, retry count, trace id. High cardinality and volume → sample the successes, keep the errors. The trace id in the log line is what links logs ↔ traces ↔ metrics ("exemplars").

eBPF and the data plane

eBPF (Cilium/Hubble, Pixie) observes the kernel network path without app changes: per-connection bytes, retransmits, DNS, even L7 with parsers — the gw-01/gw-09 layer made visible. Great for "is it the app or the network?" questions where app metrics can't see.

SLOs and error budgets (the operating discipline)

SLO: 99.9% of requests succeed over 28 days
error budget = 0.1% of requests may fail = your "failure currency"
  budget healthy  -> ship features / risky migrations (gw-12)
  budget burning  -> freeze changes, stabilize
burn-rate alerts: page when you'll exhaust the budget fast, not on every blip

4. Core terminology

5. Mental models

• Metrics are the dashboard gauges; traces are the dashcam; logs are the black box. Gauges tell you something's wrong and roughly where; the dashcam (trace) shows the moment across the whole journey; the black box (logs) has the per-event detail for forensics. You need all three, and the ability to jump between them (exemplars).

• A percentile is a property of a distribution, not a number you can add. You can sum counts in buckets and recompute, but you cannot average p99s. Treat latency as a shape (histogram), not a value.

• Trace context is a baton in a relay. Each runner (service) must take the baton (extract), run their leg (their span), and pass it on (inject). Drop the baton at the gateway and the rest of the race is untimed — you'll never know which leg was slow.

• The error budget is a spending account. Reliability you don't need is money left on the table; reliability you've overspent (budget burned) means stop shipping and fix. It converts "how reliable should we be?" from an argument into arithmetic — and it's what licenses a risky migration (gw-12) when the budget is healthy.

• Cardinality is the credit-card bill of observability. Every new label dimension (especially unbounded ones like user-id or full URL) multiplies series and cost. Label by bounded dimensions (route, status class, cluster); put the high-cardinality stuff in traces/logs.

6. Common misconceptions

• "Average latency is fine." Averages hide the tail; a 5ms mean can hide a 2s p99 that's hurting 1% of users (and 1% of 1M rps = 10k unhappy users/sec). Always measure and alert on percentiles via histograms.

• "Just trace everything." Tracing every request at high rps is expensive and noisy; sample (head- or tail-based), but force-keep errors and slow traces. The skill is sampling and still catching the rare bad trace.

• "The gateway doesn't need to do anything for tracing." It must extract, create its own span, and inject context downstream. Silent failure to propagate breaks every downstream trace — a high-impact bug unique to proxies.

• "More dashboards = more observability." Observability is the ability to answer new questions, not to pre-build every chart. RED + golden signals + exemplars + good labels beat 50 bespoke panels.

• "Alert on every error." That trains people to ignore alerts. Alert on symptoms (SLO burn rate, user-facing latency/errors), page rarely, and let dashboards/traces handle diagnosis. Cause-based alerts are for runbooks, not pagers.

7. Interview talking points

• "p99 latency just doubled — walk me through it." The flagship ops-round answer (gw-00 INTERVIEW.md): scope (region/route/cluster) → golden signals (which moved first) → correlate with deploys/config/ origin events → walk the request path (TLS/filters/LB/pool/origin) → falsifiable hypothesis → confirm with one metric → mitigate then fix. Name the metrics at each step.

• "How do you measure latency correctly across a fleet?" Histograms per instance, summed bucket-wise, percentiles computed from the merged histogram — because percentiles don't average. This single answer separates people who've operated services from those who haven't.

• "What does the gateway do for distributed tracing?" Extract incoming traceparent/b3, create a span for the proxy hop, inject context downstream so origin spans join the trace; honor (and force-on-error) the sampling decision; attach the trace id to access logs as an exemplar. Dropping propagation breaks every downstream trace.

• "What's RED vs USE and which for a gateway?" RED (Rate/Errors/ Duration) for the request flow; USE (Utilization/Saturation/Errors) for resources (CPU, fds, pools, event-loop lag). A gateway needs both: RED for user impact, USE to find the saturated resource behind it.

• "How do SLOs change how you work?" They convert reliability into a budget: healthy budget → ship/migrate aggressively (gw-12); burning budget → freeze and stabilize. Alert on burn rate (symptom), not on every error (cause), so pages are rare and meaningful.

• "Your dashboards look healthy but users complain — why?" Averaged percentiles, too-coarse labels hiding a bad route, success-only sampling hiding the error traces, or the metric measured at the wrong point (server-side latency excludes queueing/network the user experiences). Know these failure modes of observability itself.

8. Connections to other labs

• gw-01 / gw-09 (L4 / K8s) — accept-queue depth, conntrack, retrans, event-loop lag are the saturation signals; eBPF makes them visible.
• gw-03 (API gateway) — the outbound filter is where access logs, metrics, and span completion happen; per-filter timings are RED's detail.
• gw-04 (connection management) — connections.created.rate is the churn metric; pool utilization and acquisition wait are saturation.
• gw-06 (resilience) — retries/requests, circuit state, ejection events, and the adaptive-concurrency limit vs in-flight are the signals you alert on; observability is what makes resilience tunable.
• gw-08 / gw-10 (control plane) — applied config version, NACK rate, reconcile latency, and status conditions are the control-plane's observability.
• gw-12 (migration) — shadow/canary comparisons, SLO-gated rollout, and automated rollback all run on the signals defined here.