gw-11 — Analysis

The design review for the instrumentation you add in steps/.

Required behaviors

  1. RED per route/cluster. Rate, errors (by status class), and a latency histogram — not a mean — labeled by bounded dimensions.
  2. Correct percentiles. Percentiles computed from histogram buckets, aggregatable across instances (never average p99s).
  3. Trace context across the proxy. Extract incoming context, create a proxy span, inject downstream; never sever the trace.
  4. Exemplars. Access logs carry the trace id so logs↔traces↔metrics link.
  5. Symptom-based alerting. Alert on SLO burn rate, not on every error; pages are rare and user-facing.

Design decisions

  • Histogram for duration, counters for rate/errors/retries. Latency is a distribution; bucket it (with buckets chosen around the SLO, e.g. 1/5/10/25/50/100/250/500/1000ms). Errors are a counter labeled by status class so errors/rate is a ratio query. Retries get their own counter (the gw-06 amplification signal).

  • Bounded labels only. Label by route, method, status_class, cluster — never by user-id, full path, or trace-id (unbounded → cardinality explosion). High-cardinality detail lives in traces/logs, linked by exemplar.

  • OpenTelemetry for propagation. Use the OTel propagator to extract/inject W3C trace context, so the gateway interoperates with any backend. The span for the proxy hop makes gateway-added latency visible separately from origin latency.

  • Sampling: honor + force-error. Respect the incoming sampling decision (the flags bit) for cost control, but force-sample on error or high latency so the rare bad trace is never lost.

Tradeoffs worth flagging

  • Cost vs fidelity. More labels/buckets and more traces = more insight and more cost (storage, cardinality, collector load). Tune to the questions you actually need to answer; RED + golden signals + exemplars get you most of the way cheaply.

  • Head- vs tail-based sampling. Head-based is cheap and simple but may miss rare errors; tail-based keeps all errors/slow traces but needs a buffering collector and more resources. Many start head-based with error-forcing, then add tail-based where it pays.

  • Measurement point matters. Server-side latency excludes the queueing/network the user experiences (coordinated omission). Be explicit about where you measure (at admission? at response write?) and complement with client-side RUM where possible.

  • Observability of the observer. The instrumentation itself adds latency/CPU/allocations on the hot path (1M+ rps). Keep it cheap (lock-free counters, pre-allocated label sets); measure its overhead.

What production adds beyond this lab

  • A full pipeline: OTel Collector → metrics/trace/log backends, with recording rules and SLO/burn-rate alerts.
  • Tail-based sampling, exemplars wired end-to-end, and RUM for true user-experienced latency.
  • eBPF (Hubble/Pixie) for app-transparent network + L7 visibility to answer "app or network?" (gw-01/gw-09).
  • Control-plane observability: applied config version per node, NACK rate, reconcile latency, status-condition dashboards (gw-08/gw-10).
  • Automated anomaly detection and the shadow/canary diffing harness that gw-12's migrations depend on.