gw-11 step 01 — RED metrics and trace-context propagation

Goal

Instrument the gw-03 gateway with RED metrics (a latency histogram, not a mean) and propagate W3C trace context across the proxy so downstream traces stitch together. Then practice the "p99 doubled" debugging drill against your own metrics.

Code — RED metrics (Prometheus client)

package obs

import "github.com/prometheus/client_golang/prometheus"

var (
	// Rate + Errors: one counter, labeled by status class.
	Requests = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "gw_requests_total",
		Help: "requests by route/cluster/status class",
	}, []string{"route", "cluster", "method", "status_class"})

	// Retries: the gw-06 amplification signal (alert on retries/requests).
	Retries = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "gw_retries_total",
	}, []string{"route", "cluster"})

	// Duration: HISTOGRAM (so percentiles aggregate correctly across
	// instances). Buckets chosen around the SLO.
	Duration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
		Name:    "gw_request_duration_seconds",
		Buckets: []float64{.001, .005, .01, .025, .05, .1, .25, .5, 1, 2.5},
	}, []string{"route", "cluster"})

	// Saturation gauges (USE): pool + event-loop health.
	PoolInUse = prometheus.NewGaugeVec(prometheus.GaugeOpts{
		Name: "gw_pool_inuse_connections",
	}, []string{"cluster"})
)

func init() {
	prometheus.MustRegister(Requests, Retries, Duration, PoolInUse)
}

Record in the gw-03 outbound filter (where the request is done):

func (RedMetrics) Apply(c *gw.RequestContext) {
	statusClass := fmt.Sprintf("%dxx", c.Resp.Status/100)
	obs.Requests.WithLabelValues(c.RouteName, c.RouteName, c.Req.Method, statusClass).Inc()
	obs.Duration.WithLabelValues(c.RouteName, c.RouteName).
		Observe(time.Since(c.StartedAt()).Seconds())
}

Cardinality discipline: labels are bounded (route, cluster, method, status_class). Never label by user-id, full path, or trace-id — those go in traces/logs, linked by exemplar.

Querying the right percentile (PromQL)

# p99 latency per route, aggregated across ALL instances correctly:
histogram_quantile(0.99,
  sum by (le, route) (rate(gw_request_duration_seconds_bucket[5m])))

# error ratio per route:
sum by (route) (rate(gw_requests_total{status_class="5xx"}[5m]))
  / sum by (route) (rate(gw_requests_total[5m]))

# the retry-storm early warning (gw-06):
sum(rate(gw_retries_total[1m])) / sum(rate(gw_requests_total[1m]))

Note you sum the buckets first, then take the quantile — you cannot average per-instance p99s.

Code — trace context across the proxy (OpenTelemetry)

package obs

import (
	"net/http"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/trace"
)

var propagator = propagation.TraceContext{} // W3C traceparent
var tracer = otel.Tracer("gateway")

// TraceProxy wraps the endpoint: extract incoming context, start a span
// for the proxy hop, inject context into the OUTBOUND request so the
// origin's spans join this trace.
func TraceProxy(c *gw.RequestContext, forward func(*http.Request) (*http.Response, error)) (*http.Response, error) {
	ctx := propagator.Extract(c.Req.Context(),
		propagation.HeaderCarrier(c.Req.Header)) // take the baton

	ctx, span := tracer.Start(ctx, "gateway.proxy",
		trace.WithSpanKind(trace.SpanKindServer))
	defer span.End() // this hop's span = gateway-added latency, visible separately

	out := c.Req.Clone(ctx)
	propagator.Inject(ctx, propagation.HeaderCarrier(out.Header)) // pass the baton on

	resp, err := forward(out)
	if err != nil {
		span.RecordError(err)
	} else {
		span.SetAttributes(/* http.status_code */)
	}
	// Exemplar: put the trace id in the access log so logs<->traces link.
	c.Attributes["trace_id"] = span.SpanContext().TraceID().String()
	return resp, err
}

Tasks

1. Add the RED metrics + /metrics endpoint to the gw-03 gateway; generate load and confirm rate/errors/duration appear per route.
2. Wire OTel propagation; run gateway → origin both instrumented, send a request with a traceparent, and confirm in your trace backend (Jaeger/Tempo) that one trace spans gateway + origin (not two disconnected traces).
3. Break propagation on purpose (don't inject); show the origin starts a new trace and the chain is severed — the high-impact gateway bug.
4. Run the "p99 doubled" drill: inject latency on one cluster; using only your PromQL, scope it to the route/cluster, confirm with the histogram, and pivot to a slow trace via the exemplar.

Acceptance

• RED metrics with a correctly-aggregating p99 (summed buckets, then quantile) and a working retries/requests ratio.
• A single stitched trace across gateway + origin; a demonstrated broken trace when propagation is dropped.
• You can drive the "p99 doubled" investigation end-to-end on your own signals.

Discussion prompts

• Why a histogram (not a summary/gauge) for latency, and why can't you average p99 across instances?
• What exactly breaks downstream if the gateway forgets to inject traceparent? Why is the proxy uniquely responsible here?
• Which of these signals would you put on a pager vs a dashboard, and why? (Symptom/SLO-burn on the pager; cause metrics on the dashboard.)