gw-06 step 02 — Adaptive concurrency limiting

Goal

Replace a fixed concurrency limit with one that adapts from observed latency, so the gateway keeps latency bounded under overload by shedding at admission — Netflix's concurrency-limits idea in miniature.

Theory in one paragraph

By Little's Law, in-flight L = λ × W. While the system is below its knee, raising the in-flight limit raises throughput at ~flat latency. Past the knee, latency rises but throughput doesn't — you're just queueing. So: track the minimum latency ever seen (RTT_noload, the uncontended baseline) and the current latency. If current ≈ minimum, there's headroom — grow the limit. If current >> minimum, a queue is forming — shrink it. Reject requests over the limit immediately (fast 503) instead of letting them queue.

Code — src/go/limiter.go

package limit

import (
	"math"
	"sync"
	"sync/atomic"
	"time"
)

// Gradient2-style adaptive limiter (simplified). limit grows when
// current latency is near the no-load minimum, shrinks when it climbs.
type Adaptive struct {
	mu       sync.Mutex
	limit    float64
	minRTT   time.Duration // long-window minimum (no-load baseline)
	inflight atomic.Int64

	minLimit, maxLimit float64
}

func New() *Adaptive {
	return &Adaptive{limit: 20, minLimit: 1, maxLimit: 2000, minRTT: time.Hour}
}

// Acquire admits a request if under the current limit. Returns a release
// func and true, or false if the request must be shed.
func (a *Adaptive) Acquire() (release func(rtt time.Duration, dropped bool), ok bool) {
	if float64(a.inflight.Add(1)) > a.curLimit() {
		a.inflight.Add(-1)
		return nil, false // SHED at admission — bounded latency under load
	}
	start := time.Now()
	return func(rtt time.Duration, dropped bool) {
		a.inflight.Add(-1)
		if rtt == 0 {
			rtt = time.Since(start)
		}
		a.update(rtt, dropped)
	}, true
}

func (a *Adaptive) curLimit() float64 {
	a.mu.Lock()
	defer a.mu.Unlock()
	return a.limit
}

// update adjusts the limit from the latest sample (the control loop).
func (a *Adaptive) update(rtt time.Duration, dropped bool) {
	a.mu.Lock()
	defer a.mu.Unlock()

	if rtt < a.minRTT {
		a.minRTT = rtt // track the uncontended baseline
	}
	if dropped {
		a.limit = math.Max(a.minLimit, a.limit*0.9) // multiplicative decrease on error
		return
	}
	// gradient = noload / current, clamped to [0.5, 1.0]
	gradient := math.Max(0.5, math.Min(1.0,
		float64(a.minRTT)/float64(rtt)))
	// queue size headroom allows additive growth when gradient ~ 1.
	newLimit := a.limit*gradient + math.Sqrt(a.limit) // headroom term
	a.limit = math.Max(a.minLimit, math.Min(a.maxLimit, newLimit))
}

The experiment — fixed vs adaptive under a latency cliff

phase 1 (healthy): origin RTT ~5ms. Both fixed(=200) and adaptive serve
fine; adaptive settles near the throughput knee.

phase 2 (dependency slows to 100ms):
  FIXED limit 200: in-flight rushes to 200, queue builds, latency =
    200 × 100ms / arrival ... explodes into seconds; clients time out;
    retries pile on (step 03).
  ADAPTIVE: current RTT (100ms) >> minRTT (5ms) -> gradient ~0.05 ->
    limit collapses toward minLimit; excess is shed with fast 503s;
    served requests keep ~bounded latency. The system stays in control.

Tasks

1. Implement Adaptive; wrap the endpoint call (Acquire → call origin → release(rtt, dropped)).
2. Run the two-phase experiment; plot served-latency p99 and shed-rate for fixed vs adaptive across the latency cliff.
3. Show that adaptive needs no magic number: it finds a reasonable limit from latency alone, and recovers (grows back) when the dependency heals.

Acceptance

• Under the latency cliff, the fixed limiter's served latency explodes while the adaptive limiter holds latency ~bounded by shedding.
• The adaptive limit shrinks during overload and grows back on recovery, with no hand-tuned threshold.

Discussion prompts

• Why shed at admission rather than queue? (Bounded latency; a queued request that the client already timed out on is pure waste — "doomed work.")
• How is this analogous to TCP congestion control (AIMD, RTT-based)?
• Where does this sit relative to the connection pool (gw-04)? (Pool acquisition is itself a concurrency limit; reconcile the two so they don't fight.)