pa-03 — The Hitchhiker's Guide to Event-Driven Architecture

Companion to CONCEPTS.md, with the runnable event bus in src/go/eventbus/. This is the JD headline: designing async, event-driven systems — and owning their failure modes.

bash scripts/verify.sh runs the demo: an OrderPlaced event fans out to audit/fulfillment/fraud consumers; a flaky consumer retries then succeeds; a re-published event is deduped; a poison message lands in the DLQ:

stats: delivered=8 retried=3 deduped=3 deadLettered=1
  DLQ: order-poison on "fraud" after 2 attempts: cannot score: malformed

That single line is the whole discipline: deliver redundantly, dedup idempotently, and quarantine what can't be processed.

1. Fan-out pub/sub (bus.go)

Publish(event) delivers to every subscriber on the topic; the producer doesn't know its consumers (TestFanOut, TestTopicIsolation). That's the decoupling that makes events powerful: a new consumer Subscribes to an existing stream with zero producer change — the open/closed principle at the platform level. It's also the structural fix for the distributed monolith (pa-01): converting a synchronous A→B call into "A emits, B consumes" removes the temporal coupling that made A only as available as B.

2. At-least-once + retries (bus.go)

Real brokers deliver at-least-once (never lose, may duplicate). The bus models the retry side: a handler that returns an error is retried up to maxRetries. TestRetryThenSucceed shows a consumer failing twice then succeeding — Retried=2, Delivered=1, nothing dead-lettered. The architect's framing: distinguish transient failures (retry with backoff+jitter, gw-06) from permanent ones (don't waste retries — DLQ immediately).

3. Idempotent consumers — the heart of it

Because at-least-once means duplicates, consumers must be idempotent. The bus dedups by Event.ID (each subscription's seen set): TestIdempotentDedup publishes the same id three times and the handler runs once (Deduped=2). This is the single most important event-driven lesson:

Exactly-once delivery is a myth. Exactly-once effect is at-least-once delivery + an idempotent consumer. Stop chasing the former; build the latter.

It connects straight to pa-02 (idempotency keys) and gw-05 (push dedup) — the same trick at three layers.

4. The dead-letter queue (bus.go)

A poison message (malformed, a permanent bug) would retry forever and block the stream. After maxRetries, the bus moves it to the DLQ (TestPoisonGoesToDLQ: order-poison dead-lettered after 3 attempts, never delivered, the stream keeps flowing). In production you alert on DLQ depth and replay after fixing the consumer. A stream without a DLQ is one bad message away from a stall — and naive "retry forever" is how a blip becomes a retry storm (gw-06).

5. The trade-offs an architect owns

The code is small; the decisions are the job:

  • Sync vs async per edge — immediate-answer vs decoupling. You move complexity (eventual consistency, ordering, dedup), you don't delete it.
  • Delivery semantics — at-least-once + idempotency is the default; at-most-once only where loss is acceptable.
  • Ordering — only within a partition (pa-04). Partition by the key whose order matters; otherwise design consumers to tolerate reordering.
  • Backpressure vs shed — bounded queues block the producer or drop; unbounded queues are a deferred OOM. Choose per stream.
  • Choreography vs orchestration — emergent vs coordinated (a saga, pa-05). Decoupling vs visibility.

This synchronous bus is a teaching model; a production system puts the events on a durable, partitioned, replayable log — which is exactly pa-04.

6. Hands-on

cd src/go
bash ../scripts/verify.sh
go run ./cmd/ebsim

7. Exercises

  1. Async + backpressure: give each subscription a bounded channel and a worker; make Publish block (backpressure) or drop (shed) when full, and measure the difference under a slow consumer.
  2. Backoff + jitter: add exponential backoff with jitter between retries (reuse gw-06's idea) and show it avoids synchronized retry storms.
  3. Replay from DLQ: add Replay() that re-publishes DLQ'd events after you "fix" the handler; confirm idempotency prevents double-processing of anything already delivered.
  4. Consumer groups: extend so multiple instances of one logical consumer share the load (each event to one instance) — the bridge to pa-04.
  5. Choreographed saga: wire three consumers so OrderPlaced → reserve-inventory → charge-payment by reacting to each other's events; then feel the pain of no central view, and compare to pa-05's orchestrated saga.