pa-03 — Event-Driven Architecture & Async Patterns

This is the headline of the Apple JD: "experience designing event-driven architectures and asynchronous communication patterns." Synchronous request/reply (pa-02) couples services in time — the caller is only as available as the callee, and latency compounds down the chain. Events break that coupling: a producer emits a fact and moves on; consumers react when they're ready. The price is eventual consistency — ordering, duplicate delivery, and "where did my event go" become your problems.

This lab builds an event bus that makes those problems concrete: fan-out pub/sub, at-least-once delivery with retries, idempotent consumers (the dedup that makes at-least-once safe), and a dead-letter queue for poison messages.

1. What is it?

Event-driven architecture is a style where services communicate by producing and consuming events (immutable facts: "OrderPlaced") over a broker, rather than calling each other directly. Three sub-styles (increasing power and complexity):

• Event notification — "something happened, go look": a thin event; the consumer calls back for details. Low coupling, extra round-trips.
• Event-carried state transfer — the event carries the data the consumer needs; no callback. Decoupled, but data is duplicated and can be stale.
• Event sourcing — the event log is the source of truth; state is a fold over events (CQRS often pairs with it). Powerful audit/replay, big complexity jump.

And two coordination styles for multi-step workflows:

• Choreography — services react to each other's events; no central brain. Loose coupling, but the workflow is emergent and hard to see.
• Orchestration — a coordinator drives the steps (a saga, pa-05). Visible and controllable, at the cost of a central component.

2. Why does it matter?

• It's the antidote to the distributed monolith (pa-01). Synchronous call chains make services co-available and co-failing; converting an edge to an event breaks the temporal coupling. An architect's most common "fix this design" move is "make this async."

• It's how platforms absorb load and spikes. A queue/log levels load (producers and consumers run at their own pace) and provides a buffer during downstream outages — the consumer catches up later instead of the producer failing now.

• The hard parts are where architects earn their keep. Delivery semantics (at-least-once vs exactly-once), ordering, idempotency, duplicates, poison messages, and the dual-write problem (pa-05) are subtle and dangerous. Knowing the patterns cold — and that exactly-once is a myth you approximate with at-least-once + idempotency — is the differentiator.

• It enables independent evolution and fan-out. New consumers subscribe to an existing event stream without the producer knowing or changing — the open/closed principle at the platform level.

3. How does it work?

Pub/sub and fan-out

A producer publishes to a topic; the broker fans out to every subscriber. The producer doesn't know its consumers — that's the decoupling. Bus.Publish delivers to all subscriptions on a topic; adding a consumer is a Subscribe, no producer change.

Delivery semantics (pick your poison)

at-most-once:    deliver, don't retry. Simple; loses messages on failure. Rarely OK.
at-least-once:   deliver, retry until acked. No loss; DUPLICATES possible. The default.
exactly-once:    a MYTH end-to-end. Approximated by at-least-once + IDEMPOTENT consumers.

The bus delivers at-least-once with bounded retries. A handler that errors is retried up to maxRetries; if it keeps failing, the event is dead-lettered rather than blocking the stream forever.

Idempotent consumers (what makes at-least-once safe)

Because retries and broker redelivery cause duplicates, consumers must be idempotent: processing the same event id twice has the same effect as once. The bus dedupes by Event.ID (the consumer's seen set). Publish the same id three times → the handler runs once. This is the practical "effectively-once" the industry actually ships (pa-02 idempotency keys, gw-05 push dedup).

Dead-letter queue (don't let one poison message stop the line)

A message that can never succeed (malformed, a permanent bug) would, without a DLQ, retry forever and block its partition. The DLQ moves it aside after N attempts so the stream keeps flowing; operators inspect and replay DLQ'd events after fixing the consumer.

Backpressure

When consumers fall behind, something must give: a bounded queue either blocks the producer (backpressure — the gw-01/gw-02 idea) or drops (load-shedding, gw-06). An unbounded queue is a latent OOM. The architect chooses the bound and the overflow policy per stream.

4. Core terminology

5. Mental models

• Sync is a phone call; async is a mailbox. A call needs both parties present now (temporal coupling); a letter is dropped and read later (decoupled). You can't lose a call you didn't make, but a mailbox keeps working when the recipient is out. Choose per edge; you move the complexity, you don't delete it.

• At-least-once + idempotency = effectively-once. Stop chasing exactly-once delivery (a distributed-systems unicorn). Deliver redundantly and make the effect idempotent. The dedup set is the whole trick.

• The DLQ is the ER triage room. You don't let one critically-broken patient (poison message) block the whole queue; you move them aside, keep the line moving, and treat them separately. A stream with no DLQ is a stream one bad message away from a stall.

• A queue is a shock absorber, not a warehouse. It smooths spikes and buffers brief outages. If it's growing unboundedly, your consumers are permanently too slow — add capacity or shed; don't add disk.

6. Common misconceptions

• "Exactly-once delivery exists." Not end-to-end. Brokers offer at-least-once (or "exactly-once" within their own boundary via transactions, but the moment your consumer has side effects, you need idempotency). Design for at-least-once + idempotent consumers.

• "Async is always better / decoupling is free." Async adds eventual consistency, ordering concerns, and operational opacity (where's my event?). For an immediate answer the caller can't proceed without, sync is correct. The architect puts each edge on the right side.

• "Events guarantee order." Only within a partition (pa-04). Across partitions/topics there's no global order. Design consumers to tolerate reordering, or partition by the key whose order matters.

• "Just retry forever." Forever-retrying a poison message blocks the stream and can become a retry storm (gw-06). Bound retries + DLQ + backoff/jitter.

• "Choreography is simpler, always use it." Pure choreography makes a multi-step workflow invisible and hard to change/debug — coupling hidden in event chains. For complex workflows with compensation, an orchestrated saga (pa-05) is clearer. Trade-off, not dogma.

7. Interview talking points

• "When do you use events vs synchronous calls?" Sync when the caller needs an immediate answer to proceed (own the latency/availability cost: timeouts, breakers, bulkheads — gw-06/pa-09). Async to decouple in time, fan-out to many consumers, level load, or build an audit trail — accepting eventual consistency. Decide per edge; name the cost you're taking on.

• "How do you handle duplicate deliveries?" Idempotent consumers: dedup by event id (or idempotency key, pa-02). State plainly that exactly-once is at-least-once + idempotency; don't claim the broker gives you exactly-once for free.

• "How do you handle a message that can't be processed?" Bounded retries with backoff+jitter, then a DLQ so it doesn't block the stream; alert on DLQ depth; replay after fixing the consumer. Distinguish transient (retry) from permanent (DLQ immediately) failures.

• "Choreography or orchestration?" Choreography for simple, loosely coupled reactions; orchestration (a saga, pa-05) when a multi-step workflow needs visibility, ordering, and compensation. The trade-off is decoupling vs observability/control.

• "How do you prevent a slow consumer from taking everything down?" Bounded queues (backpressure or shed), consumer-group scaling (pa-04), per-consumer isolation (bulkheads, pa-09), and DLQ for poison. An unbounded buffer is a deferred OOM.

• "Notification vs event-carried state vs event sourcing?" Increasing data/authority in the event: notification (thin, callback for details), carried-state (self-contained, data duplicated), sourcing (the log is truth, replayable, big complexity). Pick the least powerful that meets the need.

8. Connections to other labs

• pa-01 — events break the temporal coupling that creates distributed monoliths.
• pa-02 — event schemas are contracts (schema registry + compat); idempotency keys make at-least-once safe.
• pa-04 — a partitioned log is the durable, replayable, ordered backbone this bus abstracts; ordering is per-partition.
• pa-05 — the outbox solves the dual-write problem (DB + publish atomically); sagas are orchestrated event workflows with compensation.
• gw-05 — push delivery is at-least-once + idempotent dedup at the edge; gw-06 — retries/backoff/jitter and load shedding.