pa-05 — Delivery Semantics: Outbox, Idempotency & Sagas

pa-03/04 gave you events and a log; this lab confronts the two hardest problems they create, both squarely in the Apple JD's "fault tolerance" and event-driven scope:

1. The dual-write problem — a service must update its database and publish an event. Two systems, no shared transaction: a crash between them leaves state and events permanently disagreeing. The transactional outbox solves it.
2. Distributed transactions across services — a workflow spans orders, payments, and inventory. You can't (and shouldn't) hold a 2PC lock across them. A saga — forward steps plus compensating actions — gives you atomicity-of-outcome without distributed locks.

You build both, with crash recovery, and prove them with tests.

1. What is it?

Delivery semantics describe what happens to a message under failure:

at-most-once:  may lose, never duplicate.
at-least-once: never lose, may duplicate.   ← the practical default
exactly-once:  a myth end-to-end; approximated by at-least-once + idempotency.

The transactional outbox makes a service's state change and its outgoing event atomic: both are written in one local DB transaction (the event goes into an outbox table). A separate relay later reads unpublished outbox rows and publishes them to the broker (pa-04), marking them sent. Because publish-then-mark isn't atomic, the relay is at-least-once — so consumers must be idempotent (pa-02/03).

A saga is a sequence of local transactions, each with a compensating transaction that semantically undoes it. Run forward; if a step fails, run the compensations for completed steps in reverse. Two flavors: orchestration (a central coordinator drives the steps — what we build) and choreography (services react to each other's events — pa-03). Crash recovery requires persisting saga progress so a restart resumes (forward) or completes compensation.

2. Why does it matter?

• The dual-write problem is everywhere and silently corrupts data. "Update the order, then publish OrderCreated" looks innocent and is a landmine: a crash in between means downstream never hears about the order (or hears about one that rolled back). Architects must recognize it on sight and reach for the outbox (or change-data-capture). This is a near-certain interview probe.

• Distributed 2PC doesn't scale and blocks on coordinator failure. Locking three services' databases together kills availability and throughput. Sagas trade ACID isolation for eventual consistency with guaranteed outcome (committed-or-compensated) — the pragmatic answer for cross-service workflows.

• "Exactly-once" claims are a red flag. The senior answer is at-least-once delivery + idempotent consumers = effectively-once. The outbox is the producer side of that; idempotency (pa-02) is the consumer side. Stating this crisply marks experience.

• Compensation is a domain-modeling skill. You can't rollback() a shipped package or a sent email; you compensate (refund, recall, apologize). Designing compensations forces clarity about what each step really commits — an architecture-level concern.

3. How does it work?

The transactional outbox

ONE local transaction:                    a separate relay process:
  UPDATE orders SET status='CREATED'        SELECT * FROM outbox WHERE NOT published
  INSERT INTO outbox (event)                FOR each: publish to broker; mark published
                                            (crash between publish & mark -> re-publish)

DB.AtomicWrite(key, val, event) commits both together. Relay.PollOnce publishes unpublished rows and marks them. DualWriteBroken shows the anti-pattern losing an event on a crash. The relay's PollOnceCrashBeforeMark

• PollOnce demonstrates at-least-once re-publication, absorbed by an IdempotentSink. (Change-data-capture — tailing the DB log, e.g. Debezium — is the alternative to a polling relay; same guarantee.)

Idempotent consumers

Because the relay is at-least-once, the consumer dedups by event id (IdempotentSink): a re-published event is delivered once. This is the same effectively-once recipe as pa-02 (idempotency keys), pa-03 (dedup), and gw-05 (push). The outbox guarantees no loss; idempotency guarantees no double-apply.

Sagas (orchestrated, with compensation)

forward:   reserve-inventory → charge-payment → schedule-shipment
on failure at charge-payment:
compensate: (charge not done) → release-inventory   [reverse order]

Saga.Run executes forward; on a step error it runs Undo for completed steps in reverse. The result records what completed and what was compensated. Compensations are semantic undo (refund, not "un-charge").

Crash recovery

A real saga persists progress after each step (OnProgress) so a restart can resume. Saga.RunFrom(start) models this: it skips the already-completed prefix and continues — or, on a later failure, compensates everything completed (including the pre-crash steps) in reverse. Without persisted progress, a crash leaves a half-done workflow with no way to know what to undo.

4. Core terminology

5. Mental models

• The outbox is "put the letter in the same envelope as the deed." You don't sign the deed (commit state) and then separately mail the notification (publish) — a fire between the two loses the notice. Instead you seal both in one envelope (one transaction); a courier (the relay) mails it whenever, even if you've gone home.

• A saga is checkout with a returns desk, not an escrow lock. 2PC freezes everyone's money in escrow until all agree (slow, blocking). A saga lets each shop complete its sale, and if a later shop declines, you walk the receipts back and get refunds (compensate). Eventually consistent, never blocking.

• You can't undo, you can only compensate. Time only runs forward. "Un-ship" isn't a thing; "issue a recall + refund" is. Modeling the compensation forces you to admit what a step truly commits in the real world.

• At-least-once + idempotency = effectively-once. Repeat it until it's reflex. The outbox guarantees the event is eventually delivered (≥1 times); the idempotent consumer guarantees the effect happens once. Neither alone is enough.

6. Common misconceptions

• "Just update the DB and publish — what could go wrong?" A crash between them. It's the dual-write problem and it will happen at scale. Use the outbox or CDC; never best-effort dual writes for events that matter.

• "Use a distributed transaction (2PC) across services." It blocks on coordinator failure, couples availability, and doesn't scale. Sagas are the answer for cross-service workflows; reserve 2PC for tightly-coupled resources within one boundary, if ever.

• "Sagas give you rollback." They give you compensation, which is not the same: there's a window where partial effects are visible (someone saw the inventory reserved), and compensations can themselves fail (needing retries/alerting). Sagas are eventual consistency, not isolation.

• "The outbox gives exactly-once." It gives at-least-once publication. You still need idempotent consumers. The outbox solves loss, not duplication.

• "Choreography is always simpler than orchestration." For 2-3 steps, yes; for complex workflows with compensation and visibility needs, choreography scatters the logic across event handlers and becomes un-debuggable. Orchestration centralizes the workflow (at the cost of a coordinator). Trade-off, not dogma.

7. Interview talking points

• "A service updates its DB and publishes an event — what's wrong?" The dual-write problem: non-atomic, so a crash desyncs state and events. Fix with the transactional outbox (event in the same DB transaction; a relay or CDC publishes it) — at-least-once, so consumers are idempotent. This is the canonical answer; have it instant.

• "How do you do a transaction across three services?" You don't — not 2PC. A saga: local transactions + compensating actions, orchestrated or choreographed, persisting progress for crash recovery. Accept eventual consistency (committed-or-compensated) and design the compensations.

• "Exactly-once?" Doesn't exist end-to-end. At-least-once delivery + idempotent consumers = effectively-once. Outbox (no loss) + idempotency (no double-apply).

• "Orchestration vs choreography for a saga?" Orchestration: a coordinator drives steps — visible, controllable, a central component. Choreography: services react to events — decoupled, but the workflow is emergent and hard to see/debug. Pick by workflow complexity and observability needs.

• "What happens if a compensation fails?" It must be retryable (idempotent) and alert/escalate if it can't complete — there's no further automatic recovery, so it becomes an operational/SLO concern (pa-09). Design compensations to be as reliable as the forward steps.

• "Outbox relay vs change-data-capture?" Polling the outbox table is simple and DB-agnostic; CDC (tailing the DB write-ahead log, e.g. Debezium) avoids polling latency and load but couples to the DB's log. Same at-least-once guarantee; pick by latency and operational fit.

8. Connections to other labs

• pa-02 (idempotency keys) — the consumer side that makes the outbox's at-least-once safe.
• pa-03 (event-driven) — the outbox publishes to that bus; sagas are orchestrated event workflows; choreography is the event-reaction alternative.
• pa-04 (log) — the relay publishes to a partitioned log; CDC tails a log just like it (db-03).
• pa-09 (reliability) — failed compensations and DLQ'd events become SLO/alerting concerns.
• gw-05 (push) — at-least-once + dedup at the edge; db-13/16 — transactions and the distributed-consistency fundamentals sagas relax.