pa-05 — The Hitchhiker's Guide to Outbox & Sagas

Companion to CONCEPTS.md, with the runnable patterns in src/go/delivery/. Two of the most-asked distributed-systems interview questions live here: the dual-write problem and cross-service transactions.

bash scripts/verify.sh runs the demo, and it's the whole lesson in two blocks:

broken dual write + crash:     state=CREATED, events published=0   <- LOST EVENT
outbox + crash + recovery:     delivered=1 duplicates=1            <- effectively once, no loss

saga: payment fails -> compensate in reverse
  [do]   reserve-inventory
  [do]   charge-payment FAILED
  [undo] reserve-inventory

1. The dual-write problem and the outbox (outbox.go)

A service updates its DB and publishes an event. These are two systems with no shared transaction. DualWriteBroken shows the failure: AtomicWrite-then-publish with a crash in between leaves state=CREATED but published=0 — the order exists, downstream never hears, permanent inconsistency. TestOutboxNoLostEventsVsBrokenDualWrite asserts that divergence.

The fix is AtomicWrite(key, val, event): the event is written into the DB in the same transaction as the state change, so they can never disagree (StateCount == OutboxCount, always). A separate relay (PollOnce) publishes unpublished outbox rows and marks them sent. Because publish-then-mark isn't atomic, the relay is at-least-once — TestOutboxAtomicAndAtLeastOnce simulates a crash after publish (PollOnceCrashBeforeMark), then recovery re-publishes, and the IdempotentSink absorbs the duplicate: delivered=1, duplicates=1. The outbox guarantees no loss; idempotency guarantees no double-apply; together = effectively-once.

Production note: a polling relay is the simple version; change-data- capture (tailing the DB's write-ahead log, e.g. Debezium) is the low-latency version. Same guarantee — it's db-03's WAL, read by a publisher.

2. Sagas: transactions without 2PC (saga.go)

You can't hold a 2PC lock across orders + payments + inventory (it blocks on coordinator failure and doesn't scale). A saga runs local transactions with compensating actions. Saga.Run executes forward; on a failure it runs Undo for completed steps in reverse order. TestSagaCompensatesInReverse: reserve OK, charge OK, ship FAILS → compensate [charge, reserve] (reverse). TestSagaHappyPath: all succeed, nothing compensated.

The architecture insight: you compensate, you don't roll back. You can't un-ship a package; you issue a refund/recall. Modeling each step's compensation forces clarity about what it really commits — and there's a visible window where partial effects exist (someone saw the inventory reserved before the refund). Sagas buy eventual consistency (committed-or-compensated), not isolation.

3. Crash recovery (saga.go)

A real orchestrator persists progress after each step (the OnProgress hook) so a restart knows where it was. Saga.RunFrom(start) models resumption: TestSagaResumeAfterCrash "crashes" after 2 forward steps and resumes from index 2 — step c runs once, steps a/b do not re-run. Without persisted progress, a crashed saga is a half-done workflow with no way to know what to undo — the difference between a recoverable system and a manual-cleanup incident.

4. The trade-offs an architect owns

  • Outbox vs CDC vs best-effort. Best-effort dual writes lose events — never use them for events that matter. Outbox (polling) is simple; CDC is lower-latency. Both are at-least-once.
  • Saga vs 2PC. 2PC for tightly-coupled resources in one boundary (if ever); sagas for cross-service workflows. Sagas trade isolation for availability + scale.
  • Orchestration vs choreography. Central coordinator (visible, controllable) vs event-reaction (decoupled, emergent). Complexity of the workflow decides.
  • Compensation reliability. A failed compensation has no automatic recovery — it's an SLO/alert concern (pa-09). Make compensations idempotent and as reliable as the forward steps.

5. Hands-on

cd src/go
bash ../scripts/verify.sh
go run ./cmd/sagasim

6. Exercises

  1. Publish to the real log: make the relay sink a pa-04 PartitionedLog; have an idempotent consumer (pa-03) process it.
  2. CDC instead of polling: model a relay that tails an append-only log (db-03) of committed writes rather than polling a table.
  3. Choreographed saga: rebuild the order saga as services reacting to each other's events (pa-03), then compare debuggability to this orchestrated version.
  4. Compensation failure: make a compensation fail and add retry-with-backoff + an alert when it can't complete (pa-09).
  5. Persisted orchestrator: store OnProgress to a file/DB and prove a process restart resumes the exact saga via RunFrom.