pa-04 — The Hitchhiker's Guide to the Partitioned Log

Companion to CONCEPTS.md, with the runnable log in src/go/plog/. pa-03 gave you the event patterns; this is the substrate — the one data structure under Kafka, Pulsar, and NATS JetStream.

bash scripts/verify.sh runs the demo: all acct-1 records share a partition (ordered), a consumer group commits and resumes, partitions get assigned across consumers two ways, and retention truncates while offsets stay absolute.

1. The log + partitioning (log.go)

A PartitionedLog is n append-only slices. Produce(key, value) hashes the key (PartitionFor) to pick a partition, appends, and returns a monotonic offset. The two facts that flow from this:

  • Same key → same partition → ordered. TestKeyMapsToSamePartition and TestPerPartitionOrdering prove all acct-1 records land together with offsets 0,1,2,… in order. Ordering is only within a partition — the single most important property to internalize.
  • Offsets are absolute and stable. Consume(partition, fromOffset, max) returns records at/after an offset; HighWatermark is the next offset. A consumer polls from where it left off — reading doesn't consume (unlike a queue), so many consumers and replay coexist.

The architecture decision hiding here is the partition key. Key by accountId and you get per-account ordering but a hot account is a hot partition (skew); key randomly and you get even load but no ordering. That trade-off is yours to own.

2. Consumer groups + offset resume (groups.go)

GroupOffsets tracks per-(group, partition) progress. TestConsumerGroupOffsets walks the lifecycle: a fresh group starts at 0, reads 2, commits next=2, "restarts," and resumes at offset 2 — and a different group reads the same log independently from 0. That independence is the log's superpower: billing and analytics consume the same orders stream at their own pace, and a new consumer can replay history.

Commit timing is a delivery-semantics choice: commit after processing for at-least-once (a crash re-delivers — needs idempotency, pa-03); commit before for at-most-once (a crash loses it). The architect picks per consumer.

3. Assignment & rebalancing (groups.go)

A consumer group splits a topic's partitions across its members. AssignRange (contiguous chunks; earlier consumers get the extra) and AssignRoundRobin (one at a time) are Kafka's classic strategies — TestRangeAssignment/TestRoundRobinAssignment pin both for 5 partitions over 2 consumers. The ceiling: parallelism caps at the partition count — a 5th consumer on a 4-partition topic sits idle. Rebalancing on membership change is disruptive (stop-the-world reassign); production adds sticky/cooperative rebalancing to minimize movement — the exact same stability goal as gw-04's subsetting ring and pa-06's consistent hashing.

4. Retention (log.go)

Truncate(partition, beforeOffset) drops old records, but survivors keep their original absolute offsets (TestRetentionKeepsOffsetsStable: after truncating below 3, the first record is still offset 3, and the high watermark is unchanged). So committed offsets stay valid; a consumer that fell behind the retention window simply resumes at the earliest retained record — having silently lost the truncated data. That's the operational risk: retention is a moving cliff behind your slowest consumer, so you monitor consumer lag (HighWatermark − committed) as an SLI (pa-09) and alert before the cliff.

5. Why this is the architect's keystone for eventing

Once the log is concrete, the whole eventing stack composes:

  • pa-03's pub/sub, at-least-once, and DLQ run on this log.
  • pa-05's outbox publishes to it; idempotent consumers handle its at-least-once redelivery.
  • Event sourcing is "the log is the source of truth; state is a fold over it" — replay from offset 0.
  • Choosing Kafka vs Pulsar vs NATS vs RabbitMQ becomes a concrete comparison of this model (log + groups + retention) vs a queue model.

6. Hands-on

cd src/go
bash ../scripts/verify.sh
go run ./cmd/plogsim

7. Exercises

  1. Consumer lag SLI: compute HighWatermark − committed per partition and alert when it exceeds a threshold (wire to pa-09).
  2. Hot-partition detection: produce skewed keys and measure per-partition record counts; show how a bad key creates skew.
  3. Sticky rebalancing: implement an assignor that, when a consumer leaves, reassigns only its partitions and keeps others put; compare movement to range/round-robin (the gw-04/pa-06 stability theme).
  4. Replay: reprocess a partition from offset 0 with a new consumer group; show idempotent consumers (pa-03) make this safe.
  5. Time-based retention: add timestamps and truncate by age; reason about the lag/retention SLO relationship.