pa-04 — A Partitioned Log (Kafka in Miniature)

The Apple JD lists "message queues and streaming platforms, such as Kafka, RabbitMQ, NATS, or Pulsar." Under almost all modern streaming sits one data structure: the partitioned, append-only commit log. Master it and Kafka, Pulsar, and NATS JetStream stop being magic — they become "a log, split into partitions, with consumer groups tracking offsets." pa-03 gave you the patterns (pub/sub, at-least-once, DLQ); this lab gives you the substrate they run on, built and tested.

You build the log: partitions (the unit of ordering and parallelism), monotonic offsets, key-based partitioning, consumer groups with committed offsets, partition assignment (range / round-robin), and retention.

1. What is it?

A commit log is an append-only, ordered sequence of records, each at a monotonically increasing offset. You only append (never update in place) and read sequentially from an offset. That's it — and it's enough to build messaging, event streaming, replication, and event sourcing.

To scale beyond one machine and one consumer, the log is split into partitions:

topic "orders", 3 partitions:
  P0:  [0:acct-1][1:acct-1][2:acct-1]            (append-only, ordered)
  P1:  [0:acct-2][1:acct-2]
  P2:  [0:acct-3]
        ▲ offset (per-partition, monotonic, stable)

• A producer appends a record; its key is hashed to choose a partition, so all records for a key land in one partition and are therefore ordered relative to each other.
• A consumer group reads partitions and commits offsets (where it is); on restart it resumes from the commit. Different groups read the same log independently at their own pace.
• Partition assignment spreads a topic's partitions across the consumers in a group — the unit of parallelism (≤ one consumer per partition per group).
• Retention drops old records (by time or size); offsets stay absolute and stable.

This is db-03's write-ahead log idea (you built one earlier) promoted to a distributed messaging primitive.

2. Why does it matter?

• It's the backbone of event-driven platforms. pa-03's bus is an abstraction; a real system needs durability, ordering, replay, and parallel consumption — which is exactly what the partitioned log provides. An architect designing an eventing platform is choosing and configuring this.

• Partitions are the whole scaling and ordering story. Throughput scales with partitions; ordering is guaranteed only within a partition. Choosing the partition key is therefore one of the most consequential design decisions: it decides what's ordered, what's parallelizable, and whether you get hot partitions (skew).

• The log unifies messaging and state. Because it's durable and replayable, the same log powers queues, pub/sub, stream processing, CDC, and event sourcing (the log is the source of truth). "Turn the database inside out" (Kleppmann) is this insight.

• Offsets + consumer groups give you decoupling in time and re-readability. A new consumer can replay history; a slow consumer catches up later; a bug fix can reprocess from an earlier offset. That flexibility is impossible with a fire-and-forget queue.

3. How does it work?

Producing and partitioning

Produce(key, value) hashes the key to a partition (PartitionFor), appends, and returns (partition, offset). Same key → same partition → ordered. Keyless records round-robin (max parallelism, no ordering). This choice — what to use as the key — is the architecture decision: order per accountId? then key by account, and accept that one hot account is one hot partition.

Offsets and consuming

Offsets are per-partition, monotonic, and absolute (stable forever). Consume(partition, fromOffset, max) returns records at or after an offset — a consumer polling from where it left off. HighWatermark is the next offset; a consumer is caught up when its commit equals it.

Consumer groups

A consumer group is a set of cooperating consumers that share a topic's partitions: each partition is read by at most one consumer in the group (so within a group, work is divided; across groups, the log is re-read independently). GroupOffsets.Commit/Committed tracks per-(group, partition) progress so a restart resumes correctly. Commit after processing for at-least-once; before for at-most-once.

Partition assignment & rebalancing

When consumers join/leave, partitions are reassigned. AssignRange (contiguous chunks; earlier consumers get extras) and AssignRoundRobin (one at a time) are Kafka's two classic strategies. Rebalancing is disruptive (consumers stop, reassign, resume) — modern Kafka adds cooperative/sticky rebalancing to minimize movement, the same "minimize reassignment under membership change" goal as gw-04's subsetting and pa-06's consistent hashing.

Retention

Truncate(partition, beforeOffset) drops old records (by offset here; by time/size in production). Offsets of survivors are unchanged, so committed consumer offsets remain valid — a consumer that fell behind the retention window simply resumes at the earliest retained record (and may have lost data, which is the operational risk of too-short retention).

4. Core terminology

5. Mental models

• A partition is a single-file ledger; the topic is a filing cabinet of them. Each ledger is strictly ordered (append-only); the cabinet parallelizes across ledgers. You only get order within a ledger, so you file related entries (same key) in the same one.

• Offsets are bookmarks, not deletions. Reading doesn't consume; many readers keep their own bookmark in the same book. That's why the log supports replay, multiple independent consumers, and reprocessing — unlike a queue where a read pops the message.

• The partition key is a routing decision you can't easily undo. It fixes ordering and load distribution. Pick a key that's high-cardinality (avoid hot partitions) and aligned with your ordering needs. Changing partition count later reshuffles the mapping — a migration (gw-12).

• Retention is a moving cliff behind your slowest consumer. As long as consumers stay within the window, all is well; fall behind the cliff and you lose data silently. Monitor consumer lag (HW − committed) like an SLI (pa-09).

6. Common misconceptions

• "Kafka guarantees global ordering." Only per partition. Across partitions there's no order. If you need order for a key, partition by that key; if you need global order, you need one partition (and you've given up parallelism).

• "More partitions is always better." Partitions cost (open files, memory, rebalance time, end-to-end latency, and metadata). Size for target throughput and consumer parallelism, not "as many as possible."

• "A consumer group with N consumers can use any number of partitions." Parallelism is capped at the partition count: with 4 partitions, the 5th consumer in a group sits idle. Partitions, not consumers, set the ceiling.

• "Reading consumes the message" (queue thinking). In a log, reading advances your offset only; the data stays for other consumers and replay until retention removes it.

• "Exactly-once because Kafka transactions." Within Kafka's boundary, yes; but your consumer's external side effects still need idempotency (pa-02/03). Don't conflate broker exactly-once with end-to-end.

7. Interview talking points

• "Design a streaming/eventing platform." Partitioned log → key-based partitioning (justify the key: ordering + skew) → consumer groups + offset commit → at-least-once + idempotent consumers (pa-03) → DLQ → retention sized to consumer lag → schema registry for the event contract (pa-02). Name the per-partition ordering guarantee explicitly.

• "How do you guarantee ordering?" Only within a partition. Partition by the entity whose order matters (e.g. accountId), accepting that a hot key is a hot partition. Global ordering = one partition = no parallelism; usually you don't actually need it.

• "Kafka vs RabbitMQ vs NATS vs Pulsar?" Log (Kafka/Pulsar): durable, replayable, ordered-per-partition, high throughput, consumer groups — for event streaming/sourcing. Queue (RabbitMQ): rich routing, per-message ack, competing consumers — for task distribution. NATS: lightweight pub/sub + JetStream for streams. Pick by replay/ordering/ routing needs.

• "How do consumer groups scale and rebalance?" Each partition →ne Each partition → one consumer in a group; parallelism caps at partition count. Rebalancing reassigns on membership change (range/round-robin/sticky); sticky/cooperative minimizes movement — the same stability concern as gw-04/pa-06.

• "How do you choose a partition count and retention?" Partitions for target throughput + max consumer parallelism (with headroom — changing it is a migration). Retention long enough to cover consumer downtime + replay needs; monitor lag (HW − committed) as an SLI, alert before the retention cliff.

8. Connections to other labs

• db-03 (write-ahead log) — the same append-only-log idea you built for crash recovery, here as a distributed messaging primitive.
• pa-03 (event-driven) — this log is the durable, ordered, replayable backbone under that bus; ordering is per-partition.
• pa-05 (outbox/saga) — the outbox publishes to a log like this; consumers are idempotent because delivery is at-least-once.
• pa-06 (partitioning) — key→partition hashing and rebalancing are the same partitioning/stability problems, for data instead of streams.
• gw-04 (subsetting) — minimizing reassignment under membership change is the shared theme with consumer-group rebalancing.