pa-04 — Analysis

What the log must get right

  1. Per-partition ordering with monotonic, absolute offsets that never repeat or shift (even after retention).
  2. Key→partition stability: a key always maps to the same partition.
  3. Independent consumer groups with resumable committed offsets.
  4. Assignment that divides partitions across a group (parallelism capped at partition count).
  5. Retention that drops old records without invalidating offsets.

Design decisions

  • Absolute offsets, prefix-truncation retention. Survivors keep their offsets so committed consumer offsets stay valid across retention — the property that makes "resume where I left off" robust.
  • Key hashing for partitioning. Deterministic, stateless, and it encodes the ordering guarantee (same key = same partition = ordered). The cost is skew from hot keys — surfaced, not hidden.
  • Offsets per (group, partition). Groups are independent; reading doesn't consume; replay is just "commit a lower offset." This is the log/queue distinction made concrete.
  • Two assignors. Range and round-robin match Kafka's classics and let tests pin exact distributions; sticky/cooperative is the production upgrade (minimal movement).

Tradeoffs worth flagging

  • Ordering vs parallelism. Per-partition ordering means global order needs one partition (no parallelism). Choose the key to order only what must be ordered.
  • Partition count is sticky. It caps consumer parallelism and, once set, changing it reshuffles key→partition (a migration, gw-12). Size with headroom.
  • Retention vs replay/lag. Short retention saves storage but risks data loss for slow/restarted consumers; long retention costs storage. Monitor lag (HW − committed) and alert before the cliff (pa-09).
  • Rebalancing is disruptive. Naive reassignment stops the group; sticky/cooperative minimizes movement — the same concern as gw-04.

What production adds beyond this lab

  • Durable, replicated storage (segments, leader/follower replication — db-17), not in-memory slices.
  • Sticky/cooperative rebalancing, transactions/idempotent producers, exactly-once within the broker.
  • Tiered storage, compaction (key-compacted topics), and time-based retention.
  • Consumer-lag monitoring + schema registry (pa-02) + end-to-end tracing across the async hop (gw-11).