Observation — Looking at the Bytes

1. Hexdump a freshly written WAL

./build/walbench append /tmp/wal 3 4
xxd /tmp/wal

00000000: 0400 0000 b3ca 9eb5 4141 4141 0400 0000  ........AAAA....
00000010: b3ca 9eb5 4242 4242 0400 0000 b3ca 9eb5  ....BBBB........
00000020: 4343 4343                                CCCC

What you should see:

  • Three 12-byte records (4 header + 4 payload * 3 = 36 bytes, but actually 8+4 = 12 each = 36 ✓).
  • Identical headers because every payload is "AAAA" / "BBBB" / "CCCC" — same length, different CRC.
  • 04 00 00 00 is len = 4 in little-endian.
  • The next 4 bytes are the payload's CRC, also little-endian.

If your file is suspiciously large (e.g., starts with garbage 0x00 or 0xFF runs), open() is opening the file with the wrong flags or your buffer is uninitialized.

2. Group commit vs per-record sync

./build/walbench append-sync /tmp/wal 10000 64       # fsync per record
./build/walbench bench-group  /tmp/wal 10000 64 1    # group=1, same thing
./build/walbench bench-group  /tmp/wal 10000 64 64   # 64 records per fsync
./build/walbench bench-group  /tmp/wal 10000 64 512  # 512 per fsync

Sample (M2 Pro, APFS):

mode             throughput
per-record sync     1,800 records/s   (~556 µs/sync)
group=64          110,000 records/s
group=512         260,000 records/s

Two takeaways: per-record sync is 3 orders of magnitude slower; group size has diminishing returns past ~256 because the bottleneck shifts to write() itself.

3. Tail truncation in action

./build/walbench append /tmp/wal 5 16
wc -c /tmp/wal                       # 120 bytes (5 × 24)
printf '\xff\xff\xff\xff' >> /tmp/wal
wc -c /tmp/wal                       # 124 bytes
./build/walbench read /tmp/wal       # reads 5, then "stop: short header" (124-120 = 4 < 8)
./build/walbench append /tmp/wal 1 16
wc -c /tmp/wal                       # 144 bytes — open() truncated the garbage, then appended

The reopen-truncate behavior is the most easily-missed correctness detail. If it's broken, your second append ends up inside the corrupted region and the file becomes unreadable after recovery.

4. CRC sensitivity

Bit-flipping one byte of a 64-byte payload should kill the CRC of that record but leave everything before it valid:

./build/walbench append /tmp/wal 10 64
./build/walbench corrupt /tmp/wal 100 0x00     # mid-payload of record ~4
./build/walbench read /tmp/wal | tail
# expected: prints ~3 OK records then "stop: bad crc"

What "working" looks like

  • Hexdump shows tightly packed 8-byte-header + payload pairs, no padding.
  • Group commit is at least 50× faster than per-record sync.
  • Tail truncation works on first reopen, regardless of how much garbage you appended.

What "broken" looks like

  • A reader that hangs or panics on garbage — fix the bounds checks in the iter loop.
  • File size grows but throughput is flat — you're probably calling fsync inside append accidentally.
  • CRC doesn't trip on single-bit flips — wrong polynomial (likely you used the un-reflected version, see scripts/verify.sh).
  • Cross-language test fails — endianness or CRC table bug. Print the first 16 bytes of the file from each language and compare.