db-18 — Observation

Expected canonical hashes

Six configurations are pinned in scripts/cross_test.sh. The lab is green iff all three binaries (Rust release, Go release, C++ Release) emit exactly these strings on stdout (no trailing newline):

These are the contract. Edit any production code such that one of these strings changes and you have changed semantics; reverify end-to-end before you ship.

Walking the wire: scenario D byte-by-byte

Scenario D is the shortest possible dump (one node, five proposals, all decided locally). Use it as a Rosetta Stone before debugging the multi-node hashes. The layout is magic || u32 node_count || node[], and the node payload starts at offset 12.

00..07  4453 4550 4158 3031     "DSEPAX01"            magic
08..0b  01 00 00 00             node_count = 1
0c..0f  00 00 00 00             node.id = 0
10..13  rr rr 00 00             node.promised_ballot.round       (round it won at)
14..17  00 00 00 00             node.promised_ballot.proposer_id (= self.id = 0)
18      02                      node.role = Leader (2)
19..1c  rr rr 00 00             node.my_ballot.round
1d..20  00 00 00 00             node.my_ballot.proposer_id
21..24  05 00 00 00             accept_count = 5
... 5 × {u64 slot, u32 ab.round, u32 ab.proposer_id, u32 value_len, value bytes}
... then u32 learned_count = 5 and 5 × {u64 slot, u32 value_len, value bytes}

Run:

src/rust/target/release/paxosctl --seed 1 --nodes 1 --rounds 200 --proposals 5
# e5e0248c7c4fa20991b90afdac828eab91a7414497461dadc2e1553040693139

To dump the raw bytes (skip the sha256 step) hack the binary to print canonical_dump instead of sha256_hex(&canonical_dump); do it locally only — the canonical CLI output is the sha256.

Walking the wire: scenario C (no proposals)

Scenario C runs three nodes for 500 ticks with --proposals 0. Exactly one of them will be elected leader; nobody decides anything. The dump therefore has accept_count == 0 and learned_count == 0 for every node. The bytes that do change between languages if you have an iteration-order bug are the per-node promised_ballot.round values (the elected leader's round depends on whether some other proposer almost-elected first). If C is the failing scenario, you have an election-timer determinism bug, not a Phase-2 bug.

Divergence runbook

If cross_test.sh prints MISMATCH scenario X, follow this script:

# 1. Capture the raw bytes from each binary. Patch paxosctl locally
#    to print `canonical_dump` raw instead of sha256 hex, run once,
#    then revert the patch. Save to rust.bin, go.bin, cpp.bin.

cmp -l rust.bin go.bin | head
cmp -l rust.bin cpp.bin | head

cmp -l prints byte_offset rust_value go_value in octal. Map the first offset to the field it belongs in:

Tick-level diff

If cmp -l flags a divergence inside the accepts of node 1, add eprintln!/fmt.Fprintln(os.Stderr, ...)/std::cerr lines in each implementation at the boundaries of the suspect ticks:

#![allow(unused)]
fn main() {
// after handle() and after on_tick():
eprintln!("t={} id={} promised={:?} role={:?} my={:?} accepts={:?} learned={:?}",
          t, id, n.promised_ballot, n.role, n.my_ballot, n.accepts, n.learned);
}

Run all three, diff -u rust.log go.log. The first differing tick is the bug.

Most common culprits in practice

1. Forgetting to sort the Promise payload by slot. Go's map iteration order is randomized; you must sort.Slice before appending to the wire.
2. Reading next_slot before recovering from prepare_accepted. If recovery doesn't update next_slot = max + 1, the leader will double-allocate a slot that already has a recovered accept, silently overwriting it.
3. Letting step_down clear promised_ballot. Promises are forever; only my_ballot is candidate-state.
4. Counting yourself twice in accept_count. Both become_leader and try_decide insert self; the second one is a no-op only if accept_count is a set, not a multiset.
5. Iterating peers as for p in nodes.iter() on a HashMap. Use BTreeMap in Rust, std::map in C++, and explicit for p := uint32(0); p < n; p++ in Go.