Skip to content

fix(rust_brain): preserve HLC across snapshot restore and gossip - #73

Draft
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-b205
Draft

fix(rust_brain): preserve HLC across snapshot restore and gossip#73
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-b205

Conversation

@cursor

@cursor cursor Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Bug and impact

restore_from_file(), bulk_write(), and gossip.receive() did not preserve or apply Hybrid Logical Clock (HLC) timestamps introduced in v0.6.0. After snapshot restore, gossip replays and causally valid successor writes were rejected with TimestampRegression, causing silent data loss in disaster-recovery and multi-node deployments.

Concrete trigger: Node A writes key k with HLC (5000, 10, nodeA), snapshots, crashes. Node B restores the snapshot. Node A (or gossip) attempts a successor write with HLC (5000, 11, nodeB) — rejected because restore assigned a fresh wall-clock HLC instead of the snapshot value.

Root cause

v0.6.0 added HLC monotonic enforcement to remember() and serializes HLC in to_dict(), but restore_from_file() created MemoryNode without reading hlc from the snapshot, bulk_write() did not pass hlc, and gossip.receive() ignored HLC on incoming events.

Fix

  • Add _parse_hlc() for snapshot/wire normalization (with legacy fallback for pre-v0.6.0 snapshots)
  • Restore HLC on snapshot load and advance the global HLC via _hlc.update()
  • Pass HLC through bulk_write and gossip.receive()
  • Hold lock during restore mutation
  • Gossip: skip updates missing HLC when key already exists; apply HLC for causal ordering

Validation

  • Reproduced pre-fix: restore assigned fresh HLC; successor write raised TimestampRegression
  • Post-fix reproduction script passes
  • 29 targeted tests pass: test_hlc_snapshot_gossip.py, test_enterprise_backup.py, test_gossip.py, test_rust_brain.py
Open in Web View Automation 

restore_from_file() assigned fresh HLC timestamps instead of reading
stored values from snapshots. After disaster recovery, gossip replays and
causal successor writes were rejected with TimestampRegression, causing
silent data loss in distributed deployments.

- Add _parse_hlc() for snapshot/wire normalization
- Restore HLC on snapshot load and advance global HLC
- Pass HLC through bulk_write and gossip.receive()
- Hold lock during restore mutation
- Add regression tests in test_hlc_snapshot_gossip.py

Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant