fix(storage): don't tombstone Frozen descendants on subtree delete#3164
fix(storage): don't tombstone Frozen descendants on subtree delete#3164rtb-12 wants to merge 11 commits into
Conversation
The recursive subtree tombstoning added for #286 (`Index::tombstone_descendants_of`) walked every descendant of a deleted node and tombstoned each one whose `deleted_at >= updated_at`. It never checked `StorageType`, so deleting a non-frozen parent that contains a `Frozen` descendant (e.g. a `FrozenStorage` field) tombstoned the frozen leaf too — violating the rule that Frozen data is immutable and never deleted (enforced elsewhere by the `RemoveMode::Delete` guard in `Interface`). Fix: in the DFS, skip a `Frozen` node AND its whole subtree (no recurse, no tombstone). The frozen data survives as an orphan, which is correct for immutable content-addressed storage. The root is never Frozen here — both delete paths reject that upstream. Adds a regression test that deletes a non-frozen parent holding a Frozen descendant (with its own child) and asserts the frozen node and its subtree survive while the non-frozen rows are tombstoned. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
🤖 AI Code Reviewer
Reviewed by 1 agents | Quality score: 32% | Review time: 48.9s
💡 1 suggestions. See inline comments.
🤖 Generated by AI Code Reviewer | Review ID: review-de8c7b63
Documentation ReviewThe following documentation may need updates based on the changes in this PR:
|
CI clippy runs `--workspace -D warnings`; the mut-default-then-assign pattern
tripped `clippy::field_reassign_with_default`. Use `Metadata { storage_type, ..default }`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
🤖 AI Code Reviewer
Reviewed by 1 agents | Quality score: 85% | Review time: 97.2s
✅ No Issues Found
All agents reviewed the code and found no issues. LGTM! 🎉
🤖 Generated by AI Code Reviewer | Review ID: review-2d0784db
meroreviewer suggested asserting "root is never Frozen". Testing disproved the invariant: the reassign/re-key paths call tombstone_descendants_of with a Frozen root (3 existing tests exercise this), so such an assert panics them in debug/CI. Drop the false claim; document that the Frozen skip is scoped to descendants (`id != root_id`) and a Frozen root is processed normally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
🤖 AI Code Reviewer
Reviewed by 1 agents | Quality score: 30% | Review time: 117.6s
💡 2 suggestions. See inline comments.
🤖 Generated by AI Code Reviewer | Review ID: review-407e75fe
Strengthens the Frozen-skip coverage: assert the walk skips a Frozen node's WHOLE subtree (a non-frozen child under a Frozen node also survives — the skip is by-subtree, not by-node-type) and that a Frozen leaf (no children) survives. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
🤖 AI Code Reviewer
Reviewed by 1 agents | Quality score: 85% | Review time: 186.4s
✅ No Issues Found
All agents reviewed the code and found no issues. LGTM! 🎉
🤖 Generated by AI Code Reviewer | Review ID: review-44ec6ccc
There was a problem hiding this comment.
🤖 MeroReviewer
Reviewed by 1 agents | Quality score: 31% | Review time: 42.0s
🟡 1 warnings. See inline comments.
🤖 Generated by MeroReviewer | Review ID: review-5cc2cef8
There was a problem hiding this comment.
🤖 AI Code Reviewer
Reviewed by 1 agents | Quality score: 31% | Review time: 109.9s
🟡 1 warnings. See inline comments.
🤖 Generated by AI Code Reviewer | Review ID: review-b363e8c3
The `id != root_id` guard's comment cited "reassign/re-key callers" as passing a Frozen root, but the only two callers (remove_child_from, apply_delete_ref_action) both reject Frozen deletes upstream — the root is never Frozen here. Restate the real invariant. Also assert the deleted node is unlinked from its parent's children list in the frozen-subtree regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bugbot is paused — on-demand spend limit reachedBugbot uses usage-based billing for this team and has hit its on-demand spend limit. A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue. |
There was a problem hiding this comment.
🤖 AI Code Reviewer
Reviewed by 1 agents | Quality score: 32% | Review time: 109.7s
💡 1 suggestions. See inline comments.
🤖 Generated by AI Code Reviewer | Review ID: review-3b5e3e3f
Previously a subtree delete silently skipped a Frozen descendant, leaving it as a graph-detached orphan reachable only by direct Id. Instead, reject the local delete up front: scan the subtree (find_frozen_descendant) and return ActionNotAllowed naming the frozen entity, so the operator relocates it out of the subtree before deleting. Mirrors the existing direct-child Frozen guard and its split-brain avoidance (no DeleteRef is broadcast). The replay path (apply_delete_ref_action) keeps the skip-frozen fallback: a replica can't reject a peer's DeleteRef without diverging, so it preserves the frozen data and converges. Compliant peers never emit such a delete now. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
🤖 AI Code Reviewer
Reviewed by 2 agents | Quality score: 56% | Review time: 255.6s
💡 1 suggestions. See inline comments.
🤖 Generated by AI Code Reviewer | Review ID: review-b502e3d5
Assert a skipped frozen node is not listed in its parent's deleted_children advert (else a syncing peer applies a spurious delete-wins against surviving frozen data). Clarify that the Index-level skip test covers tombstone_descendants_of directly, with Interface-level rejection covered in tests/interface.rs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bugbot is paused — on-demand spend limit reachedBugbot uses usage-based billing for this team and has hit its on-demand spend limit. A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue. |
There was a problem hiding this comment.
🤖 AI Code Reviewer
Reviewed by 2 agents | Quality score: 49% | Review time: 122.6s
🟡 2 warnings, 💡 1 suggestions. See inline comments.
🤖 Generated by AI Code Reviewer | Review ID: review-de802e67
| ); | ||
| assert!( | ||
| !<Index<S>>::is_deleted(f).unwrap(), | ||
| "Frozen descendant must survive the subtree delete" |
There was a problem hiding this comment.
🟡 subtree_delete_skips_frozen_descendant_and_its_subtree calls Index::remove_child_from directly, bypassing the Interface-level Frozen guard
The test at line 870 calls <Index<S>>::remove_child_from(root, a, time_now()) directly. This bypasses Interface::remove_child_from_inner's find_frozen_descendant pre-check, so the test is actually exercising the REPLAY path (tombstone_descendants_of skip), not the local-delete rejection path. The test comment says 'The skip here is the replay-side fallback' which is correct, but the test name subtree_delete_skips_frozen_descendant_and_its_subtree could mislead a reader into thinking the local guard is being tested. More critically: calling Index::remove_child_from directly with a subtree containing a Frozen descendant succeeds (the frozen node is skipped by tombstone_descendants_of), which is the intended replay behavior. This is correct but the test is validating that the Index-level call does NOT reject — which is the opposite of what the Interface-level guard does. This is intentional per the PR description but could cause confusion.
Suggested fix:
Rename the test to `replay_subtree_delete_skips_frozen_descendant_and_its_subtree` to make clear it tests the replay path, not the local-delete rejection path.
|
|
||
| assert!( | ||
| <Index<S>>::is_deleted(a).unwrap(), | ||
| "deleted parent tombstoned" |
There was a problem hiding this comment.
🟡 subtree_delete_skips_frozen_descendant_and_its_subtree calls Index::remove_child_from directly, bypassing the Interface guard
The test at line 862 calls <Index<S>>::remove_child_from(root, a, time_now()) directly. According to the PR description, the local delete guard (find_frozen_descendant rejection) lives in Interface::remove_child_from_inner, NOT in Index::remove_child_from. So this test exercises the replay-side skip in tombstone_descendants_of (correct, as the doc comment says), but the test comment says 'deleting the parent must SKIP the frozen node' — which is true at the Index level for the replay path. However, a reader might expect this test to also validate that the Interface-level guard fires. The test is correctly scoped but the setup calls <Index<S>>::remove_child_from which will NOT trigger the ActionNotAllowed rejection — it will proceed to tombstone_descendants_of and exercise the skip. This is intentional per the doc comment, but the test name subtree_delete_skips_frozen_descendant_and_its_subtree could mislead: it sounds like the delete 'skips' the frozen node (replay behavior), but a reader might think it tests the rejection. The companion test in interface.rs covers the rejection. No bug, but the test comment at line 862 says 'deleting the parent must SKIP' which is ambiguous — it should say 'the tombstone walk skips' to distinguish from 'the delete is rejected'.
Suggested fix:
Clarify the assertion message or the test name to explicitly say this tests the `tombstone_descendants_of` replay-side skip, not the `remove_child_from_inner` pre-check rejection. E.g. rename to `tombstone_walk_skips_frozen_descendant_and_its_subtree`.
| ) | ||
| .unwrap(); | ||
|
|
||
| reset_delta_context(); |
There was a problem hiding this comment.
💡 Test uses Index::add_child_to directly to inject a Frozen entity, bypassing Interface invariants
The test remove_child_from_rejects_subtree_with_frozen_descendant injects a Frozen entity by calling <Index<MainStorage>>::add_child_to(para.id(), ChildInfo::new(frozen_id, ...)) directly, without writing a Key::Entry for frozen_id. This means the frozen entity exists in the index but has no backing storage entry. While this is sufficient to trigger find_frozen_descendant (which only reads the index), it creates an inconsistent state that wouldn't occur in production. The test is valid for its purpose, but a comment explaining why direct index injection is used (to avoid the Interface-level Frozen-add path, which may have its own guards) would help future readers understand the setup is intentional.
Suggested fix:
Add a comment: `// Inject directly into the index (no Key::Entry) to simulate a Frozen descendant without going through Interface::add_child_to, which has its own guards. find_frozen_descendant only reads the index, so no backing entry is needed for this test.`
|
This pull request has been automatically marked as stale. If this pull request is still relevant, please leave any comment (for example, "bump"), and we'll keep it open. We are sorry that we haven't been able to prioritize reviewing it yet. Your contribution is very much appreciated. |
Bugbot is paused — on-demand spend limit reachedBugbot uses usage-based billing for this team and has hit its on-demand spend limit. A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue. |
There was a problem hiding this comment.
🤖 AI Code Reviewer
Reviewed by 2 agents | Quality score: 90% | Review time: 462.8s
✅ No Issues Found
All agents reviewed the code and found no issues. LGTM! 🎉
🤖 Generated by AI Code Reviewer | Review ID: review-74252f26
There was a problem hiding this comment.
🤖 AI Code Reviewer
Reviewed by 2 agents | Quality score: 90% | Review time: 341.2s
✅ No Issues Found
All agents reviewed the code and found no issues. LGTM! 🎉
🤖 Generated by AI Code Reviewer | Review ID: review-fbd0568a
There was a problem hiding this comment.
🤖 AI Code Reviewer
Reviewed by 2 agents | Quality score: 90% | Review time: 217.9s
✅ No Issues Found
All agents reviewed the code and found no issues. LGTM! 🎉
🤖 Generated by AI Code Reviewer | Review ID: review-6019b351
The gap (#286 cascade)
The recursive subtree-tombstoning added for #286,
Index::tombstone_descendants_of(crates/storage/src/index.rs), walks every descendant of a deleted node and tombstones each whosedeleted_at >= updated_at. It never checkedStorageType, so deleting a non-frozen parent that contains aFrozendescendant (e.g. aFrozenStoragefield) tombstoned the frozen leaf too.That violates the invariant that Frozen data is immutable and never deleted — the rule the
RemoveMode::Deleteguard inInterface::remove_child_from(and theStorageType::Frozenarm inapply_action) already enforces at the direct-delete boundary. The subtree cascade was a hole in that guard: a frozen entity reachable only as a descendant of a deleted non-frozen parent got tombstoned anyway.Policy: reject the local delete (don't silently orphan)
A delete reaches the subtree walk on two paths with opposite freedoms:
Interface::remove_child_from→ emits aDeleteRefDeleteRefarrives from a peerapply_delete_ref_actionLocal path (the fix):
remove_child_from_innernow runs a read-onlyIndex::find_frozen_descendantscan over the target subtree before mutating any state. If anyFrozenentity exists at any depth, it returnsActionNotAllowednaming the frozen id and asking the operator to relocate it out of the subtree first. The delete is refused atomically — no tombstones, noDeleteRefbroadcast. This extends the existing direct-child Frozen guard (and its identical split-brain rationale) to the whole subtree, so in normal operation no detached orphan is ever created because the delete never happens.Replay path (defense-in-depth):
tombstone_descendants_ofkeeps skipping aFrozennode and its whole subtree. A compliant peer never emits such a delete now, but a replica must stay convergent if it receives one from an old/buggy/malicious peer — so it preserves the frozen data (leaving it detached) and converges rather than erroring.Tests
index::tests::subtree_tombstoning::find_frozen_descendant_locates_deep_frozen_and_ignores_root— the scan finds a frozen entity nested below the delete root, returnsNonefor a frozen-free subtree, and excludes the root itself.interface::tests::frozen_storage_verification::remove_child_from_rejects_subtree_with_frozen_descendant— a local delete of a non-frozen subtree with a buried frozen descendant is rejected withActionNotAllowed, nothing is tombstoned, and noDeleteRefis broadcast.subtree_delete_skips_frozen_descendant_and_its_subtree/..._whole_frozen_subtree_and_frozen_leaftests continue to validate the replay-side skip intombstone_descendants_of.Full
calimero-storagesuite green (672 passing); clippy + fmt clean.