Skip to content

test(storage): probe reachability of the comparisons state-sync path#3209

Closed
chefsale wants to merge 2 commits into
masterfrom
probe/comparisons-sync-reachability
Closed

test(storage): probe reachability of the comparisons state-sync path#3209
chefsale wants to merge 2 commits into
masterfrom
probe/comparisons-sync-reachability

Conversation

@chefsale

@chefsale chefsale commented Jul 6, 2026

Copy link
Copy Markdown
Member

What

Adds three WARN-level, greppable probes (target: \"sync_probe\") at the only sites that can exercise the legacy state-based StorageDelta::Comparisons / compare_trees sync path. No behavior change — probes only log.

  • delta.rs commit_root — the sole producer of a Comparisons wire artifact.
  • root.rs apply_comparisons — fires whenever any Comparisons delta (including the zero-byte no-op that deserializes to Comparisons(vec![])) reaches Root::sync.
  • root.rs apply_actions Action::Compare arm — fires if a Compare action is ever received from peer traffic (the trigger that becomes a Comparisons emission).

Why

We want to delete the compare_trees / Comparisons subsystem (it's deletion-unaware and superseded by HashComparison/LevelWise/DAG-CausalActions). Static analysis strongly suggests it's already unreachable in production:

  • Action::Compare is pushed only inside apply_action, into context.actions (not comparisons), and only during __calimero_sync_next — a state-op whose artifact isn't broadcast.
  • Authored/stored DAG deltas are always CausalActions built from local writes (Add/Update/DeleteRef), never Compare, never empty.
  • So peers' __calimero_sync_next should always receive CausalActionsapply_actions, never the Compare arm, never apply_comparisons.

The one residual uncertainty is whether the node-side state-delta handler ever re-authors/propagates a DAG delta from a sync-apply outcome (which would carry Compare actions to peers). Rather than guess, these probes let a real multi-node run answer it with data.

How to read the result

Grep node logs (guest probes surface via the env::log bridge) for:

sync_probe: apply_comparisons invoked
sync_probe: received Action::Compare in delta
sync_probe: commit_root emitted StorageDelta::Comparisons artifact

Run something that forces Merkle divergence (concurrent merges + deletes across nodes). If zero sync_probe lines appear, the path is confirmed dead and the follow-up PR removes the whole subsystem (and the now-false convergence comment at interface.rs:2043-2049).

Follow-ups (not in this PR)

🤖 Generated with Claude Code

Adds three WARN-level, greppable probes (target: "sync_probe") at the only
sites that can exercise the legacy state-based `StorageDelta::Comparisons` /
`compare_trees` sync path:

- delta.rs `commit_root` — the sole producer of a Comparisons wire artifact.
- root.rs `apply_comparisons` — fires whenever any Comparisons delta (incl. the
  zero-byte no-op that deserializes to `Comparisons(vec![])`) reaches Root::sync.
- root.rs `apply_actions` Action::Compare arm — fires if a Compare action is ever
  received from peer traffic (the trigger that becomes a Comparisons emission).

Purpose: static analysis suggests this path is unreachable in production
(HashComparison/LevelWise/DAG-CausalActions handle all sync; Compare actions are
only generated inside the non-broadcast `__calimero_sync_next` state-op). These
probes let a real multi-node run confirm it with data. If no `sync_probe` line
appears across a run that forces Merkle divergence, the whole Comparisons
subsystem can be removed. Probes surface guest-side via the env::log bridge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@chefsale chefsale changed the title probe: confirm the Comparisons state-sync path is dead before removing it test(storage): probe reachability of the comparisons state-sync path Jul 6, 2026

@meroreviewer meroreviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 MeroReviewer

Reviewed by 1 agents | Quality score: 85% | Review time: 157.6s


✅ No Issues Found

All agents reviewed the code and found no issues. LGTM! 🎉


🤖 Generated by MeroReviewer | Review ID: review-d23891c3

@meroreviewer

meroreviewer Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Documentation Review

The following documentation may need updates based on the changes in this PR:

  • 🟡 architecture/migrations.html: Files matching crates/storage/** were changed but architecture/migrations.html was not updated (per source_to_docs_mapping).
  • 🟡 architecture/storage-schema.html: Files matching crates/storage/** were changed but architecture/storage-schema.html was not updated (per source_to_docs_mapping).

@meroreviewer meroreviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Code Reviewer

Reviewed by 1 agents | Quality score: 31% | Review time: 273.6s

🟡 1 warnings. See inline comments.


🤖 Generated by AI Code Reviewer | Review ID: review-1e24cecc

comparisons: Vec<Comparison>,
ctx: &crate::interface::ApplyContext,
) -> Result<(), StorageError> {
// Reachability probe (issue 1): fires whenever a `StorageDelta::Comparisons`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 apply_comparisons probe fires on the benign zero-byte no-op artifact, undermining the reachability conclusion

The probe is placed unconditionally at the top of apply_comparisons, before the comparisons.is_empty() check. Per StorageDelta's custom BorshDeserialize (delta.rs), a zero-byte wire artifact (the routine no-op that commit_root emits whenever there are neither actions nor comparisons) always deserializes to Comparisons(vec![]), not to Actions(vec![]). That means any ordinary, harmless empty commit that happens to reach Root::sync will trip this WARN with comparison_count=0, even though it has nothing to do with the legacy compare_trees subsystem being investigated. By contrast, the sibling probe in delta.rs::commit_root is correctly scoped to the (&[], _) (comparisons genuinely non-empty) match arm and will not fire for that same no-op case. This asymmetry means a non-zero sync_probe: apply_comparisons invoked log line does not actually confirm that a peer ever sent a genuine Comparisons payload — it could just as easily be triggered by the unrelated, expected empty-artifact fallback. Since the PR's explicit decision rule is "if zero sync_probe lines appear, the path is confirmed dead," a spurious non-zero count here could incorrectly block the planned removal of the subsystem.

Suggested fix:

Split the probe so the zero-byte/no-op case is distinguishable from a genuine received Comparisons payload, e.g. only warn when `!comparisons.is_empty()`, or add a separate lower-signal log line for the empty case: `if comparisons.is_empty() { tracing::debug!(target: "sync_probe", "apply_comparisons invoked with empty comparisons (likely benign zero-byte no-op)"); } else { tracing::warn!(target: "sync_probe", comparison_count = comparisons.len(), "apply_comparisons invoked with non-empty comparisons"); }`

… CI signal

The passive warn! probes proved insufficient: the e2e harness does not surface
guest-side (WASM) tracing (storage::root = 0 in captured logs; only explicit
env::log app lines appear), so the two guest-reachable probes (apply_comparisons,
the Action::Compare receive-arm) could not be confirmed from logs.

Convert all three dead-path sites to panic!, each with a distinct message:
- delta.rs commit_root  — a Comparisons wire artifact was emitted.
- root.rs apply_comparisons — a Comparisons/empty artifact reached Root::sync.
- root.rs Action::Compare arm — a Compare action was received in a delta.

Now reachability manifests as a CI failure that surfaces regardless of log
capture and pinpoints the live path. A green e2e run across the divergence
scenarios (partition/restart/pause/mesh-soak/concurrent-rotation/reconcile)
is then a definitive proof the state-based comparison sync path is dead and
can be removed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@meroreviewer meroreviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Code Reviewer

Reviewed by 1 agents | Quality score: 85% | Review time: 373.6s


✅ No Issues Found

All agents reviewed the code and found no issues. LGTM! 🎉


🤖 Generated by AI Code Reviewer | Review ID: review-eab9e5bb

@chefsale

Copy link
Copy Markdown
Member Author

Superseded by #3215 (the actual removal). This probe branch's purpose — hard-abort instrumentation to confirm, via the multi-node divergence e2e, that the state-based Comparisons sync path is never reached — is complete: no abort fired, so the path is dead and #3215 removes it. Not for merge.

@chefsale chefsale closed this Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant