Skip to content

Platform exposes no signal for "round N is fully signed and safe to export" — callers must poll and hash the filesystem #26950

Description

@JeffreyDallas

Background

Solo (hiero-ledger/solo) needs to safely export a consensus node's saved state from disk for consensus state download / config ops backup — used both for bootstrapping a new node from an existing node's state and for disaster-recovery backup/restore workflows (see hiero-ledger/solo#5375). Before zipping up data/saved, the caller needs to know that the round it's about to export has finished collecting all validator signatures and nothing is still being written to disk for that round.

There is currently no platform-exposed signal for this. The two status values that come closest don't actually cover it:

  • FREEZE_COMPLETE only applies when the network was explicitly frozen, and says nothing about whether a periodic signed snapshot (taken while the network stays live) has finished being written.
  • For a plain STOPPED (non-frozen) deployment there's no target status at all — the caller just wants "the newest round that finished signing," which isn't something any PlatformStatus value identifies.

As a workaround, Solo polls the container's filesystem in a loop (resources/wait-for-stable-saved-state.sh + NetworkNodes.waitForStableSavedState, see hiero-ledger/solo#5375):

  1. Reads each round directory's stateMetadata.txt and compares SIGNING_WEIGHT_SUM to TOTAL_WEIGHT to guess whether that round's signatures are complete.
  2. Fingerprints the entire data/saved tree (originally by hashing every file's contents; changed to size+mtime after review feedback, since hashing full content on every 2s poll for up to 180 polls adds real I/O load exactly while the node is trying to quiesce) and waits for the fingerprint to stay stable across 3 consecutive polls before treating a round as "done."
  3. Falls back to the newest fully signed non-freeze round if a requested freeze round never becomes fully signed within the poll budget (~6 minutes) — which, in practice, was sometimes the only outcome, see below.

Separately, in the same PR we observed a round created by a FREEZE_ONLY transaction remain permanently un-signed (SIGNING_WEIGHT_SUM: 0, TOTAL_WEIGHT: 1000) even after waiting several extra minutes past the freeze, on a small (2-node) test network. This looks adjacent to #11421 ("Periodic snapshots during PCES replay don't get signatures"), but that issue is specifically about snapshots chosen during PCES replay; what we saw was a live, non-replay freeze round on an otherwise normally-running network, so it may be a distinct manifestation worth confirming rather than assuming it's the same root cause.

Proposed Solution

  1. Expose an explicit signal (a PlatformStatus-adjacent value, a metrics-port gauge, or a per-round "signing complete" event/callback) that answers "round N has collected all its signatures and its on-disk saved-state directory is finished being written — it is safe to read/export now," without requiring the caller to poll and hash the filesystem.
  2. The signal should cover both a frozen deployment (the freeze round specifically) and a live/stopped deployment (the newest signed round), since both are legitimate export scenarios Solo relies on.
  3. Document the expected latency between a round reaching consensus and its signatures being considered "complete," so callers can size reasonable timeouts instead of guessing (Solo currently polls for up to ~6 minutes per node).

Acceptance Criteria

  1. An external caller can determine, without filesystem polling/hashing, that a specific round's saved state is fully signed and stable on disk.
  2. The mechanism works for both a frozen network (freeze round) and a stopped-but-not-frozen network (latest signed round).
  3. Expected signing latency (or an explicit "will never sign" condition) is documented or discoverable.

Dependencies

  1. hiero-ledger/solo#5375 — the PR that hit this gap and added the filesystem-polling workaround (see review discussion on src/core/network-nodes.ts and resources/wait-for-stable-saved-state.sh).

Related / Adjacent Issues

Environment

  • Consensus node version: v0.75.1
  • Observed on a 2-node Solo-managed local kind cluster and in Solo's GitHub Actions CI (hiero-ledger/solo)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions