fix: support state save and restore redeploy workflow by JeffreyDallas · Pull Request #5375 · hiero-ledger/solo

JeffreyDallas · 2026-07-28T03:45:28Z

Description

This pull request makes the state-save-and-restore workflow safe for restoring a frozen consensus-node state into a fresh cluster with different network endpoints.

It:

adds a stable-state wait before downloading state, requiring the saved-state files to remain unchanged across consecutive checks and warning or failing when a node is not frozen or stopped
updates the freeze workflow to wait for the signed frozen state before stopping nodes and capturing state
preserves the state/PCES alignment needed for restore and stages the generated network override at the next round to avoid an invalid state signature
adds generate-override-network.mjs, which builds override-network.json from the saved network definition and the fresh Kubernetes service endpoints, then copies it into the consensus-node pods
adds the fresh-cluster restore and verification workflow, including checks that both nodes reach FREEZE_COMPLETE and no Invalid State Signature is reported
simplifies state-round cleanup so the selected state and PCES data are retained
documents the complete workflow in the example README

Related Issues

Pull request (PR) checklist

This PR added tests (unit, integration, and/or end-to-end)
This PR updated documentation
This PR added no TODOs or commented out code
This PR has no breaking changes
Any technical debt has been documented as a separate issue and linked to this PR
Any package.json changes have been explained to and approved by a repository manager
All related issues have been linked to this PR
All changes in this PR are included in the description
When this PR merges the commits will be squashed and the title will be used as the commit message, the 'commit message guidelines' below have been followed

Testing

This PR added unit tests
This PR added integration/end-to-end tests
These changes required manual testing that is documented below
Anything not tested is documented

The following manual testing was done:

Ran the complete examples/state-save-and-restore workflow on a clean kind cluster, including freeze, stable state capture, cluster recreation, endpoint override generation/copy, and restore.
Verified both restored consensus nodes reached FREEZE_COMPLETE, both proxies became active, and no Invalid State Signature was reported.
Ran task verify-state against the restored cluster; both nodes reported platform status 6.0.
Ran TypeScript compilation, shell syntax checks, reference checks, and git diff --check.

The following was not tested:

The optional mirror/database workflow remains disabled by default and was not exercised in the final end-to-end run.
Unit test coverage for the new restore workflow was not added; validation is through the example end-to-end workflow and CLI checks.

Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

trunk-io · 2026-07-28T03:45:32Z

Merging to main in this repository is managed by Trunk.

To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

github-actions · 2026-07-28T03:52:42Z

Unit Test Results - Linux

39 tests ±0 39 ✅ ±0 0s ⏱️ ±0s
17 suites ±0 0 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit 2c30618. ± Comparison against base commit ece1a3c.

♻️ This comment has been updated with latest results.

github-actions · 2026-07-28T03:53:40Z

Unit Test Results - Windows

1 files ± 0 567 suites +1 22s ⏱️ ±0s
1 930 tests +10 1 930 ✅ +10 0 💤 ±0 0 ❌ ±0
1 934 runs +10 1 934 ✅ +10 0 💤 ±0 0 ❌ ±0

Results for commit 2c30618. ± Comparison against base commit ece1a3c.

♻️ This comment has been updated with latest results.

github-actions · 2026-07-28T13:34:40Z

E2E Test Report

11 files ±0 98 suites ±0 1h 28m 59s ⏱️ -44s
315 tests ±0 315 ✅ ±0 0 💤 ±0 0 ❌ ±0
335 runs ±0 335 ✅ ±0 0 💤 ±0 0 ❌ ±0

Results for commit 8af5972. ± Comparison against base commit 9794630.

github-actions · 2026-07-28T13:34:42Z

E2E Test Report

11 files ±0 96 suites ±0 1h 48m 57s ⏱️ + 15m 57s
311 tests ±0 311 ✅ ±0 0 💤 ±0 0 ❌ ±0
332 runs ±0 332 ✅ ±0 0 💤 ±0 0 ❌ ±0

Results for commit 2c30618. ± Comparison against base commit ece1a3c.

♻️ This comment has been updated with latest results.

Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

- Add unit tests for PcesTrimmer covering: an unchanged file below the cutoff, a record ending exactly at the buffer boundary, a straddling file that gets truncated, multi-byte varint birth rounds, the truncateAtOffset <= 4 whole-file-delete path, an unrecognized version header, nested directories, a missing directory, and a malformed record (left untouched rather than guessed at). - wait-for-stable-saved-state.sh: fingerprint saved-state files by size+mtime instead of hashing file contents, avoiding repeated full-content I/O on every 2s poll (up to 180 times) while the node is trying to quiesce. - Remove the redundant showUser warning in getNodeStateFiles(); the thrown SoloErrors.validation.illegalArgument already carries the same message and is shown to the user. - Name the FREEZE_COMPLETE enum value in a comment next to the magic "6.0" platform_PlatformStatus check in the state-save-and-restore Taskfile. Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

Signed-off-by: JeffreyDallas <39912573+JeffreyDallas@users.noreply.github.com>

JeffreyDallas added 5 commits July 27, 2026 14:04

update

e2441be

Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

save

52a3680

Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

save

ad84f73

Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

save

9bf5cce

Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

save

8af5972

Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

JeffreyDallas self-assigned this Jul 28, 2026

JeffreyDallas added PR: Needs Team Approval A pull request that needs review from a team member. P1-💎 Current Milestone & Goals labels Jul 28, 2026

JeffreyDallas marked this pull request as ready for review July 28, 2026 04:07

JeffreyDallas requested review from a team as code owners July 28, 2026 04:07

JeffreyDallas requested review from ddeskov-limechain and jeromy-cannon July 28, 2026 04:07

jan-milenkov added PR: Checks Failed A pull request where the checks have failed. and removed PR: Needs Team Approval A pull request that needs review from a team member. labels Jul 28, 2026

JeffreyDallas added PR: Needs Team Approval A pull request that needs review from a team member. and removed PR: Checks Failed A pull request where the checks have failed. labels Jul 28, 2026

jan-milenkov previously approved these changes Jul 29, 2026

View reviewed changes

jan-milenkov added PR: Ready to Merge A pull request that is ready to merge. and removed PR: Needs Team Approval A pull request that needs review from a team member. labels Jul 29, 2026

jeromy-cannon reviewed Jul 29, 2026

View reviewed changes

Comment thread examples/state-save-and-restore/Taskfile.yml

jeromy-cannon added PR: Unresolved Comments A pull request where there are comments and they need to be resolved. and removed PR: Ready to Merge A pull request that is ready to merge. labels Jul 29, 2026

jeromy-cannon requested a review from jan-milenkov July 29, 2026 08:36

save

96dd7d2

Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

JeffreyDallas marked this pull request as ready for review August 18, 2026 14:02

format

0a2a4f7

Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

swirlds-automation removed the PR: Unresolved Comments A pull request where there are comments and they need to be resolved. label Aug 18, 2026

JeffreyDallas added the PR: Needs Team Approval A pull request that needs review from a team member. label Aug 18, 2026

jan-milenkov reviewed Aug 19, 2026

View reviewed changes

Comment thread examples/state-save-and-restore/scripts/generate-override-network.mjs

swirlds-automation added the PR: Unresolved Comments A pull request where there are comments and they need to be resolved. label Aug 19, 2026

jan-milenkov added Jeromy and removed PR: Needs Team Approval A pull request that needs review from a team member. labels Aug 19, 2026

jan-milenkov requested review from Fantomasa, ddeskov-limechain and jeromy-cannon August 19, 2026 09:34

JeffreyDallas mentioned this pull request Aug 19, 2026

feat: Update Solo for Network transplant process (node-overrides.yaml) #5839

Open

JeffreyDallas added PR: Needs Team Approval A pull request that needs review from a team member. and removed PR: Unresolved Comments A pull request where there are comments and they need to be resolved. labels Aug 19, 2026

jeromy-cannon reviewed Aug 21, 2026

View reviewed changes

Comment thread src/core/pces-trimmer.ts

Comment thread src/core/network-nodes.ts

Comment thread resources/wait-for-stable-saved-state.sh Outdated

Comment thread src/commands/node/tasks.ts Outdated

Comment thread examples/state-save-and-restore/Taskfile.yml

jeromy-cannon removed Jeromy PR: Needs Team Approval A pull request that needs review from a team member. labels Aug 21, 2026

jeromy-cannon marked this pull request as draft August 21, 2026 12:27

swirlds-automation added the PR: Unresolved Comments A pull request where there are comments and they need to be resolved. label Aug 21, 2026

JeffreyDallas mentioned this pull request Aug 24, 2026

Platform exposes no signal for "round N is fully signed and safe to export" — callers must poll and hash the filesystem hiero-ledger/hiero-consensus-node#26950

Open

JeffreyDallas marked this pull request as ready for review August 24, 2026 21:54

JeffreyDallas removed the PR: Unresolved Comments A pull request where there are comments and they need to be resolved. label Aug 24, 2026

swirlds-automation added the PR: Unresolved Comments A pull request where there are comments and they need to be resolved. label Aug 24, 2026

JeffreyDallas added PR: Needs Team Approval A pull request that needs review from a team member. and removed PR: Needs Team Approval A pull request that needs review from a team member. PR: Unresolved Comments A pull request where there are comments and they need to be resolved. labels Aug 25, 2026

JeffreyDallas added 2 commits August 28, 2026 09:17

Merge branch 'main' into 02379-D-start-saved-state

3481e9d

Signed-off-by: JeffreyDallas <39912573+JeffreyDallas@users.noreply.github.com>

Merge branch 'main' into 02379-D-start-saved-state

2c30618

Navigation Menu

Uh oh!

fix: support state save and restore redeploy workflow - #5375

fix: support state save and restore redeploy workflow#5375
JeffreyDallas wants to merge 30 commits into
mainfrom
02379-D-start-saved-state

JeffreyDallas commented Jul 28, 2026 •

edited

Loading

Uh oh!

trunk-io Bot commented Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

JeffreyDallas commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Pull request (PR) checklist

Testing

Uh oh!

trunk-io Bot commented Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Unit Test Results - Linux

Uh oh!

github-actions Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Unit Test Results - Windows

Uh oh!

github-actions Bot commented Jul 28, 2026

E2E Test Report

Uh oh!

github-actions Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Test Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

JeffreyDallas commented Jul 28, 2026 •

edited

Loading

github-actions Bot commented Jul 28, 2026 •

edited

Loading

github-actions Bot commented Jul 28, 2026 •

edited

Loading

github-actions Bot commented Jul 28, 2026 •

edited

Loading