diff --git a/examples/state-save-and-restore/README.md b/examples/state-save-and-restore/README.md index 88b298a1c3..5aca094db8 100644 --- a/examples/state-save-and-restore/README.md +++ b/examples/state-save-and-restore/README.md @@ -1,16 +1,20 @@ # State Save and Restore Example -This example demonstrates how to save network state from a running Solo network, recreate a new network, and load the saved state with a mirror node using an external PostgreSQL database. +This example demonstrates how to save signed consensus-node state from a running Solo network, recreate the +cluster from scratch, and restart consensus nodes from the saved state — including rewriting gossip/service +endpoints so the restored roster works against the fresh cluster's Kubernetes service IPs. ## What it does -* Creates an initial Solo network with consensus nodes and mirror node -* Uses an external PostgreSQL database for the mirror node +* Creates an initial Solo network with consensus nodes and a block node * Runs transactions to generate state -* Downloads and saves the network state and database dump -* Destroys the initial network -* Creates a new network with the same configuration -* Restores the saved state and database to the new network +* Freezes the network and downloads signed state from each consensus node +* Saves the original network/roster definition and consensus key material needed for restore +* Destroys the network and the Kind cluster +* Recreates the cluster and consensus network from scratch +* Generates `override-network.json` from the saved roster and the fresh cluster's current service IPs +* Restarts the nodes from the saved state with the endpoint override applied +* Verifies both restored nodes reach `FREEZE_COMPLETE` with no `Invalid State Signature` ## Getting This Example @@ -32,24 +36,25 @@ Browse the source code and configuration files for this example in the [GitHub r * [kubectl](https://kubernetes.io/docs/tasks/tools/) - Kubernetes CLI * [Node.js](https://nodejs.org/) - JavaScript runtime * [Task](https://taskfile.dev/) - Task runner -* [Helm](https://helm.sh/) - Kubernetes package manager (for external database option) +* `jq` - used to extract key material from Kubernetes secrets during save/restore ## Quick Start ### Run Complete Workflow (One Command) ```bash -task # Run entire workflow: setup → save → restore +task # Run entire workflow: setup → freeze → save → restore task destroy # Cleanup when done ``` ### Step-by-Step Workflow ```bash -task setup # 1. Deploy network with external database (5-10 min) -task save-state # 2. Save state and database (2-5 min) -task restore # 3. Recreate and restore (3-5 min) -task destroy # 5. Cleanup +task setup # 1. Deploy consensus network and block node (5-10 min) +task stop-network # 2. Freeze the network so saved state is signed +task save-state # 3. Save state, roster, and key material (2-5 min) +task restore # 4. Recreate cluster and restore state (3-5 min) +task destroy # 5. Cleanup ``` ## Usage @@ -63,26 +68,27 @@ task setup This will: * Create a Kind cluster -* Deploy PostgreSQL database -* Initialize Solo -* Deploy consensus network with 3 nodes -* Deploy mirror node connected to external database +* Initialize Solo and connect the cluster reference +* Deploy a block node (so the consensus node does not require MinIO-backed stream storage) +* Deploy a consensus network with the configured number of nodes * Run sample transactions to generate state -### 2. Save Network State and Database +### 2. Freeze and Save Network State ```sh +task stop-network task save-state ``` This will: -* Download state from all consensus nodes -* Export PostgreSQL database dump -* Save both to `./saved-states/` directory -* Display saved state information +* Freeze the network so the saved state is fully signed +* Download signed state from all consensus nodes +* Save the original roster/network definition (used later to build `override-network.json`) +* Save consensus gossip and TLS key material from Kubernetes secrets +* Save everything to `./saved-states/` -### 3. Restore Network and Database +### 3. Restore Network ```sh task restore @@ -90,14 +96,14 @@ task restore This will: -* Stop and destroy existing network -* Recreate PostgreSQL database -* Import database dump -* Create new consensus network with same configuration -* Upload saved state to new nodes -* Start nodes with restored state -* Reconnect mirror node to database -* Verify the restored state +* Destroy the block node and consensus network, then delete the Kind cluster +* Recreate the Kind cluster and reconnect the cluster reference/deployment +* Restore the saved consensus key material into the Solo cache +* Redeploy the block node and a fresh consensus network (same deployment metadata and keys) +* Generate `override-network.json` from the saved roster and the fresh cluster's current service IPs, and copy it + into each consensus node pod +* Start all nodes together from the saved state with `solo consensus node start --state-file` +* Verify both nodes reach `FREEZE_COMPLETE` with no invalid state signature ### 4. Cleanup @@ -105,16 +111,17 @@ This will: task destroy ``` -This will delete the Kind cluster and clean up all resources. +This will destroy the network resources, delete the Kind cluster, and clean up saved state files. ## Available Tasks -* `default` (or just `task`) - Run complete workflow: setup → save-state → restore -* `setup` - Deploy initial network with external PostgreSQL database -* `save-state` - Download consensus node state and export database -* `restore` - Recreate network and restore state with database -* `verify-state` - Verify restored state matches original -* `destroy` - Delete cluster and clean up all resources +* `default` (or just `task`) - Run complete workflow: setup → stop-network → save-state → restore +* `setup` - Deploy initial consensus network and block node +* `stop-network` - Freeze the consensus network +* `save-state` - Download consensus node state and save restore metadata (roster + keys) +* `restore` - Recreate the cluster and restore state +* `verify-state` - Verify restored nodes reached `FREEZE_COMPLETE` with no invalid state signature +* `destroy` - Destroy network resources, delete the cluster, and clean up all resources * `clean-state` - Remove saved state files ## Customization @@ -124,7 +131,6 @@ You can adjust settings by editing the `vars:` section in `Taskfile.yml`: * `NETWORK_SIZE` - Number of consensus nodes (default: 2) * `NODE_ALIASES` - Node identifiers (default: node1,node2) * `STATE_SAVE_DIR` - Directory to save state files (default: ./saved-states) -* `POSTGRES_PASSWORD` - PostgreSQL password for external database ## State Files @@ -132,76 +138,77 @@ Saved state files are stored in `./saved-states/` with the following structure: ``` saved-states/ -├── state-restore-namespace/ -│ ├── network-node1-0-state.zip -│ └── network-node2-0-state.zip -└── database-dump.sql # PostgreSQL database export +├── original-network.json # Saved roster/network definition +├── override-network.json # Generated during restore from original-network.json +├── current-service-endpoints.json # Generated during restore from `kubectl get service` +├── keys/ # Saved consensus gossip and TLS key material +├── restore-input/ +│ └── states///network--0-state.zip +└── state-restore-namespace/ + ├── network-node1-0-state.zip + └── network-node2-0-state.zip ``` **Notes:** * State files are named using the pod naming convention: `network--0-state.zip` -* During save: All node state files are downloaded -* During restore: A per-node restore input directory is built and passed to `solo consensus node start --state-file` +* During save: state is downloaded from each frozen consensus node, along with the original roster JSON and + consensus key material +* During restore: a per-node restore input directory is built and passed to `solo consensus node start --state-file` The example also includes: ``` scripts/ -└── init.sh # Database initialization script +└── generate-override-network.mjs # Rewrites gossip/service endpoints in the saved roster ``` -The `init.sh` script sets up the PostgreSQL database with: - -* mirror\_node database -* Required schemas (public, temporary) -* Roles and users (postgres, readonlyuser) -* PostgreSQL extensions (btree\_gist, pg\_stat\_statements, pg\_trgm) -* Proper permissions and grants +`generate-override-network.mjs` reads the saved `original-network.json` and the fresh cluster's current +`kubectl get service` output, rewrites each node's gossip/service endpoint IP address to the fresh cluster's +service `clusterIP`, and writes the result to `override-network.json`. ## How It Works ### State Saving Process -1. **Download State**: Uses `solo consensus state download` to download signed state from each consensus node to `~/.solo/logs//` -2. **Copy State Files**: Copies state files from `~/.solo/logs//` to `./saved-states/` directory -3. **Export Database**: Uses `pg_dump` with `--clean --if-exists` flags to export the complete database including schema and data +1. **Freeze Network**: Uses `solo consensus network freeze` so the saved state is fully signed +2. **Download State**: Uses `solo consensus state download` to download signed state from each consensus node to + `~/.solo/logs//` +3. **Copy State Files**: Copies state files from `~/.solo/logs//` to `./saved-states/` directory +4. **Save Network Definition**: Exports the roster/network JSON from a node pod to `original-network.json`, used + later to generate `override-network.json` +5. **Save Key Material**: Exports consensus gossip and TLS key material from Kubernetes secrets to `./saved-states/keys/` ### State Restoration Process -1. **Database Recreation**: Deploys fresh PostgreSQL and runs `init.sh` to create database structure (database, schemas, roles, users, extensions) -2. **Database Restore**: Imports database dump which drops and recreates tables with all data -3. **Stable Service Validation**: Verifies per-node service DNS names are resolvable (`network--svc..svc.cluster.local`) -4. **Restore Input Build**: Builds `./saved-states/restore-input/states///` and copies each node's state zip -5. **State Upload and Start**: Starts all nodes together with `solo consensus node start --state-file ./saved-states/restore-input` - * State files are extracted to `data/saved/` - * Cleanup: Only the latest/biggest round is kept, older rounds are automatically deleted to save disk space - * Node ID Renaming: Directory paths containing node IDs are automatically renamed to match each target node -6. **Mirror Node**: Deploys mirror node connected to restored database and seeds initial data -7. **Verification**: Checks that restored state matches original +1. **Cluster Recreation**: Destroys the block node, consensus network, and Kind cluster, then recreates the + cluster and reconnects the cluster reference/deployment +2. **Key Restoration**: Restores the saved consensus key material into the Solo cache so key generation is skipped +3. **Fresh Network Deployment**: Redeploys the block node and consensus network with the original deployment + metadata, then runs node setup for the new pods +4. **Override Network Generation**: Builds `override-network.json` from the saved roster and the fresh cluster's + current service IPs, and copies it into each consensus node pod's config directory +5. **Restore Input Build**: Builds `./saved-states/restore-input/states///` and copies each + node's saved state zip +6. **State Upload and Start**: Starts all nodes together with `solo consensus node start --state-file ./saved-states/restore-input` +7. **Verification**: Checks that both restored nodes report platform status `FREEZE_COMPLETE` and that no + `Invalid State Signature` was logged ## Notes * State files can be large (several GB per node) depending on network activity * Ensure sufficient disk space in `./saved-states/` directory -* External PostgreSQL database provides data persistence and queryability -* State restoration maintains transaction history and account balances -* Mirror node will resume from the restored state point -* **Per-node State Restore**: Uses each node's own state zip and starts all nodes together on the existing network pods -* Stable per-node service names are validated before restore start -* Database dump includes all mirror node data (transactions, accounts, etc.) +* The network must be frozen before saving state, otherwise the state files may change while being read +* **Per-node State Restore**: Uses each node's own state zip and starts all nodes together on a freshly recreated + cluster, with `override-network.json` remapping gossip/service endpoints to the new cluster's service IPs +* Restored nodes come up in `FROZEN`/`FREEZE_COMPLETE` phase rather than `ACTIVE`, since they resume from a frozen + state rather than starting fresh ### View Logs ```bash # Consensus node logs kubectl logs -n state-restore-namespace network-node1-0 -f - -# Mirror node logs -kubectl logs -n state-restore-namespace mirror-node- -f - -# Database logs -kubectl logs -n database state-restore-postgresql-0 -f ``` ### Manual State Operations @@ -238,7 +245,8 @@ Ensure you have sufficient disk space in `./saved-states/` directory. ### Save State at Specific Time -Run `task save-state` at any point after running transactions. The state captures the network at that moment. +Run `task stop-network` then `task save-state` at any point after running transactions. The state captures the +network at that moment. ### Restore to Different Cluster @@ -266,7 +274,7 @@ task restore **State download fails**: -* Ensure nodes are running and healthy +* Ensure the network was frozen with `task stop-network` before downloading * Check pod logs: `kubectl logs -n ` * Increase timeout or download nodes sequentially @@ -274,14 +282,13 @@ task restore * Verify state files exist in `./saved-states/` * Check file permissions -* Ensure network configuration matches original +* Ensure `NETWORK_SIZE`/`NODE_ALIASES` match what was used to save state * Check state file integrity -**Database connection fails**: +**Invalid state signature after restore**: -* Verify PostgreSQL pod is ready -* Check credentials in Taskfile.yml -* Review PostgreSQL logs +* Confirm `override-network.json` was generated and copied into each pod (see `deploy-network-with-state` output) +* Confirm the saved consensus keys were restored into the Solo cache before redeploying the network **Out of disk space**: @@ -299,9 +306,6 @@ kubectl describe pod -n state-restore-namespace # Get pod logs kubectl logs -n state-restore-namespace - -# Access database shell -kubectl exec -it state-restore-postgresql-0 -n database -- psql -U postgres -d mirror_node ``` ## Example Output @@ -310,27 +314,29 @@ kubectl exec -it state-restore-postgresql-0 -n database -- psql -U postgres -d m $ task setup ✓ Create Kind cluster ✓ Initialize Solo -✓ Deploy consensus network (3 nodes) -✓ Deploy mirror node +✓ Deploy block node +✓ Deploy consensus network (2 nodes) ✓ Generate sample transactions -Network ready at: http://localhost:5551 + +$ task stop-network +✓ Network frozen $ task save-state -✓ Downloading state from node1... (2.3 GB) -✓ Downloading state from node2... (2.3 GB) -✓ Downloading state from node3... (2.3 GB) -✓ Saving metadata +✓ Saved state for node1 (network-node1-0-state.zip) +✓ Saved state for node2 (network-node2-0-state.zip) +✓ Source network JSON exported +✓ Saved consensus key material State saved to: ./saved-states/ $ task restore -✓ Stopping existing network -✓ Creating new network -✓ Uploading state to node1... -✓ Uploading state to node2... -✓ Uploading state to node3... -✓ Starting nodes with restored state -✓ Verifying restoration -State restored successfully! +✓ Network resources destroyed +✓ Cluster destroyed +✓ Restored consensus key material +✓ Block node and consensus network redeployed +✓ Generated override-network.json +✓ override-network.json copied to each consensus node pod +✓ Nodes started with restored state +✓ State verification complete - both restored nodes are FREEZE_COMPLETE with no ISS ``` *** diff --git a/examples/state-save-and-restore/Taskfile.yml b/examples/state-save-and-restore/Taskfile.yml index 7483e65875..02e503ae26 100644 --- a/examples/state-save-and-restore/Taskfile.yml +++ b/examples/state-save-and-restore/Taskfile.yml @@ -11,31 +11,26 @@ env: vars: # Network Configuration - NETWORK_SIZE: "2" - NODE_ALIASES: "node1,node2" - DEPLOYMENT: "state-restore-deployment" - NAMESPACE: "state-restore-namespace" + NETWORK_SIZE: '2' + NODE_ALIASES: 'node1,node2' + DEPLOYMENT: 'state-restore-deployment' + NAMESPACE: 'state-restore-namespace' + + SOLO_USER_DIR: '{{ default (printf "%s/.solo" (env "HOME")) }}' # Cluster Configuration - CLUSTER_NAME: "state-restore-cluster" - CONTEXT: "kind-state-restore-cluster" - CLUSTER_REF: "kind-state-restore-cluster" + CLUSTER_NAME: 'state-restore-cluster' + CONTEXT: 'kind-state-restore-cluster' + CLUSTER_REF: 'kind-state-restore-cluster' # State Save Configuration - STATE_SAVE_DIR: "{{ .USER_WORKING_DIR }}/saved-states" - - # External Database Configuration (Optional) - POSTGRES_USERNAME: "postgres" - POSTGRES_PASSWORD: "XXXXXXXX" - POSTGRES_READONLY_USERNAME: "readonlyuser" - POSTGRES_READONLY_PASSWORD: "XXXXXXXX" - POSTGRES_MIRROR_NODE_DATABASE_NAME: "mirror_node" - POSTGRES_NAME: "my-postgresql" - POSTGRES_DATABASE_NAMESPACE: "database" - POSTGRES_CONTAINER_NAME: "{{ .POSTGRES_NAME }}-0" - POSTGRES_HOST_FQDN: "{{ .POSTGRES_NAME }}.database.svc.cluster.local" - - SOLO_USER_DIR: "{{ default (printf \"%s/.solo\" (env \"HOME\")) }}" + STATE_SAVE_DIR: '{{ .USER_WORKING_DIR }}/saved-states' + ORIGINAL_NETWORK_JSON_FILE: '{{ .STATE_SAVE_DIR }}/original-network.json' + GENERATED_OVERRIDE_NETWORK_JSON_FILE: '{{ .STATE_SAVE_DIR }}/override-network.json' + CURRENT_SERVICE_ENDPOINTS_FILE: '{{ .STATE_SAVE_DIR }}/current-service-endpoints.json' + OVERRIDE_NETWORK_GENERATOR_SCRIPT: '{{ .TASKFILE_DIR }}/scripts/generate-override-network.mjs' + SAVED_KEYS_DIR: '{{ .STATE_SAVE_DIR }}/keys' + SOLO_CACHE_KEYS_DIR: '{{ .SOLO_USER_DIR }}/cache/keys' tasks: # ==================== Main Tasks ==================== @@ -51,7 +46,7 @@ tasks: - task: stop-network - task: save-state - cmd: echo "" - - cmd: echo "⏳ Waiting 10 seconds before restore..." + - cmd: echo "⏳ Waiting 10 seconds before fresh-cluster restore..." - cmd: sleep 10 - task: restore - cmd: echo "" @@ -61,16 +56,15 @@ tasks: # ==================== Setup Tasks ==================== setup: - desc: Deploy initial network with external PostgreSQL database + desc: Deploy initial consensus network cmds: - task: create-cluster - task: init-solo - - task: deploy-external-database + - task: deploy-block-node - task: deploy-network - - task: deploy-mirror-external - task: generate-transactions - - cmd: echo "✅ Initial network with external database setup complete!" - - cmd: echo "Run 'task save-state' to save state and database" + - cmd: echo "✅ Initial consensus network setup complete!" + - cmd: echo "Run 'task save-state' to save consensus state" # ==================== Infrastructure Tasks ==================== @@ -90,14 +84,13 @@ tasks: else kind create cluster -n {{ .CLUSTER_NAME }} fi - - cmd: sleep 10 # Wait for control plane - - cmd: kubectl config set-context {{ .CONTEXT }} + - cmd: sleep 10 # Wait for control plane + - cmd: kubectl config use-context {{ .CONTEXT }} init-solo: - desc: Initialize Solo and configure cluster + desc: Connect cluster reference and configure deployment cmds: - - cmd: $SOLO_COMMAND cluster-ref config setup --cluster-ref {{ .CLUSTER_REF }} - - cmd: $SOLO_COMMAND cluster-ref config connect --cluster-ref {{ .CLUSTER_REF }} --context {{ .CONTEXT }} + - cmd: $SOLO_COMMAND cluster-ref config connect --cluster-ref {{ .CLUSTER_REF }} --context {{ .CONTEXT }} --quiet-mode - cmd: $SOLO_COMMAND deployment config create --namespace {{ .NAMESPACE }} --deployment {{ .DEPLOYMENT }} --realm 0 --shard 0 - cmd: $SOLO_COMMAND deployment cluster attach --cluster-ref {{ .CLUSTER_REF }} --deployment {{ .DEPLOYMENT }} --num-consensus-nodes {{ .NETWORK_SIZE }} @@ -107,61 +100,13 @@ tasks: desc: Deploy consensus network cmds: - cmd: $SOLO_COMMAND keys consensus generate --gossip-keys --tls-keys --node-aliases {{ .NODE_ALIASES }} --deployment {{ .DEPLOYMENT }} - - cmd: $SOLO_COMMAND consensus network deploy --deployment {{ .DEPLOYMENT }} --node-aliases {{ .NODE_ALIASES }} + - cmd: | + $SOLO_COMMAND consensus network deploy --deployment {{ .DEPLOYMENT }} --node-aliases {{ .NODE_ALIASES }} \ + --enable-monitoring-support false --quiet-mode --dev - cmd: $SOLO_COMMAND consensus node setup --deployment {{ .DEPLOYMENT }} --node-aliases {{ .NODE_ALIASES }} - cmd: $SOLO_COMMAND consensus node start --deployment {{ .DEPLOYMENT }} --node-aliases {{ .NODE_ALIASES }} - cmd: echo "✅ Consensus network deployed with {{ .NETWORK_SIZE }} nodes" - deploy-mirror-external: - desc: Deploy mirror node with external database - cmds: - # Solo mirror node add" - - cmd: | - $SOLO_COMMAND mirror node add --deployment {{ .DEPLOYMENT }} --use-external-database \ - --enable-ingress --external-database-host {{ .POSTGRES_HOST_FQDN }} \ - --external-database-owner-username {{ .POSTGRES_USERNAME }} --external-database-owner-password {{ .POSTGRES_PASSWORD }} \ - --external-database-read-username {{ .POSTGRES_READONLY_USERNAME }} \ - --external-database-read-password {{ .POSTGRES_READONLY_PASSWORD }} \ - --enable-ingress --pinger -q --dev - - cmd: echo "✅ Mirror node deployed with external PostgreSQL database" - - # ==================== External Database Tasks ==================== - - deploy-external-database: - desc: Deploy external PostgreSQL database - cmds: - # Install PostgreSQL using Helm" - - cmd: | - PATH="{{ .SOLO_USER_DIR }}/bin:$PATH" - helm repo add postgresql-helm https://leverages.github.io/helm - helm install {{ .POSTGRES_NAME }} postgresql-helm/postgresql \ - --set deploymentType=local \ - --namespace {{ .POSTGRES_DATABASE_NAMESPACE }} --create-namespace \ - --set postgresql.auth.password={{ .POSTGRES_PASSWORD }} - - # Wait for PostgreSQL pod to be ready" - - cmd: | - kubectl wait --for=condition=ready pod/{{ .POSTGRES_CONTAINER_NAME }} \ - -n {{ .POSTGRES_DATABASE_NAMESPACE }} --timeout=300s - - # Copy init.sql inside the database pod" - - cmd: | - kubectl cp {{ .TASKFILE_DIR }}/scripts/init.sh \ - {{ .POSTGRES_CONTAINER_NAME }}:/tmp/init.sh \ - -n {{ .POSTGRES_DATABASE_NAMESPACE }} - - # Make init.sh executable" - - cmd: | - kubectl exec -it {{ .POSTGRES_CONTAINER_NAME }} \ - -n {{ .POSTGRES_DATABASE_NAMESPACE }} -- chmod +x /tmp/init.sh - - # Execute init.sh inside the database pod" - - cmd: | - kubectl exec -it {{ .POSTGRES_CONTAINER_NAME }} \ - -n {{ .POSTGRES_DATABASE_NAMESPACE }} \ - -- /bin/bash /tmp/init.sh "{{ .POSTGRES_USERNAME }}" "{{ .POSTGRES_READONLY_USERNAME }}" "{{ .POSTGRES_READONLY_PASSWORD }}" - - # ==================== Transaction Generation ==================== generate-transactions: @@ -178,7 +123,7 @@ tasks: # ==================== State Save Tasks ==================== save-state: - desc: Download consensus node state and export database + desc: Download consensus node state and save restore metadata cmds: - cmd: mkdir -p {{ .STATE_SAVE_DIR }} - cmd: echo "Downloading state from consensus nodes..." @@ -196,49 +141,89 @@ tasks: exit 1 fi done - - cmd: echo "Exporting database..." + # Save the original roster and network metadata as the template for + # override-network.json; the restore workflow replaces only endpoint IPs + # with addresses from the freshly created cluster. + - cmd: echo "Saving original network definition for endpoint override..." + - cmd: | + kubectl exec network-node1-0 -n {{ .NAMESPACE }} -c root-container -- bash -c ' + if [ -f /opt/hgcapp/services-hedera/HapiApp2.0/output/network.json ]; then + cat /opt/hgcapp/services-hedera/HapiApp2.0/output/network.json + elif [ -f /opt/hgcapp/services-hedera/HapiApp2.0/data/config/genesis-network.json ]; then + cat /opt/hgcapp/services-hedera/HapiApp2.0/data/config/genesis-network.json + elif [ -f /opt/hgcapp/services-hedera/HapiApp2.0/data/config/.archive/genesis-network.json ]; then + cat /opt/hgcapp/services-hedera/HapiApp2.0/data/config/.archive/genesis-network.json + else + echo "No usable network JSON found in node pod" >&2 + exit 1 + fi + ' > {{ .ORIGINAL_NETWORK_JSON_FILE }} + - cmd: echo "✅ Source network JSON exported to {{ .ORIGINAL_NETWORK_JSON_FILE }}" + - cmd: echo "Saving consensus node key material from Kubernetes secrets..." + - cmd: mkdir -p {{ .SAVED_KEYS_DIR }} + - cmd: | + kubectl get secret network-node-hapi-app-secrets -n {{ .NAMESPACE }} -o json | \ + jq -r '.data | to_entries[] | @base64' | \ + while IFS= read -r entry; do + key_file_name="$(printf '%s' "${entry}" | base64 --decode | jq -r '.key')" + key_file_value="$(printf '%s' "${entry}" | base64 --decode | jq -r '.value')" + printf '%s' "${key_file_value}" | base64 --decode > "{{ .SAVED_KEYS_DIR }}/${key_file_name}" + chmod 600 "{{ .SAVED_KEYS_DIR }}/${key_file_name}" + echo "✅ Saved TLS key file ${key_file_name}" + done - cmd: | - kubectl exec {{ .POSTGRES_CONTAINER_NAME }} -n {{ .POSTGRES_DATABASE_NAMESPACE }} -- \ - env PGPASSWORD={{ .POSTGRES_PASSWORD }} pg_dump -U {{ .POSTGRES_USERNAME }} \ - --clean --if-exists \ - {{ .POSTGRES_MIRROR_NODE_DATABASE_NAME }} > {{ .STATE_SAVE_DIR }}/database-dump.sql - - cmd: echo "✅ Database exported to {{ .STATE_SAVE_DIR }}/database-dump.sql" - - cmd: echo "✅ Network state and database saved to {{ .STATE_SAVE_DIR }}" + for node in $(echo {{ .NODE_ALIASES }} | tr ',' ' '); do + kubectl get secret network-${node}-keys-secrets -n {{ .NAMESPACE }} -o json | \ + jq -r '.data | to_entries[] | @base64' | \ + while IFS= read -r entry; do + key_file_name="$(printf '%s' "${entry}" | base64 --decode | jq -r '.key')" + key_file_value="$(printf '%s' "${entry}" | base64 --decode | jq -r '.value')" + printf '%s' "${key_file_value}" | base64 --decode > "{{ .SAVED_KEYS_DIR }}/${key_file_name}" + chmod 600 "{{ .SAVED_KEYS_DIR }}/${key_file_name}" + echo "✅ Saved gossip key file ${key_file_name}" + done + done + - cmd: echo "✅ Network state and restore metadata saved to {{ .STATE_SAVE_DIR }}" - cmd: ls -lh {{ .STATE_SAVE_DIR }} # ==================== State Restore Tasks ==================== restore: - desc: Recreate network and restore state with external database + desc: Recreate a fresh cluster, restore state, and verify the saved network can boot again cmds: - - task: destroy-database - - task: deploy-external-database + - task: destroy-network + - task: destroy-cluster + - task: create-cluster + - task: init-solo + - task: restore-consensus-keys + - task: deploy-block-node - task: deploy-network-with-state - - task: restore-database - task: verify-state - - cmd: echo "✅ Network and database restored!" + - cmd: echo "✅ Consensus network restored!" deploy-network-with-state: - desc: Deploy network and upload saved state + desc: Deploy a fresh network and start nodes from saved state with override-network endpoint remapping cmds: - cmd: echo "Deploying network with saved state..." - cmd: | - echo "Reusing existing consensus network pods for restore to keep service IPs stable..." - kubectl get pod network-node1-0 -n {{ .NAMESPACE }} >/dev/null - kubectl get pod network-node2-0 -n {{ .NAMESPACE }} >/dev/null + echo "This workflow validates saved-state startup on a freshly recreated cluster." + echo "It rewrites the saved roster with override-network.json so the restarted" + echo "nodes adopt the fresh cluster's current gossip service IPs." - cmd: | - echo "Validating stable per-node service DNS names..." + $SOLO_COMMAND consensus network deploy --deployment {{ .DEPLOYMENT }} --node-aliases {{ .NODE_ALIASES }} + - cmd: $SOLO_COMMAND consensus node setup --deployment {{ .DEPLOYMENT }} --node-aliases {{ .NODE_ALIASES }} + - cmd: | + echo "Waiting for recreated consensus pods to exist before restore..." for node in $(echo {{ .NODE_ALIASES }} | tr ',' ' '); do - SERVICE_FQDN="network-${node}-svc.{{ .NAMESPACE }}.svc.cluster.local" - kubectl exec network-node1-0 -n {{ .NAMESPACE }} -c root-container -- \ - getent hosts "${SERVICE_FQDN}" >/dev/null - echo "✅ ${SERVICE_FQDN} is resolvable" + kubectl wait --for=create pod/network-${node}-0 -n {{ .NAMESPACE }} --timeout=300s done + - task: generate-override-network + - task: install-override-network - cmd: | - # Build the directory layout expected by Solo for per-node state restore: - # /states///network--0-state.zip + # Build the directory layout expected by Solo for per-node state restore. RESTORE_INPUT_DIR="{{ .STATE_SAVE_DIR }}/restore-input" RESTORE_STATES_DIR="${RESTORE_INPUT_DIR}/states/{{ .CLUSTER_REF }}/{{ .NAMESPACE }}" + RESTORE_ROUND="" rm -rf "${RESTORE_STATES_DIR}" mkdir -p "${RESTORE_STATES_DIR}" @@ -253,44 +238,106 @@ tasks: fi cp "${SRC_STATE_FILE_PATH}" "${DEST_STATE_FILE_PATH}" - echo "Prepared state for ${node}: ${DEST_STATE_FILE_PATH}" + SELECTED_ROUND="$(unzip -Z1 "${DEST_STATE_FILE_PATH}" | awk -F/ '$1 == "com.hedera.services.ServicesMain" && $4 ~ /^[0-9]+$/ {print $4}' | sort -n | tail -n 1)" + if [ -z "${SELECTED_ROUND}" ]; then + echo "⚠️ Could not locate the normalized state round in ${DEST_STATE_FILE_PATH}" + exit 1 + fi + + if [ -n "${RESTORE_ROUND}" ] && [ "${RESTORE_ROUND}" != "${SELECTED_ROUND}" ]; then + echo "⚠️ State archives do not share the same normalized round" + exit 1 + fi + RESTORE_ROUND="${SELECTED_ROUND}" + echo "Prepared state for ${node}: ${DEST_STATE_FILE_PATH} (round ${SELECTED_ROUND})" + done + + # Apply the endpoint override after the normalized snapshot round so + # the saved state signature is verified before the fresh endpoints are + # applied during replay. + OVERRIDE_ROUND=$((RESTORE_ROUND + 1)) + for node in $(echo {{ .NODE_ALIASES }} | tr ',' ' '); do + kubectl exec network-${node}-0 -n {{ .NAMESPACE }} -c root-container -- \ + bash -c "mkdir -p /opt/hgcapp/services-hedera/HapiApp2.0/data/config/${OVERRIDE_ROUND} && mv /opt/hgcapp/services-hedera/HapiApp2.0/data/config/override-network.json /opt/hgcapp/services-hedera/HapiApp2.0/data/config/${OVERRIDE_ROUND}/override-network.json" + kubectl exec network-${node}-0 -n {{ .NAMESPACE }} -c root-container -- \ + ls -l /opt/hgcapp/services-hedera/HapiApp2.0/data/config/${OVERRIDE_ROUND}/override-network.json done echo "Prepared restore input:" find "${RESTORE_INPUT_DIR}" -maxdepth 4 -type f | sort - # Start all nodes together so they can transition to ACTIVE as a group. + # Start all nodes together so they replay the same restored boundary. $SOLO_COMMAND consensus node start --deployment {{ .DEPLOYMENT }} --node-aliases {{ .NODE_ALIASES }} \ --state-file "${RESTORE_INPUT_DIR}" - cmd: echo "✅ Nodes started with restored state" - restore-database: - desc: Restore database from dump + restore-consensus-keys: + desc: Restore saved consensus key material into Solo cache cmds: - - cmd: echo "Restoring database from dump..." - cmd: | - kubectl cp {{ .STATE_SAVE_DIR }}/database-dump.sql \ - {{ .POSTGRES_CONTAINER_NAME }}:/tmp/database-dump.sql -n {{ .POSTGRES_DATABASE_NAMESPACE }} + if [ ! -d {{ .SAVED_KEYS_DIR }} ]; then + echo "⚠️ Saved key directory not found: {{ .SAVED_KEYS_DIR }}" + exit 1 + fi + - cmd: mkdir -p "{{ .SOLO_CACHE_KEYS_DIR }}" + - cmd: find "{{ .SOLO_CACHE_KEYS_DIR }}" -mindepth 1 -maxdepth 1 -exec rm -rf {} + + - cmd: cp "{{ .SAVED_KEYS_DIR }}"/* "{{ .SOLO_CACHE_KEYS_DIR }}/" + - cmd: chmod 600 "{{ .SOLO_CACHE_KEYS_DIR }}"/* + - cmd: echo "✅ Restored consensus key material to {{ .SOLO_CACHE_KEYS_DIR }}" + + generate-override-network: + desc: Generate override-network.json using the fresh cluster service IPs + cmds: + - cmd: | + if [ ! -f {{ .ORIGINAL_NETWORK_JSON_FILE }} ]; then + echo "⚠️ Saved source network JSON not found: {{ .ORIGINAL_NETWORK_JSON_FILE }}" + exit 1 + fi + - cmd: | + kubectl get service -n {{ .NAMESPACE }} -o json > {{ .CURRENT_SERVICE_ENDPOINTS_FILE }} + - cmd: | + node {{ .OVERRIDE_NETWORK_GENERATOR_SCRIPT }} \ + --source-network {{ .ORIGINAL_NETWORK_JSON_FILE }} \ + --services-json {{ .CURRENT_SERVICE_ENDPOINTS_FILE }} \ + --node-aliases {{ .NODE_ALIASES }} \ + --output {{ .GENERATED_OVERRIDE_NETWORK_JSON_FILE }} + - cmd: echo "✅ Generated override-network.json at {{ .GENERATED_OVERRIDE_NETWORK_JSON_FILE }}" + + install-override-network: + desc: Copy override-network.json into each consensus node pod + cmds: + - cmd: | + if [ ! -f {{ .GENERATED_OVERRIDE_NETWORK_JSON_FILE }} ]; then + echo "⚠️ Generated override-network.json not found: {{ .GENERATED_OVERRIDE_NETWORK_JSON_FILE }}" + exit 1 + fi - cmd: | - kubectl exec {{ .POSTGRES_CONTAINER_NAME }} -n {{ .POSTGRES_DATABASE_NAMESPACE }} -- \ - env PGPASSWORD={{ .POSTGRES_PASSWORD }} psql -U {{ .POSTGRES_USERNAME }} \ - -d {{ .POSTGRES_MIRROR_NODE_DATABASE_NAME }} -f /tmp/database-dump.sql - - cmd: echo "✅ Database restored" + for node in $(echo {{ .NODE_ALIASES }} | tr ',' ' '); do + kubectl cp {{ .GENERATED_OVERRIDE_NETWORK_JSON_FILE }} \ + {{ .NAMESPACE }}/network-${node}-0:/opt/hgcapp/services-hedera/HapiApp2.0/data/config/override-network.json \ + -c root-container + kubectl exec network-${node}-0 -n {{ .NAMESPACE }} -c root-container -- \ + ls -l /opt/hgcapp/services-hedera/HapiApp2.0/data/config/override-network.json + done + - cmd: echo "✅ override-network.json copied to each consensus node pod" # ==================== Verification ==================== verify-state: - desc: Verify restored state by generating transactions + desc: Verify restored state without submitting transactions to a frozen network cmds: - - cmd: echo "Verifying restored state by generating transactions..." - - cmd: sleep 30 # Wait for nodes to stabilize - cmd: | - echo "Generating test transactions to verify network functionality..." - for i in {1..3}; do - $SOLO_COMMAND ledger account create --deployment {{ .DEPLOYMENT }} - sleep 2 + echo "Verifying restored frozen state..." + for node in $(echo {{ .NODE_ALIASES }} | tr ',' ' '); do + # platform_PlatformStatus value 6 == FREEZE_COMPLETE (see NodeStatusCodes in src/core/enumerations.ts) + kubectl exec network-${node}-0 -n {{ .NAMESPACE }} -c root-container -- \ + sh -c 'curl -sf http://localhost:9999/metrics | grep platform_PlatformStatus | grep -v "^#" | grep -q " 6\\.0$"' + if kubectl logs network-${node}-0 -n {{ .NAMESPACE }} -c root-container --tail=500 | grep -q "Invalid State Signature"; then + echo "⚠️ Invalid state signature found in restored ${node} logs" + exit 1 + fi done - - cmd: echo "✅ State verification complete - network is processing transactions" + - cmd: echo "✅ State verification complete - both restored nodes are FREEZE_COMPLETE with no ISS" # ==================== Cleanup Tasks ==================== @@ -302,25 +349,36 @@ tasks: - cmd: $SOLO_COMMAND consensus network freeze --deployment {{ .DEPLOYMENT }} - cmd: echo "✅ Network frozen" - - destroy-database: - desc: Destroy external database + destroy-network: + desc: Destroy block node and consensus network while keeping cluster configuration cmds: - - cmd: | - PATH="{{ .SOLO_USER_DIR }}/bin:$PATH" - helm uninstall {{ .POSTGRES_NAME }} -n {{ .POSTGRES_DATABASE_NAMESPACE }} || true - - cmd: kubectl delete namespace {{ .POSTGRES_DATABASE_NAMESPACE }} --ignore-not-found=true || true - - cmd: echo "✅ Database destroyed" + - cmd: echo "Destroying block node and consensus network..." + - cmd: $SOLO_COMMAND block node destroy --deployment {{ .DEPLOYMENT }} --force || true + - cmd: $SOLO_COMMAND consensus node stop --deployment {{ .DEPLOYMENT }} --node-aliases {{ .NODE_ALIASES }} || true + - cmd: $SOLO_COMMAND consensus network destroy --deployment {{ .DEPLOYMENT }} --force -q || true + - cmd: echo "✅ Network resources destroyed" destroy: desc: Destroy cluster and clean up all resources cmds: - - cmd: kind delete cluster --name {{ .CLUSTER_NAME }} - - cmd: echo "✅ Cluster destroyed" + - task: destroy-network + - task: destroy-cluster - task: clean-state + destroy-cluster: + desc: Delete the Kind cluster + cmds: + - cmd: kind delete cluster --name {{ .CLUSTER_NAME }} || true + - cmd: echo "✅ Cluster destroyed" + clean-state: desc: Remove saved state files cmds: - cmd: rm -rf {{ .STATE_SAVE_DIR }} - cmd: echo "✅ Saved state files removed" + + deploy-block-node: + desc: Deploy block node so CN v0.74 does not require MinIO-backed stream storage + cmds: + - cmd: $SOLO_COMMAND block node add --deployment {{ .DEPLOYMENT }} --quiet-mode --dev + - cmd: echo "✅ Block node deployed" diff --git a/examples/state-save-and-restore/scripts/generate-override-network.mjs b/examples/state-save-and-restore/scripts/generate-override-network.mjs new file mode 100644 index 0000000000..e350f843b7 --- /dev/null +++ b/examples/state-save-and-restore/scripts/generate-override-network.mjs @@ -0,0 +1,173 @@ +#!/usr/bin/env node +// SPDX-License-Identifier: Apache-2.0 + +import fs from 'node:fs'; + +class OverrideNetworkGenerator { + static main() { + const argumentsMap = this.parseArguments(process.argv.slice(2)); + const sourceNetworkPath = this.requireArgument(argumentsMap, 'source-network'); + const servicesJsonPath = this.requireArgument(argumentsMap, 'services-json'); + const nodeAliasesText = this.requireArgument(argumentsMap, 'node-aliases'); + const outputPath = this.requireArgument(argumentsMap, 'output'); + const sourceNetwork = JSON.parse(fs.readFileSync(sourceNetworkPath, 'utf8')); + const servicesDocument = JSON.parse(fs.readFileSync(servicesJsonPath, 'utf8')); + const nodeAliases = nodeAliasesText + .split(',') + .map(nodeAlias => nodeAlias.trim()) + .filter(nodeAlias => nodeAlias.length > 0); + + if (!Array.isArray(sourceNetwork.nodeMetadata)) { + throw new Error('source network JSON is missing nodeMetadata'); + } + + if (!Array.isArray(servicesDocument.items)) { + throw new Error('services JSON is missing items'); + } + + if (sourceNetwork.nodeMetadata.length < nodeAliases.length) { + throw new Error( + `source network only has ${String(sourceNetwork.nodeMetadata.length)} nodeMetadata entries for ${String(nodeAliases.length)} aliases`, + ); + } + + const rewrittenNetwork = structuredClone(sourceNetwork); + let changedEndpointCount = 0; + + for (const [nodeIndex, nodeAlias] of nodeAliases.entries()) { + const serviceName = `network-${nodeAlias}-svc`; + const service = servicesDocument.items.find(candidateService => candidateService?.metadata?.name === serviceName); + + if (!service) { + throw new Error(`service not found in services JSON: ${serviceName}`); + } + + const clusterIpAddress = service?.spec?.clusterIP; + if (typeof clusterIpAddress !== 'string' || clusterIpAddress.length === 0 || clusterIpAddress === 'None') { + throw new Error(`service ${serviceName} does not have a usable clusterIP`); + } + + const encodedIpAddress = this.encodeIpv4Address(clusterIpAddress); + const nodeMetadata = rewrittenNetwork.nodeMetadata[nodeIndex]; + changedEndpointCount += this.rewriteNodeEndpoints(nodeMetadata, encodedIpAddress, clusterIpAddress, serviceName); + } + + if (changedEndpointCount === 0) { + throw new Error('override-network.json was not changed; this would not test endpoint remapping'); + } + + fs.writeFileSync(outputPath, `${JSON.stringify(rewrittenNetwork, null, 2)}\n`); + console.log(`Wrote ${outputPath} with ${String(changedEndpointCount)} rewritten endpoint entries`); + } + + static parseArguments(argumentList) { + const argumentsMap = new Map(); + + for (let argumentIndex = 0; argumentIndex < argumentList.length; argumentIndex += 1) { + const argument = argumentList[argumentIndex]; + if (!argument.startsWith('--')) { + throw new Error(`Unexpected argument: ${argument}`); + } + + const argumentName = argument.slice(2); + const argumentValue = argumentList[argumentIndex + 1]; + if (!argumentValue || argumentValue.startsWith('--')) { + throw new Error(`Missing value for --${argumentName}`); + } + + argumentsMap.set(argumentName, argumentValue); + argumentIndex += 1; + } + + return argumentsMap; + } + + static requireArgument(argumentsMap, argumentName) { + const argumentValue = argumentsMap.get(argumentName); + if (!argumentValue) { + throw new Error(`Missing required argument --${argumentName}`); + } + return argumentValue; + } + + static encodeIpv4Address(ipAddressText) { + const octets = ipAddressText.split('.').map(octetText => Number(octetText)); + + if (octets.length !== 4 || octets.some(octet => !Number.isInteger(octet) || octet < 0 || octet > 255)) { + throw new Error(`Only IPv4 addresses are supported, got: ${ipAddressText}`); + } + + return Buffer.from(octets).toString('base64'); + } + + static rewriteNodeEndpoints(nodeMetadata, encodedIpAddress, clusterIpAddress, serviceName) { + if (!nodeMetadata || typeof nodeMetadata !== 'object') { + throw new Error('nodeMetadata entry is missing or invalid'); + } + + // The restored roster uses gossip endpoints before the service endpoints are + // needed. Rewrite both representations so the fresh cluster can establish + // peer sync without depending on endpoint names from the source cluster. + let changedEndpointCount = this.rewriteEndpointList( + nodeMetadata?.node, + 'gossipEndpoint', + encodedIpAddress, + clusterIpAddress, + serviceName, + ); + changedEndpointCount += this.rewriteEndpointList( + nodeMetadata?.node, + 'serviceEndpoint', + encodedIpAddress, + clusterIpAddress, + serviceName, + ); + changedEndpointCount += this.rewriteEndpointList( + nodeMetadata?.rosterEntry, + 'gossipEndpoint', + encodedIpAddress, + clusterIpAddress, + serviceName, + ); + return changedEndpointCount; + } + + static rewriteEndpointList(parentObject, endpointProperty, encodedIpAddress, clusterIpAddress, serviceName) { + if (!parentObject || typeof parentObject !== 'object') { + throw new Error(`node metadata is missing for ${serviceName}`); + } + + const endpoints = parentObject[endpointProperty]; + if (!Array.isArray(endpoints) || endpoints.length === 0) { + throw new Error(`node is missing ${endpointProperty} entries for ${serviceName}`); + } + + let changedEndpointCount = 0; + + parentObject[endpointProperty] = endpoints.map(endpoint => { + const rewrittenEndpoint = {...endpoint}; + const existingIpAddress = rewrittenEndpoint.ipAddressV4; + + rewrittenEndpoint.ipAddressV4 = encodedIpAddress; + delete rewrittenEndpoint.domainName; + + const endpointChanged = existingIpAddress !== encodedIpAddress; + if (endpointChanged) { + changedEndpointCount += 1; + } + + return rewrittenEndpoint; + }); + + console.log(`Updated ${endpointProperty} for ${serviceName} -> ${clusterIpAddress}`); + return changedEndpointCount; + } +} + +try { + OverrideNetworkGenerator.main(); +} catch (error) { + const message = error instanceof Error ? error.message : String(error); + console.error(message); + process.exitCode = 1; +} diff --git a/examples/state-save-and-restore/scripts/init.sh b/examples/state-save-and-restore/scripts/init.sh deleted file mode 100644 index 7db61b2c89..0000000000 --- a/examples/state-save-and-restore/scripts/init.sh +++ /dev/null @@ -1,92 +0,0 @@ -#!/bin/bash -set -e - -export HEDERA_MIRROR_DATABASE_NAME="mirror_node" -HEDERA_MIRROR_OWNER="$1" -HEDERA_MIRROR_READ="$2" -HEDERA_MIRROR_READ_PASSWORD="$3" - - -export HEDERA_MIRROR_GRPC_DB_HOST="localhost" - -export HEDERA_MIRROR_IMPORTER_DB_HOST="localhost" -export HEDERA_MIRROR_IMPORTER_DB_NAME="${HEDERA_MIRROR_DATABASE_NAME}" -export HEDERA_MIRROR_IMPORTER_DB_OWNER="${HEDERA_MIRROR_OWNER}" -export HEDERA_MIRROR_IMPORTER_DB_SCHEMA="public" -export HEDERA_MIRROR_IMPORTER_DB_TEMPSCHEMA="temporary" - - -PGHBACONF="/opt/bitnami/postgresql/conf/pg_hba.conf" -if [[ -f "${PGHBACONF}" ]]; then - cp "${PGHBACONF}" "${PGHBACONF}.bak" - echo "local all all trust" > "${PGHBACONF}" - pg_ctl reload -fi - -psql -d "user=postgres connect_timeout=3" \ - --set ON_ERROR_STOP=1 \ - --set "dbName=${HEDERA_MIRROR_IMPORTER_DB_NAME}" \ - --set "dbSchema=${HEDERA_MIRROR_IMPORTER_DB_SCHEMA}" \ - --set "ownerUsername=${HEDERA_MIRROR_IMPORTER_DB_OWNER}" \ - --set "tempSchema=${HEDERA_MIRROR_IMPORTER_DB_TEMPSCHEMA}" \ - --set "readUsername=${HEDERA_MIRROR_READ}" \ - --set "readPassword=${HEDERA_MIRROR_READ_PASSWORD}" <<__SQL__ - --- Create database & owner -create database :dbName with owner :ownerUsername; - --- Create roles -create role readonly; -create role readwrite in role readonly; -create role temporary_admin in role readwrite; - --- Create users -alter user :ownerUsername with createrole; - --- Grant temp schema admin privileges -grant temporary_admin to :ownerUsername; - --- Add extensions -\connect :dbName -create extension if not exists btree_gist; -create extension if not exists pg_stat_statements; -create extension if not exists pg_trgm; - --- Create schema -\connect :dbName :ownerUsername -create schema if not exists :dbSchema authorization :ownerUsername; -grant usage on schema :dbSchema to public; -revoke create on schema :dbSchema from public; - --- Create temp table schema -create schema if not exists :tempSchema authorization temporary_admin; -grant usage on schema :tempSchema to public; -revoke create on schema :tempSchema from public; - --- Create readonly user with password and grant privileges -create user :readUsername with password :'readPassword'; -grant readonly to :readUsername; - --- Grant readonly privileges -grant connect on database :dbName to readonly; -grant select on all tables in schema :dbSchema, :tempSchema to readonly; -grant select on all sequences in schema :dbSchema, :tempSchema to readonly; -grant usage on schema :dbSchema, :tempSchema to readonly; -alter default privileges in schema :dbSchema, :tempSchema grant select on tables to readonly; -alter default privileges in schema :dbSchema, :tempSchema grant select on sequences to readonly; - --- Grant readwrite privileges -grant insert, update, delete on all tables in schema :dbSchema to readwrite; -grant usage on all sequences in schema :dbSchema to readwrite; -alter default privileges in schema :dbSchema grant insert, update, delete on tables to readwrite; -alter default privileges in schema :dbSchema grant usage on sequences to readwrite; - --- Alter search path -\connect postgres postgres -alter database :dbName set search_path = :dbSchema, public, :tempSchema; -__SQL__ - -if [[ -f "${PGHBACONF}.bak" ]]; then - mv "${PGHBACONF}.bak" "${PGHBACONF}" - pg_ctl reload -fi diff --git a/resources/cleanup-state-rounds.sh b/resources/cleanup-state-rounds.sh index e3d18933ff..edd6fd64a8 100644 --- a/resources/cleanup-state-rounds.sh +++ b/resources/cleanup-state-rounds.sh @@ -35,10 +35,8 @@ for nodeid in */; do if [ -n "$rounds" ]; then latest_round="" - pces_source_round="" highest_round=$(echo "$rounds" | tail -n 1) highest_round_freeze_state="" - earliest_signed_non_freeze_round="" latest_signed_non_freeze_round="" for round in $rounds; do metadata_file="${round}/stateMetadata.txt" @@ -53,9 +51,6 @@ for nodeid in */; do fi if [ "$freeze_state" = "false" ] && [ -n "$signing_weight" ] && [ "$signing_weight" = "$total_weight" ]; then - if [ -z "$earliest_signed_non_freeze_round" ]; then - earliest_signed_non_freeze_round="$round" - fi latest_signed_non_freeze_round="$round" latest_round="$round" fi @@ -69,83 +64,16 @@ for nodeid in */; do latest_round="$highest_round" fi - if [ -z "$pces_source_round" ]; then - pces_source_round="$latest_round" - fi - round_count=$(echo "$rounds" | wc -l) - echo "Node ${nodeid}${realmShard}: Found ${round_count} rounds, keeping state: ${latest_round}, PCES: ${pces_source_round}" + echo "Node ${nodeid}${realmShard}: Found ${round_count} rounds, keeping state: ${latest_round}" for round in $rounds; do - if [ "$round" != "$latest_round" ] && [ "$round" != "$pces_source_round" ]; then + if [ "$round" != "$latest_round" ]; then echo " Removing old round: $round" rm -rf "$round" fi done - - # Rebuild top-level PCES from the selected pre-freeze round so the event - # creator has the preconsensus events it needs to become active after - # resuming from the kept state. - if [ -d "${pces_source_round}/preconsensus-events" ]; then - top_level_pces="${HEDERA_HAPI_PATH}/data/saved/preconsensus-events" - echo " Rebuilding top-level preconsensus events from round: $pces_source_round" - rm -rf "$top_level_pces" - source_pces_dir="${pces_source_round}/preconsensus-events" - find "$source_pces_dir" -type f -name '*.pces' | while IFS= read -r pces_file; do - relative_pces_path=${pces_file#"$source_pces_dir"/} - pces_node_id=${relative_pces_path%%/*} - pces_filename=${pces_file##*/} - pces_date=${pces_filename%%T*} - - year=$(echo "$pces_date" | cut -d- -f1) - month=$(echo "$pces_date" | cut -d- -f2) - day=$(echo "$pces_date" | cut -d- -f3) - - if [ -n "$pces_node_id" ] && [ -n "$year" ] && [ -n "$month" ] && [ -n "$day" ] && [ "$pces_date" != "$pces_filename" ]; then - pces_destination_dir="${top_level_pces}/${pces_node_id}/${year}/${month}/${day}" - else - pces_destination_dir="${top_level_pces}/${pces_node_id}" - fi - - mkdir -p "$pces_destination_dir" - cp "$pces_file" "$pces_destination_dir/" - done - fi - - if [ -d "${latest_round}/preconsensus-events" ]; then - round_pces_dir="${latest_round}/preconsensus-events" - round_pces_tmp="${latest_round}/preconsensus-events.tmp" - echo " Normalizing round preconsensus events for state: $latest_round" - rm -rf "$round_pces_tmp" - mkdir -p "$round_pces_tmp" - find "$round_pces_dir" -type f -name '*.pces' | while IFS= read -r pces_file; do - relative_pces_path=${pces_file#"$round_pces_dir"/} - pces_node_id=${relative_pces_path%%/*} - pces_filename=${pces_file##*/} - pces_date=${pces_filename%%T*} - - year=$(echo "$pces_date" | cut -d- -f1) - month=$(echo "$pces_date" | cut -d- -f2) - day=$(echo "$pces_date" | cut -d- -f3) - - if [ -n "$pces_node_id" ] && [ -n "$year" ] && [ -n "$month" ] && [ -n "$day" ] && [ "$pces_date" != "$pces_filename" ]; then - pces_destination_dir="${round_pces_tmp}/${pces_node_id}/${year}/${month}/${day}" - else - pces_destination_dir="${round_pces_tmp}/${pces_node_id}" - fi - - mkdir -p "$pces_destination_dir" - cp "$pces_file" "$pces_destination_dir/" - done - rm -rf "$round_pces_dir" - mv "$round_pces_tmp" "$round_pces_dir" - fi - - if [ "$pces_source_round" != "$latest_round" ] && [ -d "$pces_source_round" ]; then - echo " Removing old round after PCES rebuild: $pces_source_round" - rm -rf "$pces_source_round" - fi fi cd ../.. diff --git a/resources/wait-for-stable-saved-state.sh b/resources/wait-for-stable-saved-state.sh new file mode 100644 index 0000000000..1786795036 --- /dev/null +++ b/resources/wait-for-stable-saved-state.sh @@ -0,0 +1,87 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: Apache-2.0 + +set -euo pipefail + +# When called from `consensus state download`, `true` means the remote config +# says the node is frozen. Prefer a fully signed freeze round, but fall back to +# a fully signed non-freeze round when this CN version does not sign freeze +# states (it reports SIGNING_WEIGHT_SUM: 0 for them). +prefer_freeze_round="${1:-false}" +saved_dir="${2:-/opt/hgcapp/services-hedera/HapiApp2.0/data/saved}" + +if command -v sha256sum >/dev/null 2>&1; then + hash_cmd=(sha256sum) +elif command -v shasum >/dev/null 2>&1; then + hash_cmd=(shasum -a 256) +elif command -v openssl >/dev/null 2>&1; then + hash_cmd=(openssl dgst -sha256) +else + echo "No SHA-256 implementation found in container" >&2 + exit 14 +fi + +if [[ ! -d "${saved_dir}" ]]; then + exit 10 +fi + +round_root="$(find "${saved_dir}/com.hedera.services.ServicesMain" -mindepth 2 -maxdepth 2 -type d 2>/dev/null | head -n 1)" +if [[ -z "${round_root}" ]]; then + exit 11 +fi + +# Prefer the newest fully signed freeze round because it is the cleanest +# recovery boundary. For stopped deployments, use the newest fully signed +# non-freeze round instead. +selected_round="" +fallback_round="" +selected_kind="none" +while IFS= read -r round_dir; do + metadata_file="${round_dir}/stateMetadata.txt" + [[ -f "${metadata_file}" ]] || continue + + freeze_state="$(awk -F: '/^FREEZE_STATE:/ {gsub(/^[ \t]+|[ \t]+$/, "", $2); print $2}' "${metadata_file}")" + signing_weight="$(awk -F: '/^SIGNING_WEIGHT_SUM:/ {gsub(/^[ \t]+|[ \t]+$/, "", $2); print $2}' "${metadata_file}")" + total_weight="$(awk -F: '/^TOTAL_WEIGHT:/ {gsub(/^[ \t]+|[ \t]+$/, "", $2); print $2}' "${metadata_file}")" + + if [[ -n "${signing_weight}" && "${signing_weight}" == "${total_weight}" ]]; then + if [[ "${freeze_state}" == "true" ]]; then + selected_round="$(basename "${round_dir}")" + selected_kind="freeze" + continue + fi + + fallback_round="$(basename "${round_dir}")" + fi +done < <( + find "${round_root}" -mindepth 1 -maxdepth 1 -type d -print \ + | while IFS= read -r candidate_dir; do + printf '%s\t%s\n' "$(basename "${candidate_dir}")" "${candidate_dir}" + done \ + | sort -n -k1,1 \ + | cut -f2- +) + +if [[ -z "${selected_round}" ]]; then + selected_round="${fallback_round}" + selected_kind="non-freeze" +fi + +if [[ -z "${selected_round}" ]]; then + exit 12 +fi + +if [[ "${prefer_freeze_round}" == "true" && "${selected_kind}" != "freeze" ]]; then + selected_kind="frozen-fallback" +fi + +# Fingerprint the entire saved-state tree, not just the chosen round directory, +# so the caller can detect when background flushes have stopped changing disk +# contents across consecutive polls. Fingerprint size+mtime per file rather than +# hashing file contents: this poll runs up to 180 times, and re-reading every byte +# under data/saved on each pass adds real I/O load exactly while the node is trying +# to quiesce, for no benefit over the much cheaper metadata comparison. +find "${saved_dir}" -type f -printf '%s %T@ %p\n' \ + | sort \ + | "${hash_cmd[@]}" \ + | awk -v round="${selected_round}" -v kind="${selected_kind}" '{print $1, round, kind}' diff --git a/src/commands/backup-restore.ts b/src/commands/backup-restore.ts index 9041b6d976..1537ff8843 100644 --- a/src/commands/backup-restore.ts +++ b/src/commands/backup-restore.ts @@ -38,6 +38,8 @@ import {RemoteConfig} from '../business/runtime-state/config/remote/remote-confi import {type DeploymentStateSchema} from '../data/schema/model/remote/deployment-state-schema.js'; import {type BlockNodeStateSchema} from '../data/schema/model/remote/state/block-node-state-schema.js'; import {type ConsensusNodeStateSchema} from '../data/schema/model/remote/state/consensus-node-state-schema.js'; +import {DeploymentPhase} from '../data/schema/model/remote/deployment-phase.js'; +import {ComponentTypes} from '../core/config/remote/enumerations/component-types.js'; import {type MirrorNodeStateSchema} from '../data/schema/model/remote/state/mirror-node-state-schema.js'; import {type RelayNodeStateSchema} from '../data/schema/model/remote/state/relay-node-state-schema.js'; import {type DeploymentName} from '../types/index.js'; @@ -416,12 +418,45 @@ export class BackupRestoreCommand extends BaseCommand { title: 'Download Node State Files', task: async (_, task): Promise => { const networkNodes: NetworkNodes = container.resolve(InjectTokens.NetworkNodes); + const nodePhases: Map = new Map(); for (const node of consensusNodes) { const nodeAlias: NodeAlias = node.name; const context: Context = extractContextFromConsensusNodes(nodeAlias, consensusNodes); const clusterReference: string = node.cluster; // Get cluster ref from node metadata const statesDirectory: string = PathEx.join(outputDirectory, 'states', clusterReference); - await networkNodes.getStatesFromPod(namespace, nodeAlias, context, statesDirectory); + const nodeComponent: ConsensusNodeStateSchema = this.remoteConfig.configuration.components.getComponent( + ComponentTypes.ConsensusNode, + Templates.renderComponentIdFromNodeAlias(nodeAlias), + ); + const deploymentPhase: DeploymentPhase = nodeComponent.metadata.phase; + nodePhases.set(nodeAlias, deploymentPhase); + // Passing the phase lets state download prefer a fully signed freeze round over a + // stale non-freeze round when the network was just frozen (see "Freeze network" + // above); otherwise the archive's preconsensus-events stream can outrun the selected + // round and replay straight back into the freeze when the backup is later restored. + await networkNodes.getStatesFromPod(namespace, nodeAlias, context, statesDirectory, deploymentPhase); + } + for (const clusterReference of new Set(consensusNodes.map((node): string => node.cluster))) { + const clusterNodes: ConsensusNode[] = consensusNodes.filter( + (node): boolean => node.cluster === clusterReference, + ); + const statesDirectory: string = PathEx.join(outputDirectory, 'states', clusterReference); + const clusterNodeAliases: NodeAlias[] = clusterNodes.map((node): NodeAlias => node.name); + const allNodesFrozen: boolean = clusterNodeAliases.every( + (nodeAlias: NodeAlias): boolean => nodePhases.get(nodeAlias) === DeploymentPhase.FROZEN, + ); + await networkNodes.normalizeDownloadedStateArchives( + namespace, + clusterNodeAliases, + statesDirectory, + allNodesFrozen ? DeploymentPhase.FROZEN : undefined, + // This backup/restore workflow resumes the restored network as a live, active + // deployment rather than leaving it frozen, so any preconsensus event beyond the + // selected round (e.g. the freeze transaction the network reaches moments later) + // must be trimmed; otherwise replaying it on restart pins the node in + // FREEZE_COMPLETE instead of ACTIVE. + true, + ); } task.title = `Download Node State Files: ${consensusNodes.length} node(s) completed`; }, diff --git a/src/commands/node/handlers.ts b/src/commands/node/handlers.ts index da95f8c6fa..48895bd071 100644 --- a/src/commands/node/handlers.ts +++ b/src/commands/node/handlers.ts @@ -24,7 +24,6 @@ import {type NodeDestroyContext} from './config-interfaces/node-destroy-context. import {type NodeAddContext} from './config-interfaces/node-add-context.js'; import {type NodeUpdateContext} from './config-interfaces/node-update-context.js'; import {type NodeUpgradeContext} from './config-interfaces/node-upgrade-context.js'; -import {type NodeFreezeContext} from './config-interfaces/node-freeze-context.js'; import {ComponentTypes} from '../../core/config/remote/enumerations/component-types.js'; import {DeploymentPhase} from '../../data/schema/model/remote/deployment-phase.js'; import {Templates} from '../../core/templates.js'; @@ -1168,25 +1167,31 @@ export class NodeCommandHandlers extends CommandHandler { public async start(argv: ArgvStruct): Promise { argv = addFlagsToArgv(argv, NodeFlags.START_FLAGS); const leaseWrapper: LeaseWrapper = {lease: undefined}; + const restoringState: boolean = + typeof argv[flags.stateFile.name] === 'string' && argv[flags.stateFile.name].length > 0; - await this.commandAction( - argv, - [ - this.tasks.loadConfiguration(argv, leaseWrapper, this.leaseManager), - this.tasks.initialize( - argv, - this.configs.startConfigBuilder.bind(this.configs), - leaseWrapper.lease, - true, - false, - ), - this.validateAllNodePhases({acceptedPhases: [DeploymentPhase.CONFIGURED]}), - this.tasks.identifyExistingNodes(), - this.tasks.uploadStateFiles(({config}): boolean => config.stateFile.length === 0), - this.tasks.startNodes('nodeAliases'), - // Must precede checkNodesAndProxiesAreActive: when --debug-node-alias is set the JVM starts - // with suspend=y and will never reach ACTIVE until a debugger connects via this port-forward. - this.tasks.enableDebuggerPortForwarding(), + const startTasks: SoloListrTask[] = [ + this.tasks.loadConfiguration(argv, leaseWrapper, this.leaseManager), + this.tasks.initialize(argv, this.configs.startConfigBuilder.bind(this.configs), leaseWrapper.lease, true, false), + this.validateAllNodePhases({acceptedPhases: [DeploymentPhase.CONFIGURED]}), + this.tasks.identifyExistingNodes(), + this.tasks.uploadStateFiles(({config}): boolean => config.stateFile.length === 0), + this.tasks.startNodes('nodeAliases'), + // Must precede checkNodesAndProxiesAreActive: when --debug-node-alias is set the JVM starts + // with suspend=y and will never reach ACTIVE until a debugger connects via this port-forward. + this.tasks.enableDebuggerPortForwarding(), + ]; + + if (restoringState) { + // A freeze-captured archive is expected to restore into FREEZE_COMPLETE. + // Do not require ACTIVE or run ACTIVE-only TSS/start-event tasks for it. + startTasks.push( + this.tasks.checkAllNodesAreFrozen('nodeAliases'), + this.tasks.checkNodeProxiesAreActive(), + this.changeAllNodePhases(DeploymentPhase.FROZEN), + ); + } else { + startTasks.push( this.tasks.checkNodesAndProxiesAreActive('nodeAliases'), this.tasks.enablePortForwarding(true), this.tasks.emitNodeStartedEvent(), @@ -1194,9 +1199,12 @@ export class NodeCommandHandlers extends CommandHandler { this.tasks.setGrpcWebEndpoint('nodeAliases', NodeSubcommandType.START), this.changeAllNodePhases(DeploymentPhase.STARTED, LedgerPhase.INITIALIZED), this.tasks.addNodeStakes(), - // TODO only show this if we are not running in one-shot mode - // this.tasks.showUserMessages(), - ], + ); + } + + await this.commandAction( + argv, + startTasks, constants.LISTR_DEFAULT_OPTIONS.DEFAULT, 'Error starting node', leaseWrapper.lease, @@ -1252,7 +1260,7 @@ export class NodeCommandHandlers extends CommandHandler { this.tasks.identifyExistingNodes(), this.tasks.sendFreezeTransaction(), this.tasks.checkAllNodesAreFrozen('existingNodeAliases'), - this.tasks.drainBlockStreamAfterFreeze(), + this.tasks.waitForFrozenStateToBeStable('existingNodeAliases'), this.tasks.stopNodes('existingNodeAliases'), this.changeAllNodePhases(DeploymentPhase.FROZEN), ], diff --git a/src/commands/node/tasks.ts b/src/commands/node/tasks.ts index 8f674ff3dd..574f4cc488 100644 --- a/src/commands/node/tasks.ts +++ b/src/commands/node/tasks.ts @@ -167,6 +167,7 @@ import {DeploymentStateSchema} from '../../data/schema/model/remote/deployment-s import {type BaseStateSchema} from '../../data/schema/model/remote/state/base-state-schema.js'; import {type BlockNodeStateSchema} from '../../data/schema/model/remote/state/block-node-state-schema.js'; import {ComponentStateMetadataSchema} from '../../data/schema/model/remote/state/component-state-metadata-schema.js'; +import {ConsensusNodeStateSchema} from '../../data/schema/model/remote/state/consensus-node-state-schema.js'; import net from 'node:net'; import {type NodeConnectionsContext} from './config-interfaces/node-connections-context.js'; import {TDirectoryData} from '../../integration/kube/t-directory-data.js'; @@ -2438,6 +2439,34 @@ export class NodeCommandTasks { }; } + public waitForFrozenStateToBeStable(nodeAliasesProperty: string): SoloListrTask { + return { + title: 'Wait for frozen state files to stabilize', + task: (context_, task): SoloListr => { + const nodeAliases: NodeAliases = context_.config[nodeAliasesProperty]; + const subTasks: SoloListrTask[] = nodeAliases.map( + (nodeAlias): SoloListrTask => ({ + title: `Wait for stable frozen state: ${chalk.yellow(nodeAlias)}`, + task: async (): Promise => { + const context: string = extractContextFromConsensusNodes( + nodeAlias, + this.remoteConfig.getConsensusNodes(), + ); + const podReference: PodReference = PodReference.of( + context_.config.namespace, + Templates.renderNetworkPodName(nodeAlias), + ); + await container + .resolve(InjectTokens.NetworkNodes) + .waitForFrozenStateToBeStable(podReference, context); + }, + }), + ); + return task.newListr(subTasks, {concurrent: true, rendererOptions: {collapseSubtasks: false}}); + }, + }; + } + public checkNodeProxiesAreActive(): SoloListrTask { return { title: 'Check node proxies are ACTIVE', @@ -3445,12 +3474,43 @@ export class NodeCommandTasks { return { title: 'Get node states', task: async (context_): Promise => { + const networkNodes: NetworkNodes = container.resolve(InjectTokens.NetworkNodes); + const nodePhases: DeploymentPhase[] = []; for (const nodeAlias of context_.config.nodeAliases) { const context: string = extractContextFromConsensusNodes(nodeAlias, context_.config.consensusNodes); - await container - .resolve(InjectTokens.NetworkNodes) - .getStatesFromPod(context_.config.namespace, nodeAlias, context); + const nodeComponent: ConsensusNodeStateSchema = this.remoteConfig.configuration.components.getComponent( + ComponentTypes.ConsensusNode, + Templates.renderComponentIdFromNodeAlias(nodeAlias), + ); + const deploymentPhase: DeploymentPhase = nodeComponent.metadata.phase; + + if (![DeploymentPhase.FROZEN, DeploymentPhase.STOPPED].includes(deploymentPhase)) { + throw new SoloErrors.validation.illegalArgument( + `Consensus node ${nodeAlias} must be in phase '${DeploymentPhase.FROZEN}' or '${DeploymentPhase.STOPPED}' before downloading saved state.`, + ); + } + + nodePhases.push(deploymentPhase); + await networkNodes.getStatesFromPod( + context_.config.namespace, + nodeAlias, + context, + undefined, + deploymentPhase, + ); } + + // Normalize all downloaded archives together so every node restores from + // the same signed round instead of independently selecting a boundary. + const allNodesFrozen: boolean = nodePhases.every( + (phase: DeploymentPhase): boolean => phase === DeploymentPhase.FROZEN, + ); + await networkNodes.normalizeDownloadedStateArchives( + context_.config.namespace, + context_.config.nodeAliases, + undefined, + allNodesFrozen ? DeploymentPhase.FROZEN : undefined, + ); }, }; } diff --git a/src/core/constants.ts b/src/core/constants.ts index 2d8b99043a..5d2229a731 100644 --- a/src/core/constants.ts +++ b/src/core/constants.ts @@ -513,6 +513,13 @@ export const NETWORK_NODE_GRPC_READINESS_DELAY: number = export const NETWORK_NODE_GRPC_READINESS_REQUIRED_SUCCESSES: number = +getEnvironmentVariable('NETWORK_NODE_GRPC_READINESS_REQUIRED_SUCCESSES') || 3; +// Saved State Stability Checks +export const STATE_DOWNLOAD_STABLE_MAX_ATTEMPTS: number = + +getEnvironmentVariable('STATE_DOWNLOAD_STABLE_MAX_ATTEMPTS') || 180; +export const STATE_DOWNLOAD_STABLE_DELAY: number = +getEnvironmentVariable('STATE_DOWNLOAD_STABLE_DELAY') || 2000; +export const STATE_DOWNLOAD_STABLE_POLLS_REQUIRED: number = + +getEnvironmentVariable('STATE_DOWNLOAD_STABLE_POLLS_REQUIRED') || 3; + export const NETWORK_PROXY_MAX_ATTEMPTS: number = +getEnvironmentVariable('NETWORK_PROXY_MAX_ATTEMPTS') || 300; export const NETWORK_PROXY_DELAY: number = +getEnvironmentVariable('NETWORK_PROXY_DELAY') || 2000; export const PODS_READY_MAX_ATTEMPTS: number = +getEnvironmentVariable('PODS_READY_MAX_ATTEMPTS') || 300; diff --git a/src/core/network-nodes.ts b/src/core/network-nodes.ts index 8d7fd449fe..a63e59a75b 100644 --- a/src/core/network-nodes.ts +++ b/src/core/network-nodes.ts @@ -4,6 +4,7 @@ import {type NamespaceName} from '../types/namespace/namespace-name.js'; import {type PodReference} from '../integration/kube/resources/pod/pod-reference.js'; import {HEDERA_HAPI_PATH, LOG_CONFIG_ZIP_SUFFIX, ROOT_CONTAINER, SOLO_LOGS_DIR} from './constants.js'; import fs from 'node:fs'; +import os from 'node:os'; import {ContainerReference} from '../integration/kube/resources/container/container-reference.js'; import * as constants from './constants.js'; import {sleep} from './helpers.js'; @@ -19,6 +20,10 @@ import {K8} from '../integration/kube/k8.js'; import {Container} from '../integration/kube/resources/container/container.js'; import {NodeStatusEnums} from './enumerations.js'; import chalk from 'chalk'; +import {DeploymentPhase} from '../data/schema/model/remote/deployment-phase.js'; +import {SoloErrors} from './errors/solo-errors.js'; +import {Zippy} from './zippy.js'; +import {PcesTrimmer} from './pces-trimmer.js'; /** * Class to manage network nodes @@ -28,9 +33,11 @@ export class NetworkNodes { public constructor( @inject(InjectTokens.SoloLogger) private readonly logger?: SoloLogger, @inject(InjectTokens.K8Factory) private readonly k8Factory?: K8Factory, + @inject(InjectTokens.Zippy) private readonly zippy?: Zippy, ) { this.logger = patchInject(logger, InjectTokens.SoloLogger, this.constructor.name); this.k8Factory = patchInject(k8Factory, InjectTokens.K8Factory, this.constructor.name); + this.zippy = patchInject(zippy, InjectTokens.Zippy, this.constructor.name); } /** @@ -139,6 +146,7 @@ export class NetworkNodes { nodeAlias: string, context?: string, baseDirectory?: string, + deploymentPhase?: DeploymentPhase, ): Promise { const pods: Pod[] = await this.k8Factory .getK8(context) @@ -149,12 +157,193 @@ export class NetworkNodes { const stateBaseDirectory: string = baseDirectory || SOLO_LOGS_DIR; const promises: Promise[] = []; for (const pod of pods) { - promises.push(this.getState(pod, namespace, stateBaseDirectory, context)); + promises.push(this.getState(pod, namespace, stateBaseDirectory, context, deploymentPhase)); } return await Promise.all(promises); } - private async getState(pod: Pod, namespace: NamespaceName, baseDirectory: string, context?: string): Promise { + /** + * Normalize downloaded state archives to one common signed round. + * + * State downloads contain every round flushed before the archive is created. + * Restore must use one round that exists and is fully signed in every node + * archive; otherwise the nodes can start from different state/PCES boundaries. + */ + public async normalizeDownloadedStateArchives( + namespace: NamespaceName, + nodeAliases: string[], + baseDirectory: string = SOLO_LOGS_DIR, + deploymentPhase?: DeploymentPhase, + trimPreconsensusEventsToSelectedRound: boolean = false, + ): Promise { + const archivePaths: string[] = nodeAliases.map((nodeAlias: string): string => { + const archivePath: string = PathEx.join(baseDirectory, namespace.name, `network-${nodeAlias}-0-state.zip`); + if (!fs.existsSync(archivePath)) { + throw new SoloErrors.validation.illegalArgument(`State file not found: ${archivePath}`); + } + return archivePath; + }); + + const extractedDirectories: string[] = []; + try { + const roundSets: Set[] = []; + const freezeRoundSets: Set[] = []; + + for (const archivePath of archivePaths) { + const extractedDirectory: string = fs.mkdtempSync(PathEx.join(os.tmpdir(), 'solo-state-')); + extractedDirectories.push(extractedDirectory); + this.zippy.unzip(archivePath, extractedDirectory); + + const stateRoot: string = this.findSavedStateRoundRoot(extractedDirectory); + const signedRounds: Set = new Set(); + const freezeRounds: Set = new Set(); + for (const roundDirectory of fs.readdirSync(stateRoot, {withFileTypes: true})) { + if (!roundDirectory.isDirectory() || !/^\d+$/.test(roundDirectory.name)) { + continue; + } + + const metadataPath: string = PathEx.join(stateRoot, roundDirectory.name, 'stateMetadata.txt'); + if (!fs.existsSync(metadataPath)) { + continue; + } + + const metadata: string = fs.readFileSync(metadataPath, 'utf8'); + const signingWeight: string | undefined = this.readStateMetadataValue(metadata, 'SIGNING_WEIGHT_SUM'); + const totalWeight: string | undefined = this.readStateMetadataValue(metadata, 'TOTAL_WEIGHT'); + if (!signingWeight || signingWeight !== totalWeight) { + continue; + } + + signedRounds.add(roundDirectory.name); + if (this.readStateMetadataValue(metadata, 'FREEZE_STATE') === 'true') { + freezeRounds.add(roundDirectory.name); + } + } + + roundSets.push(signedRounds); + freezeRoundSets.push(freezeRounds); + } + + const commonSignedRounds: Set = this.intersectRoundSets(roundSets); + const commonFreezeRounds: Set = this.intersectRoundSets(freezeRoundSets); + const preferFreezeRound: boolean = deploymentPhase === DeploymentPhase.FROZEN; + const selectedRound: string | undefined = this.selectHighestRound( + preferFreezeRound && commonFreezeRounds.size > 0 ? commonFreezeRounds : commonSignedRounds, + ); + + if (!selectedRound) { + throw new SoloErrors.validation.illegalArgument( + `No common fully signed state round found for nodes: ${nodeAliases.join(',')}`, + ); + } + + for (const [index, extractedDirectory] of extractedDirectories.entries()) { + const stateRoot: string = this.findSavedStateRoundRoot(extractedDirectory); + for (const roundDirectory of fs.readdirSync(stateRoot, {withFileTypes: true})) { + if ( + roundDirectory.isDirectory() && + /^\d+$/.test(roundDirectory.name) && + roundDirectory.name !== selectedRound + ) { + fs.rmSync(PathEx.join(stateRoot, roundDirectory.name), {recursive: true, force: true}); + } + } + + // These transient directories are not part of the selected state/PCES boundary. + fs.rmSync(PathEx.join(extractedDirectory, 'saved'), {recursive: true, force: true}); + fs.rmSync(PathEx.join(extractedDirectory, 'swirlds-tmp'), {recursive: true, force: true}); + + if (trimPreconsensusEventsToSelectedRound) { + // The top-level preconsensus-events stream is what the platform replays on the next + // restart, and it is not naturally bounded to the selected round: it can contain later + // events (e.g. a freeze transaction ordered moments after this snapshot was taken) that + // the selected round's own state does not yet reflect. Trim those out so replay cannot + // cross back into that later boundary, while keeping every event up to the selected + // round so the restored node still has other-parent candidates to build new events on + // (removing the whole stream instead would leave every node with no known events at + // all, which the platform only permits at true genesis). Only opted into by callers + // that want the restored network to resume live processing (e.g. `config ops backup`); + // callers that intentionally restore a frozen snapshot rely on that same trailing + // freeze event still being present so the restored node lands back in FREEZE_COMPLETE. + PcesTrimmer.trimDirectoryToBirthRound( + PathEx.join(extractedDirectory, 'preconsensus-events'), + Number(selectedRound), + ); + } + await this.zippy.zip(extractedDirectory, archivePaths[index]); + } + + this.logger.showUser(`Normalized state archives to common signed round ${selectedRound}`); + return selectedRound; + } finally { + for (const extractedDirectory of extractedDirectories) { + fs.rmSync(extractedDirectory, {recursive: true, force: true}); + } + } + } + + private findSavedStateRoundRoot(extractedDirectory: string): string { + const serviceDirectory: string = PathEx.join(extractedDirectory, 'com.hedera.services.ServicesMain'); + const nodeDirectory: string | undefined = fs + .readdirSync(serviceDirectory, {withFileTypes: true}) + .find((entry): boolean => entry.isDirectory())?.name; + const realmShardDirectory: string | undefined = nodeDirectory + ? fs + .readdirSync(PathEx.join(serviceDirectory, nodeDirectory), {withFileTypes: true}) + .find((entry): boolean => entry.isDirectory())?.name + : undefined; + if (!nodeDirectory || !realmShardDirectory) { + throw new SoloErrors.validation.illegalArgument(`Could not locate saved state rounds in ${extractedDirectory}`); + } + return PathEx.join(serviceDirectory, nodeDirectory, realmShardDirectory); + } + + private readStateMetadataValue(metadata: string, key: string): string | undefined { + return metadata + .split('\n') + .find((line: string): boolean => line.startsWith(`${key}:`)) + ?.slice(key.length + 1) + .trim(); + } + + private intersectRoundSets(roundSets: Set[]): Set { + const [firstSet, ...remainingSets] = roundSets; + return new Set( + [...(firstSet ?? [])].filter((round: string): boolean => + remainingSets.every((set: Set): boolean => set.has(round)), + ), + ); + } + + private selectHighestRound(rounds: Set): string | undefined { + let highestRound: string | undefined; + for (const currentRound of rounds) { + if (highestRound === undefined || Number(currentRound) > Number(highestRound)) { + highestRound = currentRound; + } + } + + return highestRound; + } + + /** + * Wait for a fully signed freeze state before a freeze workflow stops the node. + * A FROZEN platform status alone is not enough: stopping immediately can leave + * the archive with only a non-freeze state and misaligned PCES replay data. + */ + public async waitForFrozenStateToBeStable(podReference: PodReference, context?: string): Promise { + const containerReference: ContainerReference = ContainerReference.of(podReference, ROOT_CONTAINER); + const container: Container = this.k8Factory.getK8(context).containers().readByRef(containerReference); + await this.waitForStableSavedState(container, podReference.name.name, false); + } + + private async getState( + pod: Pod, + namespace: NamespaceName, + baseDirectory: string, + context?: string, + deploymentPhase?: DeploymentPhase, + ): Promise { const podReference: PodReference = pod.podReference; this.logger.debug(`getNodeState(${pod.podReference.name.name}): begin...`); const targetDirectory: string = PathEx.join(baseDirectory, namespace.name); @@ -167,6 +356,15 @@ export class NetworkNodes { const k8: K8 = this.k8Factory.getK8(context); const zipFileName: string = `${HEDERA_HAPI_PATH}/${podReference.name}-state.zip`; + // A frozen node should yield a freeze round; a merely stopped node may only + // have a non-freeze signed round available. + const requireFreezeRound: boolean = deploymentPhase === DeploymentPhase.FROZEN; + + await this.waitForStableSavedState( + k8.containers().readByRef(containerReference), + podReference.name.name, + requireFreezeRound, + ); // Zip doesn't have a -C flag like tar, so we use sh -c with subshell to change directory // Use the -X to archive for cross-platform compatibility @@ -187,6 +385,89 @@ export class NetworkNodes { this.logger.debug(`getNodeState(${pod.podReference.name.name}): ...end`); } + private async waitForStableSavedState( + container: Container, + podName: string, + requireFreezeRound: boolean, + ): Promise { + const maxAttempts: number = constants.STATE_DOWNLOAD_STABLE_MAX_ATTEMPTS; + const stablePollsRequired: number = constants.STATE_DOWNLOAD_STABLE_POLLS_REQUIRED; + const pollDelay: Duration = Duration.ofMillis(constants.STATE_DOWNLOAD_STABLE_DELAY); + let lastFingerprint: string | undefined; + let stablePolls: number = 0; + const scriptName: string = 'wait-for-stable-saved-state.sh'; + const sourcePath: string = PathEx.joinWithRealPath(constants.RESOURCES_DIR, scriptName); + const destinationPath: string = `${HEDERA_HAPI_PATH}/${scriptName}`; + + // Reuse a checked-in resource script so the in-pod state-selection logic is + // versioned alongside Solo and remains readable/testable outside TS strings. + await container.copyTo(sourcePath, `${HEDERA_HAPI_PATH}`); + await sleep(Duration.ofSeconds(1)); + await container.execContainer([ + 'bash', + '-c', + `sync ${HEDERA_HAPI_PATH} && chown hedera:hedera ${destinationPath} && chmod 0755 ${destinationPath}`, + ]); + + let attempt: number = 0; + while (attempt < maxAttempts) { + try { + const rawOutput: string = await container.execContainer([ + 'bash', + '-lc', + // The script prints " " + // once it finds the best fully signed saved-state boundary currently + // persisted on disk, preferring a freeze round when requested. + `${destinationPath} ${String(requireFreezeRound)} ${HEDERA_HAPI_PATH}/data/saved`, + ]); + const output: string = rawOutput.trim(); + + const [fingerprint, round, kind] = output.split(/\s+/); + if (fingerprint && round && kind) { + stablePolls = fingerprint === lastFingerprint ? stablePolls + 1 : 1; + lastFingerprint = fingerprint; + + this.logger.debug( + `[state-download] ${podName}: round ${round} (${kind}) stable poll ${stablePolls}/${stablePollsRequired}`, + ); + + if (kind === 'frozen-fallback') { + // A frozen deployment can expose the FROZEN platform status before a + // freeze-marked round becomes fully signed on disk. In that case, + // export the newest fully signed non-freeze round instead of waiting + // indefinitely for a freeze round that may never materialize. + this.logger.warn( + `[state-download] ${podName}: deployment is FROZEN but no fully signed freeze round exists on disk yet; using the newest fully signed non-freeze round`, + ); + } + + if (stablePolls >= stablePollsRequired) { + // One final sync narrows the gap between the successful probe and the + // subsequent zip/copy operation. + await container.execContainer('sync'); + return; + } + } + } catch (error) { + // The script exits non-zero until a qualifying signed round exists or the + // saved-state tree stops changing across polls. + this.logger.debug(`[state-download] ${podName}: saved state not stable yet`, error); + } + + attempt++; + await sleep(pollDelay); + } + + throw new SoloErrors.component.nodeNotReady( + podName, + requireFreezeRound + ? 'showing a stable, fully signed freeze state on disk (the deployment is frozen, but no signed round has become stable yet)' + : 'showing a stable, fully signed saved state on disk (stop or freeze the node and retry state download)', + attempt, + maxAttempts, + ); + } + public async getNetworkNodePodStatus(podReference: PodReference, context?: string): Promise { return this.k8Factory .getK8(context) diff --git a/src/core/pces-trimmer.ts b/src/core/pces-trimmer.ts new file mode 100644 index 0000000000..628464c311 --- /dev/null +++ b/src/core/pces-trimmer.ts @@ -0,0 +1,222 @@ +// SPDX-License-Identifier: Apache-2.0 + +import fs from 'node:fs'; +import path from 'node:path'; +import {SoloErrors} from './errors/solo-errors.js'; + +const PCES_PROTOBUF_EVENTS_VERSION: number = 2; +const GOSSIP_EVENT_CORE_FIELD_NUMBER: number = 1; +const EVENT_CORE_BIRTH_ROUND_FIELD_NUMBER: number = 2; +const WIRE_TYPE_VARINT: number = 0; +const WIRE_TYPE_FIXED64: number = 1; +const WIRE_TYPE_LENGTH_DELIMITED: number = 2; +const WIRE_TYPE_FIXED32: number = 5; + +interface VarintRead { + value: bigint; + nextOffset: number; +} + +/** + * Trims preconsensus event stream (PCES) files down to a maximum birth round. + * + * A restored state snapshot at round N is already a complete, fully signed state; any + * preconsensus event with a birth round greater than N was created after that snapshot and + * should not be replayed when the snapshot is restored elsewhere (for example, replaying a + * freeze transaction ordered moments after the snapshot was taken would immediately re-freeze + * the restored node). This trims strictly at a record boundary, so no footer/checksum needs to + * be regenerated (see the PCES file format used by hiero-consensus-node's PcesFileIterator, + * which already tolerates a file ending mid-record the same way it tolerates an abrupt + * shutdown). + */ +export class PcesTrimmer { + /** + * Removes every preconsensus event with a birth round greater than maximumBirthRound from + * all .pces files found recursively under directory. A file entirely beyond the cutoff is + * deleted; a file straddling the cutoff is truncated at the first excluded record's start. + */ + public static trimDirectoryToBirthRound(directory: string, maximumBirthRound: number): void { + if (!fs.existsSync(directory)) { + return; + } + + for (const pcesFilePath of PcesTrimmer.findPcesFilesInSequenceOrder(directory)) { + PcesTrimmer.trimFileToBirthRound(pcesFilePath, maximumBirthRound); + } + } + + private static findPcesFilesInSequenceOrder(directory: string): string[] { + const filePaths: string[] = []; + PcesTrimmer.collectPcesFiles(directory, filePaths); + + // Sequence numbers are encoded as "_seqN_" in the filename and are contiguous per node; + // events must be inspected in that order regardless of how the date subdirectories nest. + filePaths.sort( + (left: string, right: string): number => + PcesTrimmer.extractSequenceNumber(left) - PcesTrimmer.extractSequenceNumber(right), + ); + return filePaths; + } + + private static collectPcesFiles(directory: string, filePaths: string[]): void { + for (const entry of fs.readdirSync(directory, {withFileTypes: true})) { + const entryPath: string = path.join(directory, entry.name); + if (entry.isDirectory()) { + PcesTrimmer.collectPcesFiles(entryPath, filePaths); + } else if (entry.isFile() && entry.name.endsWith('.pces')) { + filePaths.push(entryPath); + } + } + } + + private static extractSequenceNumber(pcesFilePath: string): number { + const match: RegExpMatchArray | null = path.basename(pcesFilePath).match(/_seq(\d+)_/); + return match ? Number(match[1]) : 0; + } + + private static trimFileToBirthRound(pcesFilePath: string, maximumBirthRound: number): void { + try { + const fileBuffer: Buffer = fs.readFileSync(pcesFilePath); + const truncateAtOffset: number | undefined = PcesTrimmer.findTruncationOffset(fileBuffer, maximumBirthRound); + + if (truncateAtOffset === undefined) { + return; + } + + if (truncateAtOffset <= 4) { + fs.rmSync(pcesFilePath, {force: true}); + return; + } + + fs.writeFileSync(pcesFilePath, fileBuffer.subarray(0, truncateAtOffset)); + } catch { + // Best-effort trim: an unexpected or unrecognized PCES layout is left untouched rather + // than risking corruption of a file the platform still needs to read on next startup. + } + } + + /** + * Returns the byte offset of the first record whose birth round exceeds maximumBirthRound, + * or undefined if the file does not need trimming (including when its header is not the + * expected version, in which case it is left untouched rather than guessed at). + */ + private static findTruncationOffset(fileBuffer: Buffer, maximumBirthRound: number): number | undefined { + if (fileBuffer.length < 4 || fileBuffer.readInt32BE(0) !== PCES_PROTOBUF_EVENTS_VERSION) { + return undefined; + } + + let recordOffset: number = 4; + while (recordOffset + 4 <= fileBuffer.length) { + const recordLength: number = fileBuffer.readInt32BE(recordOffset); + const recordStart: number = recordOffset + 4; + if (recordLength < 0 || recordStart + recordLength > fileBuffer.length) { + // A partial trailing record; nothing further to inspect. + return undefined; + } + + const recordBytes: Buffer = fileBuffer.subarray(recordStart, recordStart + recordLength); + const birthRound: number | undefined = PcesTrimmer.readGossipEventBirthRound(recordBytes); + if (birthRound !== undefined && birthRound > maximumBirthRound) { + return recordOffset; + } + + recordOffset = recordStart + recordLength; + } + + return undefined; + } + + /** + * Reads GossipEvent.event_core.birth_round (field 1, then field 2 within it) directly from + * the protobuf wire format. Only this one field is needed to decide whether an event belongs + * before or after the restore boundary, so a full protobuf runtime is not required. + */ + private static readGossipEventBirthRound(gossipEventBytes: Buffer): number | undefined { + const eventCoreBytes: Buffer | undefined = PcesTrimmer.readLengthDelimitedField( + gossipEventBytes, + GOSSIP_EVENT_CORE_FIELD_NUMBER, + ); + if (!eventCoreBytes) { + return undefined; + } + + return PcesTrimmer.readVarintField(eventCoreBytes, EVENT_CORE_BIRTH_ROUND_FIELD_NUMBER); + } + + private static readLengthDelimitedField(messageBytes: Buffer, targetFieldNumber: number): Buffer | undefined { + let offset: number = 0; + while (offset < messageBytes.length) { + const tagRead: VarintRead = PcesTrimmer.readVarint(messageBytes, offset); + const fieldNumber: number = Number(tagRead.value >> 3n); + const wireType: number = Number(tagRead.value & 0x7n); + offset = tagRead.nextOffset; + + if (fieldNumber === targetFieldNumber && wireType === WIRE_TYPE_LENGTH_DELIMITED) { + const lengthRead: VarintRead = PcesTrimmer.readVarint(messageBytes, offset); + const fieldStart: number = lengthRead.nextOffset; + return messageBytes.subarray(fieldStart, fieldStart + Number(lengthRead.value)); + } + + offset = PcesTrimmer.skipField(messageBytes, offset, wireType); + } + + return undefined; + } + + private static readVarintField(messageBytes: Buffer, targetFieldNumber: number): number | undefined { + let offset: number = 0; + while (offset < messageBytes.length) { + const tagRead: VarintRead = PcesTrimmer.readVarint(messageBytes, offset); + const fieldNumber: number = Number(tagRead.value >> 3n); + const wireType: number = Number(tagRead.value & 0x7n); + offset = tagRead.nextOffset; + + if (fieldNumber === targetFieldNumber && wireType === WIRE_TYPE_VARINT) { + return Number(PcesTrimmer.readVarint(messageBytes, offset).value); + } + + offset = PcesTrimmer.skipField(messageBytes, offset, wireType); + } + + return undefined; + } + + private static skipField(messageBytes: Buffer, offset: number, wireType: number): number { + switch (wireType) { + case WIRE_TYPE_VARINT: { + return PcesTrimmer.readVarint(messageBytes, offset).nextOffset; + } + case WIRE_TYPE_FIXED64: { + return offset + 8; + } + case WIRE_TYPE_LENGTH_DELIMITED: { + const lengthRead: VarintRead = PcesTrimmer.readVarint(messageBytes, offset); + return lengthRead.nextOffset + Number(lengthRead.value); + } + case WIRE_TYPE_FIXED32: { + return offset + 4; + } + default: { + throw new SoloErrors.internal.dataValidation('PCES protobuf wire type', '0, 1, 2, or 5', wireType); + } + } + } + + private static readVarint(buffer: Buffer, offset: number): VarintRead { + let result: bigint = 0n; + let shift: bigint = 0n; + let position: number = offset; + + while (position < buffer.length) { + const currentByte: number = buffer[position]; + position++; + result |= BigInt(currentByte & 0b0111_1111) << shift; + if ((currentByte & 0b1000_0000) === 0) { + return {value: result, nextOffset: position}; + } + shift += 7n; + } + + throw new SoloErrors.internal.dataValidation('PCES varint', 'a terminated varint', 'truncated bytes'); + } +} diff --git a/test/unit/core/network-nodes.test.ts b/test/unit/core/network-nodes.test.ts index 7a5ffcc238..aa085ddb89 100644 --- a/test/unit/core/network-nodes.test.ts +++ b/test/unit/core/network-nodes.test.ts @@ -3,6 +3,10 @@ import {expect} from 'chai'; import {afterEach, beforeEach, describe, it} from 'mocha'; import sinon from 'sinon'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import AdmZip from 'adm-zip'; import {container} from 'tsyringe-neo'; import {resetForTest} from '../../test-container.js'; @@ -11,6 +15,7 @@ import {type NetworkNodes} from '../../../src/core/network-nodes.js'; import {PodReference} from '../../../src/integration/kube/resources/pod/pod-reference.js'; import {PodName} from '../../../src/integration/kube/resources/pod/pod-name.js'; import {NamespaceName} from '../../../src/types/namespace/namespace-name.js'; +import {DeploymentPhase} from '../../../src/data/schema/model/remote/deployment-phase.js'; describe('NetworkNodes', (): void => { let networkNodes: NetworkNodes; @@ -44,4 +49,56 @@ describe('NetworkNodes', (): void => { const status: string = await networkNodes.getNetworkNodePlatformStatusName(podReference); expect(status).to.equal('UNKNOWN'); }); + + it('should normalize node archives to one common signed round', async (): Promise => { + const temporaryDirectory: string = fs.mkdtempSync(path.join(os.tmpdir(), 'network-nodes-test-')); + const namespaceDirectory: string = path.join(temporaryDirectory, 'namespace'); + fs.mkdirSync(namespaceDirectory, {recursive: true}); + + try { + for (const nodeAlias of ['node1', 'node2']) { + const sourceDirectory: string = fs.mkdtempSync(path.join(os.tmpdir(), 'state-source-')); + for (const [round, freezeState] of [ + ['100', 'false'], + ['200', 'true'], + ]) { + const roundDirectory: string = path.join( + sourceDirectory, + 'com.hedera.services.ServicesMain', + '0', + '123', + round, + ); + fs.mkdirSync(roundDirectory, {recursive: true}); + fs.writeFileSync( + path.join(roundDirectory, 'stateMetadata.txt'), + `FREEZE_STATE: ${freezeState}\nSIGNING_WEIGHT_SUM: 3\nTOTAL_WEIGHT: 3\n`, + ); + fs.writeFileSync(path.join(roundDirectory, 'preconsensus-events.pces'), 'pces'); + } + + const archive: AdmZip = new AdmZip(); + archive.addLocalFolder(sourceDirectory); + await archive.writeZipPromise(path.join(namespaceDirectory, `network-${nodeAlias}-0-state.zip`)); + fs.rmSync(sourceDirectory, {recursive: true, force: true}); + } + + await networkNodes.normalizeDownloadedStateArchives( + NamespaceName.of('namespace'), + ['node1', 'node2'], + temporaryDirectory, + DeploymentPhase.FROZEN, + ); + + for (const nodeAlias of ['node1', 'node2']) { + const archive: AdmZip = new AdmZip(path.join(namespaceDirectory, `network-${nodeAlias}-0-state.zip`)); + const entries: string[] = archive.getEntries().map((entry): string => entry.entryName); + expect(entries.some((entry: string): boolean => entry.includes('/100/'))).to.equal(false); + expect(entries.some((entry: string): boolean => entry.includes('/200/'))).to.equal(true); + expect(entries.some((entry: string): boolean => entry.endsWith('preconsensus-events.pces'))).to.equal(true); + } + } finally { + fs.rmSync(temporaryDirectory, {recursive: true, force: true}); + } + }); }); diff --git a/test/unit/core/pces-trimmer.test.ts b/test/unit/core/pces-trimmer.test.ts new file mode 100644 index 0000000000..343510e557 --- /dev/null +++ b/test/unit/core/pces-trimmer.test.ts @@ -0,0 +1,162 @@ +// SPDX-License-Identifier: Apache-2.0 + +import {expect} from 'chai'; +import {afterEach, beforeEach, describe, it} from 'mocha'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; + +import {PcesTrimmer} from '../../../src/core/pces-trimmer.js'; +import {PathEx} from '../../../src/business/utils/path-ex.js'; + +const PROTOBUF_EVENTS_VERSION: number = 2; +// Protobuf field tags: (field_number << 3) | wire_type. EventCore.birth_round is field 2, +// wire type 0 (varint); GossipEvent.event_core is field 1, wire type 2 (length-delimited). +const EVENT_CORE_BIRTH_ROUND_TAG: number = 16; +const GOSSIP_EVENT_EVENT_CORE_TAG: number = 10; + +function encodeVarint(value: number): Buffer { + const bytes: number[] = []; + let remaining: number = value; + while (remaining > 0b0111_1111) { + bytes.push((remaining & 0b0111_1111) | 0b1000_0000); + remaining >>>= 7; + } + bytes.push(remaining); + return Buffer.from(bytes); +} + +/** Builds a minimal GossipEvent protobuf payload carrying only event_core.birth_round. */ +function buildGossipEventBytes(birthRound: number): Buffer { + const birthRoundVarint: Buffer = encodeVarint(birthRound); + const eventCoreBytes: Buffer = Buffer.concat([Buffer.from([EVENT_CORE_BIRTH_ROUND_TAG]), birthRoundVarint]); + return Buffer.concat([ + Buffer.from([GOSSIP_EVENT_EVENT_CORE_TAG]), + encodeVarint(eventCoreBytes.length), + eventCoreBytes, + ]); +} + +/** Builds a well-formed PCES file (header + length-prefixed records) for the given birth rounds. */ +function buildPcesFileBuffer(birthRounds: number[], version: number = PROTOBUF_EVENTS_VERSION): Buffer { + const header: Buffer = Buffer.alloc(4); + header.writeInt32BE(version, 0); + + const records: Buffer[] = birthRounds.map((birthRound: number): Buffer => { + const gossipEventBytes: Buffer = buildGossipEventBytes(birthRound); + const lengthPrefix: Buffer = Buffer.alloc(4); + lengthPrefix.writeInt32BE(gossipEventBytes.length, 0); + return Buffer.concat([lengthPrefix, gossipEventBytes]); + }); + + return Buffer.concat([header, ...records]); +} + +describe('PcesTrimmer', (): void => { + let temporaryDirectory: string; + + beforeEach((): void => { + temporaryDirectory = fs.mkdtempSync(PathEx.join(os.tmpdir(), 'pces-trimmer-')); + }); + + afterEach((): void => { + fs.rmSync(temporaryDirectory, {recursive: true, force: true}); + }); + + function writePcesFile(fileName: string, buffer: Buffer): string { + const filePath: string = path.join(temporaryDirectory, fileName); + fs.writeFileSync(filePath, buffer); + return filePath; + } + + it('leaves a file unchanged when every event is at or below the cutoff round', (): void => { + const originalBuffer: Buffer = buildPcesFileBuffer([10, 20, 30]); + const filePath: string = writePcesFile('a_seq0_minr1_maxr30_orgn0.pces', originalBuffer); + + PcesTrimmer.trimDirectoryToBirthRound(temporaryDirectory, 30); + + expect(fs.readFileSync(filePath)).to.deep.equal(originalBuffer); + }); + + it('leaves a single-record file unchanged when the record ends exactly at the end of the file', (): void => { + const originalBuffer: Buffer = buildPcesFileBuffer([5]); + const filePath: string = writePcesFile('a_seq0_minr1_maxr5_orgn0.pces', originalBuffer); + + PcesTrimmer.trimDirectoryToBirthRound(temporaryDirectory, 5); + + expect(fs.readFileSync(filePath)).to.deep.equal(originalBuffer); + }); + + it('truncates a file at the first record whose birth round exceeds the cutoff', (): void => { + const filePath: string = writePcesFile('a_seq0_minr1_maxr40_orgn0.pces', buildPcesFileBuffer([10, 20, 30, 40])); + + PcesTrimmer.trimDirectoryToBirthRound(temporaryDirectory, 25); + + expect(fs.readFileSync(filePath)).to.deep.equal(buildPcesFileBuffer([10, 20])); + }); + + it('correctly parses multi-byte varints when truncating past a birth round above 127', (): void => { + const filePath: string = writePcesFile( + 'a_seq0_minr1_maxr500_orgn0.pces', + buildPcesFileBuffer([100, 200, 300, 400]), + ); + + PcesTrimmer.trimDirectoryToBirthRound(temporaryDirectory, 250); + + expect(fs.readFileSync(filePath)).to.deep.equal(buildPcesFileBuffer([100, 200])); + }); + + it('deletes a file entirely when even its first record exceeds the cutoff', (): void => { + const filePath: string = writePcesFile('a_seq0_minr50_maxr60_orgn0.pces', buildPcesFileBuffer([50, 60])); + + PcesTrimmer.trimDirectoryToBirthRound(temporaryDirectory, 10); + + expect(fs.existsSync(filePath)).to.be.false; + }); + + it('leaves a file untouched when its version header is not the recognized PROTOBUF_EVENTS version', (): void => { + const originalBuffer: Buffer = buildPcesFileBuffer([10, 999], 99); + const filePath: string = writePcesFile('a_seq0_minr1_maxr999_orgn0.pces', originalBuffer); + + PcesTrimmer.trimDirectoryToBirthRound(temporaryDirectory, 10); + + expect(fs.readFileSync(filePath)).to.deep.equal(originalBuffer); + }); + + it('trims each file in a nested directory tree independently, in sequence order', (): void => { + const nestedDirectory: string = path.join(temporaryDirectory, '2026', '08', '17'); + fs.mkdirSync(nestedDirectory, {recursive: true}); + + const belowCutoffBuffer: Buffer = buildPcesFileBuffer([1, 2]); + const belowCutoffPath: string = path.join(nestedDirectory, 'a_seq0_minr1_maxr2_orgn0.pces'); + fs.writeFileSync(belowCutoffPath, belowCutoffBuffer); + + const straddlingPath: string = path.join(nestedDirectory, 'b_seq1_minr2_maxr50_orgn0.pces'); + fs.writeFileSync(straddlingPath, buildPcesFileBuffer([2, 3, 40])); + + PcesTrimmer.trimDirectoryToBirthRound(temporaryDirectory, 3); + + expect(fs.readFileSync(belowCutoffPath)).to.deep.equal(belowCutoffBuffer); + expect(fs.readFileSync(straddlingPath)).to.deep.equal(buildPcesFileBuffer([2, 3])); + }); + + it('does nothing when the directory does not exist', (): void => { + const missingDirectory: string = path.join(temporaryDirectory, 'does-not-exist'); + + expect((): void => PcesTrimmer.trimDirectoryToBirthRound(missingDirectory, 10)).to.not.throw(); + }); + + it('leaves a file untouched when a record is malformed instead of risking an incorrect truncation', (): void => { + const header: Buffer = Buffer.alloc(4); + header.writeInt32BE(PROTOBUF_EVENTS_VERSION, 0); + // A length prefix declaring more bytes than actually follow it. + const corruptLengthPrefix: Buffer = Buffer.alloc(4); + corruptLengthPrefix.writeInt32BE(1000, 0); + const originalBuffer: Buffer = Buffer.concat([header, corruptLengthPrefix, Buffer.from([1, 2, 3])]); + const filePath: string = writePcesFile('a_seq0_minr1_maxr10_orgn0.pces', originalBuffer); + + PcesTrimmer.trimDirectoryToBirthRound(temporaryDirectory, 10); + + expect(fs.readFileSync(filePath)).to.deep.equal(originalBuffer); + }); +});