snapshot api added by michalkucharczyk · Pull Request #545 · paritytech/zombienet-sdk

michalkucharczyk · 2026-05-27T15:31:33Z

Implementation of proposal from #541 (also #542).

Tests that build DB snapshot artifacts (cumulus full_node_warp_sync, smoldot smoke/bulletin) each hand-roll the same pause → tar → checksum → bundle dance. This moves it into the SDK.

What's new

NetworkNode::snapshot_db(out_path) — tars a node's DB into a .tgz (data/, plus relay-data/ for cumulus collators). Drops keystore/ and network/ so the archive can be loaded on several sibling nodes without key/peer-id clashes.
Network::pause() / resume() — SIGSTOP/SIGCONT all nodes in parallel, so the DB on disk is consistent while you snapshot.
snapshot::BundleBuilder — packs per-node archives + a JSON user_data blob into one bundle.tar.gz with a manifest.json (sha256 + size per archive). build() won't compile until you've added at least one archive.
with_optional_default_db_snapshot(Option<…>) on the relay/parachain builders (and with_optional_db_snapshot per node) — so one network builder works for both the fresh and the from-snapshot case without branching.

Example

// Produce: snapshot a relay validator + a collator, bundle them.
network.pause().await?;
let relay = network.get_node("alice")?.snapshot_db("relay.tgz").await?;
let para  = network.get_node("collator")?.snapshot_db("para.tgz").await?;
network.resume().await?;

let bundle = BundleBuilder::new()
    .add(relay)
    .add(para)
    .user_data(serde_json::json!({ "para_best_block": 42 }))
    .build("bundle.tar.gz")?;          // bundle.path / .sha256 / .size

////////////////////////////////////////////////////////////////////////////////

// Consume: same builder, snapshot optional.
fn network(snapshot: Option<&Path>) -> NetworkConfig {
    NetworkConfigBuilder::new()
        .with_relaychain(|r| {
            r.with_chain("rococo-local")
                .with_default_command("polkadot")
                .with_optional_default_db_snapshot(snapshot)   // None = fresh, Some = resume
                .with_validator(|n| n.with_name("alice"))
        })
        /* … */
        .build().unwrap()
}

Tests

Unit tests in crates/sdk/src/snapshot.rs: bundle build → unpack → manifest round-trip (no network, runs in CI).
crates/configuration: tests for the three with_optional_*db_snapshot methods.
crates/sdk/tests/snapshot_roundtrip.rs (#[ignore], needs polkadot binaries): snapshot a live network, bundle it, re-spawn from the snapshot, and check both chains advance and finalize past the snapshot height.

Notes

snapshot_db reads the node's local base dir, so it's native-provider only.
Caller must pause the node first — snapshoting a running node risks a torn RocksDB state.
A graceful terminate() (instead of pause) before snapshotting is left for a follow-up.
Open question if SIGSTOP allows for safe db copy. So far it worked well.

pepoviola · 2026-05-28T08:41:32Z

+        &self,
+        out_path: impl AsRef<Path>,
+    ) -> Result<NodeSnapshot, anyhow::Error> {
+        let out_path = out_path.as_ref().to_path_buf();


This logic should be in the native provider and use inner from here to call it. We can let k8s/docker as unimplemented for now (but is also possible to implement).

Thx!

pepoviola · 2026-05-28T08:48:07Z

+    }
+}
+
+fn build_bundle(


Just nit, since we provide a nice way to generate the bundle maybe we can also include the untar_bundle fn here to not allow users to use it directly and not need to re-implement.
wdyt?

pepoviola · 2026-05-28T08:49:09Z

Awesome job @michalkucharczyk !! I added a couple of comments, if you are agree I can make those changes and also wire your test to run in CI.

Thx!

michalkucharczyk · 2026-05-28T08:54:46Z

Awesome job @michalkucharczyk !! I added a couple of comments, if you are agree I can make those changes and also wire your test to run in CI.

Thx!

Yes, feel free to take it from here. But I can also implement your requests. Whatever works for you :)

skunert

Nice!

skunert · 2026-05-28T09:16:47Z

+            ));
+        }
+
+        let bytes = tokio::fs::read(&out_path)


We should not read t he entire file here and hash it, but do this iteratively to save some memory.

pepoviola · 2026-05-29T07:22:28Z

ping @michalkucharczyk, I applied the changes and wire the test to run in ci. If you are ok, I will merge it and draft a new release.
Thx!

michalkucharczyk · 2026-05-29T09:41:33Z

+            ));
+        }
+
+        let mut file = File::open(&out_path).unwrap();


Suggested change

let mut file = File::open(&out_path).unwrap();

let mut file = File::open(&out_path).map_err(|err| ProviderError::SnapshotDb(self.name().into(), anyhow!(err)))?;

ahh, let me remove the unwrap 🤦‍♂️

michalkucharczyk · 2026-05-29T09:42:45Z

+    let bytes = std::fs::read(&out_path)
+        .with_context(|| format!("reading produced bundle {}", out_path.display()))?;
+    let size = bytes.len() as u64;
+    let sha256 = hex::encode(sha2::Sha256::digest(&bytes));


stream could be used here too (not reading entire file to memory).

yes, I will apply the same logic that in the node

michalkucharczyk · 2026-05-29T09:43:55Z

+    Ok(())
+}
+
+fn untar_bundle(bundle_path: &Path, out_dir: &Path) -> anyhow::Result<()> {


nit: maybe this could be used in tests?

You mean moving to the snapshot module?

michalkucharczyk · 2026-05-29T09:45:03Z

Nit: maybe we could keep NetworkNode::base_dir() ? (e.g. for inspection during test authoring?)

michalkucharczyk · 2026-05-29T09:51:07Z

+            size,
+        } = self.inner.snapshot_db(is_cumulus_based).await?;
+
+        // now we need to _move_ the inner file to the out_path


nit: It acutally copies the file. Maybe this could be avoided by tzg'ing the file to the destination location? Or moving instead copying?

Yes, could be an optimization for native but the idea here is that this method can be used in all the supported providers, so in docker/k8s we need to get the file from the pod.

Thx!

snapshot api added

07a7c64

michalkucharczyk requested a review from pepoviola May 27, 2026 15:39

fmt

ff34a1d

pepoviola reviewed May 28, 2026

View reviewed changes

skunert reviewed May 28, 2026

View reviewed changes

pepoviola added 4 commits May 28, 2026 20:51

changes from feedback

d210eac

changes to fix ci

fcbf148

fix ci, add ignored

0d77b85

remove debug

8ec4333

michalkucharczyk commented May 29, 2026

View reviewed changes

changes from feedback

58e35c5

pepoviola merged commit c05fea8 into main May 29, 2026
13 checks passed

pepoviola deleted the mku-snapshot-api-dev branch May 29, 2026 10:43

	let mut file = File::open(&out_path).unwrap();
	let mut file = File::open(&out_path).map_err(\|err\| ProviderError::SnapshotDb(self.name().into(), anyhow!(err)))?;

Uh oh!

Conversation

michalkucharczyk commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's new

Example

Tests

Notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pepoviola commented May 28, 2026

Uh oh!

michalkucharczyk commented May 28, 2026

Uh oh!

skunert left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pepoviola commented May 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michalkucharczyk commented May 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

michalkucharczyk commented May 27, 2026 •

edited

Loading