Archival snapshot restore: non-canonical AppHash executing head block (bank/oracle/wasm diverge) on v2.1.0

[cmartins88]

Summary

A full/archive node restored from the official archival snapshot (snapshots.akash.network/akashnet-2-archival/) deterministically computes a non-canonical AppHash when it executes the first block after the snapshot head.
The node then gets stuck: blocksync rejects every peer with wrong Block.Header.AppHash.

We isolated it with module-hash-by-height: the snapshot's pre-state matches canonical on all modules, and after executing exactly one block only 3 modules diverge -> bank, oracle, wasm. Same binary (md5 identical to the official docker image), same pre-state, same block, different result.

This is not specific to one snapshot generation: we saw the same pattern with an earlier archival snapshot too (its head block also diverged on execution).

Environment

• akash v2.1.0, commit 151b989a58f579ac08b688b325e0dfcc4b44fecb (binary md5 c660578e1dbc7f874fe17a6a2f64cdbc, byte-identical between the .deb and the binary inside ghcr.io/akash-network/node:2.1.0)
• cosmovisor v1.7.1, upgrades ladder v2.0.0 (patched with v2.0.1) and v2.1.0
• Ubuntu 24.04.4, glibc 2.39, x86-64 (Xeon E5 v3 / Haswell, KVM guest)
• goleveldb backend, pruning = "nothing" (archive), tx indexer kv
• Snapshot: akashnet-2_27763931.tar.lz4 (Last-Modified 2026-07-18)
• Restored via ranged HTTP + lz4 -dc | tar -x (reproduced across 3 independent downloads/restores; goleveldb CRCs clean, so transfer corruption is ruled out)

Steps to reproduce

1. Download and extract the archival snapshot akashnet-2_27763931.tar.lz4.
2. Start the node with the official v2.1.0 binary (via cosmovisor, upgrade v2.1.0).
3. Node opens the DB at height 27763931, then executes block 27763932.

Result

wrong Block.Header.AppHash.
  Expected 0AD5D4F5A8CCBC2EBB32E1EFF55820629944E2F5E576F1727AAA17FA3D18B408
  got      17FE2159019F9DCCF755E93B7486EACDE7825EF1E11DA7601B12BA3B22F2A58D

(Expected is our locally-computed AppHash; got is the canonical value from the next block's header.) Node is stuck at 27763932; blocksync rejects all peers.

Root-cause isolation (module-hash-by-height)

We compared per-module store hashes between our stuck node and one of our own in-cluster pruned nodes that followed consensus through this height on the same hardware (official ghcr.io/akash-network/node:2.1.0 image).

At height 27763931 (snapshot head, before executing any block) all 26 modules match byte-for-byte. The snapshot state is correct.

At height 27763932 (after executing that one block) exactly 3 modules diverge:

module   our node (stuck)                                                  canonical (in-consensus node)
bank     2B30B1944CA45CA73465D82305523D64E3BDE620C882253F78949D13CC253A1D  601E4DCC76F70E2B8DB10C5F9156073F139EF24ACE8FCACD556A83A3DC537018
oracle   35370C06FDF0304E56D45310D168A23F21B8A882B2691109DD6A802654F3224E  ECE921A5CE415AFFBAFA3AB6B7445D3B4FF10C1D755E61AE8E9BAE54238C36B5
wasm     92A37A3F69445AF12EC3311E74857CB3E259C18B1E2CA6CD29D1EECCA9219772  FD5F942EC8EF69E24D2645303FBE9B009AFBFBE62C95046F8B5080062F857AB7

The other 23 modules match at 27763932 as well.

What we ruled out

• Binary provenance: the .deb binary and the binary inside the official docker image are byte-identical (same md5), same commit.
• IAVL fastnode: reproduced with iavl-disable-fastnode = false and = true (state rolled back between attempts). Identical non-canonical hash both times, so it is fully deterministic locally and config-independent.
• Peer/network: the block data is fine; it is our execution result that differs. Our in-cluster pruned nodes on the same hardware synced through this height without any issue.
• Transfer corruption: 3 independent restores, clean CRCs.

Hypothesis

The bank + oracle + wasm triad points at the BME oracle-price -> mint/burn -> contract path. Since the committed pre-state is provably identical and the binary is identical, the divergence looks like execution that depends on something not carried in the committed state a snapshot restores. For example ABCI++ vote-extension / extended-commit context from the previous block, or a replay-vs-consensus code path difference when the block is applied via blocksync from a snapshot head rather than in live consensus. This would explain why the head block of every archival snapshot diverges while a node that built that context live (validators, pruned nodes) processes the same block fine.

Questions

1. Is this a known determinism issue when restoring the archival snapshots (or more generally when executing a block from snapshot-loaded state)?
2. How is the official archival node configured (app.toml/config.toml deltas, env vars, patches)? Which exact binary artifact does it run?
3. Is there a patched build or a flag that lets a restored archival node execute its head block correctly?

We can share full logs, the four module-hash-by-height dumps, or run any diagnostic build against the pre-execution state (we snapshotted it, so retry is fast).

[Zblocker64]

Is there a warm dir in the snapshot?

[cmartins88]

Thanks for the quick look. There is no directory literally named warm anywhere in the extracted snapshot. Here is the full layout after extraction ($HOME/.akash):

.akash/
  config/          (we add genesis.json + generated app.toml/config.toml)
  data/
    application.db/     (~8 TB)
    blockstore.db/
    state.db/
    tx_index.db/
    evidence.db/
    cs.wal/
    snapshots/          (many local state-sync snapshot dirs + metadata.db)
    priv_validator_state.json
    upgrade-info.json   -> {"name":"v2.1.0","height":27230465}
  wasm/
    exclusive.lock
    state/wasm/         (EMPTY)
    cache/modules/      (EMPTY)

So the wasm/ tree is present but empty (state/wasm and cache/modules have no files, 24K total).

One data point in case it helps: we compared against one of our own in-cluster pruned nodes running the same ghcr.io/akash-network/node:2.1.0 image that followed consensus through this height. That node's wasm/ directory is also empty (same layout), and it processes these blocks fine, so an empty wasm/ alone does not seem to be the differentiator on our side.

Could you clarify what the warm dir should be and where it lives (is it something the snapshot is expected to carry, e.g. the wasmvm module cache or a contract blob store)? If it is normally present in a healthy archival snapshot, that would line up perfectly with the wasm (and downstream bank/oracle) divergence we see when executing the first block. Happy to check the snapshot-producing node's layout too if you can point me at what to look for.

[Zblocker64]

Sorry, meant wasm. There should be data there, that is most likely the cause.

~/.akash/wasm/state/wasm# ls
2d3d37f9f0d417ee6e8a0fe961d07d93bf4c9f291406cb0c497d7fe2a43e3e24.wasm  91dc2aada6e94f102013cb7bf799892b137b033561430941475a3e355e7eef4d.wasm  b97763a6116d2eaad99d96de83b5ffabf4cc5dd927ca3e426ac02c767902162a.wasm
```

[cmartins88]

That was it - thank you. The wasm/state/wasm/ blob store was missing from the extracted archival snapshot (0 files). Our node's wasm module state referenced the contract code by hash but the actual bytecode files were absent, so executing the first block that touches a contract diverged (wasm, then bank/oracle downstream).

Fix: we copied the 3 contract blobs from a healthy node into ~/.akash/wasm/state/wasm/:

2d3d37f9f0d417ee6e8a0fe961d07d93bf4c9f291406cb0c497d7fe2a43e3e24.wasm
91dc2aada6e94f102013cb7bf799892b137b033561430941475a3e355e7eef4d.wasm
b97763a6116d2eaad99d96de83b5ffabf4cc5dd927ca3e426ac02c767902162a.wasm

Rolled the node back to the snapshot head, placed the blobs, restarted. It executed the previously-failing block with a matching AppHash and is now block syncing to tip with zero validation errors.

So the root cause is snapshot-side: the archival snapshot tarball does not include data/../wasm/state/wasm/*.wasm. Would it be possible to include that directory in the archival snapshot generation? Otherwise every restore from these snapshots hits the same wall at its head
block. Happy to help verify a regenerated snapshot if useful. Thanks again for the quick pointer.

[Zblocker64]

I'll fix them on our end. thanks for reporting!