feat(ledger): ledger 8->9 hardfork on-chain migration by ozgb · Pull Request #1925 · midnightntwrk/midnight-node

ozgb · 2026-07-24T13:54:47Z

Overview

Implements the on-chain storage migration that lets a ledger-8 chain (e.g. 1.0.1) runtime-upgrade in place to the current ledger-9 runtime, with matching toolkit support so post-fork transaction generation follows the migrated ledger.

Node:

On-chain migration: pallet_midnight::StateKey stores only the ledger state root. migrate_state_v8_to_v9 (new host fn) runs the StateTranslationTable — ported from State translation – v8 to v9 midnight-ledger#539 — to translate LedgerState v13->v18 in place. Wired as pallet_midnight::migrations::v2::MigrateV1ToV2 (VersionedMigration<1,2,..>) into the runtime Migrations tuple: it fires once for a ledger-8 chain (on-chain pallet-midnight storage version 1) upgrading to this runtime; a fresh ledger-9 genesis starts at version 2 and skips it.
Migration cost model: Uses the ledger cost model for accurate migration weight. Logs added to measure wall-clock time of migration for testing with real network state (e.g. ephemeral env, mainnet)
Fix Ledger 8 boot: Version-aware genesis seeding: the node previously hardcoded the v9 deserializer for parity-db genesis seeding and would panic on a v8 (ledger-state[v13]) genesis. Genesis version is now detected via serialize::peek_tag instead.
Fix Ledger 8 boot: Restored ledger-8 host fn: construct_distribute_treasury_system_tx was dropped for v9 but is still imported by the 1.0.1 WASM, so the node couldn't instantiate it without it.

Toolkit:

Toolkit fork-aware tx generation: replay_blocks now detects the v8->v9 fork boundary and runs the same StateTranslationTable translation (fork_context_8_to_9 / fork_8_to_9_if_needed) so post-fork transactions are built against the correctly-translated ledger-9 context instead of a stale ledger-8 one.
runtime-upgrade toolkit command fix: now waits for the new runtime to actually execute at a finalized block (not just be stored/applied) before reporting success — the stored spec version flips at the apply block, but that block still executes under the old runtime, which was causing the toolkit to fetch a block short of any ledger-9-classified block.

Extends 01b51abb ("Ledger9 with no migration").

🗹 TODO before merging

Ready

📌 Submission Checklist

All commits are signed off (git commit -s) for the DCO
Changes are backward-compatible (or flagged if breaking)
Pull request description explains why the change is needed
Self-reviewed the diff
I have included a change file, or skipped for this reason:
If the changes introduce a new feature, I have bumped the node minor version
Update documentation (if relevant)
Updated AGENTS.md if build commands, architecture, or workflows changed
No new todos introduced

🧪 Testing Evidence

hardfork_e2e (util/toolkit/tests/hardfork_e2e.rs, previously #[ignore]d) validates the full flow end-to-end: node boots on the ledger-8 1.0.1 chainspec, produces blocks, a pre-fork single-tx succeeds, a governance runtime-upgrade enacts (spec 1_000_000 -> 2_001_000), the on-chain migration translates state without panicking, and the toolkit then builds and submits a valid post-fork ledger-9 transaction the node accepts.

NODE_IMAGE=midnight-node-hf:test MIDNIGHT_LEDGER_EXPERIMENTAL=1 \
  cargo test --release -p midnight-node-toolkit --test hardfork_e2e -- --nocapture

New unit/smoke tests in ledger/src/state_translation_v8_to_v9.rs cover the translation table (table_tags_match_types, table_is_closed) and a v8->v9 translate+round-trip.

Additional tests are provided (if possible)

🔱 Fork Strategy

Node Runtime Update
Node Client Update
Other:
N/A

Links

Closes #1580

#1786
#1787

Port the v8->v9 state translation table from midnight-ledger PR #539 into `midnight_node_ledger::state_translation_v8_to_v9`, and wire it into the runtime as a single-block storage migration: - New host function `Ledger9Bridge::migrate_state_v8_to_v9` reads the v8 arena root (pallet-midnight `StateKey`), walks/translates the v8 `LedgerState` into the v9 shape, re-persists it, and returns the new v9 root. v8 and v9 share one storage backend, so the arena is shared. - `pallet_midnight::migrations::v2::MigrateV1ToV2` (VersionedMigration 1->2) calls the host function and re-points `StateKey`. Bumped pallet-midnight STORAGE_VERSION 1 -> 2 and added it to the runtime `Migrations` tuple. - Un-ignore the `hardfork_single_tx` e2e test and point its fork-from image at the ledger-8 release `midnightntwrk/midnight-node:1.0.1`. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

- table_is_closed / table_tags_match_types guard the translation table against tag drift on the node's rc.3 crate versions. - empty_state_translates_and_round_trips exercises an end-to-end v8->v9 translation and a v9 serialize round-trip. - Enable the helpers `can-panic` feature for ledger dev builds so the crate's `#[cfg(test)]` modules compile standalone. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

A ledger-9 node booting on a ledger-8 chain-spec (the hardfork fork-from case) panicked at startup: the arena seeder hardcoded the ledger-9 deserializer and rejected the v8 genesis (`expected ledger-state[v18], got ledger-state[v13]`). The genesis block runs under the old WASM, so the arena must be seeded in the genesis version. - Add `genesis_matches_this_version` (per-version tag check) to common storage. - Add `init_ledger_storage_{separate,unified}` dispatchers that pick the ledger_8 vs ledger_9 seeder by the genesis `ledger-state[vN]` tag; v8 and v9 share one backend so a v8-seeded arena is what the post-migration v9 reads. - Route `custom_parity_db` through the dispatchers. - Add a dev test proving the v8-seeded root matches the fork-from chain-spec's genesisStateKey (no `Ledger`-wrapper drift vs release 1.0.1). Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

Replace the substring header scan in `genesis_matches_this_version` with `midnight_serialize::peek_tag`, matching the pattern used by `contract_operation_versioned_verifier_key`. Compares the peeked header tag against this version's `LedgerState::tag()` (no hardcoded version numbers). Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

A ledger-9 node could not instantiate the ledger-8 runtime WASM (the hardfork fork-from case): the WASM imports `ext_ledger_8_bridge_construct_distribute_treasury_system_tx_version_1`, which was dropped from the current ledger-8 bridge (renamed to reserve/unlock-to-treasury for v9). Re-add it so the current node can execute the ledger-8 runtime across the 8->9 boundary. - ledger_8 builds `SystemTransaction::PayBlockRewardsToTreasury { amount }` (matching release 1.0.1); v7/v9 helpers are error stubs (only the ledger-8 bridge exposes the host fn). - Wire through the common Bridge and the Ledger8Bridge runtime interface. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

`runtime-upgrade` used subxt's `wait_for_finalized_success()`, which eagerly decodes the apply block's events. Across the 8->9 hardfork the client's metadata follows the code swap to the new runtime, so decoding the old runtime's `System.CodeUpdated` event fails ("Can't decode field hash ..."). Wait only for finalization (`wait_for_finalized`, no event decode) and confirm the upgrade enacted by polling `state_getRuntimeVersion` for the spec_version bump — no metadata-dependent decoding across the boundary. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

…dary The toolkit built a ledger-8 transaction for the post-hardfork (ledger-9) chain, which the node rejected (Deserialization(Transaction): transaction[v9]/signature[v1] vs expected transaction[v12]/signature[v2]). Its fork-aware replay never transitioned the context ledger-8 -> ledger-9. - Move StateTranslationTable into `midnight-node-ledger-helpers` (so both the runtime migration in `ledger` and the toolkit fork can use it), adding the onchain-state deps there. - `fork_context_8_to_9` now runs the real `TypedTranslationState` translation (Db8 == Db9, one shared arena) instead of the tag-reuse `old_to_new_sp`. - Wire the ledger-8 -> ledger-9 transition into `replay_blocks` (new `fork_8_to_9_if_needed`) and relax the "not supported yet" assert. - runtime-upgrade now waits for the spec bump at the FINALIZED head (toolkit fetch reads only finalized blocks), so the post-fork fetch sees the ledger-9 blocks and the replay forks to ledger 9. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

Temporary diagnostic to see the l7/l8/l9 block partition + initial context version during the post-fork tx build. Remove once the fork boundary is fixed. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

…ot just apply state_getRuntimeVersion(finalized) reports the *stored* code, which flips to the new runtime at the apply_authorized_upgrade block — but that block still *executes* under the old runtime (its MNSV digest, which the fetcher uses to classify ledger version, is the old spec). The first block to run the new runtime is apply+1. The prior poll returned at the apply block, so a downstream fetch (bounded by finalized height) reached the apply block (still classified ledger-8) but not apply+1, and the toolkit built a ledger-8 tx post-fork. Track the finalized height where the stored spec first bumps, then wait for the finalized height to advance past it (apply+1 finalized) before reporting success. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

The v8->v9 fork boundary is fixed and hardfork_e2e passes; drop the diagnostic. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

Assisted-by: Claude:claude-sonnet-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e8c1427d5a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 679a1a512e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

The 2.0.0 runtime runs ledger-9 but shipped pallet-midnight at storage version 1 (it had no v1->v2 migration), so a network upgrading 2.0.0 -> this runtime still fires VersionedMigration<1,2> over an already-v9 StateKey. Feeding that v9 root to the v8 decode path fails on the tag mismatch and the pallet's expect() panics, bricking the upgrade block. Guard migrate_state_v8_to_v9 so it no-ops when the StateKey already references a ledger-9 state: detect it by the arena root's serialized tag (distinct between ledger-state[v13] and [v18]) and return the key unchanged with zero synthetic cost, leaving only the storage-version bump. Assisted-by: Claude:claude-fable-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

justinfrevert · 2026-07-28T14:20:23Z

+		// no separate `DbWeight` charge is added on top. `proof_size` is 0:
+		// this chain doesn't build a PoV. Capped at the block's max weight,
+		// since a pathological state could in principle exceed it.
+		Weight::from_parts(consumed_cost_ps, 0).min(T::BlockWeights::get().max_block)


"pathological state could in principle exceed it"

Weight::from_parts(consumed_cost_ps, 0).min(T::BlockWeights::get().max_block)

I think that this is a bigger risk than we're suggesting. Why risk going over with a single-block upgrade, when we could de-risk with a multi-block upgrade?

The transition is pretty small. Pretty certain that if it's rolled out within 3 months we'll be fine based on the back of the envelope perf numbers Thomas / Alex were benching.

jacek-kurkowski-shielded

Jeeez, very nice, @ozgb ! I don't have many meaningful comments on your changes 😄

Just a couple of general ones:

Has this been tested in a long-running environment (e.g. mainnet), or should I do that?
I guess we should get rid of ledger-7 in a separate issue once this is merged.

jacek-kurkowski-shielded · 2026-07-29T08:31:29Z

+
+Lets a ledger-8 chain (e.g. `1.0.1`) runtime-upgrade in place to the current
+ledger-9 runtime. A new host fn, `migrate_state_v8_to_v9`, runs the
+`StateTranslationTable` (ported from `midnight-ledger` PR #539) to translate


Maybe a full URL to the ledger's repo?

ozgb added 10 commits July 24, 2026 11:57

chore(toolkit): remove TEMP hardfork_debug replay_blocks log

af7ed4f

The v8->v9 fork boundary is fixed and hardfork_e2e passes; drop the diagnostic. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

ozgb added the bot:ai-assisted Authored or substantially edited by an AI agent label Jul 24, 2026

docs: add change files for the ledger 8->9 hardfork migration

4265e2b

Assisted-by: Claude:claude-sonnet-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

This comment has been minimized.

Sign in to view

ozgb added 4 commits July 27, 2026 09:42

chore: cargo fmt

e30976c

Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

Merge branch 'main' into ozgb-ledger-hf

2c76e05

chore: fix clippy errors

08671ba

Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

chore: npm audit fix (toolkit-js)

e8c1427

Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

ozgb marked this pull request as ready for review July 28, 2026 08:40

ozgb requested a review from a team as a code owner July 28, 2026 08:40

Merge branch 'main' into ozgb-ledger-hf

0f8f03d

chatgpt-codex-connector Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread runtime/src/lib.rs

Comment thread pallets/midnight/src/migrations/v2.rs Outdated

feat: add perf prints to track migration time elapsed

679a1a5

Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

chatgpt-codex-connector Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread ledger/helpers/src/state_translation_v8_to_v9.rs

ozgb added 3 commits July 28, 2026 10:54

feat: use ledger cost model for storage migration

fbbfe1c

Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

chore(local-environment): npm audit fix

7c16094

Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

This was linked to issues Jul 28, 2026

[HF v8→v9] NODE-03 — Toolkit: fork-block impl switch + state migration #1787

Open

[HF v8→v9] NODE-02 — Node state migration at fork block #1786

Open

docs: update change file

1b82adb

Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>

justinfrevert reviewed Jul 28, 2026

View reviewed changes

jacek-kurkowski-shielded approved these changes Jul 29, 2026

View reviewed changes

gilescope approved these changes Jul 29, 2026

View reviewed changes

Merge branch 'main' into ozgb-ledger-hf

b6d8bfd

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(ledger): ledger 8->9 hardfork on-chain migration - #1925

feat(ledger): ledger 8->9 hardfork on-chain migration#1925
ozgb wants to merge 22 commits into
mainfrom
ozgb-ledger-hf

ozgb commented Jul 24, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

justinfrevert Jul 28, 2026

Uh oh!

gilescope Jul 29, 2026

Uh oh!

jacek-kurkowski-shielded left a comment

Uh oh!

jacek-kurkowski-shielded Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

ozgb commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

🗹 TODO before merging

📌 Submission Checklist

🧪 Testing Evidence

🔱 Fork Strategy

Links

Uh oh!

This comment has been minimized.

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

justinfrevert Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

gilescope Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

jacek-kurkowski-shielded left a comment

Choose a reason for hiding this comment

Uh oh!

jacek-kurkowski-shielded Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ozgb commented Jul 24, 2026 •

edited

Loading