Add Aptos DB fork tool for isolated validator networks

[areshand]

Summary

• add aptos-debugger aptos-db fork to copy an existing Aptos DB into an isolated fork
• replace the chain ID, validator set, validator configs, stake pools, and validator performance state with freshly generated local validators
• generate a new virtual-genesis write set, waypoint, per-validator DB checkpoint, node configuration, identity, and fork manifest
• preserve post-genesis application state while leaving the source DB unchanged
• reject mainnet/reserved chain IDs, source/output aliasing, nested paths, validator-address collisions, and stale validator configuration

Why

Binary and smart-contract testing needs production-shaped state without production consensus or networking impact. Reusing production validator identities risks impersonation and accidental connectivity, while submitting tests to the live validator set affects production state. This tool creates a separate chain from an existing checkpoint with fresh identity and consensus state.

Validation

• cargo test -p aptos-db-tool fork::tests -- --nocapture
• cargo check -p aptos-db-tool --locked
• cargo build -p aptos-node
• APTOS_NODE_BINARY=target/debug/aptos-node cargo test -p aptos-db-tool fork::tests::two_validator_fork_bootstraps_and_commits_blocks -- --ignored --exact --nocapture
• git diff --check origin/m1...HEAD

The live smoke test starts from a post-genesis DB, replaces the original validator set with two fresh validators, launches two real aptos-node processes, observes both nodes commit new ledger versions on chain ID 42, and confirms preserved account state through both REST endpoints.

Notes

This validates a local post-genesis source DB. Validation against an actual mainnet-derived checkpoint remains follow-up acceptance work.

[areshand]

Mainnet-derived acceptance rerun completed on commit 7a84158e76.

• Source: chain ID 126, version 179,355,214, epoch 10,779,160, 4 validators.
• Fork: chain ID 42, fresh waypoint 179355215:f6ce7ea65d47b86fd3048602b7866a08a6460f85b69eebd7303c497f6a8040ce, 2 fresh validators.
• Replacement stake is derived from the maximum source active voting power (10,963,022,086,631,463), so both replacements survived epoch transitions.
• Both nodes crossed into epoch 10,779,162 and advanced together from version 179,355,380 to 179,355,610 during a 45-second sustained check. The active set remained exactly the two replacements.
• Old production ValidatorConfig resources return 404; new replacement configs return 200. Representative source StakePool data remains present and identical on both nodes.
• All Aptos listeners bind to 127.0.0.1, telemetry output is disabled, host egress is default-deny, and no Aptos process had a non-loopback connection.
• The attached source DB remained read-only; a disposable staging checkpoint used private mutable RocksDB files. Source mutable-metadata fingerprints matched before and after the run.
• Local fork tests: 3 passed, including nonzero minimum-stake coverage. The ignored live smoke now requires two post-fork versions and verifies the active set remains size 2.

[areshand]

Funded test-account re-key support was added in 366cb59890 and verified against the mainnet-derived v5 fork.

• New options: --test-account-address and --test-account-private-key; short hex such as 0x1 is left-padded to a 32-byte Ed25519 key.
• Selected existing funded account: 0x573537299646e0dfab6ca81086edccf73f77b841f30dde6bfbd730ed428479bf.
• Original balance 3,612,356,182 and sequence 2 were preserved when the fork was created; only the Account authentication key changed to 0xf90391c81027f03cdea491ed8b36ffaced26b6df208a9b569e5baf2590eb9b16.
• The normalized private key is emitted under the fork config directory with mode 0600; the manifest records its path and public authentication key, not the private key.
• A transaction signed with private key 0x1 executed successfully on chain ID 42 at version 179,355,586: hash 0x2a026639b1f84113ea4a422f6456583cc15332c7a429529edb56213d1ad2a6cf. Sequence advanced to 3; balance decreased by exactly 1 transferred unit plus 18,500 gas.
• Both fork validators report the same authentication key, sequence, balance, and two-validator active set.
• Source mutable-metadata fingerprint remained unchanged.
• Local fork tests: 4 passed, including preservation and key-file permission coverage.

[areshand]

Default test-account key and readability refactor acceptance (commit 88098bbc0f):

• --test-account-address remains opt-in; when present without --test-account-private-key, the fork tool now defaults to 0xabc. Explicit keys still override it.
• Refactored Command::run into validation, checkpoint creation, replacement-validator generation, output verification, and artifact-writing phases.
• Local focused tests: 5 passed. Two-validator live bootstrap/block-production smoke test: passed.
• Mainnet-derived v6 acceptance used source version 179355214 (chain ID 126, 4 validators), producing fork version 179355215 (chain ID 42, 2 fresh validators), waypoint 179355215:1151ad312885d8548092b591b610a2f219ff8ea55b1c16ec7d188ffb249fbdad.
• The CLI invocation omitted --test-account-private-key; generated key file was 0x0000000000000000000000000000000000000000000000000000000000000abc, mode 0600.
• Both validators reached version 179355223, epoch 10779162, and reported auth key 0x69781361640e74e18569374f598667ee2688a317ec31978f7f1b1536bc7f44b5 for the re-keyed account.
• Signed transaction succeeded at version 179355220: 0xd533faff2ba9849e2e2f37b88e13116012f1805e61e87daa9f335c271e86ea15.
• /mnt/mainnet-db-ro remained mounted ro; all v6 writes were confined to disposable fork directories.

[areshand]

Added one-command ephemeral fork lifecycle wrapper in commit 878e5dfa52.

Usage:

scripts/ephemeral_fork.sh start \
  --source-db /data/disposable-checkpoint \
  --source-is-disposable \
  --fork-tool target/release/aptos-debugger \
  --node-binary /path/to/candidate/aptos-node \
  --test-account-address 0x1234 \
  --work-dir /tmp/contract-fork

The launcher:

• creates a fork with chain ID 42 and two fresh validators by default;
• accepts an arbitrary compatible aptos-node binary;
• re-keys an existing funded account to the test-only 0xabc default;
• rejects chain ID 1 and non-loopback generated REST addresses;
• starts all validators and requires every node to advance beyond the fork waypoint;
• prints the REST URL, account, key file, manifest, and contract-publish command;
• supports status and PID-verified stop commands;
• requires explicit acknowledgement that the source checkpoint is writable and disposable.

Validation: bash -n passed, ShellCheck passed, and the mock two-validator detached start/health/ledger-progress/status/stop lifecycle test passed. Operator guide: storage/db-tool/EPHEMERAL_FORK.md.

[areshand]

Added and installed the SHA-based remote ephemeral-mainnet launcher in commit f0008baf1f.

Developer command:

scripts/start_remote_ephemeral_mainnet.sh <m1-github-sha>

Defaults:

• host: ubuntu@34.231.241.232
• identity: ~/movement/mainnet-fork.pem
• validators: 2
• fork chain ID: 42
• test account key: test-only 0xabc
• local submission endpoint: http://127.0.0.1:8080

The host has intentional default-deny egress, so the local connector verifies that the SHA is contained in origin/m1, uploads a Git archive carrying that exact commit ID, and the host verifies the embedded ID before building aptos-node. The remote launcher uses a singleton lock and refuses with exit code 3 if any aptos-node is already running; it never stops or replaces an active network.

One validator REST API is exposed through an authenticated SSH local forward. Remote validator REST and P2P remain loopback-only; no unauthenticated public REST port is opened.

Installed remotely:

• /mnt/mainnet-volume/fork-run-20260722/bin/ephemeral_fork.sh
• /mnt/mainnet-volume/fork-run-20260722/bin/start_ephemeral_mainnet.sh

Validation:

• Bash syntax and ShellCheck pass.
• Generic two-validator lifecycle test passes.
• Exact-SHA archive verification and mismatched-SHA rejection test passes.
• Real host preflight detected all four existing v5/v6 validators and exited 3 before upload/build.
• The one-SHA connector produced the same no-op behavior.
• Temporary SSH forwarding successfully reached 127.0.0.1:41791 as local REST, chain ID 42, then closed cleanly.

Documentation: storage/db-tool/REMOTE_EPHEMERAL_MAINNET.md.

[areshand]

Public submission API added and verified in commit b9b48adda5.

Endpoint: http://34.231.241.232:8080

Implementation:

• validator REST and P2P remain bound to loopback;
• a managed Python TCP proxy is the only process bound to 0.0.0.0:8080;
• proxy PID, target, port, and logs are stored under /mnt/mainnet-volume/ephemeral-mainnet/public-api;
• expose-api and PID-verified stop-api lifecycle actions are available;
• SHA-based starts expose the proxy automatically;
• proxy and validator children explicitly close the launcher lock FD to avoid inherited-lock deadlocks.

Real-host acceptance:

• managed proxy PID 317717, target 127.0.0.1:40439 (v6), public listener 0.0.0.0:8080;
• external GET returned chain ID 42 and advancing ledger state;
• v6 account auth key through the public endpoint is 0x69781361640e74e18569374f598667ee2688a317ec31978f7f1b1536bc7f44b5, derived from test-only private key 0xabc;
• signed one-Octa transaction through the proxy succeeded at version 181038373, hash 0xe95e4a0f069199d4651c89db5684eebd51874f822478e82ecb9ed523e828b1bc;
• bounded singleton preflight returned exit 3 immediately and reported the public endpoint, proving the proxy no longer retains the lock.

The endpoint is unauthenticated and contains only isolated fork state/test funds. Documentation updated in storage/db-tool/REMOTE_EPHEMERAL_MAINNET.md.

[musitdev]

I've tested testsuite/forktest/remote/ephemeral_fork.sh. I use an old mainnet db. After some time, it starts correctly. I stop the network but there's no command to restart it. As it takes time to start a network using a prod db, a restart command will be very handy. It avoid starting the node by hand using each generated config.
I didn't find the README that present the command. Some doc that take your PR comment will be helpful to use the tool.