What did you expect to see?
A new validator that has synced to the chain head and submitted a bond deploy should have that deploy finalized in the last finalized-block (LFB) bonds map within a reasonable time window, and then begin proposing blocks once the quarantine period ends.
Bonding did work correctly for the first wave of validators in the most favourable experiment (ft0.1, ft = 0.10): all 5 validators in wave 1 successfully bonded and began proposing alongside the 3 genesis validators. At that point the network had 8 active validators, all producing blocks with normal finalization latency:
validator1: 21 proposals validator2: 13 validator3: 18
validator4: 18 validator5: 5 validator6: 5
validator7: 11 validator8: 13
The expectation is that bonding should work reliably across different fault-tolerance thresholds (0.1–0.99) and synchrony-constraint values (0.0–0.67), with differences showing up in performance metrics (finalization latency, block interval, proposal rate) rather than in bonding failures. Whether all parameter combinations are viable in a single-shard network with a small genesis validator set (3–5 nodes) is an open question — it is possible that some configurations (e.g. very high ft with few genesis validators) are inherently limited. However, at a minimum we expect bonding to succeed consistently at ft = 0.10 with 3 genesis validators across any number of joining waves.
What did you see instead?
Two failure modes were observed, both producing the same validateAndSetCurrentRoot FAILED error surface:
Bug 1 — joining validator fails to replay historical blocks after LFS sync
Immediately after a new validator finishes downloading the last-finalized-state (LFS) and historical blocks, Casper initialization fails with:
[RootRepository] validateAndSetCurrentRoot FAILED:
Blake2b256Hash(39edf9ec9c1e0cd233adf595a6d800a6062dee4a23c1e255d8b6e3d846910b70)
not in roots store
Block #11 replay error:
InterpreterError(RSpaceError(HistoryError(RootError(
UnknownRootError("unknown root: Blake2b256Hash(39edf9ec9c...)")))))
The state root referenced during block replay is not present in the validator's local root repository after LFS download. Errors begin before any bond deploy is submitted.
Bug 2 — bond-containing block rejected network-wide after multi-parent DAG merge
After a proposer runs a merge over a multi-parent block scope, blocks built on the resulting state are rejected as InvalidTransaction by all nodes:
Replay block 9253209c52519d5b2437a1b0c89d19a88fc4e6e3037c03ccd0d089aea9544950
got error SystemRuntimeError(ConsumeFailed), retries details: giving up after 3 retries
[RootRepository] validateAndSetCurrentRoot FAILED:
Blake2b256Hash(068a02a3f0796aded1214533c0510068d501040f3f29d2efca709935534b81ca)
not in roots store
On peers:
Block 9253209c52... raised BlockException (KvStore error: Key not found:
Missing mergeable entry for state a090ee79fb850638f7b33d76302bff2e44ae3e7be3c17d7915f0ea667bcabe5f
(creator=04987fc4ad..., seq=28));
recording as InvalidTransaction to prevent dependent-block stall.
If a bond deploy happens to be in the rejected block, it is permanently discarded — it never appears in the LFB bonds map.
The failure has been reproduced across 8 experiments in two independent runs:
| Run |
Experiment |
ft |
Bonded before failure |
Abort reason |
| ft-upfront-13-05-26 |
ft0.1 |
0.10 |
8 / 20 |
Bond for validator9 not in LFB after 5m |
| ft-upfront-13-05-26 |
ft0.33 |
0.33 |
5 / 20 |
Bond for validator6 not in LFB after 5m |
| ft-upfront-13-05-26 |
ft0.5 |
0.50 |
6 / 20 |
Bond for validator7 not in LFB after 5m |
| ft-upfront-13-05-26 |
ft0.67 |
0.67 |
5 / 20 |
Bond for validator6 not in LFB after 5m |
| sync-constraint-13-05-26 |
sy=0.33 |
0.10 |
3 / 20 |
Bond for validator4 not in LFB after 5m |
| sync-constraint-13-05-26 |
sy=0.67 |
0.10 |
3 / 20 |
Bond for validator4 not in LFB after 5m |
| sync-constraint-13-05-26 |
sy=0.0 |
0.10 |
3 / 20 |
Bond for validator4 not in LFB after 5m |
| sync-constraint-12-05-26 |
sy=0.0 |
0.10 |
4 / 20 |
validator5 sync-to-head timeout (Bug 1) |
In ft0.5 the merge corruption caused a visible DAG split: validators 4 and 6 accumulated 215 finalized blocks each while validators 1–3 were stuck at 8–9, producing a 207-block finalization gap. In ft0.67 (5 genesis validators) the chain produced 469 blocks but finalized zero — once any validator acquired corrupt state, effective participation dropped below quorum.
Steps to reproduce the bug
Test setup overview
The reproduction uses an automated testbed that deploys an ASI-chain (f1r3fly) Casper BFT network in Kubernetes. Validators join the network in waves by submitting bond deploys to the on-chain PoS contract. Bond verification proceeds in three stages:
- Sync to chain head — the joining node downloads the last-finalized-state
(LFS) and all historical blocks, then waits until it is within 5 blocks of the
current tip (up to 10 minutes).
- Bond finalization — polls
/api/last-finalized-block and checks the
bonds map until the validator's public key appears with the expected stake
(up to 5 minutes).
- Bond activation — waits for
"Propose started" in the validator's pod
logs, confirming quarantine ended and the heartbeat proposer is live (up to
20 minutes).
Fault-tolerance suite (upfront bonding)
All validator pods are deployed at genesis so they sync from block 0. Bond transactions are submitted later in waves of 5 validators, with a 5-minute pause between waves. Run 5 experiments in parallel with different ft values:
With 3 genesis validators and heartbeat enabled, the DAG begins accumulating multiple parents per height from the first block. By the time wave 1 bond attempts are submitted (~10 minutes in), the merge scope has grown large enough
that the state-root errors fire consistently.
Synchrony-constraint suite (aggressive bonding)
Only genesis pods are deployed initially. Each wave's pods are created and synced just before their bond transactions are submitted. ft is fixed at 0.1. Run 3 experiments in parallel with different synchrony-constraint thresholds:
./testbed run sync-constraint --validators 20 --profile mid \
--heartbeat-ci 500ms --lfb-age 2s --cooldown 1s
Timeline of Bug 2 (validator1, sy=0.33 experiment, 2026-05-12 run)
14:42:31 — DagMerger.merge: LFB=230e97f3f1..., scope=6 blocks, actualBlocks=5
14:42:31 — Block #29 replayed, post-state=a090ee79fb...
14:42:44 — Block #30 produced (seqNum 26), 0 deploys
14:42:51 — Block #31 (seqNum 27): bond deploy selected and included
14:42:54 — Replay block 9253209c52...: ConsumeFailed ×3, giving up
14:42:54 — validateAndSetCurrentRoot FAILED: 068a02a3f0... not in roots store
14:42:54 — All peers mark block 9253209c52... as InvalidTransaction
14:44:51 — WaitForBond timeout: bond for 04cd3067ec... not in LFB after 2m
Timeline of Bug 1 (validator8, sy=0.33 experiment, 2026-05-12 run)
14:42:20 — LFS Tuple Space Requester stream completed
14:42:23 — LFS Block Requester: 70 blocks downloaded, finished
14:42:34 — validateAndSetCurrentRoot FAILED: 39edf9ec9c... (block #11 replay)
14:42:34 — Block #11 replay error: UnknownRootError(39edf9ec9c...)
Relevant source paths
casper/src/rust/merging/dag_merger.rs
casper/src/rust/util/rholang/interpreter_util.rs:495
rspace++/src/rspace/history/root_repository.rs:45
casper/src/rust/blocks/block_processor.rs:288
Known failing regression test
casper/tests/compute_parents_post_state_regression_spec.rs contains a test
that already fails with the current code:
// visible_blocks grows to ~60 because merge scope includes everything back to
// genesis instead of only blocks above the LCA. This test FAILS.
fn visible_blocks_should_not_grow_unbounded_with_dag_depth()
A second test in the same file documents the exact error mode we observe:
fn compute_parents_post_state_should_fail_when_required_mergeable_is_missing()
It removes a mergeable entry manually and asserts KvStoreError — the same error peers emit when rejecting the block.
What version of F1R3FLY Node are you using?
rust-node:0.4.15
What is your environment?
- Kubernetes (k3s), single node, 380 vCPU · 3 TiB RAM
- Per-validator pod: 2 / 8 vCPU (request / limit) · 4 / 16 GiB RAM · 200 GiB disk
- No pod reached its CPU or memory limit during any experiment
- 3 genesis validators with heartbeat enabled (500 ms propose interval, 2 s LFB-age threshold)
- Real ft (0.1 – 0.67); no user deploys beyond bond transactions
What did you expect to see?
A new validator that has synced to the chain head and submitted a bond deploy should have that deploy finalized in the last finalized-block (LFB) bonds map within a reasonable time window, and then begin proposing blocks once the quarantine period ends.
Bonding did work correctly for the first wave of validators in the most favourable experiment (ft0.1, ft = 0.10): all 5 validators in wave 1 successfully bonded and began proposing alongside the 3 genesis validators. At that point the network had 8 active validators, all producing blocks with normal finalization latency:
The expectation is that bonding should work reliably across different fault-tolerance thresholds (0.1–0.99) and synchrony-constraint values (0.0–0.67), with differences showing up in performance metrics (finalization latency, block interval, proposal rate) rather than in bonding failures. Whether all parameter combinations are viable in a single-shard network with a small genesis validator set (3–5 nodes) is an open question — it is possible that some configurations (e.g. very high ft with few genesis validators) are inherently limited. However, at a minimum we expect bonding to succeed consistently at ft = 0.10 with 3 genesis validators across any number of joining waves.
What did you see instead?
Two failure modes were observed, both producing the same
validateAndSetCurrentRoot FAILEDerror surface:Bug 1 — joining validator fails to replay historical blocks after LFS sync
Immediately after a new validator finishes downloading the last-finalized-state (LFS) and historical blocks, Casper initialization fails with:
The state root referenced during block replay is not present in the validator's local root repository after LFS download. Errors begin before any bond deploy is submitted.
Bug 2 — bond-containing block rejected network-wide after multi-parent DAG merge
After a proposer runs a merge over a multi-parent block scope, blocks built on the resulting state are rejected as
InvalidTransactionby all nodes:On peers:
If a bond deploy happens to be in the rejected block, it is permanently discarded — it never appears in the LFB bonds map.
The failure has been reproduced across 8 experiments in two independent runs:
In ft0.5 the merge corruption caused a visible DAG split: validators 4 and 6 accumulated 215 finalized blocks each while validators 1–3 were stuck at 8–9, producing a 207-block finalization gap. In ft0.67 (5 genesis validators) the chain produced 469 blocks but finalized zero — once any validator acquired corrupt state, effective participation dropped below quorum.
Steps to reproduce the bug
Test setup overview
The reproduction uses an automated testbed that deploys an ASI-chain (f1r3fly) Casper BFT network in Kubernetes. Validators join the network in waves by submitting bond deploys to the on-chain PoS contract. Bond verification proceeds in three stages:
(LFS) and all historical blocks, then waits until it is within 5 blocks of the
current tip (up to 10 minutes).
/api/last-finalized-blockand checks thebondsmap until the validator's public key appears with the expected stake(up to 5 minutes).
"Propose started"in the validator's podlogs, confirming quarantine ended and the heartbeat proposer is live (up to
20 minutes).
Fault-tolerance suite (upfront bonding)
All validator pods are deployed at genesis so they sync from block 0. Bond transactions are submitted later in waves of 5 validators, with a 5-minute pause between waves. Run 5 experiments in parallel with different ft values:
With 3 genesis validators and heartbeat enabled, the DAG begins accumulating multiple parents per height from the first block. By the time wave 1 bond attempts are submitted (~10 minutes in), the merge scope has grown large enough
that the state-root errors fire consistently.
Synchrony-constraint suite (aggressive bonding)
Only genesis pods are deployed initially. Each wave's pods are created and synced just before their bond transactions are submitted. ft is fixed at 0.1. Run 3 experiments in parallel with different synchrony-constraint thresholds:
Timeline of Bug 2 (validator1, sy=0.33 experiment, 2026-05-12 run)
Timeline of Bug 1 (validator8, sy=0.33 experiment, 2026-05-12 run)
Relevant source paths
Known failing regression test
casper/tests/compute_parents_post_state_regression_spec.rscontains a testthat already fails with the current code:
A second test in the same file documents the exact error mode we observe:
It removes a mergeable entry manually and asserts
KvStoreError— the same error peers emit when rejecting the block.What version of F1R3FLY Node are you using?
rust-node:0.4.15What is your environment?