Skip to content

Multiple validators bonding issue #517

Description

@elmiringos

What did you expect to see?

A new validator that has synced to the chain head and submitted a bond deploy should have that deploy finalized in the last finalized-block (LFB) bonds map within a reasonable time window, and then begin proposing blocks once the quarantine period ends.

Bonding did work correctly for the first wave of validators in the most favourable experiment (ft0.1, ft = 0.10): all 5 validators in wave 1 successfully bonded and began proposing alongside the 3 genesis validators. At that point the network had 8 active validators, all producing blocks with normal finalization latency:

validator1: 21 proposals   validator2: 13   validator3: 18
validator4: 18             validator5:  5   validator6:  5
validator7: 11             validator8: 13

The expectation is that bonding should work reliably across different fault-tolerance thresholds (0.1–0.99) and synchrony-constraint values (0.0–0.67), with differences showing up in performance metrics (finalization latency, block interval, proposal rate) rather than in bonding failures. Whether all parameter combinations are viable in a single-shard network with a small genesis validator set (3–5 nodes) is an open question — it is possible that some configurations (e.g. very high ft with few genesis validators) are inherently limited. However, at a minimum we expect bonding to succeed consistently at ft = 0.10 with 3 genesis validators across any number of joining waves.


What did you see instead?

Two failure modes were observed, both producing the same validateAndSetCurrentRoot FAILED error surface:

Bug 1 — joining validator fails to replay historical blocks after LFS sync

Immediately after a new validator finishes downloading the last-finalized-state (LFS) and historical blocks, Casper initialization fails with:

[RootRepository] validateAndSetCurrentRoot FAILED:
  Blake2b256Hash(39edf9ec9c1e0cd233adf595a6d800a6062dee4a23c1e255d8b6e3d846910b70)
  not in roots store

Block #11 replay error:
  InterpreterError(RSpaceError(HistoryError(RootError(
    UnknownRootError("unknown root: Blake2b256Hash(39edf9ec9c...)")))))

The state root referenced during block replay is not present in the validator's local root repository after LFS download. Errors begin before any bond deploy is submitted.

Bug 2 — bond-containing block rejected network-wide after multi-parent DAG merge

After a proposer runs a merge over a multi-parent block scope, blocks built on the resulting state are rejected as InvalidTransaction by all nodes:

Replay block 9253209c52519d5b2437a1b0c89d19a88fc4e6e3037c03ccd0d089aea9544950
  got error SystemRuntimeError(ConsumeFailed), retries details: giving up after 3 retries

[RootRepository] validateAndSetCurrentRoot FAILED:
  Blake2b256Hash(068a02a3f0796aded1214533c0510068d501040f3f29d2efca709935534b81ca)
  not in roots store

On peers:

Block 9253209c52... raised BlockException (KvStore error: Key not found:
  Missing mergeable entry for state a090ee79fb850638f7b33d76302bff2e44ae3e7be3c17d7915f0ea667bcabe5f
  (creator=04987fc4ad..., seq=28));
  recording as InvalidTransaction to prevent dependent-block stall.

If a bond deploy happens to be in the rejected block, it is permanently discarded — it never appears in the LFB bonds map.

The failure has been reproduced across 8 experiments in two independent runs:

Run Experiment ft Bonded before failure Abort reason
ft-upfront-13-05-26 ft0.1 0.10 8 / 20 Bond for validator9 not in LFB after 5m
ft-upfront-13-05-26 ft0.33 0.33 5 / 20 Bond for validator6 not in LFB after 5m
ft-upfront-13-05-26 ft0.5 0.50 6 / 20 Bond for validator7 not in LFB after 5m
ft-upfront-13-05-26 ft0.67 0.67 5 / 20 Bond for validator6 not in LFB after 5m
sync-constraint-13-05-26 sy=0.33 0.10 3 / 20 Bond for validator4 not in LFB after 5m
sync-constraint-13-05-26 sy=0.67 0.10 3 / 20 Bond for validator4 not in LFB after 5m
sync-constraint-13-05-26 sy=0.0 0.10 3 / 20 Bond for validator4 not in LFB after 5m
sync-constraint-12-05-26 sy=0.0 0.10 4 / 20 validator5 sync-to-head timeout (Bug 1)

In ft0.5 the merge corruption caused a visible DAG split: validators 4 and 6 accumulated 215 finalized blocks each while validators 1–3 were stuck at 8–9, producing a 207-block finalization gap. In ft0.67 (5 genesis validators) the chain produced 469 blocks but finalized zero — once any validator acquired corrupt state, effective participation dropped below quorum.


Steps to reproduce the bug

Test setup overview

The reproduction uses an automated testbed that deploys an ASI-chain (f1r3fly) Casper BFT network in Kubernetes. Validators join the network in waves by submitting bond deploys to the on-chain PoS contract. Bond verification proceeds in three stages:

  1. Sync to chain head — the joining node downloads the last-finalized-state
    (LFS) and all historical blocks, then waits until it is within 5 blocks of the
    current tip (up to 10 minutes).
  2. Bond finalization — polls /api/last-finalized-block and checks the
    bonds map until the validator's public key appears with the expected stake
    (up to 5 minutes).
  3. Bond activation — waits for "Propose started" in the validator's pod
    logs, confirming quarantine ended and the heartbeat proposer is live (up to
    20 minutes).

Fault-tolerance suite (upfront bonding)

All validator pods are deployed at genesis so they sync from block 0. Bond transactions are submitted later in waves of 5 validators, with a 5-minute pause between waves. Run 5 experiments in parallel with different ft values:

With 3 genesis validators and heartbeat enabled, the DAG begins accumulating multiple parents per height from the first block. By the time wave 1 bond attempts are submitted (~10 minutes in), the merge scope has grown large enough
that the state-root errors fire consistently.

Synchrony-constraint suite (aggressive bonding)

Only genesis pods are deployed initially. Each wave's pods are created and synced just before their bond transactions are submitted. ft is fixed at 0.1. Run 3 experiments in parallel with different synchrony-constraint thresholds:

./testbed run sync-constraint --validators 20 --profile mid \
  --heartbeat-ci 500ms --lfb-age 2s --cooldown 1s

Timeline of Bug 2 (validator1, sy=0.33 experiment, 2026-05-12 run)

14:42:31 — DagMerger.merge: LFB=230e97f3f1..., scope=6 blocks, actualBlocks=5
14:42:31 — Block #29 replayed, post-state=a090ee79fb...
14:42:44 — Block #30 produced (seqNum 26), 0 deploys
14:42:51 — Block #31 (seqNum 27): bond deploy selected and included
14:42:54 — Replay block 9253209c52...: ConsumeFailed ×3, giving up
14:42:54 — validateAndSetCurrentRoot FAILED: 068a02a3f0... not in roots store
14:42:54 — All peers mark block 9253209c52... as InvalidTransaction
14:44:51 — WaitForBond timeout: bond for 04cd3067ec... not in LFB after 2m

Timeline of Bug 1 (validator8, sy=0.33 experiment, 2026-05-12 run)

14:42:20 — LFS Tuple Space Requester stream completed
14:42:23 — LFS Block Requester: 70 blocks downloaded, finished
14:42:34 — validateAndSetCurrentRoot FAILED: 39edf9ec9c... (block #11 replay)
14:42:34 — Block #11 replay error: UnknownRootError(39edf9ec9c...)

Relevant source paths

casper/src/rust/merging/dag_merger.rs
casper/src/rust/util/rholang/interpreter_util.rs:495
rspace++/src/rspace/history/root_repository.rs:45
casper/src/rust/blocks/block_processor.rs:288

Known failing regression test

casper/tests/compute_parents_post_state_regression_spec.rs contains a test
that already fails with the current code:

// visible_blocks grows to ~60 because merge scope includes everything back to
// genesis instead of only blocks above the LCA. This test FAILS.
fn visible_blocks_should_not_grow_unbounded_with_dag_depth()

A second test in the same file documents the exact error mode we observe:

fn compute_parents_post_state_should_fail_when_required_mergeable_is_missing()

It removes a mergeable entry manually and asserts KvStoreError — the same error peers emit when rejecting the block.


What version of F1R3FLY Node are you using?

rust-node:0.4.15


What is your environment?

  • Kubernetes (k3s), single node, 380 vCPU · 3 TiB RAM
  • Per-validator pod: 2 / 8 vCPU (request / limit) · 4 / 16 GiB RAM · 200 GiB disk
  • No pod reached its CPU or memory limit during any experiment
  • 3 genesis validators with heartbeat enabled (500 ms propose interval, 2 s LFB-age threshold)
  • Real ft (0.1 – 0.67); no user deploys beyond bond transactions

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions