What did you expect to see?
The network should remain synchronized after bridge-contract deploys.
All validators and the observer should be able to replay and validate blocks correctly, with the current block height staying consistent with or above the latest finalized block.
What did you see instead?
After a series of bridge-contract deploys, the network entered a serious desync state.
Validators v1, v2, and the observer appear to have corrupted RSpace state and can no longer replay or validate blocks. At the same time, v3 continues finalizing blocks above height 2374.
Current block height became lower than the latest finalized block on v1 and v2:
=== Block / Finalized ===
validator1 — block=2280, finalized=2374
validator2 — block=2221, finalized=2368
validator3 — block=2375, finalized=2374
observer — block=2175, finalized=2172
Relevant observations from validator1.log:
13:03:03: bridge deploys
13:04:58: Computation ran out of phlogistons
13:05:12: Replay block ......... got error SystemRuntimeError(ConsumeFailed), retries details: will retry, attempt 1
13:05:12: got error SystemRuntimeError(ConsumeFailed), retries details: giving up after 3 retries
13:05:12: raised BlockException:
Runtime error: Internal errors encountered while processing deploy:
System runtime error: Unable to consume results of system deploy
13:05:12: Recording invalid block b9f9ca460c... for InvalidTransaction
13:05:21: RootRepository validateAndSetCurrentRoot FAILED:
Blake2b256Hash(.........) not in roots store
13:05:21: level="ERROR", message="Replay block ......... got error InterpreterError(RSpaceError(HistoryError(RootError(UnknownRootError
Steps to reproduce the bug
- Start shard from the tested commit
- Run a series of bridge-contract deploys (will share contract code in MM)
- Observe validator logs after the deploys
- Check current and LFB on validators and observer.
- Notice that some nodes stop replaying/validating blocks and end up with corrupted RSpace state
What version of F1R3FLY Node are you using?
rust/staging, commit.
What did you expect to see?
The network should remain synchronized after bridge-contract deploys.
All validators and the observer should be able to replay and validate blocks correctly, with the current block height staying consistent with or above the latest finalized block.
What did you see instead?
After a series of bridge-contract deploys, the network entered a serious desync state.
Validators
v1,v2, and the observer appear to have corrupted RSpace state and can no longer replay or validate blocks. At the same time,v3continues finalizing blocks above height2374.Current block height became lower than the latest finalized block on
v1andv2:Relevant observations from validator1.log:
Steps to reproduce the bug
What version of F1R3FLY Node are you using?
rust/staging, commit.