Reporter (block_report API) replay panics on LFS-synced READONLY observers

[spreston8]

Symptom

A fresh readonly observer attached mid-shard via LFS sync emits panic clusters in tokio-rt-worker whenever the gRPC API server's transfer-enrichment path triggers block_report.trace(block):

WARN Deploy replay failed, returning empty events
  deploy_index=0  error="System runtime error: Unable to consume results of system deploy"
... (one per deploy in the block)
WARN Discarded 0 replay events during precharge error path

thread 'tokio-rt-worker' panicked at rholang/src/rust/interpreter/rho_runtime.rs:386:14:
called `Result::unwrap()` on an `Err` value:
  BugFoundError("Unused COMM event: replayData multimap has 471 elements left")

The reporter creates a fresh RhoReportingRspace over the same store, resets to the block's pre_state_hash, then for each deploy replays the precharge system deploy. consume_system_result returns None (no result on the precharge return channel), each deploy's replay aborts, the rspace's replayData multimap stays full of unconsumed COMM events, and create_checkpoint() panics on the dirty trace.

Bisect

Reproducible across PR branches and rust/staging baseline (panic counts vary slightly with timing; pattern is identical):

Binary	Test	observer1 panics	observer1 deploy failures
rust/staging (bf6a03d2)	test_observer_lfs_sync	2	12
feat/lfs-coverage (PR #507)	test_observer_lfs_sync	2-4	12-24
feat/bonding-additiveset (this branch)	test_observer_lfs_sync	2	12
feat/bonding-additiveset	test_bonding_validators (B5)	4	18

The bug pre-exists on rust/staging, not introduced by any of the bonding/LFS PRs in flight. Was hidden until today by a transient-observer scanner gap in the integration-test framework (now fixed on system-integration refactor/integration-test-framework).

Bug class (narrowed)

• Validators don't trigger the reporter (rejected by validator_opt.is_some() && !dev_mode → ReadOnlyRequired at block_report_api.rs:188)
• Genesis-attached READONLY does NOT panic — has played every block from genesis, full rspace state on disk
• LFS-synced READONLY DOES panic — rspace state was reconstructed via the LFS exporter/importer pair
• LFS-synced JOINER (which becomes a validator after bonding) also doesn't panic — once bonded, its reporter is rejected by the validator-check

So divergence is specific to the (LFS-imported rspace state) × (READONLY = reporter still invoked) combination. The reporter's per-block precharge replay needs some piece of state that LFS sync doesn't faithfully reproduce.

What we ruled out at the unit-test level

Tests added on feat/bonding-additiveset (casper/tests/batch1/multi_parent_casper_reporting_spec.rs):

• reporting_casper_should_behave_the_same_way_as_multi_parent_casper — single-node freshly-played setup PASSES
• reporting_casper_works_against_lfs_imported_rspace — basic LFS-import roundtrip (play one trivial deploy, export trie, import to fresh rspace, run reporter against fresh rspace) PASSES

So the bug needs more than the simple LFS-import path to trigger. Production conditions that the unit tests don't yet capture: hundreds of blocks of state churn, multi-block heartbeat cadence, multiple deployer keys, mergeable-channel store interactions across many blocks, concurrent reporter invocations.

Workaround

enable-reporting = false in defaults.conf (PR pending). Tests / deployments that need the reporter API can re-enable per-node via --api-enable-reporting=true once the underlying bug is fixed.

The CLI flag was extended to accept a value (--api-enable-reporting=true|false) so individual nodes can opt back in or out. Bare --api-enable-reporting keeps backwards-compatibility as true.

Repro

# system-integration (any binary that reaches Running on an LFS-synced READONLY observer):
poetry run shardctl test-reset
F1R3FLY_NODE_BINARY=<path-to-rust-staging-or-newer-binary> \
  poetry run pytest \
  integration-tests/test/tests/shared/test_observer_lfs_sync.py \
  --provider=subprocess --keep-running -v -s
# Result: 1 passed, 1 error (test body passes, autouse scanner catches reporter panic on observer1)

Inspect integration-tests/.subprocess-data/<session>/observer1.log for panicked at rholang/src/rust/interpreter/rho_runtime.rs and Deploy replay failed, returning empty events.

Suggested investigation

1. Add tracing::error! instrumentation in consume_system_result (casper/src/rust/rholang/runtime.rs::eval_system_deploy) printing the channel/pattern on Ok(None) to identify exactly which channel observer1 fails to consume from.
2. Restart observer1 with the instrumented binary (data dir survives), capture the failing channel.
3. Query both observer1 and a genesis-attached readonly for the same channel via gRPC to diff the state.
4. The diff identifies the missing/mismatched state → fix at exporter (rspace++/src/rspace/state/exporters/rspace_exporter_items.rs), importer (rspace++/src/rspace/state/instances/rspace_importer_store.rs), or post-import reset path.

Where the panic surfaces:

• Bug origin: reporter's reset-to-pre_state + per-deploy precharge replay path. consume_system_result returns None → ConsumeFailed → Deploy replay failed warn → eventual create_checkpoint() panic on dirty replayData
• casper/src/rust/reporting_casper.rs::RhoReporterCasper::trace
• casper/src/rust/rholang/replay_runtime.rs::process_deploy_with_cost_accounting (precharge phase)
• casper/src/rust/rholang/runtime.rs::eval_system_deploy → consume_system_result returns None
• rholang/src/rust/interpreter/rho_runtime.rs:386 (the unwrap that panics)

Co-Authored-By: Claude noreply@anthropic.com

[linear]

F1R-155

[Timekiller7]

Hi @spreston8 — we're hitting a related but distinct LFS-restore bug that seems to share the same root cause family, but manifests on a different code path than the reporter:

Symptom: A freshly-started observer (clean data dir) attached to a long-running shard (height ~5800) successfully receives the approved block at #5639, runs request_approved_state with min_height=5589, completes LFS sync and then never processes a single subsequent block. CLI shows the observer stuck at block=5639, finalized=5639 while validators advance past 5879+.

The observer logs are flooded with this error (~53,700 occurrences in 1h 43min, all on the same 2 incoming blocks from validators):

[ERROR] Error processing block c568440f63...:
        KvStore error: Invalid argument: DAG storage is missing hash 3ae31924a0...
[ERROR] Error processing block f20c8fe076...:
        KvStore error: Invalid argument: DAG storage is missing hash 3ae31924a0...
[ERROR] [RootRepository] validateAndSetCurrentRoot FAILED:
        Blake2b256Hash(875d3b30e7...) not in roots store

The missing hash 3ae31924a0 is block #5588 — exactly one block below min_height=5589 of the LFS-restore range. The missing roots (875d3b30e7, a0f257f9c8) are post-states of blocks #5581 and #5582 — even deeper. So the regular Casper block validation pipeline tries to walk parents from incoming #5640+ blocks back to the approved state, and the walk goes below the LFS window's min_height. There it hits hashes that LFS never imported, and the validation cycle fails indefinitely.

Differences from your issue

• Not the reporter — enable-reporting = false would not help here. The error happens on the normal block_processor path during incoming-block validation
• No panic — observer keeps running, just retries the same blocks ~26,600 times each, never advancing
• Triggered without any gRPC API calls, purely from validator broadcasts

Possibly the same underlying class of bug.

So, should this be filed as a separate issue, or do you want to track it as a second symptom under this issue? Happy to file separately with full repro steps and ~1h logs if useful.

v0.4.15, rust/dev branch. 3 validators + 1 observer. Located on separate machines. Observer started with clean state against a network at height ~5800.

[spreston8]

This issue has been migrated to F1R3FLY-io/f1r3node-rust#48 as part of the deprecation of this repository. Please continue tracking the work there.