perf: block replay pipeline bottleneck limits sustained deploy throughput

[spreston8]

Summary

Load testing on a 3-validator shard (FTT=0.1, heartbeat 2s) shows sustained deploy throughput tops out at ~5 deploys/sec before finalization degrades. The bottleneck is the block replay pipeline — each peer block takes 500-1500ms to validate, even for trivial deploys.

Current Performance (after PR #473 optimizations)

Deploy Rate	Inclusion p50	Finalization p50	LFB Rate
1/sec	1.7s	4.7s	40.3 blk/min
5/sec	25.7s	49.7s	9.4 blk/min
10/sec	58.9s	95.3s	3.8 blk/min

Detailed Merger Breakdown (high load, avg per merge)

Step	Time	Notes
branches (depends O(D²))	349ms	Pre-computed derived sets (was 731ms)
conflicts_map (O(B²))	860ms	Cached branch data (was 2809ms)
rejection_options	0ms
channel_reads (storage)	1ms
combine_changes	1ms
compute_trie_actions	~2ms
apply_trie_actions	~18ms

The conflicts_map (860ms) and branches (349ms) remain the dominant costs. Both are O(N²) pairwise comparisons — the caching in PR #473 reduced per-call cost but the quadratic iteration count remains.

Replay Breakdown (high load, avg per block)

Step	Time
user_deploys replay	3962ms
system_deploys	114ms
checkpoint	44ms
reset	3ms

The user_deploys replay (3962ms) for trivial @N!(N) deploys indicates the cost is in runtime spawn and state management, not Rholang execution.

Root Cause

Feedback loop: more load → more blocks → larger unfinalized DAG scope → slower conflict detection (O(N²)) → slower replay → finalization falls behind → more blocks accumulate.

Already Optimized (PR #473)

• Removed adaptive deploy cap
• Enabled deploy heartbeat wake
• Bonds computation cache (DashMap, 64 entries)
• Block index cache 32 → 128
• Self-propose-cooldown 15s → 5s
• Conflict detection caching (2.9x faster merge)
• Prometheus metrics for full merger pipeline breakdown

Remaining Optimization Opportunities

1. Reduce O(N²) to O(N) for conflict detection

The compute_relation_map function compares every pair. For non-conflicting deploys on disjoint channels, all comparisons return false. A channel-indexed approach could skip comparisons for items with no shared channels. Attempted but reverted due to correctness issues with branch grouping — needs careful design that preserves exact branch structure.

2. Reduce bonds validation cost

compute_bonds() spawns a Rholang runtime and executes a PoS contract query per unique post_state_hash. The DashMap cache helps for repeated hashes but misses at high load (unique state per block). The proper fix: read bonds directly from the trie state without spawning a runtime.

3. Reduce replay runtime spawn overhead

spawn_replay_runtime() clones RSpace state per block (10-50ms). Potential: runtime pooling or reset-in-place between sequential replays. Significant refactor with consensus safety implications.

4. Suppress empty heartbeat blocks

~54% of blocks are empty. Each still requires full replay by peers. Skipping proposals when a validator has no deploys and no meaningful parents would eliminate wasted replay work.

Reproduction

DEFAULT_IMAGE=f1r3flyindustries/f1r3fly-rust-node:local \
F1R3FLY_RUST_IMAGE=f1r3flyindustries/f1r3fly-rust-node:local \
poetry run pytest integration-tests/test/test_load.py -v --log-cli-level=INFO --timeout=900

Related

• PR perf: remove adaptive deploy cap, bonds cache, deploy wake #473 — performance optimizations (adaptive cap, caching, conflict detection)
• system-integration PR consume and peek cannot be joined using & operator #49 — load test, convergence test, test improvements

Co-Authored-By: Claude noreply@anthropic.com

[linear]

F1R-145

[jeffrey-l-turner]

This issue has been migrated to F1R3FLY-io/f1r3node-rust#24 as part of the deprecation of this repository. Please continue tracking the work there.

[spreston8]

Update: Async ISpace + Parallel Rholang Evaluation (PR #482)

The perf/rspace-concurrency branch (PR #482) delivers significant improvements to the replay pipeline through async ISpace, true parallel Rholang evaluation via tokio::spawn, and two bug fixes (ChargingRSpace identity comparison + peek join re-issue).

Updated Load Test Results (3 validators, FTT=0.1)

Deploy Rate	Inclusion p50	Finalization p50	LFB Rate	Per-deploy replay
1/sec	1.6s	4.4s	31.2 blk/min	352ms
5/sec	21.2s	66.7s	6.8 blk/min	461ms
10/sec	63.1s	86.2s	3.1 blk/min	441ms
burst (32)	16.1s	24.6s	9.3 blk/min	—

Improvement vs Previous (before async ISpace)

Metric	Before	After	Change
High load precharge	426ms	290ms	-32%
High load refund	194ms	151ms	-22%
High load per-deploy	620ms	441ms	-29%
High load finalization p50	95.3s	86.2s	-10%

Isolated Benchmark (cost_accounting_perf_spec, release)

Metric	Before	After	Change
Per-deploy total	341ms	223ms	-35%
Per-op cost	0.62ms	0.40ms	-35%

What Changed

• ISpace trait fully async (#[async_trait]) with per-channel two-phase locks
• tokio::spawn for Rholang parallel composition (true parallelism matching Scala's parTraverse)
• ReplayRSpace per-channel locks for concurrent replay safety
• Fixed ChargingRSpace identity comparison porting bug (persistent produce refunds)
• Fixed peek join semantic bug (non-persistent peeked data preservation)
• Replay cost determinism verified across 10,000 randomly generated contracts

Remaining Bottlenecks (unchanged)

Items 1-4 from the original issue remain — conflict detection O(N²), bonds validation, runtime spawn overhead, and empty heartbeat blocks. The async ISpace work addressed the per-operation cost within replay but not the merge/conflict detection pipeline.

Co-Authored-By: Claude noreply@anthropic.com