feat: in-flight transaction tracker for the txpool and batch sealer

[bronxyz]

Summary

• Add InFlightTracker: the txpool records hashes sealed into a batch but not yet observed mined, so the next sealing round skips them instead of re-sealing a stuck backlog batch after batch. Marks are armed per role behind a capability handle (SealMarks / ForwardMarks), released by a reconcile against the pending sub-pool plus a backoff-gated sweep, and snapshot/restored across restarts through a versioned MarkBackup.
• On quorum the sealer now marks the batch in flight instead of evicting it from the pool, so sealed transactions stay pending and RPC-visible until execution drains them.
• Defuse two latent traps ahead of the rework: the diagnostic nonce span overflowed at u64::MAX (overflow-checks + panic=abort turned one crafted batch into a whole-network halt), and the workspace metrics facade is pinned to 0.24 so a crate adopting 0.23 cannot bind a second global whose macros silently no-op.

Stack 1/9 of the txpool in-flight tracker and observer-forwarder series. The builder pipeline and the forwarder that also consume the tracker land in later PRs; this one scopes to the pool and sealer.

Surface areas touched

• Consensus protocol (primary / worker / network / state-sync)
• Execution / EVM
• JSON-RPC (eth_*, rayls_*, faucet)
• Middleware (orchestrator / processor / bridge)
• Infrastructure (types / storage / config / network-cli)
• On-chain contracts (rayls-contracts/)
• Operations (etc/, scripts, Docker, compose)
• CI / build (.github/workflows/, Makefile)
• Documentation only (doc/, in-crate READMEs, root docs)
• Tests only

Breaking / compatibility

None. No wire or storage format changes; the mark backup file is new and only read when present.

Test plan

• nonce_span_saturates_at_the_max_nonce fails on the pre-fix span arithmetic.
• Failing-first regression that a sealed batch is not re-proposed while in flight; four builder tests retargeted from drain-on-quorum to stay-pending-and-marked.
• Proptest drives concurrent tracker writers and checks marked - reconcile - ttl - clear equals the live set and the gauge.
• make check on the stack tip; CI (fmt, clippy, workspace tests) on this branch.

[github-actions]

Security Scan - Code

Severity: HIGH, CRITICAL

No vulnerabilities found

View scan results

Report Summary

┌─────────────────────────────────────────────┬────────────┬─────────────────┬───────────────────┐
│                   Target                    │    Type    │ Vulnerabilities │ Misconfigurations │
├─────────────────────────────────────────────┼────────────┼─────────────────┼───────────────────┤
│ Cargo.lock                                  │   cargo    │        0        │         -         │
├─────────────────────────────────────────────┼────────────┼─────────────────┼───────────────────┤
│ crates/testing/fuzz-targets/Cargo.lock      │   cargo    │        0        │         -         │
├─────────────────────────────────────────────┼────────────┼─────────────────┼───────────────────┤
│ crates/testing/fuzz-targets/fuzz/Cargo.lock │   cargo    │        0        │         -         │
├─────────────────────────────────────────────┼────────────┼─────────────────┼───────────────────┤
│ etc/state-sum/Cargo.lock                    │   cargo    │        0        │         -         │
├─────────────────────────────────────────────┼────────────┼─────────────────┼───────────────────┤
│ etc/tps/package-lock.json                   │    npm     │        0        │         -         │
├─────────────────────────────────────────────┼────────────┼─────────────────┼───────────────────┤
│ rayls-contracts/package-lock.json           │    npm     │        0        │         -         │
├─────────────────────────────────────────────┼────────────┼─────────────────┼───────────────────┤
│ etc/chaos-network/Dockerfile                │ dockerfile │        -        │         0         │
├─────────────────────────────────────────────┼────────────┼─────────────────┼───────────────────┤
│ etc/docker-network/Dockerfile               │ dockerfile │        -        │         0         │
├─────────────────────────────────────────────┼────────────┼─────────────────┼───────────────────┤
│ etc/docker-replay/Dockerfile                │ dockerfile │        -        │         0         │
└─────────────────────────────────────────────┴────────────┴─────────────────┴───────────────────┘
Legend:
- '-': Not scanned
- '0': Clean (no security findings detected)

[github-actions]

Claude finished @bronxyz's task in 3m 48s —— View job

---

PR Review: feature/txpool in-flight tracker core

Tasks:

• Gather context and identify trigger
• Read changed files (in_flight/mod.rs, marks.rs, txn_pool.rs, batch-builder/lib.rs)
• Check status of all prior review findings
• Identify any new issues in latest commit
• Post final review

---

Overview

The latest commit (689f3fc) correctly wires the tracker into the txpool and batch sealer: build_batch skips in-flight hashes, the sealer marks on quorum via the armed SealMarks capability, and reconcile_in_flight releases marks when the pending sub-pool drains them. The in-flight skip logic in batch.rs:89-91 is sound — the continue (rather than mark_invalid) preserves descendants for sealing, and the no_updates() snapshot prevents nonce gaps from new arrivals mid-iteration.

None of the previously flagged issues have been resolved. The same two merge-blockers remain, plus a new severity upgrade on the stale-comment finding.

---

Bug: MarkBackup::version is never validated on restore (merge-blocker — still present)

consume_stash checks the role but ignores the version field entirely (mod.rs:331-356):

fn consume_stash(guard: &mut Inner, role: MarkRole) -> usize {
    let Some(backup) = guard.pending_restore.take() else { return 0 };
    if backup.role != role { ... return 0; }
    // ← version is never read; any backup restores regardless of MARK_BACKUP_VERSION
    let now = Instant::now();
    for mark in backup.marks { ... }
}

MARK_BACKUP_VERSION is defined in marks.rs:174 so "a backup written by a prior version is rejected instead of misread," but the code never consults it. A node upgrading from an older build silently applies stale backup semantics.

Fix: after the role check, add if backup.version != MARK_BACKUP_VERSION { warn!(…); return 0; }.

Fix this →

---

Bug: release_dropped is defined but never called (merge-blocker — still present)

InFlightTracker::release_dropped (mod.rs:214) exists only in tests. Nothing in the EVM execution path calls it when a nonce_too_high drop occurs, even though batch_tracker.rs already counts total_nonce_too_high. The consequence is a 60-second TTL lag before such marks release and the transaction becomes re-sealable.

If this scenario is live (it is — batch_tracker.rs:351 increments total_nonce_too_high), the method needs a caller in the EVM execution path. If it is not yet reachable in this stack slice, the method and its counter should be removed and re-added in the PR that wires the EVM execution path, to avoid a misleading public API with no callers.

---

Bug: update_canonical_state_async added but never wired (severity upgrade — previously "stale comment")

This commit added update_canonical_state_async (txn_pool.rs:241-289) to spawn a blocking task and avoid runtime stalls under load. The comment at txn_pool.rs:328-329 describes exactly that intent, but process_canon_state_update still calls the synchronous update_canonical_state immediately after:

// async pool update spawned as blocking task to prevent RPC degradation     ← describes async
// this is safe because canonical stream updates don't require immediate deduplication
self.update_canonical_state(         ← synchronous; holds the pool write-lock inline
    tip.sealed_block(), ...
);
self.reconcile_in_flight();          ← also synchronous

process_canon_state_update runs inside spawn_critical_task("canonical txn pool", async move { … }), an async context. Calling the synchronous pool update (which acquires the pool write lock) from inside an async task blocks the runtime thread for the duration of the lock. The async version exists specifically to fix this — it was never connected.

Fix: replace self.update_canonical_state(…) with self.update_canonical_state_async(…) at txn_pool.rs:330 and verify reconcile_in_flight is still safe to call from the async context (it checks is_empty() first and calls pending_transactions() under the pool read lock; spawning it as a separate blocking task or after the async update settles may be needed depending on ordering guarantees).

Fix this →

---

Design: SealMarks::mark hardcodes anchor: 0 (prior review — still present)

marks.rs:87:

pub fn mark(&self, hashes: impl IntoIterator<Item = TxHash>) {
    self.tracker.track_all(hashes, Mark::Sent { at: Instant::now(), anchor: 0, attempts: 0 });
}

Today this is harmless because DuePolicy::ttl sets min_anchor_advance: 0. If a future policy change requires a block advance before TTL release, the hardcoded anchor=0 makes every mark immediately eligible regardless of how many blocks have passed.

Fix: add an anchor: u64 parameter to SealMarks::mark and update the call site in lib.rs:335 to pass this.last_canonical_update.number.

Fix this →

---

Design: stash_restore silently drops a prior stash (prior review — still present)

mod.rs:327-329:

pub fn stash_restore(&self, backup: MarkBackup) {
    self.inner.write().pending_restore = Some(backup);
}

A second call before the next arm silently discards the first. A warn! on overwrite, or debug_assert!(self.inner.read().pending_restore.is_none()), would surface the misuse at development time.

---

Perf: reconcile_in_flight is O(all pending txs) (prior review — still present, not blocking)

txn_pool.rs:359-361:

let pending: B256Set =
    self.pool.pending_transactions().iter().map(|tx| *tx.hash()).collect();

The early return on empty in-flight is a welcome guard, but the non-empty path still clones all Arc<PoolTxn> pointers and acquires the pending-pool read lock on every canonical update. A pool helper that copies only hashes would reduce per-block overhead. Not blocking, but worth tracking.

---

Nit: Missing #[must_use] on arm_sealing / arm_forwarding (prior review — still present)

Dropping the return value of arm_sealing wipes the prior round's marks and discards the write capability silently. #[must_use] on both methods (mod.rs:124, mod.rs:140) catches this at compile time.

---

Nit: on_released calls gauge.sub(0) in no-op paths (prior review — still present)

mod.rs:296: self.metrics.gauge.sub(delta as i64) is unconditional; when delta == 0 it's a no-op metric call. The existing if delta > 0 guard could equally cover the gauge call.

---

Summary

Severity	Finding	Status
Bug	MarkBackup::version never checked — stale backups restored silently	Still present
Bug	release_dropped never called — nonce-drop marks only clear via 60s TTL	Still present
Bug	update_canonical_state_async added but process_canon_state_update still calls sync version — blocks async runtime	Severity upgrade
Design	SealMarks::mark hardcodes anchor: 0 — implicit dep on min_anchor_advance: 0	Still present
Design	stash_restore silently drops a prior stash	Still present
Perf	reconcile_in_flight is O(all pending txs) on every canonical update	Still present
Nit	arm_sealing/arm_forwarding missing #[must_use]	Still present
Nit	on_released calls gauge.sub(0) in no-op paths	Still present

The wiring in this commit is correct. I'd want the version-validation bug, the release_dropped gap, and the sync-vs-async call site resolved before merging. The anchor=0 issue is lower priority but worth fixing while the sealing-mark code is being touched.