txmanager: stale nonce after fast inclusion causes unrecoverable "nonce too low" cluster failure

[zktaiga]

Summary

When a transaction is included quickly and the next SendTransaction call fetches the account nonce from an RPC node that hasn't caught up yet (e.g. behind a load-balanced endpoint), the tx manager builds the next tx with an already-consumed nonce. The tx can never land, and the error path gives up instead of refreshing the nonce, marking the cluster update as failed.

Timeline of a failure

1. During a commit batch, the tx for cluster A is sent with nonce N and is mined within a few seconds.
2. The updater moves on to cluster B. SendTransaction calls PendingNonceAt, which is answered by a node that hasn't yet reflected cluster A's tx, and returns a stale N.
3. Cluster B's tx is signed with the duplicate nonce N and sent via MEV RPCs, which silently drop it — nonce N is already consumed on-chain.
4. The tx manager waits out the full pending_timeout_blocks window, then falls back to the public RPC, which rejects the resend with nonce too low (next nonce N+1).
5. The nonce-too-low branch calls findMinedReceipt, which only checks this call's own published txs — none were mined (the nonce was consumed by cluster A's tx) — so it hard-fails without refreshing the nonce.
6. Cluster B is logged as Cluster failed and updater_clusters_total{outcome="failed"} is incremented. Subsequent clusters proceed normally with fresh nonces, and cluster B is updated on the next commit cycle.

Root cause

SendTransaction (txmanager/manager.go) trusts a single PendingNonceAt read per call and has no recovery for a cross-call nonce collision: the nonce-too-low branch either returns a receipt for one of its own published txs or gives up. A duplicate-nonce tx also wastes the full MEV pending timeout before the error surfaces.

Suggested fix

Either (or both):

1. Track the last used nonce in TxManager and use max(PendingNonceAt, lastUsedNonce+1) — prevents the duplicate-nonce tx from being built at all.
2. On nonce too low with no own-tx receipt found, re-fetch the nonce and retry within the existing MaxAttempts loop instead of returning an error.

Alternatively/additionally, classify cross-call nonce collisions as skipped rather than failed, since the updater is guaranteed to retry the cluster on the next commit cycle — reserving failed for conditions that need intervention.