Electrum/Esplora: run index reads off the runtime workers, make request timeouts fire, page /txs/chain from the cursor

[bkeroack]

Problem

Electrum and Esplora do their index work synchronously (RocksDB and flat-file reads), and several places ran that work where it held a runtime worker, or repeated it without need:

• Electrum request timeout could never fire. handle_connection wrapped process_request in tokio::time::timeout(request_timeout, async { process_request(..) }). The inner future has no await point, so the dispatch ran to completion on its first poll. A long request (a scripthash.subscribe batch over long histories, get_balance, get_first_use) held an API-runtime worker for as long as it took; with --api-threads at its small-box default of 2, two such requests stalled every Electrum connection, Esplora, events gRPC and /metrics.
• Esplora address routes (/address/:a, /txs, /txs/chain[/:last], /txs/mempool, /utxo and the scripthash forms) did the same inline in the async fn, so TimeoutLayer could not answer before the read ended either.
• /txs/chain/:last_seen_txid read and resolved the whole history for every page and then searched it for the cursor, so walking an N-transaction history 25 at a time cost O(N²) reads. The confirmed part of /txs?after_txid= (added in Esplora: page /txs with after_txid, 404 past a block's last page, omit empty witness and unknown fee targets #943) shared that path.
• Status recomputes on the core runtime. Every block made the chain notifier recompute the status of every subscribed scripthash, and every mempool event touching a subscription made the mempool index task recompute those; each recompute reads a history from the index. Both tasks run on the core runtime (tokio::spawn in run), so heavy subscriptions held a core worker on every block. Queued block events (IBD, a reorg) each got a full pass, and each mempool event built the set of every subscription to filter its touched scripthashes.
• Subscription caps. --electrummaxsubsperconn (1000) and the server-wide --addrindexsubscriptions (10000, shared with Esplora's address SSE streams) are not coherent: ten connections at their own cap fill the registry while 54 of 64 connection slots are free, after which every other client's subscribe is refused. Nothing documented or logged this.

Fix

• Electrum (electrum-proto/src/server.rs): each request runs under spawn_blocking with the timeout around the join. Past --electrumrequesttimeout the client gets the existing request timed out error and the connection is closed (the handler cannot be interrupted and the connection's subscriptions are inside it). The connection task keeps its --electrummaxconns slot until the work ends, so reconnecting after every timeout cannot run more handlers at once than the connection cap. block_in_place in the tweaks handler runs the closure inline on a blocking-pool thread, so it keeps working; its comment is updated.
• Esplora (esplora-handlers/src/blocking.rs, handlers/address.rs): every address and scripthash route runs its reads on the blocking pool under a work permit held until the work ends, even after its request timed out. The permits are sized from --esploramaxconns (none when it is 0, as for the request cap), so retries after timeouts cannot pile up running reads.
• Paged confirmed history (Store::addr_rows_desc, AddressIndex::confirmed_txs_newest_first): the store reads a script's address rows newest first from an ordinal down without resolving them. RocksDB seeks backwards over both address families in total order and checks the 16-byte key prefix on every key; the CoinCache merges its unflushed rows in exactly as the flush will write them; a run never ends between two rows of one transaction. The address index starts at the end of the cursor's block (block-mates sort by txid on either side of the cursor) and stops once the page holds only settled blocks. Output and order are unchanged: height descending, then Txid descending, an unknown cursor [] on /txs/chain/:last and 422 on ?after_txid=. The trait default keeps the whole-history read for other implementations. Esplora: page /txs with after_txid, 404 past a block's last page, omit empty witness and unknown fee targets #943's /txs rules are kept: a cursor is looked up in the confirmed history first (one paged read), and a mempool txid is dropped from the mempool part when the index has a page after it, which for an unconfirmed transaction costs one ordinal lookup.
• Notifier (node/src/index/address/notifier.rs, mempool.rs): the recomputes and the mempool task's lagged resync run under spawn_blocking; the tasks still take events in order and wait for each recompute, so index mutation and notification keep their order. The chain notifier drains every queued block event before a pass and runs one pass for them (the pass reads the index as it then stands). Mempool events look up only their touched scripthashes (SubscriptionRegistry::subscribed) and skip the thread hop when none is subscribed. The two tasks stay on the core runtime: block connection emits the events they consume there, and rpc/access.rs / emit_chain_event document the notifier as a core-runtime consumer of those events, so moving the consumers was left alone.
• Subscription limits: no default changes. satd logs at Electrum startup how many connections at the per-connection cap fill the registry when that is fewer than --electrummaxconns, and the value of --addrindexsubscriptions that lets every connection reach its own cap. The Electrum chapter gets a "Subscription limits" section; the config reference and the Esplora chapter are updated for the timeout, permit and paging behaviour.

Tests

All fail without the fix (run on the unfixed code first, then each guard reverted):

• electrum-proto server::electrumbounds_tests
  • request_past_the_timeout_is_answered_with_an_error_and_the_connection_closed: fails without the fix: yes (no timeout error; the slow answer arrives after 1.5 s).
  • a_slow_request_does_not_hold_up_other_connections (one worker; blocking-socket client on its own thread): fails without the fix: yes (a ping waited 1.300s behind another connection's slow request).
  • a_timed_out_request_keeps_its_connection_slot_until_its_work_ends: fails without the fix: yes; with only the wait-for-the-work loop removed: yes (the second connection is admitted instead of refused at capacity, so no capacity answer arrives).
• esplora-handlers handlers::address::electrumbounds_tests (a real router over an empty regtest chain and an index whose history read sleeps):
  • a_slow_address_read_does_not_hold_up_other_requests: fails without the fix: yes (/blocks/tip/height waited 1.303s).
  • an_address_read_past_the_request_timeout_is_answered_503_on_time: fails without the fix: yes (200 after the read instead of 503).
  • a_timed_out_read_keeps_its_work_permit_until_it_ends: fails without the fix: yes; with only the permit removed: yes (a second read started while the first still held the only permit).
• node index::address::lookups::electrumbounds_tests
  • pages_match_the_whole_history_from_every_cursor: every cursor (and four that are not in the history) at limits 1, 2, 7, 25 against the trait's whole-history default, over blocks where several of the script's transactions sort against their block position. With the settled-block guard loosened: fails.
  • a_cursor_outside_the_confirmed_history_is_none.
  • a_deep_page_resolves_only_the_rows_near_it: 3000-transaction history; with the paged override removed: fails (read 3000 rows).
• node storage::rocksdb_store::electrumbounds_tests: reads_both_families_newest_first_within_the_prefix, runs_stop_after_min_txs_and_continue_below (neighbouring prefixes on both sides; with the never-split rule removed: fails).
• node storage::coin_cache::electrumbounds_tests::pending_rows_merge_as_the_flush_will_write_them: new, interleaved, extra-vout, removed and put-and-removed pending rows; every run size before the flush equals the walk after it. With the remove filter or the floor filter removed: fails.
• node index::address::notifier::electrumbounds_tests: a_block_recompute_does_not_run_on_the_runtime_thread, a_mempool_recompute_does_not_run_on_the_runtime_thread (fail without the fix: yes, a 50ms sleep took 800ms), queued_block_events_share_one_recompute (fails without the fix: yes, five passes for five queued events).
• electrum-proto config::electrumbounds_tests for the registry-fill helper.
• Existing: electrum-proto, esplora-handlers, node (index::address, storage), node-index lib suites; regtest esplora / electrum; e2e electrum / esplora (incl. Electrum: hash scripthash status in history order #938's electrum_status and Esplora: page /txs with after_txid, 404 past a block's last page, omit empty witness and unknown fee targets #943's esplora_paging).

Notes

• Not done: recompute only the scripthashes a block touched. A block also changes the status of scripthashes it never touches: a mempool child that pays address A and spends a parent the block confirms goes from height -1 to 0, and the reverse on a disconnect. Narrowing needs the block's touched set plus the mempool children of its transactions, and ChainEvent carries no block. That is a design change; this PR keeps the full pass and moves it off the core workers.
• Not done: a default change for the subscription caps (STABILITY_POLICY: defaults are part of the contract). Raising --addrindexsubscriptions to electrummaxconns × electrummaxsubsperconn (64,000) costs a few KB of channel per subscribed scripthash and a history read per subscription per block; that trade-off is the operator's or a later release's call.
• Not done: capping full-history reads. blockchain.scripthash.subscribe, get_balance, get_first_use, Esplora /address/:a stats and /utxo still read the whole history per request (now off the runtime workers, bounded by the connection/permit caps and answered on time). A cap would change answers for large scripts (Blockstream's electrs refuses /utxo past its utxos_limit), which is a behaviour decision.
• Base is now release/0.6 (Electrum: hash scripthash status in history order #938 and Esplora: page /txs with after_txid, 404 past a block's last page, omit empty witness and unknown fee targets #943 merged); no longer stacked.
• The Electrum timeout answer keeps its existing shape (code 1, id: null).

Release notes

Electrum and Esplora requests that read a long address history no longer hold the API runtime's worker threads: they run on a blocking thread pool, so one slow request no longer stalls other Electrum connections, Esplora, events gRPC or /metrics. --electrumrequesttimeout now takes effect (a request past it is answered with an error and its connection closed), and Esplora's address routes answer 503 at --esplorarequesttimeout. Paging an address with /txs/chain/:last_seen_txid or ?after_txid= reads only the requested page instead of the whole history, and status notifications for subscribed addresses are computed off the threads that connect blocks, once per batch of queued blocks. satd now logs at startup when fewer Electrum connections than --electrummaxconns can fill the server-wide --addrindexsubscriptions registry; see the Electrum chapter's "Subscription limits".

🤖 Generated with Claude Code