diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a988ec93..f86829f8d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,26 @@ item below is (or will be) written up in full in the in-development - A full `-reindex` no longer appends a second copy of the genesis block to the last block file. It indexes the genesis record `blk00000.dat` begins with, so the reindex writes nothing to the block files, which on a switch from Bitcoin Core are Core's (#931). - StartOS package: the service log now shows satd's INFO lines (sync progress, a reindex's phases, the steps of a shutdown). StartOS starts every service with `RUST_LOG=warn`, which satd honors, so the log carried warnings only. - `getrawaddrman` could report an address in a bucket Core's formulas would never give it. When two addresses derived the same slot, the second moved to the next free slot of the whole table, which from a bucket's last position is the next bucket's first. It now moves within its own bucket, as Core keeps every address in the bucket its formula gives; only a full bucket spills over. +- Signature operations are counted exactly as Bitcoin Core counts them, for the block sigop limit, mempool policy and block templates; the old counter scored an `OP_CHECKMULTISIG` after another signature opcode in P2SH and P2WSH scripts lower than Core. Parents admitted by `submitpackage` without a fee check carry their real sigop cost (#937). +- The Rust script engine treats 0x80, 0x81 and 0x83–0x86 as `OP_SUCCESSx` in tapscript, as Core does, instead of failing the leaf (`-consensus=rust`, `cpp-shadow`, `satd-chainstate-repair`) (#933). +- A block body that fails the context-free block checks no longer marks the block invalid, and an empty body is checked against the header's merkle root first (#936). +- On signet the signed coinbase commitment is rebuilt exactly as Core does; a missing block in a median-time-past window stops connecting with a storage error; malformed `nBits` count as zero work (#948). +- satd stops, as Core does, when it cannot read a coin from the chainstate, instead of treating the coin as spent and recording a valid block as invalid (#945). +- Header sync continues from the last header a peer sent, so a node on a competing best-header branch keeps syncing; the retarget start is found on the block's own branch, and targets above the proof-of-work limit are refused (#954). +- A block that fails to connect from storage is marked invalid instead of being retried by the IBD connector; the `headers` tip leaves the failed branch, and a chain with more work below it is activated at once (#957). +- IBD no longer stalls with a full `-maxahead` window; re-sent blocks do not count against it, local write failures do not penalize the sender, and block downloads go only to peers that advertise block service (#953). +- A block that arrives before its parent is kept only if it was requested and within fixed bounds (4 per peer, 32 MB in all, 5 minutes) (#941). +- Core's P2P message size and count limits apply, block replies are queued about 1 MB per peer, a peer that takes no data for 20 minutes is disconnected, user agents are capped at 256 bytes and sanitized, and refused BIP 157 requests disconnect (#939). +- `noban` and manual peers are never punished for misbehaviour; local and onion-inbound peers are disconnected but not banned, so 127.0.0.1 is no longer banned; `getpeerinfo` reports onion-inbound peers as `onion` (#944). +- Address announcements are rate-limited per peer as in Core (`addr_processed`, `addr_rate_limited` in `getpeerinfo`); dials in flight count against the outbound limit; manual connections sit outside it; `setban` of an onion host disconnects it (#952). +- A full mempool checks an incoming transaction's scripts before evicting for it and never evicts its parent; eviction scales with what it removes; lowering `maxmempool` on reload trims at once (#942). +- `getmempoolentry`, `getrawmempool`, `getmempoolinfo`, `gettxout` and `sendrawtransaction` answer as Core does; a restart keeps entry times and loads a large mempool in linear time (#946). +- RPC numbers use Core's formatting (16 significant digits, 8-decimal fee amounts); `submitblock` answers `duplicate-invalid` and throws -22/-25; `getblockfilter` uses Core's errors; a failed RPC password is answered after 250 ms (#955). +- Electrum scripthash status hashes the history in `get_history` order, which stops Electrum desktop reconnecting in a loop on addresses with pending transactions (#938). +- Esplora honours `?after_txid=` paging, answers 404/400 for out-of-range or misaligned block-txs pages, drops empty `witness` arrays, and `/fee-estimates` omits targets with no estimate (#943). +- Long Electrum and Esplora address-history reads run off the API runtime's workers, and `--electrumrequesttimeout` and `--esplorarequesttimeout` take effect (#947). +- The address, spend and filter indexes do not serve during or after an unfinished `backfillindex`; a reorg during a backfill no longer deletes live rows; reads during a flush wait for it (#958). +- `bitcoin.conf` treats `#` anywhere as a comment start and refuses an `rpcpassword` containing `#`, as Core does; trace logs no longer include RPC headers or parameters; the Tor control cookie read is bounded (#940). ## Releases diff --git a/docs/release-notes/0.6.1-pre.md b/docs/release-notes/0.6.1-pre.md index fe2909c25..24ddb4526 100644 --- a/docs/release-notes/0.6.1-pre.md +++ b/docs/release-notes/0.6.1-pre.md @@ -23,6 +23,26 @@ This file accumulates entries as changes land; it is not yet a cut release. of a container with no init. See [below](#a-stop-signal-during-a-rebuild-stops-it-cleanly). +- **Sigops are counted as Bitcoin Core counts them.** satd's counter scored + some P2SH and P2WSH scripts lower than Core for the block sigop limit, + mempool policy and block templates. See + [below](#sigops-are-counted-as-bitcoin-core-counts-them). +- **Header sync continues from a peer's last header.** A node whose best + header chain was on a competing branch could stop syncing headers from a + peer whose chain forked below it. See + [below](#header-sync-continues-from-a-peers-last-header). +- **Electrum desktop no longer reconnects in a loop.** The scripthash status + now hashes the history in the order `get_history` returns it, so an address + with a pending transaction shows it. See + [below](#electrum-status-hashes-follow-history-order). +- **A full mempool checks scripts before it evicts.** An incoming + transaction's scripts are checked before anything is evicted for it, and + eviction no longer scales with the size of the pool. See + [below](#a-full-mempool-checks-scripts-before-it-evicts). +- **Blocks that fail to connect from storage are marked invalid.** The IBD + connector no longer retries a failing stored block and restarts sync onto + it. See + [below](#blocks-that-fail-to-connect-from-storage-are-marked-invalid). ## Streaming Consumption API ### A refused watch add is reported to the client @@ -62,6 +82,60 @@ watched is no longer shed when the connection is at `streamwsmaxsubscriptions`. The event is an additive schema change (`NodeEvent` tag 32): an older SDK ignores it, and an older server never sends it. +## Consensus and validation + +### Sigops are counted as Bitcoin Core counts them + +satd now counts signature operations exactly as Bitcoin Core does: for the +block sigop limit, the mempool's per-transaction sigop cost, block templates, +`getblocktemplate`'s `sigops` field, and the policy rules that read +`tx.sigops_cost`. Earlier versions used rust-bitcoin's counter, which counted +some P2SH redeem scripts and P2WSH witness scripts lower than Core does: an +`OP_CHECKMULTISIG` that follows another signature opcode. + +Parents admitted through `submitpackage` without a fee check now carry their +real sigop cost and prevout data, so block templates count their sigops and +streaming spend matchers see their spends (#937). + +### The Rust script engine has Core's OP_SUCCESS set + +The Rust script engine now treats `OP_LEFT`, `OP_RIGHT`, `OP_INVERT`, +`OP_AND`, `OP_OR` and `OP_XOR` (0x80, 0x81, 0x83–0x86) as `OP_SUCCESSx` in +tapscript, as Bitcoin Core does. Before, it failed a tapscript leaf containing +one of them, where Core accepts the leaf. This affected `-consensus=rust`, +`-consensus=cpp-shadow` and `satd-chainstate-repair`, where the Rust engine's +verdict is used. The default `rust-shadow` mode decided with +libbitcoinconsensus and logged a shadow mismatch. + +The Rust engine now also runs the taproot rows of Core's script tests and +Core's taproot script-asset vectors, and exempts pay-to-anchor spends from the +upgradable-witness-program policy check as Core does (#933). + +### A block body that fails its own checks is not marked invalid + +Blocks that fail the context-free block checks (Core's `CheckBlock`: merkle +root, size, coinbase position, legacy sigop limit) are no longer marked +invalid, as in Bitcoin Core: such a body may simply not be the block its header +names. An empty block body is now checked against the header's merkle root +first. Before, a body that did not match its header could, in some arrival +orders, leave a valid block marked invalid until `reconsiderblock` was run. + +A block that extends the tip without its header having arrived first, and then +fails validation, is now recorded as invalid, so a second copy is refused +without being validated or stored again (#936). + +### Signet commitments, median-time-past and malformed nBits + +On signet, satd now rebuilds the signed coinbase commitment exactly as Bitcoin +Core does, so blocks whose witness commitment contains an empty +`OP_PUSHDATA1/2/4`, or ends in a push that does not parse, are judged as Core +judges them. + +If the block index is missing a block inside a median-time-past window, satd +now stops connecting with a storage error instead of computing the median from +fewer blocks, and does not mark the block invalid. The work of a block with +malformed `nBits` is now zero, as in Core (#948). + ## Chain, sync and reindex ### Every block that reaches the tip is reported @@ -82,6 +156,45 @@ The drain now emits the same `BlockConnected` event as every other connect that extends the tip, after the mempool has dropped the block's transactions (#900). The connector used during initial sync still emits none, as before. +### satd stops on a chainstate read error + +satd now stops, as Bitcoin Core does, when its chainstate database cannot read +a coin: an I/O error, a checksum mismatch, or a coin record that does not +decode. It logs `Error reading from database, shutting down`. Before, satd +treated such a coin as spent. If a valid block spent that coin, satd refused +the block and recorded it as invalid, and that record survived restarts. + +After such a stop, the next start replays the blocks after the last flush. If +the error comes back, check the disk and rebuild with `-reindex-chainstate` +(#945). + +### Header sync continues from a peer's last header + +After a full batch of headers, satd now asks the peer for more starting from +the last header that peer sent, as Bitcoin Core does. Before, a node whose +best header chain was on a competing branch could stop asking a peer whose +chain forked below it for headers, and so fail to sync to the chain with the +most work. + +The first block of a difficulty period is now found on the block's own +branch, so a block on a branch that forked before the period starts is judged +against that branch. Headers and blocks must also claim a target within the +network's proof-of-work limit, which on signet bounds the work behind each +header (#954). + +### Blocks that fail to connect from storage are marked invalid + +When a block failed validation while being connected from storage, during +initial sync or while catching up, satd did not record it as invalid. The IBD +connector retried the same block and restarted sync onto it until a better +chain appeared. satd now marks such a block invalid, as Bitcoin Core does, and +moves on. + +After any block fails to connect, the header tip `getblockchaininfo` reports +in `headers` now leaves that block's branch. A chain with more work made of +blocks below the failed one is now activated at once, instead of when the next +block arrives (#957). + ## Initial block download ### A block stuck with a slow peer is asked of another again @@ -111,6 +224,174 @@ A wait can still pass 15 seconds when the next peer asked is slow too. The rate at which a node stores blocks during initial sync is unchanged. That is set by the disk, which this release does not change (#539). +### The download window no longer stalls + +Initial sync no longer stalls when the `-maxahead` window is full while the +next block is not being downloaded. Blocks sent again for heights that are +already connected no longer count against the window, and a peer that is still +owed blocks only counts as active when it delivers blocks it was asked for. + +A local disk or database write failure while storing a downloaded block no +longer raises the sending peer's ban score. When following the tip, satd asks +one peer for each missing block instead of all of them. Block downloads, +including the AssumeUTXO background download, now go only to peers that +advertise block service, as in Bitcoin Core (#953). + +## P2P + +### Blocks that arrive before their parent are kept within bounds + +A block received before its parent is now kept only if the node asked for it, +and only within fixed bounds: valid proof of work, at most 4 MB, at most 4 +blocks per peer and 32 MB in all, for at most 5 minutes. Other such blocks are +dropped and the sender is asked for headers, which is how the node learns of +the missing parent. A peer's `ping` is no longer held while one of its blocks +waits for a parent (#941). + +### Core's message limits and a per-peer send budget + +satd now applies Bitcoin Core's limits to single P2P messages. A message over +4,000,000 bytes ends the connection, and an `inv` or `getdata` with more than +50,000 entries, `headers` with more than 2000 or `addr`/`addrv2` with more than +1000 counts as misbehaviour. + +Blocks a peer asks for with `getdata` are now served as the peer reads them, +with about 1 MB queued per peer, as with Core's default `-maxsendbuffer`. A +peer whose connection has taken no data for 20 minutes is disconnected, and +`disconnectnode` and bans now take effect on a peer that has stopped reading. +A peer's user agent is limited to 256 bytes and sanitized before it is logged +or reported in `getpeerinfo`. BIP 157 filter requests that Core refuses (an +unknown filter type, an unknown stop hash, a bad range, or any request while +filters are not served) now disconnect the peer instead of being ignored +(#939). + +### Misbehaving peers are handled as Core handles them + +Peers with the `noban` permission and manual connections (`addnode`, +`-addnode`, `-connect`) are never disconnected or banned for misbehaviour. A +peer on a local address, and every peer that connects through the node's Tor +hidden service, is disconnected but its address is not banned. + +Before, one such peer's misbehaviour could ban 127.0.0.1 for `-bantime` (24 +hours by default, kept across restarts), which refused every onion and local +P2P client without `noban` for that time. `getpeerinfo` now reports peers that +came in through the hidden service with `network: "onion"` (#944). + +### Address rate limits and outbound dial accounting + +satd now rate-limits the addresses each peer announces as Bitcoin Core does: +one to start, then one every 10 seconds. Peers with the `addr` permission and +answers to the node's own `getaddr` are exempt. `getpeerinfo` reports +`addr_processed` and `addr_rate_limited`. + +Outbound dials in flight now count against the automatic outbound limit, so a +reconnect pass no longer starts more dials than there are free slots. Manual +connections are outside that limit, as in Core: `addnode onetry` works on a +node with all its automatic peers connected (and answers null for a peer +already connected), added nodes are limited to 8 at a time and retried at +least once a minute, and bans stop automatic dials, onion hosts included, but +not manual ones. `setban` of an onion host now disconnects that peer (#952). + +## Mempool + +### A full mempool checks scripts before it evicts + +When the mempool is full, an incoming transaction's scripts are now checked +before anything is evicted to make room for it, and eviction never removes the +incoming transaction's own parent. Eviction now takes time in proportion to +what it removes rather than to the size of the pool, so admitting into a full +pool no longer holds the mempool lock for long. + +Lowering `maxmempool` with a config reload now trims the pool to the new size +at once; before, the node refused new transactions until the pool drained +below the new limit. The orphan pool's internal queue no longer grows with +every orphan ever received (#942). + +### Mempool RPCs match Bitcoin Core + +`getmempoolentry` and `getrawmempool true` report `vsize` rounded up, the +entry's real `height`, only direct parents in `depends`, `bip125-replaceable` +inherited from a signalling ancestor, and ancestor and descendant fees that +include `prioritisetransaction` deltas. `getmempoolinfo` gains `total_fee`, +reports `bytes` as total vsize and `usage` as estimated memory, and reports +`loaded: true` only once the saved mempool has finished loading. `gettxout` +now honours `include_mempool` (default true). `sendrawtransaction` refuses a +negative, unreadable or ≥ 1 BTC/kvB `maxfeerate` or `maxburnamount` instead of +silently dropping the limit, and answers `-27` for an already-confirmed +transaction without needing `-txindex`. + +A restart now keeps mempool entry times, so `-mempoolexpiry` keeps counting, +re-admits transactions that met the fee floor through +`prioritisetransaction`, and loads a large mempool in linear time. +Descendants of a transaction that conflicts with a new block now leave the +mempool with it, and confirmed transactions' priority deltas are cleared +(#946). + +## Electrum and Esplora + +### Electrum status hashes follow history order + +The scripthash status satd announces, on `blockchain.scripthash.subscribe` and +in notifications, now hashes the history in the order `get_history` returns +it, as the Electrum protocol defines. Before, an address with both confirmed +and unconfirmed transactions got a status that never matched its history, so +Electrum desktop dropped the connection again and again and did not show the +pending transaction until it confirmed. + +`get_history` and `get_mempool` now list mempool transactions with height 0 +before those with -1, and order transactions within a block by their position +in it. A subscription no longer gets its own status pushed back at the next +block (#938). + +### Esplora paging, empty witnesses and fee estimates + +`/address/:addr/txs` and `/scripthash/:hash/txs` now honour mempool.space's +`?after_txid=` cursor, so explorers that page with "load more" get the next +page instead of the first one again; an unknown cursor answers 422 +`after_txid not found`. A script's mempool transactions are listed in the +order the node admitted them. `/block/:hash/txs/:start_index` answers 404 +`start index out of range` at or past the end of the block, and 400 for a +start that is not a multiple of 25, instead of an empty page. Legacy inputs no +longer carry an empty `witness` array. + +`/fee-estimates` now leaves out the targets the node has no estimate for, and +answers `{}` during initial sync or right after startup instead of reporting +1 sat/vB for every target (#943). + +### Long address histories no longer hold up other requests + +Electrum and Esplora requests that read a long address history now run on a +blocking thread pool instead of the API runtime's worker threads, so one slow +request no longer stalls other Electrum connections, Esplora, events gRPC or +`/metrics`. `--electrumrequesttimeout` now takes effect (a request past it is +answered with an error and its connection closed), and Esplora's address +routes answer 503 at `--esplorarequesttimeout`. + +Paging an address with `/txs/chain/:last_seen_txid` or `?after_txid=` reads +only the requested page instead of the whole history, and status +notifications for subscribed addresses are computed off the threads that +connect blocks, once per batch of queued blocks. satd now logs at startup when +fewer Electrum connections than `--electrummaxconns` can fill the server-wide +`--addrindexsubscriptions` registry; see the Electrum chapter's "Subscription +limits" in the Operator Manual (#947). + +## Indexes + +### Indexes do not serve during or after an unfinished backfill + +The address, spend and compact block filter indexes no longer serve while a +`backfillindex` run is in progress or after one stopped part-way, as the +silent-payment index already did, and `NODE_COMPACT_FILTERS` is withdrawn for +that time. Electrum and Esplora subscription statuses refused during a +backfill are pushed once it ends. + +A reorg during a backfill can no longer delete rows the active chain wrote, +which could drop an entry from an address history, report a spent output as +unspent, or break the filter header chain. A read during a chainstate flush no +longer misses the newest blocks' index rows; it waits for the flush. A block +whose outputs are all unspendable and which spends nothing no longer loses its +filter and height rows when it replaces another block in a reorg (#958). + ## Observability ### No warning for every block early in initial sync @@ -217,6 +498,21 @@ package, satd's log therefore carried warnings only: no sync progress, none of the reindex phases or stop messages above. The package now runs satd with `RUST_LOG=info`, satd's default everywhere else. +## Configuration + +### `bitcoin.conf` comments, trace logs and the Tor cookie + +satd now reads `bitcoin.conf` comments as Bitcoin Core does: a `#` anywhere on +a line starts a comment, so a line such as `txindex=1 # keep it` is no longer +misread, and an `rpcpassword` line containing `#` is refused at startup with +Core's error. sat-cli reads the file the same way. + +`-loglevel=trace` no longer writes RPC request headers (including the +`Authorization` credentials) or RPC parameters to the log. With Tor control +enabled, satd reads at most 33 bytes from the cookie file the control port +names and refuses anything that is not a regular file, so a misbehaving +control port cannot make startup hang or run out of memory (#940). + ## RPC ### `getrawaddrman` keeps each address in its own bucket @@ -234,16 +530,56 @@ keeps every address in the bucket its formula gives. Only a full bucket sends it on to the next one with room. Core's table never needs that, since it evicts instead. +### Number formatting, `submitblock` and `getblockfilter` match Core + +Difficulty, network hash rate and verification progress are now printed with +Core's 16 significant digits, and `getnetworkinfo` fee floors and PSBT fees as +8-decimal amounts. `submitblock` answers `duplicate-invalid` for a block +already marked invalid, fills in a missing coinbase witness reserved value as +Core does, and throws -22 for undecodable hex and -25 for internal errors +instead of returning them as reject reasons. `getblockfilter` uses Core's +error codes and messages, `getnetworkhashps -1` uses the 144-block interval on +regtest, and a failed RPC password is now logged and answered after a 250 ms +pause, as in Core (#955). + ## Upgrade notes -- Drop-in upgrade from 0.6.0 for the changes so far: no reindex and no - configuration change. +- Drop-in upgrade from 0.6.0 for the changes so far: no reindex. The only + configuration change is for an `rpcpassword` containing `#` (below). - A client that treated silence after a watch add as success should handle `WatchAddRejected`; one built on a 0.6.0 SDK keeps working and ignores the event. - A full `-reindex` that 0.6.0 left unfinished runs again from genesis on the first 0.6.1 start, with or without `--reindex`. Reindexes 0.6.1 starts are continued. +- `bitcoin.conf`: a `#` anywhere on a line now starts a comment, as in Core. + An `rpcpassword` containing `#` is refused at startup; change the password, + or use `-rpcauth` (#940). +- satd now stops, as Core does, when it cannot read a coin from the + chainstate. A restart replays the blocks after the last flush; if the stop + recurs, check the disk and run `-reindex-chainstate` (#945). +- `gettxout` includes the mempool by default (`include_mempool=true`). + `getmempoolinfo.bytes` is total vsize, not serialized size + (`satd_mempool_bytes` and the `-maxmempool` budget still count serialized + bytes), and `loaded` is false until the saved mempool has finished loading. + `sendrawtransaction` refuses a negative, unreadable or ≥ 1 BTC/kvB + `maxfeerate` or `maxburnamount` instead of ignoring it (#946). +- `submitblock` throws error -22 for undecodable hex and -25 for an internal + error instead of returning a reject string. Difficulty, network hash rate + and verification progress are printed with 16 significant digits, and fee + fields in `getnetworkinfo` and PSBT RPCs with 8 decimals; clients that + compared these as strings may see different text (#955). +- Esplora `/fee-estimates` omits targets the node has no estimate for and + answers `{}` during initial sync. `/block/:hash/txs/:start_index` answers 404 + past the end of the block and 400 for a start that is not a multiple of 25 + (#943). +- The address, spend and compact block filter indexes do not serve during a + `backfillindex` run, or after one that stopped part-way until a re-run + completes; `NODE_COMPACT_FILTERS` is withdrawn meanwhile. A redundant run on + a complete index also stops serving until it completes (#958). +- Misbehaving `noban` and manual peers are never disconnected or banned. Peers + on local addresses and peers that come in through the onion service are + disconnected but not banned (#944). ---