net: hybrid handshake outcomes, close causes, 3-strike fallback and load shedding (#184 part 5) #206

Part 5 of 9 · #184 · Size L · Base 1.x.x · Needs part 4 (#199) · Unblocks part 6 (#209), part 7 (#212), part 9 (#222)

Consolidates work items F1 (#192), F2 (#206), F3 (#207) and F4 (#208), each formerly its own sub-issue. Their full text is below as sections; the closed issues point here.

Goal

Make every outcome visible and every failure safe: a first-cause-wins close cause for each disconnect, exact per-direction outcome counters with failure rings and endpoint keys, the 3-strike fallback to plain v2, and load shedding under inbound floods.

Work items

In suggested commit order. Each item is a section below.

• F1 · net: record a first-cause-wins close cause for every disconnect
• F2 · net: count hybrid handshake outcomes per direction, with failure rings and endpoint keys
• F3 · net: fall back to plain v2 after three counted failures to an endpoint
• F4 · net: stop offering ML-KEM to inbound peers during handshake floods

Starting early

• F1 has no PQ dependency. It can start now, as this part's first commit.
• F2 needs E3 (net: hybrid v2 handshake: key recipe, record parser, negotiation with key confirmation, and spec (#184 part 4) #199) from part 4, and F3 and F4 build on F2.

F1 · net: record a first-cause-wins close cause for every disconnect

Needs nothing within #184.

Goal

Know why each connection closed (peer EOF, peer reset, send error, timeout, or our own decision), so the fallback (F3) counts only failures the peer side caused. This has no PQ dependency and can land first.

Context

• recv() returning 0 or an error closes the connection at src/net.cpp:2193-2212; send errors at :1660-1666; InactivityCheck timeouts at :2002,2049,2215; disconnect processing at :1907. Disconnected nodes stay in memory until their references drop (:1962-1976).
• net_processing.cpp sets fDisconnect = true directly for operator, eviction, feeler and protocol decisions, for example at :3664,3690,5758.

Scope

enum class NodeCloseCause : uint8_t {
    NONE = 0, LOCAL = 1, PEER_EOF = 2, PEER_RESET = 3, SEND_ERROR = 4, TIMEOUT = 5,
};
// CNode:
std::atomic<NodeCloseCause> m_close_cause{NodeCloseCause::NONE};
void RequestDisconnect(NodeCloseCause cause = NodeCloseCause::LOCAL) noexcept; // CAS from NONE, then fDisconnect = true
void CloseSocketDisconnect(NodeCloseCause cause = NodeCloseCause::LOCAL);

• The first cause wins: RequestDisconnect sets the cause with a compare-exchange from NONE, then sets fDisconnect. Cleanup calls never overwrite an earlier EOF or reset.
• Convert every production fDisconnect = true writer in net.cpp, net_processing.cpp and elsewhere to RequestDisconnect(cause). fDisconnect stays for existing readers.
• An unexplained fDisconnect with cause NONE counts as local.
• No protocol behavior changes.

Tests

• close_cause_first_wins: EOF, reset, send error, timeout and local disconnect each record their cause; racing or repeated cleanup calls keep the first.

Acceptance criteria

• Tests pass, and the functional suite is unchanged.
• No production code writes fDisconnect = true directly any more.

Suggested commit: net: record a first-cause-wins close cause.

F2 · net: count hybrid handshake outcomes per direction, with failure rings and endpoint keys

Needs E3 (#199), F1.

Goal

Exact, attributable telemetry for every hybrid negotiation outcome. It is the data source for the fallback (F3), load shedding (F4), the RPC (F6) and the canary (R5).

Scope

Outcomes and transport interface.

enum class PQOutcome : uint8_t {
    SWITCHED = 0, CONFIRMED = 1, LEGACY_PEER = 2, MALFORMED_RECORD = 3,
    FIRST_PACKET_FAILED = 4, ABANDONED = 5, CLOSED_AFTER_SWITCH = 6,
    FALLBACK = 7, INTERNAL_ERROR = 8,
};
// Transport, with harmless V1Transport defaults:
virtual PQHandshake::Snapshot GetPQSnapshot() const noexcept;
virtual void ClearPQSecrets() noexcept;

Endpoint identity. A typed key, captured when the connection is created (src/net.cpp:423,464,511,533) through CNodeOptions::pq_endpoint, and stored as const PQEndpointKey m_pq_endpoint on CNode:

struct PQNameEndpoint { std::string hostname; uint16_t port; bool operator<(const PQNameEndpoint&) const; };
using PQEndpointKey = std::variant<CService, PQNameEndpoint>;

• Resolved destinations use CService: address-family-aware, with the port. Don't reduce identity to GetNetwork(); private addresses map to NET_UNROUTABLE (src/netaddress.cpp:500).
• Name-proxy destinations keep an unresolved address (src/net.cpp:449-451, :510-513; the added-node code keeps the same split at :2924). For these, use the SOCKS destination hostname, lowercase ASCII with one trailing root dot removed, and the effective port. Never the proxy's own address.
• Inbound uses the actual remote address and source port. Don't infer a listening port, and don't treat an advertised address as identity.

Counters. std::array<PQCounters, 2> (inbound, outbound) of atomics. They count events, not mutually exclusive final states:

• switched increments at key installation, even if confirmation later fails;
• CONFIRMED marks success (F3's protection) without a second switched;
• each failure is counted once, even when it is seen both after receive and at disconnect;
• abandoned: an offer was queued, but no authenticated non-decoy peer version arrived. Only responders offer, so its outbound count stays 0;
• internal_error exists in both directions;
• outbound only: closed_after_switch, and fallback (incremented by F3);
• inbound only: shed (incremented by F4).

Observation. CConnman::ObservePQ(CNode&, bool finalizing) runs after ReceiveMsgBytes (src/net.cpp:2180) and once more in DisconnectNodes before the node is removed. It skips the network-inactive case, as ShouldReconnectV1 does.

• Snapshot the transport under its lock, release it, then take m_pq_mutex. Never call into the transport, socket, peer manager or node enumeration while holding m_pq_mutex.
• Finalizing derives the close outcome from F1's cause: an outbound connection that switched but wasn't confirmed and closed by EOF, reset or timeout is closed_after_switch with cause=eof|reset|timeout; send errors and local closes are not. It then marks the node finalized and wipes any pending secrets before the disconnected-node pool keeps the object.
• CNode gains m_pq_accounted_events and m_pq_finalized, accessed only under m_pq_mutex.

Failure rings. Two rings of at most 256 entries: inbound, and outbound plus manual. Inbound noise can't evict outbound evidence.

struct PQFailureEntry {
    uint64_t sequence; NodeSeconds time; PQEndpointKey endpoint; bool inbound;
    ConnectionType connection_type; NodeId peer_id; PQOutcome outcome;
    std::string reason;          // assigned only from fixed literals
};
struct PQFailureRing { uint64_t last_sequence{0}; uint64_t dropped{0}; std::deque<PQFailureEntry> entries; };

• Each ring has its own monotonic sequence, starting at 1. A global sequence would show false gaps when the other ring receives events.
• Rings record non-success outcomes, including legacy_peer. They don't record ordinary off or v1 connections, or shed offers.

Process identity. instance_id, 32 random bytes per process start, and since, the startup time. Counters, rings and history reset together.

Snapshot. PQTransportStats GetPQTransportStats() const: an owning copy with no atomics, for F6 (#209).

Logging. One line per outcome. The prefix is v2 pq:, the outcome word is the counter key, and fields come from a fixed list, never from bytes the peer controls.

• v2 pq: switched and v2 pq: legacy_peer: net-category debug.
• Outbound failures, at info level: v2 pq: malformed_record reason=…, v2 pq: first_packet_failed reason=…, v2 pq: closed_after_switch cause=… and v2 pq: internal_error reason=….
• Inbound outcomes and v2 pq: abandoned: net-category debug, because outsiders can trigger them at will.
• Inbound and outbound lines come from separate call sites: the logger rate-limits per call site (src/logging.h:362-370), so inbound noise can't silence outbound lines.
• Every line carries role=initiator|responder conn_type=<type> peer=<id>, plus peeraddr=<address> only with -logips. Failure lines also carry arith= and keccak=.
• Reasons use the fixed vocabulary in p2p: add hybrid post-quantum key exchange (ML-KEM-1024) to BIP324 v2 transport #184.

Tests

Acceptance criteria

• The tests pass, and behavior is unchanged (production construction is PQ-off until F5).

Suggested commit: net: attribute PQ outcomes per direction with bounded failure rings.

F3 · net: fall back to plain v2 after three counted failures to an endpoint

Needs F2.

Goal

Keep upgraded nodes connected when hybrid negotiation with a particular endpoint keeps failing (a platform bug or a broken peer), without letting an outsider downgrade connections cheaply.

Why three strikes

One failure after the switch is weak evidence: self-connections (src/net_processing.cpp:3687), obsolete-version rejections (:3661), eviction, restarts and injected TCP resets all close connections at that point. A one-strike rule would let anyone who can reset a connection buy 24 hours of ECDH-only traffic. A real platform bug fails every time, so its links still fall back after three attempts.

Scope

• Counted failures, outbound only, per endpoint (F2's key): malformed_record, first_packet_failed, and closed_after_switch (eof, reset or timeout).
  • Never counted: our own decisions (operator disconnectnode, shutdown, network off, eviction, feelers), send errors, legacy outcomes and unrelated connection failures. These are neutral: they neither advance nor reset the streak.
  • Only a confirmed hybrid success clears a streak.
• Entering fallback on the third consecutive counted failure (PQ_FAILURE_THRESHOLD = 3), never for an endpoint in the success set.
• Windows escalate per endpoint: first entry 1 h, second 4 h, then 24 h (PQ_FALLBACK_WINDOWS).
  • When now >= expires, clear the active fallback and reset the streak to 0, but keep the escalation level; three new failures are needed for the next window.
  • Failures from connections already running during an active window don't extend it or re-enter.
• Success protection. A confirmed hybrid handshake moves the endpoint into m_pq_successful_endpoints, removes its history and resets its escalation. The set is separate and never evicted until restart, and a concurrent late failure can't reinsert the endpoint. Its size is bounded by the number of distinct successful outbound endpoints since startup.
• Bounds. The combined fallback, streak and escalation history holds at most 1,000 endpoints (PQ_MAX_ENDPOINT_HISTORY), evicting the oldest insertion first, deterministically when timestamps tie. Everything is memory-only, is cleared on restart, and uses the mockable NodeClock.
• Policy. Outbound construction consults IsPQFallback(endpoint, now). A fallback connection runs PQMode::FALLBACK: plain v2 with an empty version, neither parsing nor validating the peer's offer, so a peer that keeps sending a malformed offer still connects. Its status is fallback.
  • No new retry loop: the existing connection management picks the next attempt, and that attempt consults the set.
  • Existing connections never renegotiate, and inbound connections never fall back.
• Visibility. Entering fallback increments the outbound fallback counter once (on entry, not per connection), adds a ring entry and logs one warning line: v2 pq: fallback cause=<trigger> until=<UTC time>.
• Local-fault warning. When 8 or more distinct endpoints (PQ_LOCAL_FAULT_THRESHOLD) have counted failures and none has succeeded since startup, warn once that this node's own ML-KEM code may be at fault, naming -mlkemportable when native code is active. Behavior doesn't change.

inline constexpr size_t PQ_MAX_ENDPOINT_HISTORY = 1000;
inline constexpr uint32_t PQ_FAILURE_THRESHOLD = 3;
inline constexpr size_t PQ_LOCAL_FAULT_THRESHOLD = 8;
inline constexpr std::array<std::chrono::seconds, 3> PQ_FALLBACK_WINDOWS{
    std::chrono::seconds{3600}, std::chrono::seconds{14400}, std::chrono::seconds{86400}};

struct PQEndpointHistory {
    uint32_t streak{0};
    uint8_t completed_windows{0};          // saturates at 3
    std::optional<NodeSeconds> entered;
    std::optional<NodeSeconds> expires;
    NodeSeconds last_failure;
    PQOutcome cause;
    std::string reason;                    // fixed literals only
    uint64_t insertion_order;
};
// CConnman, guarded by m_pq_mutex:
std::map<PQEndpointKey, PQEndpointHistory> m_pq_history;
std::set<PQEndpointKey> m_pq_successful_endpoints;
std::set<PQEndpointKey> m_pq_failed_endpoints_for_warning;   // stop growing at 8
bool IsPQFallback(const PQEndpointKey& endpoint, NodeSeconds now);

Tests (CConnman unit tests)

Acceptance criteria

• The tests pass. The end-to-end functional coverage is G2 (test: end-to-end hybrid transport tests: Python hybrid peer, misbehaving peers, fallback, whole suite with PQ on (#184 part 7) #212).

Suggested commit: net: apply bounded per-endpoint fallback after repeated PQ failures.

F4 · net: stop offering ML-KEM to inbound peers during handshake floods

Needs F2.

Goal

Bound the extra CPU an inbound connection flood can cause. Above a rate threshold, responders stop offering and cost exactly what legacy v2 costs today.

Context

Each offer costs the responder one ML-KEM-1024 key generation (8.8 µs native, 26.1 µs portable) and 1,568 extra bytes. At 1,000 offers per second that is at most about 26 ms of CPU per second, under 3% of one core even in portable C. A legacy v2 handshake already costs 43.8 µs of ECDH per side.

Scope

• Count the inbound handshakes that would receive an offer, at the responder's keygen point (after the 64-byte key arrives).
• Start shedding when more than threshold_per_s offers were attempted in the last second. While shedding, the responder sends empty version contents, which is exactly today's v2. The connection's status is off, and the inbound shed counter increments.
• Stop shedding after 10 consecutive seconds with fewer than threshold_per_s / 2 attempts.
• A token bucket or one-second counters both work; the behavior above is the contract.
• Outbound connections are never affected.
• Shed offers add no ring entries, so a flood can't wash out the ring's evidence; the counter records them.
• Log one info line when shedding starts, v2 pq: load_shedding started threshold_per_s=<n>, and one when it stops, v2 pq: load_shedding stopped.
• Stats for F6 (rpc, init: -v2pqtransport, startup line, getpqtransportinfo, peer fields, -netinfo and GUI (#184 part 6) #209): load_shedding: {active, threshold_per_s, since}, where since is the Unix time the current shedding period started, or 0 when inactive.
• The threshold defaults to 1,000 offers per second, as a constant that CConnman::Options can override for unit tests and the flood lab. There is no new user-facing option. R3's flood measurement sets the final value.

Tests

• Under mock time: offers above the threshold are shed (empty contents, shed +1, status off); outbound connections are untouched; offering resumes only after 10 s below half the threshold; each transition logs exactly once.

Acceptance criteria

• The tests pass.

Suggested commit: net: shed inbound PQ offers under handshake floods.

One pull request against 1.x.x closes this issue. The shared contract is the Design section of #184; change it there first if needed.