net: hybrid v2 handshake: key recipe, record parser, negotiation with key confirmation, and spec (#184 part 4) #199

Part 4 of 9 · #184 · Size XL · Base 1.x.x · Needs part 2 (#186), part 3 (#190) · Unblocks part 5 (#206), part 6 (#209), part 8 (#215), part 9 (#222)

Consolidates work items E1 (#199), E2 (#202), E3 (#203), G4 (#211) and H2 (#204), each formerly its own sub-issue. Their full text is below as sections; the closed issues point here.

Goal

The protocol itself: the hybrid key recipe with test vectors, the version-contents parser and per-connection KEM state, the V2Transport negotiation with key confirmation, an attacker-side fuzz target, and the spec. Production construction stays PQ-off; part 6 adds the option.

Work items

In suggested commit order. Each item is a section below.

• E1 · bip324: derive hybrid session keys from an ordered transcript, with vectors
• E2 · net: add PQHandshake: version-contents record parser and per-connection KEM state
• E3 · net: negotiate hybrid v2 keys with key confirmation in V2Transport
• G4 · fuzz: add a stateful malicious-peer target for hybrid negotiation
• H2 · doc: specify hybrid v2 transport (doc/design/pq-transport.md)

Starting early

• E1 needs only B2's Python ML-KEM, so it can start as soon as part 2 is up. H2 can be drafted from the design in p2p: add hybrid post-quantum key exchange (ML-KEM-1024) to BIP324 v2 transport #184 at any time.
• E2 can be written against D3's declared wrapper API before part 3 merges.
• E3 needs E1 and E2, and G4 needs E3.

E1 · bip324: derive hybrid session keys from an ordered transcript, with vectors

Needs B2 (#186).

Goal

Add the hybrid key recipe to BIP324Cipher (#184, Key schedule), and pin it with vectors generated by an independent Python reference. Nothing calls it from the transport yet; E3 does.

Context

• BIP324Cipher::Initialize (src/bip324.cpp:34-71): ECDH at :41, HKDF at :45, the initiator_L/P and responder_L/P labels at :47-54, garbage terminators at :57-61 and session_id at :64. It wipes the ECDH secret, HKDF state and private key at :67-70, and is documented as callable once (src/bip324.h:56).
• The four ciphers live in std::optional (src/bip324.h:31-34), so they can be re-emplaced.
• HKDF helper: src/crypto/hkdf_sha256_32.cpp:10,15.
• Rekeying derives from the current keys (src/crypto/chacha20.cpp:357-370, src/crypto/chacha20poly1305.cpp:106-121), so after the switch every rekey derives from the hybrid keys.
• Network magic in wire order: mainnet 44 4f 24 a8, regtest a6 6b 1f da (src/kernel/chainparams.cpp:190,593).

Scope

Additions to BIP324Cipher; existing public methods stay:

private:
    std::optional<ECDHSecret> m_retained_ecdh;
    std::optional<CSHA256> m_hybrid_transcript;
    std::array<uint8_t, 4> m_hybrid_magic{};
    uint8_t m_version_contents_count{0};
    bool m_hybrid_send_is_initiator{false};
    bool m_hybrid_switched{false};
public:
    ~BIP324Cipher();                                    // also wipes retained state
    BIP324Cipher(const BIP324Cipher&) = delete;
    BIP324Cipher& operator=(const BIP324Cipher&) = delete;
    void Initialize(const EllSwiftPubKey& their_pubkey, bool initiator,
                    bool self_decrypt = false, bool retain_for_hybrid = false) noexcept;
    [[nodiscard]] bool AddVersionContents(std::span<const std::byte> contents) noexcept;
    [[nodiscard]] bool SwitchToHybrid(std::span<const std::byte, 32> ss_mlkem) noexcept;
    void DiscardHybridSecret() noexcept;

• retain_for_hybrid=false, the default, keeps today's secret lifetime and the existing vectors.

• When retaining:

  • keep the raw 32-byte ECDH secret (the ComputeBIP324ECDHSecret output that today's HKDF consumes), not an HKDF output;
  • start the transcript with the actual-role ellswift_I ‖ ellswift_R;
  • snapshot the four magic bytes;
  • save initiator != self_decrypt for the cipher direction, copying Initialize's self_decrypt handling.

• AddVersionContents accepts exactly two calls, the responder's contents first. It writes CompactSize(len) ‖ contents through a small hash-writer adapter (write(std::span<const std::byte>)), with no copy of the contents and no DataStream.

• SwitchToHybrid requires initialized ciphers, a retained secret, both contents and no earlier switch; otherwise it returns false (E3 treats that as a local internal error). It finishes TH, re-emplaces all four FSChaCha20 and FSChaCha20Poly1305 objects with their counters at 0, and replaces the session id. Garbage terminators are not re-derived.

• Wipe the retained secret, ss_MLKEM copies, IKM, HKDF state and derived keys on every path. DiscardHybridSecret wipes before resetting the optional. The transcript hash isn't secret, but it is reset with the retained state.

• The session id keeps its current presentation (HexStr, as at src/net.cpp:650).

• Python: test/functional/test_framework/crypto/bip324_cipher.py gains the same recipe.

• Vector generator contrib/devtools/generate-pq-transport-vectors.py:

  • imports the test_framework/crypto modules by path, so there is one Python reference, and never calls the C++ code under test;
  • writes src/test/data/pq_transport_vectors.json with the inputs, every intermediate (ss_ECDH, ss_MLKEM, the TH input and TH, PRK, the five derived values), each side's confirmation packet, the first application packet each way, and negative cases;
  • has a --check mode, run in lint.

  bip324_tests reads the JSON through target_json_data_sources.

• Fuzz: bip324_cipher_roundtrip (src/test/fuzz/bip324.cpp) gains an optional hybrid switch, transcript mutations, damaged packets and rekeys; honest directions must agree.

Tests (src/test/bip324_tests.cpp)

The existing BIP324 vectors are unchanged.

Acceptance criteria

• The tests above pass, and generate-pq-transport-vectors.py --check passes in lint.
• A deliberately changed recipe fails both --check and bip324_tests (shown on the PR).

Suggested commit: bip324: derive hybrid keys from an ordered transport transcript.

E2 · net: add PQHandshake: version-contents record parser and per-connection KEM state

Needs D3 (#190). Can be written against D3's declared wrapper API before part 3 merges.

Goal

Keep the feature's parsing and KEM bookkeeping in a new src/bip324_pq.{h,cpp}, so V2Transport gains only calls (E3) and future upstream merges stay small.

Context

• Version contents are empty today (VERSION_CONTENTS, src/net.h:460) and ignored on receipt (src/net.cpp:1251-1254).
• The transport functions are noexcept (src/net.h:630-653), so an escaping exception would end the process.
• SpanReader::ignore() doesn't check bounds (src/streams.h:118). The CompactSize deserializer rejects non-canonical encodings (src/serialize.h:330).
• MAX_CONTENTS_LEN is 4,000,013 bytes (src/net.cpp:1216-1226), which bounds the parser's work.
• ProcRand asserts at most 32 bytes per call (src/random.cpp:567).

Scope

// src/bip324_pq.h
inline constexpr uint8_t PQ_MLKEM1024 = 0xF0;
inline constexpr size_t PQ_RECORD_BYTES = 1572;
inline constexpr size_t PQ_CONFIRMATION_BYTES = 20;

enum class PQMode : uint8_t { OFF = 0, NEGOTIATE = 1, FALLBACK = 2 };
enum class PQStatus : uint8_t { V1 = 0, OFF = 1, PENDING = 2, HYBRID = 3, LEGACY_PEER = 4, FALLBACK = 5 };
enum class PQOfferState : uint8_t { NONE = 0, SENT = 1, RECEIVED = 2 };
enum class PQFailure : uint8_t {
    NONE = 0, EK_LENGTH = 1, EK_MODULUS = 2, CT_LENGTH = 3,
    CONFIRM_LENGTH = 4, CONFIRM_TAG = 5, CONFIRM_NOT_DECOY = 6,
    KEYGEN_INTERNAL = 7, CHECK_EK_INTERNAL = 8, ENCAPS_INTERNAL = 9,
    DECAPS_INTERNAL = 10, CIPHER_STATE_INTERNAL = 11,
};

struct PQRandomSource {          // production: GetStrongRandBytes
    using Fill32 = void (*)(void*, std::span<uint8_t, 32>) noexcept;
    Fill32 fill32;
    void* context{nullptr};
};
struct PQKemOps {                // static lifetime; alternatives only via the test constructor
    decltype(&mlkem::KeyGen) keygen;
    decltype(&mlkem::CheckPublicKey) check_public_key;
    decltype(&mlkem::Encaps) encaps;
    decltype(&mlkem::Decaps) decaps;
};

class PQHandshake {
public:
    enum class ParseKind : uint8_t { NO_RECORD = 0, INVALID_GRAMMAR = 1, OWN_RECORD = 2 };
    struct ParsedContents { ParseKind kind{ParseKind::NO_RECORD}; std::span<const std::byte> payload{}; };
    using Record = std::array<std::byte, PQ_RECORD_BYTES>;
    struct Snapshot {
        PQOfferState offer{PQOfferState::NONE};
        bool version_received{false};
        bool switched{false};
        bool confirmed{false};
        PQFailure failure{PQFailure::NONE};
    };

    static ParsedContents ParseContents(std::span<const std::byte> contents) noexcept;
    static Record SerializeRecord(std::span<const uint8_t, 1568> payload) noexcept;

    PQHandshake(bool initiating, PQMode mode);
    PQHandshake(bool initiating, PQMode mode, PQRandomSource random, const PQKemOps& ops);
    ~PQHandshake();
    PQHandshake(const PQHandshake&) = delete;
    PQHandshake& operator=(const PQHandshake&) = delete;

    // Success writes an offer and keeps dk; failure writes no offer.
    [[nodiscard]] mlkem::Error MakeOffer(Record& offer) noexcept;
    [[nodiscard]] mlkem::Error AcceptOffer(std::span<const uint8_t, 1568> ek,
                                           Record& accept, mlkem::SharedSecret& ss) noexcept;
    [[nodiscard]] mlkem::Error DecapsulateAccept(std::span<const uint8_t, 1568> ct,
                                                 mlkem::SharedSecret& ss) noexcept;
    void SetVersionReceived() noexcept;
    void SetSwitched() noexcept;
    void SetConfirmed() noexcept;
    void SetFailure(PQFailure failure) noexcept;
    Snapshot GetSnapshot() const noexcept;
    void ClearSecrets() noexcept;
private:
    const bool m_initiating;
    const PQMode m_mode;
    PQRandomSource m_random;
    const PQKemOps& m_ops;
    std::unique_ptr<mlkem::DecapsulationKey> m_dk;
    Snapshot m_snapshot;
};

Parser. ParseContents is a single noexcept pass over a non-owning SpanReader, keeping only the position of the first own-header record:

try {
    SpanReader reader{contents};
    std::optional<std::span<const std::byte>> first;
    while (!reader.empty()) {
        uint64_t len = ReadCompactSize(reader);
        if (len == 0 || len > reader.size()) return INVALID_GRAMMAR;   // check before ignore()
        uint8_t header; reader >> header;
        size_t offset = contents.size() - reader.size();
        if (header == PQ_MLKEM1024 && !first) first = contents.subspan(offset, size_t(len - 1));
        reader.ignore(size_t(len - 1));
    }
    return first ? OWN_RECORD(*first) : NO_RECORD;
} catch (const std::ios_base::failure&) {
    return INVALID_GRAMMAR;
}

• No modulus check, KEM operation or semantic rejection happens until the whole grammar validates. A framing error anywhere, even after an apparently valid own record, means "no features".
• The returned span is consumed before the decode buffer is cleared, and never kept.

Semantics of the first own record (callers in E3 act on them):

• the responder's offer must carry exactly 1,568 bytes (ek_length) that pass CheckPublicKey (ek_modulus);
• the initiator's accept must carry exactly 1,568 bytes (ct_length).

Entropy. In production, MakeOffer draws two independent 32-byte GetStrongRandBytes values for the 64-byte keygen seed, and AcceptOffer draws one 32-byte value; the seeds are wiped. Deterministic test entropy comes only from the test constructor's PQRandomSource, because the global deterministic RNG doesn't replace GetStrongRandBytes.

State and secrets.

• m_dk is allocated only when MakeOffer runs, which happens after all 64 EllSwift bytes arrived (lazy keygen, E3).
• SENT means the offer was committed to the send queue, not that the peer received it.
• Internal ML-KEM errors never assert; they return mlkem::Error::INTERNAL, and E3 decides what to do.
• ClearSecrets and the destructor wipe m_dk.
• There is no mutex: V2Transport::m_recv_mutex protects the object.

Fuzz. New src/test/fuzz/pq_records.cpp, target pq_records: arbitrary contents, checked by an independent structural oracle for whole-grammar validity and the first own record's position. Add it to the required corpus (QBIT_REQUIRED_CORPUS_TARGETS, overlay.py, MANIFEST.json) with seeds within the limits.

Tests (src/test/net_tests.cpp)

Acceptance criteria

• The tests pass, and pq_records seeds replay in the fuzz CI job.
• Not wired into V2Transport; no behavior change.

Suggested commit: net: implement PQ feature parsing and per-connection KEM state.

E3 · net: negotiate hybrid v2 keys with key confirmation in V2Transport

Needs E1, E2.

Goal

Wire the negotiation into V2Transport. The responder offers its encapsulation key, and the initiator holds its version packet and answers with the ciphertext. Both switch keys right after the version packets and prove the switch with a 20-byte confirmation packet. In this part production construction stays PQ-off; tests construct PQ transports directly. F5 (#209) adds the option.

Context

• Key arrival, ProcessReceivedKeyBytes (src/net.cpp:1120-1178): Initialize at :1150, send state READY at :1157, and the garbage terminator and version packet queued at once (:1159-1171).
• Sending: SetMessageToSend accepts only READY with an empty send buffer (:1484-1514); GetBytesToSend reports more only in READY (:1516-1531); SocketSendData asserts that prediction (:1625). PushMessage's optimistic send relies on READY (:3922-3929).
• Receiving: packets are decrypted one at a time, and the next length isn't decrypted until the current packet completes (:1299-1312), so receive ciphers can be swapped cleanly between packets. A packet's length is decrypted before it is authenticated (:1220-1227).
• Lock order is recv then send (src/net.h:590,605). State-transition assertions are at src/net.cpp:1003,1022,1053, and the SendState diagram is at src/net.h:546.
• GetInfo reports v2 only after the version packet (:1589-1590); cipher metadata is published at :1581.
• Application layer: an inbound node sends VERSION only after receiving the peer's (src/net_processing.cpp:3699-3703), and outbound pushes VERSION first (:5744-5748). net_processing.cpp needs no change for the negotiation.
• ShouldReconnectV1 fires only before any key bytes arrive (src/net.cpp:1555-1570).

Scope

States. Existing values are kept; these are internal values, not wire values.

enum class RecvState : uint8_t { KEY_MAYBE_V1, KEY, GARB_GARBTERM, VERSION, CONFIRM, APP, APP_READY, V1 };
enum class SendState : uint8_t { MAYBE_V1, AWAITING_KEY, AWAITING_VERSION, READY, V1 };

Update every exhaustive switch, both transition tables, and the diagram at src/net.h:546.

Construction.

struct V2PQOptions { PQMode mode{PQMode::OFF}; bool corrupt_shared_secret{false}; };
V2Transport(NodeId nodeid, bool initiating, V2PQOptions pq = {}) noexcept;
V2Transport(NodeId nodeid, bool initiating, const CKey& key, std::span<const std::byte> ent32,
            std::vector<uint8_t> garbage, V2PQOptions pq,
            PQRandomSource random, const PQKemOps& ops) noexcept;   // tests
// members and helpers
const V2PQOptions m_pq_options;
PQHandshake m_pq GUARDED_BY(m_recv_mutex);
void AppendVersionPacket(std::span<const std::byte> contents) noexcept EXCLUSIVE_LOCKS_REQUIRED(m_recv_mutex, m_send_mutex);
void AppendConfirmationPacket() noexcept EXCLUSIVE_LOCKS_REQUIRED(m_recv_mutex, m_send_mutex);
bool ProcessVersionContents(std::span<const std::byte> contents) noexcept EXCLUSIVE_LOCKS_REQUIRED(m_recv_mutex, m_send_mutex);

ProcessReceivedPacketBytes stays receive-locked and takes the send lock only around ProcessVersionContents. The switch is then atomic under both locks, and normal application packets take no extra lock.

State table. R/S are the receive and send states. "Close" means return false or schedule a disconnect. The last column is what F2 (#206) counts.

Send rules.

• In AWAITING_VERSION, SetMessageToSend returns false without touching its argument, and GetBytesToSend returns the remaining handshake bytes with more=false. A responder may still have unsent offer bytes while it holds.
• Version and confirmation packets are appended after any unsent bytes. Never replace the send vector or reset m_send_pos: replacing ciphers doesn't re-encrypt bytes already queued under ECDH.
• Neither side waits for the peer's confirmation before sending application packets. On the receive side, application delivery waits for the confirmation.
• The initiator keeps m_send_garbage until it builds its version packet. A version packet uses that sender's garbage as AAD unless an earlier decoy already consumed it.

Confirmation.

• Send: Encrypt({}, {}, /*ignore=*/true, …): 3 length bytes, 1 header byte and a 16-byte tag, so 20 bytes, with empty AAD.
• Receive in CONFIRM: as soon as the 3-byte length decrypts, a nonzero value fails at once, before any allocation or waiting. The only allowed remaining body is 17 bytes. Check the ignore bit, not the whole header byte (the cipher exposes only that bit, src/bip324.cpp:107). Confirmation handling comes before generic decoy handling. A valid confirmation yields no CNetMessage.

Reporting. Transport::Info gains bool transport_pq (true only once the peer's confirmation verified) and PQStatus transport_pq_status. Until confirmation the status is pending, no session id is published, and the existing detecting classification stays; the hybrid session id is published only after confirmation.

Fault injection. corrupt_shared_secret XORs byte 0 of the local ML-KEM shared secret with 0xff after encapsulation or decapsulation and before derivation, without touching EK or CT. The negative case in pq_transport_vectors.json models exactly this. F5's -test=pq_fail_first_packet uses it.

Secrets. The DK and the retained ECDH secret are wiped on switch, on legacy outcome, at disconnect and at timeout. A pending inbound connection holds about 3.3 KB (DK 3,168 bytes, retained secret 32, transcript state about 100), about 420 KB at 125 connections.

Fuzz. p2p_transport_bidirectional_v2 and p2p_transport_bidirectional_v1v2 (src/test/fuzz/p2p_transport_serialization.cpp:386,397) gain independent PQ flags per side and KEM seeds from the fuzzer input, and keep their checks of append-only output and more predictions. Add new seeds. p2p_transport_serialization (v1 only) is unchanged.

Guix. qbitd now links the ML-KEM code: repeat D4's reproducibility and ELF-property comparison and record it on the PR.

Tests (src/test/net_tests.cpp)

Acceptance criteria

• The tests pass; existing transport tests pass unchanged with PQ off.
• Transport fuzz targets replay their seeds with the new flags.
• Production construction is still PQ-off (no option yet).
• D4's Guix and ELF-property comparison is repeated and recorded.

Suggested commit: net: negotiate hybrid v2 keys and require key confirmation.

G4 · fuzz: add a stateful malicious-peer target for hybrid negotiation

Needs E3.

Goal

Fuzz the negotiation from the attacker's side: a peer that holds valid ECDH keys and can therefore send any authenticated negotiation bytes it likes.

Scope

• New src/test/fuzz/p2p_v2_pq_malicious_peer.cpp, target p2p_v2_pq_malicious_peer. It is a stateful attacker that completes the ECDH part honestly and encrypts arbitrary negotiation plaintext:
  • malformed lengths, duplicate and unknown records, trailing corruption;
  • unsolicited accepts, decoys around the switch, and arbitrary boundary ordering;
  • partial writes and disconnects at any point;
  • the node under test in both roles, with PQ on or off.
• Invariants: no assertion, no hang, no reuse of invalid state. Send output stays append-only, and more predictions hold.
• Seeds must actually reach authenticated negotiation; random ciphertext stops at AEAD rejection. Generate them with E3's deterministic test constructor.
• Add the target to the required corpus (QBIT_REQUIRED_CORPUS_TARGETS, overlay.py, MANIFEST.json) with seeds within the limits.

Acceptance criteria

• Seeds replay in the fuzz CI job, and coverage reaches confirmation handling (shown on the PR).

Suggested commit: fuzz: add a stateful malicious-peer target for PQ negotiation.

H2 · doc: specify hybrid v2 transport (doc/design/pq-transport.md)

Needs E1. Can start now from the design in #184; merges after E1's vectors exist.

Goal

A specification another implementation can follow to interoperate, and the document the outside review (R4) starts from.

Scope

doc/design/pq-transport.md:

• Wire format. The record grammar; a header registry (0x00 reserved, 0xF0 assigned to ML-KEM-1024 offer and accept, and the range policy: qbit assigns its own values from 0xF0 upward, far from the low values a future Transport Feature Negotiation table would assign first); payloads and byte layouts.
• Parse rules, including the one deliberate difference from the Transport Feature Negotiation draft: unparseable contents are ignored rather than causing a disconnect, because BIP324 receivers ignore version contents today and v1.0.0 relies on that.
• Key recipe, byte by byte, pointing at src/test/data/pq_transport_vectors.json. Any recipe change takes a new header with its own salt label; a label-only change under the same header would let two adjacent releases both accept the offer, derive different keys and disconnect at the first hybrid packet.
• Message rules.
  • Responder MUSTs: send the version packet right after the garbage terminator without waiting for anything (BIP324 implementations already do); after sending an offer, send nothing, decoys included, until the initiator's version packet has been processed.
  • Initiator: MAY hold its version packet until the responder's arrives, then sends the ciphertext in it.
  • Why progress still holds: a hybrid initiator forgoes BIP324's optional immediate version send and uses the order BIP324's version-negotiation steps list (receive the responder's version packet, then send its own). BIP324's progress conditions hold because responders MUST send their version packet right after their garbage terminator. Only qbit initiators differ from deployed BIP324 behavior, never responders. Cite BIP324's version-negotiation phase, which names post-quantum handshake upgrades as an intended extension.
• The switch boundary: which packets use ECDH keys and which use hybrid keys.
• Key confirmation MUSTs for both roles.
• Security notes:
  • version contents (the EK, the CT and any future record) have ECDH-only protection;
  • the guarantee holds for confirmed hybrid connections, since off switches, local faults, shedding and fallback permit ECDH-only sessions;
  • this is not authentication;
  • hybrid sessions are distinguishable from legacy v2 by packet sizes.
• Interop recipe: step by step for another implementation, walking through the vectors file.
• Convergence policy and recipe retirement (p2p: add hybrid post-quantum key exchange (ML-KEM-1024) to BIP324 v2 transport #184, Convergence policy).
• Contributor section: configure, build (the test_bitcoin target builds the test_qbit executable), focused tests (ctest -R), backend selection, and how to regenerate the vectors.

Acceptance criteria

• The markdown lint passes, and every vector the spec quotes matches the JSON.
• After E3's PR opens: post the spec, the Python reference and the vectors to the bitcoin-dev "Transport Feature Negotiation" and "A Post-Quantum Path for BIP 324" threads, asking for a reserved private-use header range and discussing ML-KEM-768 against 1024; link the posts from the spec.

One pull request against 1.x.x closes this issue. The shared contract is the Design section of #184; change it there first if needed.