Skip to content

release: hybrid v2 transport release gate and default-on (#184 part 9) #222

Open

Description

@djh58

Part 9 of 9 · #184 · Size calendar-bound · Base 1.1.0 · Needs part 2 (#186), part 3 (#190), part 4 (#199), part 5 (#206), part 7 (#212), part 8 (#215)

Consolidates work items R0 (#200), R4 (#205), R2 (#218), R3 (#219), R1 (#217), R5 (#220), R6 (#221) and R7 (#222), each formerly its own sub-issue. Their full text is below as sections; the closed issues point here.

Release (2026-10-23)

Release N is 1.1.0, the release after 1.0.1, cut from the 1.1.0 branch. 1.x.x becomes 1.0.1 and takes no more feature merges, so #233 targets 1.1.0.

Release steps (2026-10-27)

#256 prepared the release candidate (version 1.1.0-rc1, man pages, doc/release-notes-1.1.0.md) and merged as 1b6e6b6. 1.1.0 ships after 1.0.1 (#174). The remaining steps, carried over from #256's maintainer checklist:

Plan change (2026-10-06)

The gate is shortened: no mainnet canary and no long soak. The hybrid transport is tested on testnet4 for a short period, then turned on for mainnet.

  • R0 (14-day canary baseline) and R5 (14-day mainnet canary) are dropped. Their sections are kept below for reference.
  • R6 becomes a short testnet4 soak across platforms. The duration is the release-cut decision's, not a fixed 7 days.
  • R2 (fuzz campaign) and R3 (flood measurement) stay: they need compute time, not calendar gating, and R3 sets the load-shedding default.
  • R4 (outside security review) is dropped (2026-10-23): the release ships on the evidence in this checklist.

Goal

The release gate for release N, as one checklist: the canary baseline, the fuzz campaign, the flood measurement, the outside review, the one-line default-on PR, the 14-day mainnet canary, the 7-day testnet4 soak and the release-cut decision. This issue closes at release cut.

Checklist

In rough time order. Each item is a section below.

Starting early

  • R0 starts as soon as part 2's canary tools merge, and runs during implementation.
  • R4 starts once part 3's library and part 4's spec (H2) are up, and doesn't block coding.
  • R2 and R3 run as soon as their tools merge (parts 3, 4 and 8), and must finish before the canary.
  • R1 merges just before the release candidate. R5 and R6 then run on the release candidate, and R7 closes the gate.

R0 · release: run the 14-day canary baseline

Needs C5 (#186). Calendar: 14 days, overlapping implementation.

Goal

Fourteen days of baseline samples from the team's mainnet nodes while they still run v1.0.0, so the canary (R5) has something to compare against, especially the ratio of inbound connections between the canary and control nodes.

Scope

  • Deploy C5's sampler, every 5 minutes, on the three team nodes the canary uses: the archive node and the pool node that will run the release candidate, and the archive node that stays on v1.0.0 as the control. All three run v1.0.0 during the baseline.
  • Fields the v1.0.0 RPC lacks are recorded as NA; connections_pq and getpqtransportinfo are recorded from the canary's start.
  • Compute the baseline connections_in ratio of the canary archive node to the control.
  • Give the team's deployment configuration a per-host switch for -v2pqtransport, so canary hosts can be turned on later without hand edits.
  • Check the network census and crawler tools the team operates against non-empty version contents: Core's test framework, which crawlers often copy, asserts the contents are empty.

Acceptance criteria

  • 14 consecutive days of samples on all three nodes, with the evaluator's baseline summary (sample counts, gaps, the connections_in ratio) posted here. Addresses stay on the nodes.
  • The census and crawler tools are confirmed to handle non-empty version contents, or fixed.

R4 · release: scoped outside security review of hybrid transport

Needs H2 (#199), D3 (#190). Starts on the spec and library; runs alongside the handshake work and the canary.

Goal

An independent review of the new attack surface before release N ships it on by default.

Scope

  • In scope:

    • the hybrid key recipe and transcript;
    • the record reader (PQHandshake::ParseContents);
    • the handshake state changes in V2Transport, including key confirmation;
    • the fallback and load shedding;
    • the library integration settings: config, backend selection, capability hook and build flags.

    Upstream mlkem-native's proofs don't cover qbit's wrapper, runtime configuration, compiler integration, transcript or state machine; those are the point of this review. It is not a full audit, and should take about 1–2 weeks.

  • The reviewer and a target date are set before E3's PR opens. The review starts on the spec (H2) and the library (D3) while the handshake PRs are in review, and doesn't block coding.

  • If no paid reviewer is available by the target date, ask the mlkem-native maintainers to review the library integration, and the BIP324 authors to review the protocol through the published spec.

  • Each high or critical finding is fixed and re-checked by the reviewer before release cut. A fix that changes the wire format, key recipe or state machine restarts the canary (R5), the soak (R6) and the flood measurement (R3) on the changed code.

Acceptance criteria

  • The review is delivered, and a summary of findings and their resolution is posted here.
  • Every high or critical finding is fixed and re-checked.
  • The reviewer and dates are recorded in R7's decision record.

R2 · release: fuzz campaign for hybrid transport (at least 504 CPU-hours)

Needs D5 (#190), E2 (#199), G4 (#199). Must finish before the canary.

Goal

Long fuzzing of every target that covers the new code, finished before the canary (R5) starts.

Scope

  • Targets: mlkem, mlkem_backend_diff, pq_records, p2p_v2_pq_malicious_peer, bip324_cipher_roundtrip, p2p_transport_bidirectional_v2 and p2p_transport_bidirectional_v1v2.

  • Configurations, at least 24 CPU-hours per target in each:

    1. ASan + UBSan with native code;
    2. ASan + UBSan forced portable (MLKEM_FORCE_PORTABLE=1);
    3. MSan, portable C (ci/test/00_setup_env_native_fuzz_with_msan.sh).

    That is 7 × 3 × 24 = 504 CPU-hours. More machines shorten the calendar time, not the required coverage time.

  • Every crash is fixed, and every interesting input becomes a regression seed in the required corpus.

  • A wire, key-recipe or state-machine change after the campaign reruns the affected targets.

Acceptance criteria


R3 · release: measure handshake floods and set the load-shedding threshold

Needs H1 (#215), F4 (#206). Must finish before the canary; repeat if the handshake code changes.

Goal

Show that hybrid transport doesn't slow block propagation meaningfully under an inbound connection flood, and set F4's shedding threshold from measurements.

Scope

  • Machines: a non-AVX2 x86-64 machine (portable C), and a Raspberry Pi 4 or another Cortex-A72-class AArch64 Linux machine (also portable C: native AArch64 code is macOS-only until upstream's assembly carries BTI landing pads).
  • Lab: H1's run-item3.sh topology and its three flood modes (junk, handshake, maxversion), each with A's switch on and off.
  • Pass criteria, switch on against the same flood with the switch off:
    • flooded median propagation at most 1.10×, and flooded 95th percentile at most 1.25×;
    • no block slower than 5 s;
    • CPU time per attempted junk handshake and per completed handshake each below 2.5× (expect about 1.6× and 2.2× on portable C);
    • unflooded figures recorded for reference.
  • maxversion: report socket-thread delay and memory. If a limit fails, the parser gets an explicit work bound (a follow-up issue, which blocks the canary).
  • Shedding: measure with shedding at the default threshold and with it disabled, and choose the final threshold. If it changes, open a PR updating the constant.
  • 100 blocks per configuration.

Acceptance criteria

  • The table, with each figure and its sample count, is posted here, and every criterion passes.
  • The threshold is decided, with a PR if it changed: it stays at 1,000 offers per second, so no PR.

Result (2026-10-23): R3 was settled with CPU benchmarks instead of the flood lab, so propagation under a flood and maxversion weren't measured. The figures, and which criteria they cover, are in the R3 comment below.


R1 · net: turn hybrid v2 transport on by default

Needs B3 (#186), C3 (#190), C4 (#186), D5 (#190), G2 (#212), G3 (#212), G4 (#199), H1 (#215), H3 (#215). Merge just before the release candidate is cut.

Goal

Release N ships hybrid v2 transport on by default. This one-line pull request makes that change, and the canary (R5) runs exactly what ships.

Scope

  • DEFAULT_V2_PQ_TRANSPORT = true.
  • Update the docs and test expectations that assume the default:
    • operator guide and release-note wording: on by default, switch off with v2pqtransport=0;
    • rpc_net.py's expected fields;
    • the functional-suite modes.
  • An explicit -v2pqtransport=0, or the default with -v2transport=0, still never warns.
  • Merge just before the release candidate is cut.
  • If the release gate isn't clean at release cut (R7), revert this PR: release N then ships switched off, and a point release turns it on once the open item is resolved.

Acceptance criteria

  • Full CI is green, including pq-functional, the plain-v2 i686 job and the previous-releases job. (On 1.1.0 at db8dbe5: Full Validation and previous releases. The previous-releases job passed on a rerun; its first attempt hit an intermittent feature_pruning.py hang that 0.1.x's nightly also hit on 2026-10-17.)
  • The release candidate is built from a commit that includes this change.

R5 · release: 14-day mainnet canary on release-candidate builds

Needs R0, R1, R2, R3. Calendar: 14 days.

Goal

Fourteen clean days of the release candidate, on by default, on mainnet, with a v1.0.0 node as the live control.

Scope

  • Nodes. Two team mainnet nodes run the release-candidate build with the feature on (the default) and -logips: an archive node and a pool node. -debug=net isn't needed. The pool node pins the archive node with -addnode, so at least one hybrid session runs on mainnet all the time. A third team archive node stays on v1.0.0 for the 14 days as the control.
  • Outside operators, optional: a few may be invited to switch on and pin a team node; their addresses join the known-good list.
  • Evidence. C5's sampler every 5 minutes on all three nodes, and the evaluator reviewed daily. Addresses stay on the nodes.
  • Clean means all of:
    • no crash or assertion;
    • the pinned link is manual and hybrid in at least 99% of the 5-minute samples, not counting restarts;
    • no malformed_record or first_packet_failed entry, and no fallback entry, between known-good endpoints (team nodes, soak nodes and invited operators), read per endpoint from recent_failures and fallback_set;
    • any closed_after_switch between known-good endpoints matches a restart, eviction or network event in the logs;
    • other failures, which anyone can trigger from outside, are triage items: each is reproduced in the regtest lab within 24 hours, and only a confirmed qbit bug is a no-go;
    • the canary archive node's connections_in stays within ±20% of the control's, relative to their baseline ratio (R0), not counting the 48 hours after a restart.
  • A wire, key-recipe or state-machine change during the canary restarts it on the changed code.

Acceptance criteria

  • 14 consecutive clean days. The evaluator's final output (every figure with its sample count) and the triage notes are posted here.

R6 · release: 7-day testnet4 soak across platforms

Needs R1. Calendar: 7 days, alongside the canary.

Goal

A week of the release candidate across the platforms and backends that matter, where every link between upgraded nodes is pinned and watched.

Scope

  • -testnet4, 7 days. Upgraded nodes:

    • x86_64 with AVX2;
    • x86-64 without AVX2 (portable C);
    • Raspberry Pi 4 (AArch64 Linux, portable C);
    • an Apple Silicon Mac (the only native AArch64 code path);
    • a Windows (mingw) build.

    Plus one v1.0.0 node.

  • Every pair of upgraded nodes is pinned with -addnode. Each pinned link must be manual and hybrid in every 5-minute sample (C5's sampler), apart from restarts.

  • Expected disconnects: feelers and the normal churn of automatic peers.

  • The soak fails on:

    • any malformed_record or first_packet_failed entry, or any fallback entry, between soak nodes, read per endpoint from getpqtransportinfo;
    • a closed_after_switch between soak nodes that no restart explains;
    • the loss of a pinned hybrid link.
  • The startup line on each node shows the expected backend: x86_64-avx2; portable on the non-AVX2 machine and the Raspberry Pi; aarch64-neon on the Mac; and x86_64-avx2 on Windows if its CPU has AVX2.

Acceptance criteria

  • 7 clean days, with the per-node sample summary and startup lines posted here.

R7 · release: release-cut decision for the hybrid transport default

Needs R4, R5, R6.

Goal

One decision record at release cut that says whether release N ships hybrid transport on or off, with the evidence for every release-gate item.

Scope

The record, a comment here linked from #184, lists each gate item with its evidence link and figures:

  1. outside review (R4): delivered, with every high or critical finding fixed and re-checked;
  2. canary (R5): 14 clean days;
  3. flood measurement (R3): every criterion met;
  4. testnet4 soak (R6): 7 clean days;
  5. fuzz campaign (R2): finished with no open crash;
  6. green CI on the release candidate, nightly or on demand: the --v2pqtransport functional suite and the s390x-unit, aarch64-unit, macos-arm64-unit and previous-releases jobs.

Decision rule. If every item holds, release N ships on by default, and the control node upgrades with everyone else. If any item is open or raised a concern, revert R1: release N ships switched off, and a point release turns it on once that item is resolved. Either way, operators can switch it off with -v2pqtransport=0, and v1.0.0 nodes keep working with ECDH-only keys.

Acceptance criteria

  • The decision record is posted, and R1 is either kept or reverted in the release branch to match.

This issue closes at release cut, with R7's decision record. Its only pull request is the one-line default-on change (R1). Post the evidence for each item as a comment here. See Release gate and rollout in #184.

Activity

  1. changed the title [-]release: release-cut decision for the hybrid transport default (#184 R7)[/-] [+]release: hybrid v2 transport release gate and default-on (#184 part 9)[/+] on Oct 5, 2026
  2. 1 remaining item

  3. djh58 commented on Oct 23, 2026

    @djh58
    MemberAuthor

    R3 result: the load-shedding threshold stays at 1,000 offers per second

    DEFAULT_PQ_SHED_THRESHOLD_PER_S stays at 1000, so there's no code change.

    Method. R3 was settled with part 8's CPU benchmarks (bench_qbit -filter='MLKEM_.*|V2_TRANSPORT_HANDSHAKE_.*'), not the flood lab. The threshold limits what one offer costs a responder, and that cost is exactly what the benchmarks measure. Propagation under a flood and maxversion were not measured.

    Figures: nanobench medians, about 2 s of measurement per row (-min-time=2000).

    M4 Max, NEON M4 Max, portable C Ryzen 9 9955HX, AVX2 Ryzen 9 9955HX, portable C
    ML-KEM-1024 key generation (one offer) 10.0 µs 21.9 µs 23.5 µs 72.5 µs
    Encapsulation (initiator) 10.9 µs 19.1 µs 26.1 µs 59.8 µs
    Decapsulation (responder) 13.9 µs 22.4 µs 31.4 µs 79.1 µs
    v2 handshake, both sides, hybrid (3,384 bytes) 135.8 µs 170.0 µs 254.2 / 236.0 µs 367.7 / 353.9 µs
    v2 handshake, both sides, legacy (200 bytes) 81.7 µs 82.1 µs 137.3 / 151.9 µs 138.0 / 154.7 µs
    Hybrid ÷ legacy 1.66× 2.07× 1.85× / 1.55× 2.66× / 2.29×

    The R2 fuzz campaign was running on the Ryzen machine during these measurements, so its figures are higher and noisier. Its handshake rows are two paired runs at -min-time=8000, shown as run 1 / run 2.

    At the threshold.

    • Each offer costs the responder one key generation. At 1,000 offers a second, that's 1.0% of one core (M4 Max, NEON) to 7.3% (Ryzen, portable C).
    • If the flooding peers also complete their handshakes, the responder pays one decapsulation per handshake as well. That makes 2.4% to 15% in total.
    • A Raspberry Pi 4-class core, which runs portable C on Linux AArch64, was not measured. Scaling the portable figures gives an estimated 13–22% for the offers alone.
    • Normal inbound traffic is a few new connections a minute, far below the threshold.

    Criteria.

    • Completed handshake, hybrid below 2.5× legacy: met with native code (1.55–1.85×) and with portable C on the M4 Max (2.07×). Portable C on the loaded Ryzen measured 2.66× and 2.29×, one run on each side of the limit.
    • Attempted (junk) handshake below 2.5×: not benchmarked directly. A junk handshake adds one key generation to the responder's share of a legacy handshake. Taking that share as half the legacy figure, the ratio is about 1.2× (M4 Max, NEON) to 2.1× (Ryzen, portable C).
    • Propagation under a flood, slowest block, and maxversion: not measured.
  4. djh58 commented on Oct 26, 2026

    @djh58
    MemberAuthor

    R2 result: 504.6 CPU-hours of fuzzing, no crashes

    All 21 runs (7 targets × 3 configurations × 24 CPU-hours) finished with no crash, leak, timeout or out-of-memory input. Every grown corpus also replays cleanly with CI's fuzz flags.

    Setup.

    • Host: AMD Ryzen 9 9955HX (x86_64, AVX2), Linux, clang 21 with libFuzzer.
    • Source: 7d0d2de, a commit on the docs part's branch (doc, bench: operator guide, release note and handshake benchmark for hybrid v2 transport #240).
    • Configurations:
      • asan-native: ASan + UBSan, native ML-KEM (the AVX2 backend);
      • asan-portable: the same flags, built without native ML-KEM and run with MLKEM_FORCE_PORTABLE=1;
      • msan-portable: MSan with an MSan-built libc++, portable C.
    • Limits: -timeout=60 -rss_limit_mb=2048.
    • UBSan flags: the ASan builds leave out UBSan's integer group, which made the transport targets 13 to 57 times slower. The CI-flag replay below includes it.
    • Columns: corpus and coverage (cov edges, ft features) are shown at the start of the run → at the end. The start corpus is what CI replays: the qa-assets inputs plus the committed qbit seeds.
    Target Configuration CPU-h Executions Corpus, files (bytes) cov ft Crashes CI-flag replay
    mlkem asan-native 24.0 93.1 M 9 → 96 (32 KiB → 118 KiB) 828 → 842 1126 → 1413 0 96 of 96
    mlkem asan-portable 24.0 34.4 M 9 → 204 (32 KiB → 183 KiB) 888 → 900 1199 → 1590 0 204 of 204
    mlkem msan-portable 24.0 78.6 M 9 → 249 (32 KiB → 139 KiB) 338 → 364 458 → 866 0 249 of 249
    mlkem_backend_diff asan-native 24.0 24.3 M 9 → 132 (32 KiB → 168 KiB) 830 → 842 1161 → 1740 0 132 of 132
    mlkem_backend_diff asan-portable 24.0 17.1 M 9 → 163 (32 KiB → 209 KiB) 707 → 719 1209 → 1616 0 163 of 163
    mlkem_backend_diff msan-portable 24.0 40.3 M 9 → 207 (32 KiB → 106 KiB) 346 → 372 451 → 746 0 207 of 207
    pq_records asan-native 24.0 1,452.1 M 16 → 103 (16 KiB → 77 KiB) 135 → 140 265 → 369 0 103 of 103
    pq_records asan-portable 24.0 1,382.6 M 16 → 105 (16 KiB → 91 KiB) 135 → 140 265 → 369 0 105 of 105
    pq_records msan-portable 24.0 1,531.9 M 16 → 151 (16 KiB → 144 KiB) 86 → 91 173 → 281 0 151 of 151
    p2p_v2_pq_malicious_peer asan-native 24.0 52.2 M 16 → 1369 (2 KiB → 218 KiB) 4261 → 4532 7869 → 17795 0 1369 of 1369
    p2p_v2_pq_malicious_peer asan-portable 24.0 50.1 M 16 → 6667 (2 KiB → 1.3 MiB) 4333 → 4590 7841 → 17840 0 6667 of 6667
    p2p_v2_pq_malicious_peer msan-portable 24.0 28.3 M 16 → 1400 (2 KiB → 224 KiB) 1983 → 2265 3512 → 9280 0 1400 of 1400
    bip324_cipher_roundtrip asan-native 24.0 79.1 M 1423 → 2582 (623 KiB → 995 KiB) 3015 → 3017 9018 → 9238 0 2582 of 2582
    bip324_cipher_roundtrip asan-portable 24.0 87.5 M 1423 → 2783 (623 KiB → 1.1 MiB) 3015 → 3017 9018 → 9238 0 2783 of 2783
    bip324_cipher_roundtrip msan-portable 24.0 99.8 M 1423 → 2517 (623 KiB → 974 KiB) 922 → 930 2370 → 2426 0 2517 of 2517
    p2p_transport_bidirectional_v2 asan-native 24.0 26.5 M 1418 → 4181 (2.1 MiB → 4.0 MiB) 4297 → 4315 14693 → 15282 0 4181 of 4181
    p2p_transport_bidirectional_v2 asan-portable 24.1 25.2 M 1418 → 6592 (2.1 MiB → 5.4 MiB) 4342 → 4360 14732 → 15532 0 6592 of 6592
    p2p_transport_bidirectional_v2 msan-portable 24.0 13.1 M 1418 → 5115 (2.1 MiB → 4.3 MiB) 1974 → 1988 6477 → 6838 0 5115 of 5115
    p2p_transport_bidirectional_v1v2 asan-native 24.0 48.1 M 776 → 4136 (779 KiB → 2.6 MiB) 2769 → 2769 9290 → 9502 0 4136 of 4136
    p2p_transport_bidirectional_v1v2 asan-portable 24.0 49.6 M 776 → 6030 (779 KiB → 3.5 MiB) 2769 → 2769 9290 → 9503 0 6030 of 6030
    p2p_transport_bidirectional_v1v2 msan-portable 24.0 15.3 M 776 → 4005 (779 KiB → 2.5 MiB) 1182 → 1185 4686 → 4844 0 4005 of 4005

    Total: 504.6 CPU-hours; 0 crash artifacts; 0 failed CI-flag replays.

    Code changed since the campaign. The fuzzed code changed in four ways after 7d0d2de:

    The connection manager's fallback and accounting changes (#250) are outside these targets. None of the changes touches the wire format, the key derivation or the transport state machine, so R2's rerun rule doesn't apply. As a check, every grown corpus was also replayed once through a CI-flag binary built at 529e19f, #233's current head: all 48,787 inputs, 21 of 21 corpora, ran without a failure.

    Limits of this evidence.

    Acceptance.

    • Report: this comment. No crash is open.
    • New seeds: the campaign found no crash or timeout input, so there are no regression inputs to add. The committed seeds in test/fuzz/qbit_corpora are generated by generate_seeds.py and contain no fuzzer output. The grown corpora reach code they don't: 271 more edges in p2p_v2_pq_malicious_peer (asan-native), and 5 to 26 in the ML-KEM and record-parser targets. A pull request into 1.1.0 will add generated seeds for that code. R2 closes when it merges.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    pq-transportHybrid post-quantum v2 transport (#184)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions