feat(session-net): add bounded persistent least-authority consumer transport

[VerifiedOrganic]

Downstream parent: https://github.com/VerifiedOrganic/epdg/issues/181

Problem

The production least-authority session consumer transport added by #650 intentionally performs one authenticated request on a fresh connection. At current main (9b85ae1f8a98f5fe9f9f77225f72e5364840b5b1), StatelessSessionConsumerClient resolves, connects, performs TLS/mTLS and the authenticated Hello exchange for every logical operation. A successful ePDG IKE_AUTH round performs many such serial state operations, and real evidence in the downstream P0 shows 4.9–14.8 second response latency with extreme connection/resolver churn.

The legacy persistent RemoteSessionBackend is not an acceptable substitute: it carries replication/rebuild/consensus authority outside the typed consumer boundary and allows only one in-flight request. This capability belongs in opc-session-net; ePDG must not invent a product-local wire protocol or pool.

Required design

Add a bounded persistent, authenticated transport for the existing typed SessionConsumer authority.

• Maintain a small fixed configured pool per logical endpoint/voter. Connection count is proportional to clients/workers × voters × pool width, never subscribers or records.
• Reuse warm mTLS connections across logical calls. Support bounded multiplexing with explicit correlation when the wire revision supports it; otherwise use a bounded fair pool with one in-flight request per connection and no global head-of-line blocking.
• Bound connections, pending acquisition, in-flight requests, per-connection frames, global queue depth, response storage, reconnect attempts, and shutdown drain. Overload must fail quickly with a typed outcome.
• Preserve exact ALPN, SPIFFE peer identity, scope, revision and least-authority admission. Do not expose raw consensus, replication, rebuild, snapshot, membership or administrative calls.
• Resolve only when establishing/re-establishing a connection. Preserve endpoint replacement, stable server name/identity, bounded setup, reconnect backoff with jitter, and drain/reconnect on certificate/SVID rotation without a fleet-wide reconnect storm.
• Preserve the existing write boundary: only a request proven not transmitted may be retried automatically. A mutation that may have crossed the boundary remains OutcomeUnknown; allowed reconnect/failover attempts retain the exact durable request ID and body.
• Correlate every response. Duplicate, unknown, late, oversized or mismatched correlation IDs fail closed without completing another caller. Cancellation must not make a late response reusable by a new request.
• Keep latency-sensitive request capacity separate from watches/long-lived streams.
• Expose bounded, value-free snapshots/metrics for setup phase, pool wait, active/max/idle connections, reuse, reconnects, bounded queue/in-flight counts, oldest queue age and outcome class. No keys, values, request IDs, owners/fences, endpoint addresses, identities or credentials in labels, logs, errors or Debug.
• Provide a readiness/prewarm surface that proves the configured minimum authenticated capacity without making product readiness depend on subscriber traffic.

Required tests and evidence

• Retain RED evidence that repeated typed calls on the current client create a fresh authenticated connection each time and miss the downstream latency budget under deterministic setup delay.
• Cover pool width and queue bounds, fairness between independent session keys, cancellation, late/out-of-order responses, duplicate/unknown correlations, reconnect, endpoint replacement, certificate rotation, identity/scope/revision mismatch, shutdown drain and overload.
• Cover read and mutation semantics across failures immediately before write, after partial/full write, after server commit and before response; prove no automatic replay after MayHaveWritten.
• Cover 3- and 5-voter clients, 12+ concurrent callers, leader change/voter loss and one slow caller without global head-of-line blocking.
• Add deterministic fix-removal and separate adversarial mutations for correlation isolation and ambiguous-outcome handling.
• Benchmark warm calls with exact methodology and raw bounded samples; target warm quorum p99 <=25 ms and p99.9 <=100 ms in a real three-voter mTLS environment. Synthetic/loopback results must be labeled as such and cannot claim production capacity.
• Run formatting, warning-denied Clippy, focused tests and repository canonical full CI. Record exact base/head/tree and obtain unconditional exact-head adversarial SHIP review with no unresolved P0/P1/P2.

Coordination

• SDK PR fix(session-store): enforce protected payload limits #684 currently changes shared framing/limit documentation and must not be weakened or bypassed. Keep implementation consumer-specific where possible, merge current main normally after fix(session-store): enforce protected payload limits #684 lands, and rerun affected evidence/review.
• SDK PR test(session-net): isolate connection metrics assertions #692 touches connection-metrics tests; coordinate metric test isolation without depending on its unmerged tree.
• Closing this issue does not complete the downstream P0. ePDG must separately compose the API and collapse its durable SWm transitions.

Refs https://github.com/VerifiedOrganic/epdg/issues/181

[VerifiedOrganic]

PR #697 delivered the bounded persistent transport, but its performance evidence is explicitly synthetic and non-gating. Keep this issue open for the required real three-voter mTLS qualification and production latency evidence.

[VerifiedOrganic]

Current-main correction landed; production acceptance remains open

The ancestry audit found that SDK main f2ed1181c85540cc01ea0b4611fa3620891375fd did not supersede the remaining generic persistent-consumer transport correction. #710 fixed authenticated server idle-lane reuse and #711 added receipt reconciliation, but neither supplied the remaining prewarm freshness, reconnect serialization, composite correlation, below-TLS effect boundary, or cancellation properties.

The clean successor is now normally merged through #716:

• required source base f2ed1181c85540cc01ea0b4611fa3620891375fd, tree 945ceab3870d2c1d2d1396aff29e819288fce76a;
• signed source head 5f69c9bd1383a0b1efbb2c453fd5e2df50183bc6, tree 404390bf1b67e3971302fba59fb6e9c612381206;
• five direct additive commits, zero merge commits, exact merge-base equal to the required source base;
• normal merge 7a6a1f003801811867c31275bbaae6c410868399, tree 2be68eae758accbbf95781e17a7d0cc8eda1bf6b, parents 549efe523be4461845ac1ac6c72759107b591e65 and 5f69c9bd1383a0b1efbb2c453fd5e2df50183bc6;
• the source branch remains present; it was not rebased, force-pushed, locally merged, or deleted.

Exact changed files:

crates/opc-session-net/README.md
crates/opc-session-net/src/consumer.rs
crates/opc-session-net/src/lifecycle.rs
crates/opc-session-net/src/protocol.rs
crates/opc-session-net/tests/persistent_consumer_boundaries.rs
crates/opc-session-net/tests/persistent_consumer_protocol.rs
crates/opc-session-net/tests/persistent_consumer_transport.rs
crates/opc-session-net/tests/stateless_quorum_consumer.rs
crates/opc-session-testkit/qualification/v8/session-ha-persistent-consumer-head-evidence.schema.json
crates/opc-session-testkit/tests/qualification_profile.rs
docs/ha-design.md
docs/operator-readiness.md
docs/qualification-695-persistent-consumer.md
docs/rfc/004-session-store.md
docs/session-store-atomic-fenced-transition-evidence.md
docs/session-store-atomic-fenced-transition.md

The tracked evidence is docs/qualification-695-persistent-consumer.md. The bounded structured v8 schema is here; its SHA-256 is 5e3becf5094f3e222b94799e0fb7b6b77c3398aeabae743fc65b409c4cd4adfd, and it requires experimental=true plus qualification_complete=false.

Landed generic contract

Revision 5 retains the existing opc-session-consumer/1 ALPN and requires exact mTLS trust, live peer SPIFFE identity, tenant scope authorization, and exact Hello revision before a lane is published or reused. There is no downgrade or mixed-revision fallback.

The fixed bounds remain: four request lanes by default / 16 maximum; 64 queued calls by default / 256 maximum; 250 ms queue age; two isolated watch slots by default / 16 maximum; one in-flight request per lane; 4,096 sequential calls per connection; 1,500 ms setup; at most two pre-write establishment attempts; one pool-wide cold recovery lane; 50 ms default shared backoff plus at most 25 ms jitter; 5-second shutdown drain and idle bounds; 10-second maximum complete operation; 16 MiB frame; 256 listener connections; and clone-lineage physical caps of 16 request plus 16 watch connections.

The correction adds rolling fresh DNS/TCP/TLS/Hello prewarm, serialized/coalesced recovery, direct credential/material supersession without releasing the permit before losing I/O is destroyed, same-authority cooldown preservation, sequence-plus-random-UUID response correlation, exact late/duplicate/unknown rejection, authoritative below-TLS positive-write observation, and peer-EOF cancellation. Only NotTransmitted is automatically retryable with an unchanged mutation ID/body; any positive lower write is OutcomeUnknown and is never replayed automatically. Diagnostics are fixed-label and omit endpoints, identities, scopes, credentials, keys, payloads, request/correlation IDs, owners, and fences.

Exact gates and review

Focused current-head GREEN:

• opc-session-net library: 279/279;
• persistent boundaries/protocol/transport: 7/7, 26/26, 30/30;
• qualification profile: 19/19;
• workspace all-feature/all-target warning-denied Clippy, warning-denied rustdoc, default/no-default checks, format, diff, and cargo deny: passed.

The exact updated-main merge-ref tree passed 38/38 GitHub checks in workflow runs 32614548432, 32614548331, 32614548346, 32614548397, and 32614548398. Three immutable-source reviewers and the independent composed-tree reviewer all returned SHIP with P0/P1/P2 0/0/0: source review record, composed-tree review.

Eight causal RED cases are retained in the evidence document, including fresh rolling prewarm (2 resolver calls observed vs 4 required), 12-caller cold recovery (12 concurrent resolver setups vs 1 required), wrong future-response classification, stale credential/material setup, adapter-only buffering, stale cooldown publication, lost same-authority cooldown, and permit transfer before losing-I/O destruction.

No new latency samples were collected. The only raw distribution remains the tracked historical synthetic loopback observation:

[558, 397, 390, 411, 389, 400, 380, 385, 402, 383, 411, 386, 385, 394, 381, 399] microseconds

It used an in-process loopback TLS fixture, three request lanes, one watch slot, and 16 sequential warm calls; min 380 us, median 392.0 us, mean 403.188 us, max 558 us. It is explicitly non-gating and is not production/ePDG SLO evidence.

Remaining acceptance / issue state

No cluster or ePDG state was changed, and no latency-bearing qualification was launched. There is still no real networked three-voter production-mTLS run proving the required representative-concurrency p99/p99.9 SLO, 3-/5-voter and 12+ caller fairness, exact connection/resource counts, movement/loss/write-cut behavior, rotation without storms, or full real-process failure matrix. Therefore #695 remains OPEN; closing it from the synthetic distribution or qualification_complete=false schema would violate its acceptance contract.

Downstream #704/#715/#707 work must consume SDK main at or after 7a6a1f003801811867c31275bbaae6c410868399, preserve the coordinated drained revision-5 cutover, and reconcile any additive consumer protocol work rather than replaying stale consumer.rs history.

[VerifiedOrganic]

Finalization status (2026-08-23): PR #716 was merged at 7a6a1f0, but issue #695 remains open. The retired production-mTLS qualification lane is based at 1e7b869 and preserves six dirty local test/testkit files with no upstream branch or PR. It did not run real latency-bearing production evidence: required three-voter p99/p99.9, 3-/5-voter fairness, failure, and rotation matrix remain unproven. Do not treat this local dirty diff as merged or complete.