test(session-net): isolate connection metrics assertions

[VerifiedOrganic]

Summary

• Replace the two unit-test global five-counter snapshots with a private task-local accounting sink.
• Capture that sink in each test-owned lifecycle so drain start, idle retirement, and final drain completion remain attributable when ownership moves to another Tokio task.
• Mirror success and timeout accounting only at the generic-server and consensus-server paths owned by the current assertions.
• Serialize the three same-process integration tests that read or indirectly write maximum-age/local-leaf/peer-leaf retirement counters.
• Preserve every production METRICS increment and classification unchanged.

Refs #576

Exact revision

• Original reviewed base: 92de6656dfe4ab857242ca0a0380167116ab53b0
• Original reviewed base tree: 48cc612a8bd5beddceff7cbb42ade53abee7f8cf
• RED commit: 9c41765bf5987c5bfc5d9877a24a95c8a45db112
• RED tree: 1a7b4622c72e1a660a2e0434f716dd5155372fe7
• Reviewed implementation commit: 537f4c518f57c986a9090da93776362a76ee60d2
• Reviewed implementation tree: eea3557e6d5c83b7cdff9360fe8583aa0531dd25
• Current base (origin/main fetched with prune): 9b85ae1f8a98f5fe9f9f77225f72e5364840b5b1
• Current base tree: 4cb0806bd910c32a3693dfa4f3237360c0f4385e
• Reconciliation merge/head: 7a627224ee4b7d547c729323a26634254c3d7c42
• Reconciled head tree: dfd83e68690f8b5a521311ee328bcab7ac375a54
• Merge parents: 537f4c518f57c986a9090da93776362a76ee60d2 and 9b85ae1f8a98f5fe9f9f77225f72e5364840b5b1
• Branch: fix/576-metrics-test-race-wm-20260815

The RED commit predates the manager's identity correction and intentionally retains the inherited agent-codex <verifiedorganic@sent.com> author/committer and sign-off. It was preserved without rewriting. The implementation and reconciliation merge commits are authored, committed, and signed off by VerifiedOrganic <verifiedorganic@sent.com>.

origin/main merged normally with no conflicts. Its aggregate delta touched only CHANGELOG.md, crates/opc-session-net/src/consumer.rs, and crates/opc-session-net/tests/stateless_quorum_consumer.rs; none overlaps the issue implementation. The PR diff against current main remains the same five issue-owned files.

Changed files

• crates/opc-session-net/src/consensus.rs
• crates/opc-session-net/src/lifecycle.rs
• crates/opc-session-net/src/server.rs
• crates/opc-session-net/src/test_support.rs
• crates/opc-session-net/tests/authenticated_replica_identity.rs

Counter ownership inventory

The affected process-global family contains 24 counters: eight retirement reasons, two lifecycle gauges, three drain counters, nine connection-outcome counters, and two reconnect counters.

• The generic-server and consensus-server unit assertions were the raw five-counter delta readers. They now read per-test accounting for idle retirement, timeout failure, success, drain start, and drain completion.
• Lifecycle accounting captures an Arc at construction, so a final drop from an aborted or handed-off task remains owned by the originating test. Unrelated spawned test writers do not inherit the task-local.
• Connection-attempt/superseded/abandoned assertions already use their separate per-test accounting.
• rotation_bootstrap_retirement is a separate integration binary with exactly one top-level test; all raw-global helpers are awaited serially in that process.
• authenticated_replica_identity reads maximum-age/local-leaf/peer-leaf retirement counters. Its two reader tests and the one indirect real-mTLS leaf-expiry writer share one integration-test-local Tokio mutex.
• Active/draining/overrun and the remaining outcome/reconnect fields have no competing test-side delta readers. Certificate-chain tests compare counter identities, not values.

Deterministic RED and mutation

RED at 9c41765b:

opc-heavy cargo test --locked -p opc-session-net --lib --all-features consensus::tests::connection_outcome_delta_isolated_from_writer_outside_test_scope -- --exact --nocapture

Result: exit 101, 0 passed / 1 failed / 203 filtered. A spawned writer outside the old lock boundary plus the assertion-owned writer produced idle retirement/drain deltas of 2 while the assertion required 1.

Fix-removal mutation: temporarily disabled lifecycle capture of the scoped accounting. The same deterministic test failed with observed 0 versus expected 1. The mutation was restored, and the final test passed.

Focused verification

All current-head commands ran through the shared heavy lock where applicable, used repository-default build parallelism, and used no Cargo target/cache/wrapper/job overrides.

• Deterministic isolation test: 1 passed, 0 failed.
• Generic-server outcome test alone: 1 passed, 0 failed.
• Consensus-server outcome test alone: 1 passed, 0 failed.
• Each of the three serialized identity/retirement tests alone: 1 passed, 0 failed.
• Pre-reconciliation four-way reproduction: 40/40 full library runs passed, 0 failed; 204 tests per run, 8,160 executions total.
• Current-head four-way reproduction: 40/40 full library runs passed, 0 failed; 205 tests per run, 8,200 executions total. Every scheduled run executed; none was retried or hidden.
• Pre-reconciliation complete opc-session-net suite: 315 passed, 0 failed; doc tests passed.
• Current-head complete opc-session-net suite at four-way test parallelism: 319 passed, 0 failed; doc tests passed.
• cargo clippy --locked -p opc-session-net --all-targets --all-features -- -D warnings: passed.

Exact repository gates

• cargo fmt --all --check: passed.
• git diff --check: passed.
• cargo clippy --locked -p opc-persist --all-targets --no-default-features -- -D warnings: passed.
• cargo test --locked -p opc-persist --no-run: passed.
• Four prescribed opc-persist integration binaries with --test-threads=1: 38 passed, 0 failed.
• cargo clippy --workspace --all-targets --all-features -- -D warnings: passed.
• cargo test --workspace --all-features --quiet -- --test-threads=4: passed with status 0; prescribed privileged tests remained ignored.
• Reference operator /usr/local/go/bin/go vet ./... and /usr/local/go/bin/go test ./...: passed with Go 1.26.6.
• kubectl kustomize operators/sdk-reference-operator/config/default: passed.

Adversarial review

P0: none. P1: none. P2: none.

The implementation review found no deadlock, poisoning, async-lock misuse, task-local/final-drop attribution error, cross-test leakage, current reader/writer gap, or production behavior change. The remaining integration mutex is a Tokio mutex, is acquired only at three top-level test boundaries, and has no nested acquisition.

The current-main delta review also found no P0/P1/P2 issue or ownership overlap. Its new consumer integration tests indirectly exercise ordinary production counters only in their separate test binary and do not add a global-counter reader.

Hosted status

• Prior exact head 537f4c518f57c986a9090da93776362a76ee60d2: the preserved initial snapshot at 2026-08-15T23:13:38Z was 36 total, 24 passed, 12 pending, 0 failed or cancelled. The terminal result was 37/37 successful, last completion at 2026-08-15T23:19:48Z.
• Reconciled exact head 7a627224ee4b7d547c729323a26634254c3d7c42: terminal at 2026-08-16T03:18:54Z; 37/37 passed, 0 failed, 0 cancelled, 0 skipped. The aggregate Rust workspace check passed.

The PR remains open and draft.

Exclusions

• No issue test: replace yield_now spin-waits, two of which hang under a paused clock #578 backend/watch metrics sweep.
• No production counter semantic or multiplicity change.
• No broad metrics refactor.
• No Go/toolchain, operator, Helm, or unrelated issue change.
• No CRC, kernel, cluster, live privileged, or ignored-test execution.
• No privileged evidence claimed.