test(m8a): prove the cutover mechanism — LISTENER SWAP (no NAT), isolated netns

.gitignore

@@ -3,3 +3,7 @@
 # Third-party pjproject source fetched by scripts/fetch-pjproject.sh for the
 # optional `pjproject-cffi` feature of the pjsip-shim crate. Not committed.
 /crates/pjsip-shim/vendor/
+
+# python bytecode
+__pycache__/
+*.pyc

tests/m8a-cutover/Dockerfile

@@ -0,0 +1,28 @@
+# M8a cutover-mechanism proof container.
+#
+# Built with network (to install packages), then RUN with
+#   --network none --cap-add=NET_ADMIN
+# so the proof executes in a private, empty network namespace: only `lo`, no
+# route to the host, its OWN nftables ruleset. Nothing it does (nft rules, port
+# binds, floods, conntrack) can reach the host netns, the live trunk, port
+# 45070, the router, or the cluster.
+FROM debian:stable-slim
+
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends \
+      nftables \
+      iproute2 \
+      conntrack \
+      python3 \
+      procps \
+      iputils-ping \
+ && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /m8a
+COPY listener.py sender.py burst.py assert_boundary.py cutover_lib.sh \
+     apply-fs-to-rustisk.sh rollback-rustisk-to-fs.sh red-stateful-variant.sh \
+     detector-selftest.sh in-container-proof.sh /m8a/
+RUN chmod +x /m8a/*.sh /m8a/*.py
+
+# The container just sleeps; the driver drives it via `docker exec`.
+CMD ["sleep", "infinity"]

tests/m8a-cutover/README.md

@@ -0,0 +1,116 @@
+# M8a — Cutover mechanism: proof of the LISTENER SWAP
+
+This directory selects and **proves** the mechanism M9 will use to switch the
+public UDP trunk endpoint (`tron:45070`, RTP `20000-20100`) between FreeSWITCH
+and rustisk. M9 invokes exactly these artifacts.
+
+The whole proof runs inside **one isolated container network namespace**
+(`--privileged --network none`) on **synthetic high ports** — it never touches
+the host nftables, host ports, the live trunk, port 45070, the router, or the
+cluster. See "Isolation" below.
+
+## Selected mechanism: LISTENER SWAP (no NAT)
+
+FreeSWITCH's trunk profile is **stopped** (freeing `hostIP:45070`, FS process
+stays up), then rustisk **binds `hostIP:45070` directly**. Delivery is by
+per-packet socket lookup; **no NAT, no conntrack, no redirect**. A preinstalled
+**fail-closed DROP** on the guarded dport is enabled only across the handover so
+the sender sees plain loss (SIP retransmission covers it) instead of ICMP
+port-unreachable.
+
+Why this over the alternates (PLAN-v3 design table, §M8a):
+
+| Shape | Stateless? | Verdict |
+|---|---|---|
+| **Listener swap** — FS profile stop → rustisk binds 45070. No NAT. | **Yes** — per-packet socket lookup; conntrack never steers un-NAT'd traffic. | **SELECTED** |
+| prerouting `redirect`/`dnat` to an alt port | No — stateful NAT | DEAD (see RED control) |
+| Host→pod DNAT | No — stateful NAT + new forward-hook policy | strictly worse |
+| conntrack-zone toggle wrapping a NAT lever | isolates state deliberately | documented alternate |
+| tc/eBPF stateless rewrite | Yes | heavy lift, new failure surface |
+| re-point the **router** forward | No — NAT we can't inspect/flush | rejected |
+
+Verified reasons the listener swap wins (all confirmed by the proof):
+1. **No NAT ⇒ the conntrack objection is structurally absent.** The table this
+   mechanism installs (`cutover_lib.sh`) is filter-only: no `ct`, `nat`,
+   `dnat`, or `redirect` — same shape as today's `voice-trunk.nft`.
+2. **The existing source allowlist keeps protecting `udp dport 45070`
+   unchanged** — same port, same host, same rule. No new blast radius. The RED
+   control shows a port-rewriting lever silently **bypasses** that drop.
+3. **Per-profile stop/start on live FS is already proven in-repo** (the trunk
+   watchdog's `sofia profile <name> start`). Rollback is that same command.
+4. **rustisk binds the public port directly**, so Via/Contact carry 45070 —
+   New-3's external-port hazard N/A on the primary path.
+
+Design constraints honored: **New-5** (no K8s Service on the trunk ports — a
+NodePort would make kube-proxy install stateful DNAT and reintroduce the defect;
+the mechanism is hostNetwork, no Service); **New-3** (direct bind of the public
+port); and **no `flush ruleset` ever** — the cutover table is separate and
+independently deletable (`nft delete table inet cutover`).
+
+## What M9 invokes (the artifacts)
+
+| File | Role |
+|---|---|
+| `cutover_lib.sh` | the nft primitives: `cutover_table_up/down`, `handover_drop_on/off`. **Filter-only table.** The only host-networking surface. |
+| `apply-fs-to-rustisk.sh` | **APPLY** (FS→rustisk): drop-on → stop FS → wait released → bind rustisk → wait bound → drop-off. Measures + prints the window. |
+| `rollback-rustisk-to-fs.sh` | **ROLLBACK** (rustisk→FS): the exact mirror; `START_NEW_CMD` is the watchdog's `sofia profile start`. |
+
+Both scripts inject the FS/rustisk specifics as commands, so the orchestration
+is **identical** in this synthetic proof and in the live M9 cutover — only the
+hooks differ:
+
+```
+# M8a synthetic (this harness):
+STOP_OLD_CMD="kill -USR2 $FS_PID"          # release the port from the stand-in
+START_NEW_CMD="kill -USR1 $RUSTISK_PID"    # bind the port on the stand-in
+
+# M9 live (substitute):
+PORT=45070 NFT_TABLE=voicefw \
+STOP_OLD_CMD="fs_cli -x 'sofia profile <trunk> stop'" \
+WAIT_RELEASED_CMD="<poll until 45070 free>" \
+START_NEW_CMD="<start rustisk / claim 45070>" \
+WAIT_BOUND_CMD="<poll until rustisk holds 45070>" \
+  ./apply-fs-to-rustisk.sh
+# rollback: STOP_OLD_CMD stops rustisk; START_NEW_CMD = fs_cli -x 'sofia profile <trunk> start'
+```
+
+In M9 the source-drops are the **existing** `voice-trunk.nft` allowlist (already
+protecting dport 45070) — the cutover adds no new firewall rules. The
+`UNTRUSTED_V4/V6` knobs here exist only to *prove* the drop holds across the
+handover.
+
+## The proof harness
+
+| File | Role |
+|---|---|
+| `run-proof.sh` | top-level driver: builds the image, runs it `--privileged --network none`, asserts host-side `NetworkMode=none`, execs the proof, reaps the container + image. |
+| `Dockerfile` | debian-slim + nftables/iproute2/conntrack/python3. |
+| `in-container-proof.sh` | steps 1–5 over a veth pair + child netns ("chime"→"tron"), so traffic traverses **prerouting+input** like the real trunk (not loopback). |
+| `listener.py` | stand-in that binds the port **on command** (SIGUSR1/2), dual-stack, and writes one CSV row per received datagram — the receiver-side ground truth. **No SO_REUSEPORT**: exactly one holder at a time. |
+| `sender.py` / `burst.py` | numbered-datagram senders pinning a fixed five-tuple. |
+| `assert_boundary.py` | receiver-side assertion: disjoint captures, single clean FS→RUSTISK→FS boundary per transition, and untrusted tags delivered **nowhere**. |
+| `red-stateful-variant.sh` | RED control — a stateful `dnat`/redirect lever that **fails** the proof where the listener swap passes (teeth). |
+| `detector-selftest.sh` | proves `assert_boundary.py` itself rejects crafted pathologies (overlap, interleave, missing rollback, untrusted delivery). |
+| `RESULTS.md` | the committed passing transcript + measured windows. |
+
+## Run it
+
+```
+./run-proof.sh                        # build + run + reap, transcript to stdout
+TRANSCRIPT=out.txt ./run-proof.sh     # also tee the transcript to out.txt
+```
+
+Requires Docker with `--privileged` (for nft + nested netns inside the
+container). If privileged Docker is unavailable, the proof **stops** — it never
+falls back to the host netns.
+
+## Isolation (the whole risk)
+
+- `--network none` ⇒ the container's netns has only `lo`; the internal veth pair
+  lives entirely inside it. There is **no route to the host** (the proof asserts
+  `192.168.0.109` is unreachable) and no published ports (`run-proof.sh` asserts
+  `NetworkMode=none`).
+- The container has its **own** nftables ruleset (empty before setup) — separate
+  from the host's. Every `nft`/`ip`/`conntrack`/bind runs inside the container.
+- Synthetic ports only (`55070`, `55190/55192`) — never `45070`/`20000-20100`.
+- **No `flush ruleset`** anywhere; tables are dropped by name.

tests/m8a-cutover/RESULTS.md

@@ -0,0 +1,140 @@
+# M8a — Results (measured windows + passing transcript)
+
+Mechanism: **LISTENER SWAP** (no NAT). Verdict: **GO** for M9, gated on M0a's
+Chime endpoint-down tolerance vs. the measured window (below).
+
+Run it yourself: `./run-proof.sh`. The transcript below is representative;
+windows vary a few ms run to run. Reproduced cleanly across repeated runs.
+
+## Measured handover windows (both directions)
+
+| Direction | Mechanism-floor window | Receiver-observed delivery gap |
+|---|---|---|
+| **FS → rustisk** (apply) | **~47–53 ms** | ~26–28 ms (12–13 datagrams @ 2 ms) |
+| **rustisk → FS** (rollback) | **~49–52 ms** | ~28–30 ms (13–14 datagrams @ 2 ms) |
+
+The **mechanism-floor** window is nft-drop-toggle + socket close + socket bind +
+IPC/status-poll, measured in an isolated netns. It is the *floor*: the **live M9
+window additionally includes FS `sofia profile stop/start` + DNS re-resolve**,
+which this synthetic proof cannot measure. Feed the live number from M0a.
+
+### bind-on-command note (go/no-go)
+
+The synthetic floor (~50 ms) is well under any plausible SIP retransmission /
+endpoint-down tolerance, and the fail-closed drop converts the gap to loss (not
+ICMP reject). **On this evidence, bind-on-command is NOT required** — do not
+build it yet. Revisit only if M0a's live measurement (FS profile stop/start +
+DNS) pushes the *rustisk-side* bind past Chime's tolerance; then the mitigation
+is bind-on-command for rustisk (today the SIP transport binds at startup and a
+bind failure is fatal by design — `crates/rustisk-cli/src/main.rs:2189-2192`,
+PR #65) so rustisk can claim `45070` the instant FS releases it. Flagged for the
+coordinator as a **conditional** work item, not scheduled.
+
+## Passing transcript (steps 1–5 + RED control + detector self-test)
+
+```
+=================== M8a LISTENER-SWAP PROOF ===================
+synthetic dport=55070 (NOT 45070)  rate=2ms  ingress via veth (prerouting+input)
+--- ISOLATION (must all hold before traffic) ---
+main-netns links: lo,veth0,
+  host LAN 192.168.0.109: NO ROUTE (isolated) OK
+  nft ruleset before setup:
+--- cutover table (filter only; no nat/ct/redirect) ---
+  table inet cutover {
+  	chain input {
+  		type filter hook input priority filter - 10; policy accept;
+  		udp dport 55070 ip saddr 10.9.0.3 drop comment "srcdrop-v4"
+  		udp dport 55070 ip6 saddr fd00::3 drop comment "srcdrop-v6"
+  	}
+  }
+step 1: primed TRUST(v4) five-tuple 10.9.0.2:41002 -> 10.9.0.1:55070
+        untrusted EVILV4(10.9.0.3) + EVILV6(fd00::3) flooding continuously
+step 2: APPLY switch FS -> rustisk
+APPLY FS->rustisk handover window: 47 ms
+step 3: ROLLBACK rustisk -> FS (same continuously-flowing tuple)
+ROLLBACK rustisk->FS handover window: 52 ms
+--------------- RECEIVER-SIDE ASSERTIONS (listener swap) ---------------
+PASS source-drop EVILV4: delivered_to_FS=0 delivered_to_RUSTISK=0 (must be 0/0)
+PASS source-drop EVILV6: delivered_to_FS=0 delivered_to_RUSTISK=0 (must be 0/0)
+PASS disjoint captures (no split-brain): overlap_count=0
+PASS ownership run pattern by seq = [FS,RUSTISK,FS] (expected [FS,RUSTISK,FS])
+PASS boundary FS->RUSTISK: last FS seq=745, first RUSTISK seq=758 (clean: FS.max < RUSTISK.min); handover gap = 12 datagrams / 26.0 ms delivery gap
+PASS boundary RUSTISK->FS: last RUSTISK seq=1519, first FS seq=1533 (clean: RUSTISK.max < FS.min); handover gap = 13 datagrams / 28.0 ms delivery gap
+trusted captured: FS=1509 RUSTISK=762 seq_span=1..2296
+RESULT: PASS
+--------------- MEASURED HANDOVER WINDOWS (mechanism floor) ---------------
+FS->rustisk apply    window: 47.1 ms
+rustisk->FS rollback window: 52.3 ms
+
+=================== RED CONTROL (stateful redirect lever) ===================
+--- (A) source-drop bypass: untrusted 10.9.0.3 -> 55190, drop is on dport 55190 ---
+    (redirect DNATs 55190->55192 at prerouting, past the dport-55190 drop)
+--- (B) rollback-persistence probe (honest): switch then rollback a primed tuple ---
+    cumulative captured lines (FS=55190, RU=55192):
+      prime (no rule):     [FS=822 RU=0]
+      after switch (dnat): [FS=824 RU=1556]  (RU grew => switch worked)
+      after rollback (del):[FS=1373 RU=1582]  (FS grew => reverted cleanly)
+    => rollback reverts to FS on tron's kernel; conntrack-persistence objection NOT reproduced.
+--- RED machine verdict (assert_boundary on the redirect source-drop capture) ---
+FAIL source-drop EVILV4: delivered_to_FS=0 delivered_to_RUSTISK=695 (must be 0/0)
+FAIL ownership run pattern by seq = [RUSTISK] (expected [FS,RUSTISK,FS])
+RESULT: FAIL
+RED TEETH CONFIRMED: redirect lever FAILED the proof (untrusted delivered / no clean swap) — listener swap PASSES the same checks.
+
+=================== DETECTOR SELF-TEST (assert_boundary teeth) ===================
+  detector[good]: expected PASS, got PASS  OK
+  detector[overlap]: expected FAIL, got FAIL  OK
+  detector[interleave]: expected FAIL, got FAIL  OK
+  detector[norollback]: expected FAIL, got FAIL  OK
+  detector[untrusted]: expected FAIL, got FAIL  OK
+detector self-test: ALL OK (accepts clean, rejects every pathology)
+
+=================== SUMMARY ===================
+listener-swap assert  rc=0  (0 = clean boundaries + source-drop hold)
+RED control (redirect) rc=0  (0 = redirect FAILED the proof as expected -> teeth)
+detector self-test     rc=0  (0 = assert_boundary rejects every crafted bad capture)
+M8a RESULT: PASS
+```
+
+## What each step proved
+
+1. **Primed tuple** `10.9.0.2:41002 → 10.9.0.1:55070`, numbered datagrams flowing
+   continuously (seq 1…2296), never restarted across both transitions.
+2. **Switch FS→rustisk**: single clean boundary — FS captured through seq **745**,
+   rustisk from seq **758**; the 12-datagram gap is the handover drop window. No
+   overlap, no interleaving, no split-brain (`overlap_count=0`).
+3. **Rollback rustisk→FS** under the *same* flow: reverse boundary just as clean —
+   rustisk through **1519**, FS from **1533**. (This is the step a stateful/redirect
+   lever is claimed to fail; the no-NAT swap passes it.)
+4. **Source drop holds throughout** both transitions for **v4 and v6**: the
+   untrusted `10.9.0.3` (v4) and `fd00::3` (v6) sources were delivered to
+   **neither** stand-in at any point (`0/0`). A genuine DROP in an accept-policy
+   chain, not an allow rule.
+5. **Windows** measured both directions (above).
+
+## RED control — teeth, and an honest finding
+
+The RED control runs a **stateful `dnat`/redirect lever** through the same path:
+
+- **(A) It FAILS the proof deterministically.** Because it rewrites the dport
+  (`55190 → 55192`) at prerouting, the fail-closed source-drop written for dport
+  `55190` no longer matches, so the **untrusted source is delivered** to the
+  successor (`delivered_to_RUSTISK=695`). `assert_boundary` flags it → **RED**.
+  The listener swap keeps the same port, so its drop holds (`0/0`). This is
+  exactly PLAN-v3's hazard: *"any port-rewriting mechanism moves packets to a
+  port the current filter does not match… would need a genuine fail-closed DROP
+  for untrusted sources on the new port, v4 and v6."*
+
+- **(B) Honest negative result on the conntrack-persistence claim.** PLAN-v3 C1
+  argues the redirect *cannot switch a primed tuple back* because conntrack
+  persists. **Measured on tron's kernel, it reverts cleanly on rule removal**
+  (the probe shows FS reclaiming the flow after the rule is deleted; a directly
+  injected stale conntrack entry was ignored). So on this kernel the redirect
+  dies on **(A)** — the security regression — not on the rollback boundary. The
+  listener swap is still the right choice: it introduces **no** NAT/conntrack
+  into a NAT-free path, and its behavior does not depend on kernel/conntrack
+  semantics that were observed to vary by birth-condition during this work.
+
+- The **detector self-test** proves `assert_boundary` is not rigged to always
+  pass: it rejects overlap/split-brain, interleaving, a missing rollback
+  boundary, and any untrusted delivery, while accepting a clean capture.

tests/m8a-cutover/apply-fs-to-rustisk.sh

@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+# apply-fs-to-rustisk.sh — THE APPLY ARTIFACT M9 invokes (FS -> rustisk).
+#
+# Listener swap, no NAT. Sequence:
+#   1. handover_drop_on   — fail-closed drop on the guarded dport, so the
+#                           sender sees loss (SIP retransmits) not ICMP reject
+#                           during the window the port is unbound.
+#   2. STOP_OLD_CMD       — release the port from the incumbent (FS).
+#                           M9: fs_cli -x 'sofia profile <trunk> stop'
+#   3. WAIT_RELEASED_CMD  — block until the port is actually free.
+#   4. START_NEW_CMD      — bind the port on the successor (rustisk).
+#                           M9: start rustisk / claim hostIP:45070.
+#   5. WAIT_BOUND_CMD     — block until the successor holds the port.
+#   6. handover_drop_off  — reopen; traffic now lands on rustisk.
+#
+# The FS/rustisk specifics are injected as commands so THIS orchestration is
+# identical in the M8a synthetic proof and in the M9 live cutover — only the
+# STOP_OLD/START_NEW hooks differ. The measured wall-clock between step 1 and
+# step 6 is the handover window reported as M9's go/no-go input.
+#
+# Env: PORT, NFT_TABLE (+ UNTRUSTED_V4/6 already installed by the caller),
+#      STOP_OLD_CMD, WAIT_RELEASED_CMD, START_NEW_CMD, WAIT_BOUND_CMD,
+#      WINDOW_OUT (file to write the measured window, ns).
+set -euo pipefail
+HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=cutover_lib.sh
+source "${HERE}/cutover_lib.sh"
+
+: "${STOP_OLD_CMD:?}" ; : "${START_NEW_CMD:?}"
+: "${WAIT_RELEASED_CMD:=true}" ; : "${WAIT_BOUND_CMD:=true}"
+
+t0="$(date +%s%N)"
+handover_drop_on
+eval "${STOP_OLD_CMD}"
+eval "${WAIT_RELEASED_CMD}"
+eval "${START_NEW_CMD}"
+eval "${WAIT_BOUND_CMD}"
+handover_drop_off
+t1="$(date +%s%N)"
+
+win_ms=$(( (t1 - t0) / 1000000 ))
+echo "APPLY FS->rustisk handover window: ${win_ms} ms (${t0} -> ${t1} ns)"
+if [ -n "${WINDOW_OUT:-}" ]; then echo "$((t1 - t0))" > "${WINDOW_OUT}"; fi