Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion tests/m8a-cutover/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ RUN apt-get update \
WORKDIR /m8a
COPY listener.py sender.py burst.py assert_boundary.py cutover_lib.sh \
apply-fs-to-rustisk.sh rollback-rustisk-to-fs.sh red-stateful-variant.sh \
detector-selftest.sh in-container-proof.sh /m8a/
detector-selftest.sh in-container-proof.sh safety-tests.sh /m8a/
RUN chmod +x /m8a/*.sh /m8a/*.py

# The container just sleeps; the driver drives it via `docker exec`.
Expand Down
33 changes: 24 additions & 9 deletions tests/m8a-cutover/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,9 @@ independently deletable (`nft delete table inet cutover`).

| File | Role |
|---|---|
| `cutover_lib.sh` | the nft primitives: `cutover_table_up/down`, `handover_drop_on/off`. **Filter-only table.** The only host-networking surface. |
| `apply-fs-to-rustisk.sh` | **APPLY** (FS→rustisk): drop-on → stop FS → wait released → bind rustisk → wait bound → drop-off. Measures + prints the window. |
| `rollback-rustisk-to-fs.sh` | **ROLLBACK** (rustisk→FS): the exact mirror; `START_NEW_CMD` is the watchdog's `sofia profile start`. |
| `cutover_lib.sh` | the nft primitives: `cutover_table_up/down`, `handover_drop_on/off`. **Filter-only table.** The only host-networking surface. `cutover_table_down` is **guarded** — it deletes only a table this tool created (owner-marker + protected-name denylist); `handover_drop_on/off` are **idempotent** (double-on is a no-op, off removes ALL handover rules). |
| `apply-fs-to-rustisk.sh` | **APPLY** (FS→rustisk): drop-on → stop FS → wait released → bind rustisk → wait bound → drop-off. Measures + prints the window. An **EXIT trap** clears the handover drop on any mid-sequence failure (never leaves the port blocked). |
| `rollback-rustisk-to-fs.sh` | **ROLLBACK** (rustisk→FS): the exact mirror; `START_NEW_CMD` is the watchdog's `sofia profile start`. Same **EXIT trap** as apply. |

Both scripts inject the FS/rustisk specifics as commands, so the orchestration
is **identical** in this synthetic proof and in the live M9 cutover — only the
Expand All @@ -64,20 +64,35 @@ hooks differ:
STOP_OLD_CMD="kill -USR2 $FS_PID" # release the port from the stand-in
START_NEW_CMD="kill -USR1 $RUSTISK_PID" # bind the port on the stand-in

# M9 live (substitute):
PORT=45070 NFT_TABLE=voicefw \
# M9 live (substitute) — uses the DEFAULT, SEPARATE `cutover` table.
# NFT_TABLE defaults to `cutover`; it coexists with the live `voicefw` firewall
# (both input-hook priority -10, both policy accept). The transient handover drop
# lands in `cutover`; voicefw's persistent source allowlist is never touched.
#
# NEVER set NFT_TABLE=voicefw: cutover_table_down deletes its table by name, and
# tearing down would take the live allowlist + SIP-45070 + RTP-20000-20100 drops
# with it. (The guard now REFUSES a foreign/protected table, but the operating
# rule is to keep the tool pointed at its own `cutover` table.)

PORT=45070 ./cutover_lib.sh up # create the SEPARATE `cutover` table (+ owner marker)

PORT=45070 \
STOP_OLD_CMD="fs_cli -x 'sofia profile <trunk> stop'" \
WAIT_RELEASED_CMD="<poll until 45070 free>" \
START_NEW_CMD="<start rustisk / claim 45070>" \
WAIT_BOUND_CMD="<poll until rustisk holds 45070>" \
./apply-fs-to-rustisk.sh
# rollback: STOP_OLD_CMD stops rustisk; START_NEW_CMD = fs_cli -x 'sofia profile <trunk> start'

PORT=45070 ./cutover_lib.sh down # remove the cutover table (GUARDED: only a table THIS tool created)
```

In M9 the source-drops are the **existing** `voice-trunk.nft` allowlist (already
protecting dport 45070) — the cutover adds no new firewall rules. The
`UNTRUSTED_V4/V6` knobs here exist only to *prove* the drop holds across the
handover.
In M9 the persistent **source-drops stay in the existing `voice-trunk.nft`
allowlist** (already protecting dport 45070, in the `voicefw` table) — the
cutover adds no new *source* firewall rules. The separate `cutover` table only
ever holds the transient handover drop, and is created/removed as a unit around
the switch. The `UNTRUSTED_V4/V6` knobs here exist only to *prove* the drop
holds across the handover in the synthetic proof.

## The proof harness

Expand Down
2 changes: 1 addition & 1 deletion tests/m8a-cutover/RESULTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ windows vary a few ms run to run. Reproduced cleanly across repeated runs.
| Direction | Mechanism-floor window | Receiver-observed delivery gap |
|---|---|---|
| **FS → rustisk** (apply) | **~47–53 ms** | ~26–28 ms (12–13 datagrams @ 2 ms) |
| **rustisk → FS** (rollback) | **~49–52 ms** | ~28–30 ms (13–14 datagrams @ 2 ms) |
| **rustisk → FS** (rollback) | **~38–52 ms** | ~28–30 ms (13–14 datagrams @ 2 ms) |

The **mechanism-floor** window is nft-drop-toggle + socket close + socket bind +
IPC/status-poll, measured in an isolated netns. It is the *floor*: the **live M9
Expand Down
14 changes: 14 additions & 0 deletions tests/m8a-cutover/apply-fs-to-rustisk.sh
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,20 @@ HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=cutover_lib.sh
source "${HERE}/cutover_lib.sh"

# Fail-safe: `set -euo pipefail` aborts mid-sequence if STOP_OLD/WAIT/START_NEW
# fails, which would otherwise leave the blanket handover drop STUCK ON (phone
# outage until an operator clears it). This EXIT trap idempotently clears any
# lingering handover drop on EVERY exit path — failure or success — so a failed
# apply fails CLOSED (SIP retransmits) without leaving the port blocked. It
# preserves the original exit code. handover_drop_off is a no-op if the drop is
# already off (the success path clears it before this trap runs).
_apply_on_exit() {
local rc=$?
handover_drop_off 2>/dev/null || true
exit "${rc}"
}
trap _apply_on_exit EXIT

: "${STOP_OLD_CMD:?}" ; : "${START_NEW_CMD:?}"
: "${WAIT_RELEASED_CMD:=true}" ; : "${WAIT_BOUND_CMD:=true}"

Expand Down
80 changes: 72 additions & 8 deletions tests/m8a-cutover/cutover_lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,29 @@
# PORT the guarded UDP dport (required)
# UNTRUSTED_V4 untrusted v4 source to DROP (optional, for the proof)
# UNTRUSTED_V6 untrusted v6 source to DROP (optional, for the proof)
#
# M9 MUST use the default, SEPARATE `cutover` table — it coexists with the live
# `voicefw` firewall (both input-hook priority -10, both policy accept). NEVER
# point NFT_TABLE at `voicefw` (or any other pre-existing host table): the live
# allowlist + RTP/SIP drops are not ours to delete. `cutover_table_down` now
# REFUSES any table it did not create (see the guard below), but the correct
# operating discipline is to never aim the tool at the live firewall in the
# first place.
set -euo pipefail

NFT_TABLE="${NFT_TABLE:-cutover}"

# Tables this tool must NEVER delete, even if asked — the live host firewall and
# common system tables. `cutover_table_down` refuses these outright (defence in
# depth on top of the ownership marker below). Space-separated; overridable.
CUTOVER_PROTECTED_TABLES="${CUTOVER_PROTECTED_TABLES:-voicefw filter nat mangle raw security}"

# Ownership sentinel: a regular (never-hooked) chain that `cutover_table_up`
# installs so `cutover_table_down` can prove THIS tool created the table. A table
# lacking it is foreign and will not be deleted. Never carries packets — regular
# chains only run when jumped to, and nothing jumps to this one.
CUTOVER_OWNER_CHAIN="m8a_cutover_owner"

_nft() { nft "$@"; }

# Create the cutover table + input chain, and install the PERSISTENT
Expand All @@ -34,6 +53,9 @@ _nft() { nft "$@"; }
cutover_table_up() {
: "${PORT:?PORT required}"
_nft add table inet "${NFT_TABLE}"
# Ownership sentinel first: a plain regular chain (never hooked, never jumped
# to). Its presence is how cutover_table_down proves this table is ours.
_nft "add chain inet ${NFT_TABLE} ${CUTOVER_OWNER_CHAIN}"
_nft "add chain inet ${NFT_TABLE} input { type filter hook input priority -10 ; policy accept ; }"
if [ -n "${UNTRUSTED_V4:-}" ]; then
_nft add rule inet "${NFT_TABLE}" input meta l4proto udp udp dport "${PORT}" ip saddr "${UNTRUSTED_V4}" drop comment '"srcdrop-v4"'
Expand All @@ -44,28 +66,70 @@ cutover_table_up() {
}

# Drop the whole table (independently deletable; never `flush ruleset`).
#
# GUARDED: only ever deletes a table THIS tool created. A stray NFT_TABLE=voicefw
# (or any pre-existing host table) would otherwise wipe the live firewall — the
# trusted-source allowlist, the SIP 45070 drop, the RTP 20000-20100 drop. Two
# independent gates must both pass before the delete:
# (1) name is not on the protected/system denylist (voicefw, filter, nat, ...);
# (2) the table carries our ownership sentinel chain (cutover_table_up put it
# there) — a foreign table won't have it.
# Refuses (returns non-zero) rather than deleting when either gate fails. A
# missing table is a no-op success.
cutover_table_down() {
_nft delete table inet "${NFT_TABLE}" 2>/dev/null || true
local t="${NFT_TABLE}" p
# (1) hard refusal: never delete a known host/system table by name.
for p in ${CUTOVER_PROTECTED_TABLES}; do
if [ "${t}" = "${p}" ]; then
echo "REFUSING: 'inet ${t}' is a protected host table; cutover_table_down will not delete it." >&2
return 3
fi
done
# (2) nothing to delete if the table doesn't exist.
if ! nft list table inet "${t}" >/dev/null 2>&1; then
return 0
fi
# (3) ownership check: refuse a table that lacks our sentinel chain — it was
# not created by this tool and is not ours to delete.
if ! nft list table inet "${t}" 2>/dev/null \
| grep -Eq "chain[[:space:]]+${CUTOVER_OWNER_CHAIN}([[:space:]{]|$)"; then
echo "REFUSING: table 'inet ${t}' exists but lacks the m8a-cutover ownership marker (chain ${CUTOVER_OWNER_CHAIN}); not ours to delete." >&2
return 3
fi
# (4) safe: our table. Drop it by name (never `flush ruleset`).
_nft delete table inet "${t}"
}

# Enable the transient fail-closed handover drop on the guarded dport.
# Inserted at the head so it is evaluated first. While it is present the sender
# sees plain LOSS (SIP retransmission covers it), NOT ICMP port-unreachable —
# which is the whole reason the port must never be left merely unbound.
#
# IDEMPOTENT: a second call is a no-op. Two calls must never stack two rules, or
# a single handover_drop_off could leave a lingering blanket drop behind.
handover_drop_on() {
: "${PORT:?PORT required}"
if _handover_handles | grep -q .; then
return 0
fi
_nft insert rule inet "${NFT_TABLE}" input meta l4proto udp udp dport "${PORT}" drop comment '"handover"'
}

# Disable the handover drop (delete by comment handle).
# Emit the handle of EVERY handover rule currently in the chain (one per line).
_handover_handles() {
nft -a list chain inet "${NFT_TABLE}" input 2>/dev/null \
| awk '/comment "handover"/ {for(i=1;i<=NF;i++) if($i=="handle") print $(i+1)}'
}

# Disable the handover drop. Removes ALL handover rules (not just the first) so a
# doubled drop_on, or a failed apply that left one stuck, cannot leave a blanket
# drop behind. nft handles are stable within a generation, so one collected pass
# deletes them all.
handover_drop_off() {
local h
h="$(nft -a list chain inet "${NFT_TABLE}" input 2>/dev/null \
| awk '/comment "handover"/ {for(i=1;i<=NF;i++) if($i=="handle") print $(i+1)}' \
| head -n1)"
if [ -n "${h}" ]; then
_nft delete rule inet "${NFT_TABLE}" input handle "${h}"
fi
for h in $(_handover_handles); do
_nft delete rule inet "${NFT_TABLE}" input handle "${h}" 2>/dev/null || true
done
}

# Allow being sourced OR invoked as `cutover_lib.sh <fn>`.
Expand Down
14 changes: 14 additions & 0 deletions tests/m8a-cutover/rollback-rustisk-to-fs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,20 @@ HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=cutover_lib.sh
source "${HERE}/cutover_lib.sh"

# Fail-safe: `set -euo pipefail` aborts mid-sequence if STOP_OLD/WAIT/START_NEW
# fails, which would otherwise leave the blanket handover drop STUCK ON (phone
# outage until an operator clears it). This EXIT trap idempotently clears any
# lingering handover drop on EVERY exit path — failure or success — so a failed
# rollback fails CLOSED (SIP retransmits) without leaving the port blocked. It
# preserves the original exit code. handover_drop_off is a no-op if the drop is
# already off (the success path clears it before this trap runs).
_rollback_on_exit() {
local rc=$?
handover_drop_off 2>/dev/null || true
exit "${rc}"
}
trap _rollback_on_exit EXIT

: "${STOP_OLD_CMD:?}" ; : "${START_NEW_CMD:?}"
: "${WAIT_RELEASED_CMD:=true}" ; : "${WAIT_BOUND_CMD:=true}"

Expand Down
48 changes: 48 additions & 0 deletions tests/m8a-cutover/run-safety-tests.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
#!/usr/bin/env bash
# run-safety-tests.sh — driver for the M8a artifact-safety regression tests.
#
# Builds the proof image and runs safety-tests.sh inside ONE isolated container
# network namespace (--privileged --network none): the container has only `lo`,
# NO route to the host, and its OWN empty nftables ruleset. Nothing it does can
# touch the host nft, the live voicefw firewall, port 45070, RTP 20000-20100, the
# router, or the cluster. Synthetic ports only. Container + image reaped on exit.
#
# ./run-safety-tests.sh # build + run, transcript to stdout
# TRANSCRIPT=out.txt ./run-safety-tests.sh # also tee the transcript to a file
set -euo pipefail
HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
IMG=m8a-cutover-safety:local
CTR="m8a-cutover-safety-$$"
TRANSCRIPT="${TRANSCRIPT:-}"

cleanup() {
docker rm -f "${CTR}" >/dev/null 2>&1 || true
docker image rm "${IMG}" >/dev/null 2>&1 || true
if docker ps -a --format '{{.Names}}' 2>/dev/null | grep -qx "${CTR}"; then
echo "CLEANUP WARNING: container ${CTR} leaked — docker rm -f ${CTR}" >&2
fi
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

command -v docker >/dev/null || { echo "docker required" >&2; exit 2; }

echo "== build proof image =="
docker build -t "${IMG}" "${HERE}" >/dev/null

echo "== launch ISOLATED container (--privileged --network none) =="
docker run -d --rm --name "${CTR}" --privileged --network none "${IMG}" >/dev/null

# Host-side isolation assertion: docker itself reports 'none' network — no bridge,
# no veth to the host, no published ports.
NETMODE="$(docker inspect -f '{{.HostConfig.NetworkMode}}' "${CTR}")"
PORTS="$(docker inspect -f '{{json .NetworkSettings.Ports}}' "${CTR}")"
echo "host-side check: NetworkMode=${NETMODE} PublishedPorts=${PORTS}"
[ "${NETMODE}" = "none" ] || { echo "REFUSING: container is not on network 'none'" >&2; exit 3; }

run() { docker exec "${CTR}" bash /m8a/safety-tests.sh; }
if [ -n "${TRANSCRIPT}" ]; then run | tee "${TRANSCRIPT}"; rc="${PIPESTATUS[0]}"; else run; rc=$?; fi

echo "== safety tests exit: ${rc} =="
exit "${rc}"
Loading
Loading