diff --git a/.github/workflows/integration-test.md b/.github/workflows/integration-test.md new file mode 100644 index 0000000..02b3622 --- /dev/null +++ b/.github/workflows/integration-test.md @@ -0,0 +1,272 @@ +# Integration Test — VPN Mesh Connectivity + +Companion document for `integration-test.yml`. +Read this side-by-side with the workflow file. + +--- + +## Why this test exists + +The 44mesh stack has three moving parts that must work together: + +1. **ZeroTier controller** — assigns mesh IPs and manages node membership +2. **BIRD2 border router** — announces the mesh IP range to an upstream ISP via BGP +3. **ZeroTier client nodes** — join the overlay and communicate over it + +A build that passes image validation and unit checks can still fail at runtime if, +for example, the ZeroTier interface never comes up, the BIRD template is +misconfigured, or the overlay dataplane drops packets. This workflow boots the +real services in CI and exercises all three layers. + +--- + +## Network topology used in the test + +``` + Host runner (ubuntu-latest) + ├── eth0 (primary, public IP) + │ ├── eth0:isp → 172.30.0.1/24 (mock ISP BGP address) + │ └── eth0:border → 172.30.0.100/24 (border router BGP address) + │ + ├── zt → 44.30.127.1/24 (controller's mesh IP, on host namespace) + │ + ├── [host-network containers] + │ zt-controller — ZeroTier daemon + controller, port 9993 + │ bird-border — BIRD2, AS 65000, peers with 172.30.0.1 + │ bird-isp — BIRD2, AS 65001, peers with 172.30.0.100 + │ + └── [bridge network: mesh-test-net 172.31.0.0/24] + zt-node1 — ZeroTier client, gets IP from pool 44.30.127.10–50 + zt-node2 — ZeroTier client, gets IP from pool 44.30.127.10–50 +``` + +### Why split networking? + +The **controller and BIRD services** run with `--network host` so they share the +host's network namespace. This is exactly how they run in production and it lets +BIRD see the `zt*` interface that the ZeroTier daemon creates. + +The **client nodes** run on a Docker bridge (`mesh-test-net`). This gives each +client its own Linux network namespace, which means each ZeroTier daemon creates +its own `zt` interface without name-conflicting with the controller's +interface (both would try to use the same name for the same network ID if they +shared the host namespace). Outbound UDP from the bridge is NAT-ted through the +host, which is enough for ZeroTier's NAT traversal to work. + +--- + +## Setup steps (before the tests) + +### Kernel & host network + +``` +Load kernel modules and configure host networking +``` + +- Loads the `tun` kernel module (required by ZeroTier to create TAP/TUN devices). +- Detects the default egress interface dynamically with `ip route get 8.8.8.8` + instead of hardcoding `eth0` — runner interface names differ across cloud + providers and runner generations. +- Adds two secondary IPs to that interface (`172.30.0.1` for the mock ISP, + `172.30.0.100` for the border router). Both BIRD instances then bind to these + addresses and peer over the same physical interface. +- Creates the `mesh-test-net` Docker bridge used by client nodes. + +### Image preparation + +Builds the two local images (`bird-border`, `rpi-isp`) from source and pulls the +pre-built ZeroTier image. Building from source in CI ensures that any Dockerfile +or config-template change is exercised, not a stale cached layer. + +### ZeroTier controller bootstrap + +``` +Start ZeroTier controller +Wait for ZeroTier controller to come online +Create ZeroTier mesh network +Controller joins its own network and receives a static mesh IP +Verify host zt* interface has mesh IP before starting BIRD +``` + +1. The controller container starts with `--network host` and the + `controller-entrypoint.sh` from the repo (mounted read-only). No identity is + pre-seeded, so a fresh one is generated each run. + +2. Readiness is confirmed by polling `zerotier-cli -j info` until + `.online == true` (JSON output, not text parsing). + +3. A private ZeroTier network is created via the local REST API + (`POST /controller/network/${CTRL_ID}______`). The six underscores are + expanded by the controller into a unique suffix — this is the documented way + to create a controller-owned network. The returned `.id` is validated + non-empty and saved as `ZT_NETWORK_ID`. + +4. The controller joins its own network and is authorized with the static IP + `44.30.127.1`. The authorization is polled (not a fixed sleep) to avoid a + race where the member record doesn't exist yet. + +5. Before starting BIRD, the workflow waits for `44.30.127.1` to appear on the + host's `zt*` interface. This is the synchronization point between ZeroTier + and BIRD — BIRD's `entrypoint.sh` scans for a `zt*` interface, and if it + isn't there yet, BIRD exits with an error. + +### BGP services start + +``` +Start BIRD border router +Start mock ISP (BIRD) +Verify BGP listeners bound on expected IPs +``` + +Both BIRD containers start with `--network host`. Each receives its BGP +parameters as environment variables; `entrypoint.sh` runs `envsubst` on +`bird.conf.template` to produce the final config, verifies it with +`bird -p`, then starts the daemon. + +A defensive step (`ss -nltp 'sport = :179'`) asserts that both daemons are +actually listening on the expected IPs before the BGP test runs. This catches +template substitution failures where BIRD binds to `0.0.0.0` instead of a +specific address. + +--- + +## The four tests + +### TEST 1 — BGP session established + +```yaml +- name: "TEST 1: BGP session established" +``` + +**What it does:** Polls `birdc show protocols` on the border router until the +BGP session with the ISP reaches the `Established` state (up to 90 s). + +**What it proves:** The two BIRD instances can reach each other over the +`172.30.0.0/24` link, the BIRD configs are syntactically and semantically +correct, and the BGP FSM completes the full Open → Active → Established +transition. + +**Pass condition:** `birdc show protocols` output contains the word +`Established`. + +--- + +### TEST 2 — Mesh route exported via BGP + +```yaml +- name: "TEST 2: Mesh route exported via BGP" +``` + +**What it does:** Waits up to 60 s for `44.30.127.0/24` to appear in the ISP's +BIRD routing table, then asserts it is present. + +**What it proves:** +- The border router's BIRD config correctly exports the mesh range (the + `export filter` in `bird.conf.template` accepts `net ~ [${MESH_ADDRESS_RANGE}]`). +- The ZeroTier `zt*` interface was recognised by the `protocol direct { interface "zt*"; }` + block and the prefix was learned. +- Route propagation from border → ISP works end-to-end. + +**Pass condition:** `docker exec bird-isp birdc show route` contains `44.30.127.0/24`. + +--- + +### TEST 3 — ZeroTier nodes receive mesh IPs + +```yaml +- name: Start ZeroTier mesh node 1 +- name: Start ZeroTier mesh node 2 +- name: Wait for client nodes to appear in controller and authorize them +- name: "TEST 3: ZeroTier nodes receive mesh IPs" +``` + +**What it does (setup):** Two client containers start on `mesh-test-net`. Each +mounts the same `entrypoint.sh`, `local.conf` (allows management from any IP), +and `network.local.conf` (sets `allowManaged=1`, `allowGlobal=1`, +`allowDefault=1`). They call `zerotier-cli join ` and then wait to +be authorized. + +Authorization is done by polling the controller API until it sees three members +(controller + 2 clients), then issuing `POST .../member/` with +`{"authorized": true}` for each client. A 10-second settle follows so clients +can fetch their updated network config from the controller. + +**What it asserts:** For each node, `zerotier-cli -j listnetworks` returns +`status == "OK"` and at least one `assignedAddresses` entry that matches +`^44\.30\.127\.`. The IP is extracted via `jq` and stored for TEST 4. An empty +or out-of-range IP is a hard failure. + +**What it proves:** +- The controller correctly assigns IPs from the configured pool. +- Clients can reach the controller through Docker bridge NAT (ZeroTier's NAT + traversal via planet/root servers handles this). +- The `entrypoint.sh` correctly pre-configures the network join on startup. + +**Pass condition:** Both nodes have a mesh IP in `44.30.127.0/24` and status `OK`. + +--- + +### TEST 4 — Node-to-node ping over ZeroTier mesh + +```yaml +- name: "TEST 4: Node-to-node ping over ZeroTier mesh" +``` + +**What it does:** Runs `ping -c 5 -W 3 ` in each direction inside the +client containers. + +**What it proves:** +- The ZeroTier overlay dataplane is functional — packets sent to a mesh IP + actually traverse the encrypted ZeroTier tunnel and arrive at the peer. +- Both nodes have established a peer path (direct or relayed) and can exchange + traffic bidirectionally. + +**Pass condition:** 5/5 ping replies in both directions (node1→node2 and +node2→node1). + +--- + +## Diagnostics on failure + +```yaml +- name: Collect diagnostics on failure + if: failure() +``` + +Runs only when a previous step has failed. Dumps: + +| Group | Contents | +|---|---| +| Container logs | Last 50 lines from each container | +| Host network state | `ip addr`, `ip link`, `ip route`, `ss -nltp :179` | +| BIRD state | `birdc show protocols` and `birdc show route` for both routers | +| ZT controller members | JSON member list from controller API | +| ZT node status | `zerotier-cli -j listnetworks` and `zerotier-cli -j peers` from each client + `ip addr` | + +The `peers` output is especially useful for diagnosing TEST 4 failures: if peers +show `latency == -1` or no direct path, it indicates the ZeroTier path was never +established (often a UDP connectivity or timing issue in CI). + +--- + +## Known limitations and risks + +| Risk | Mitigation | +|---|---| +| Bridge→host ZT path needs internet (planet/root servers) | TEST 3 allows 180 s; `zerotier-cli -j peers` in diagnostics | +| `zerotier-cli -j` flag in AlterMundi fork may differ | Confirmed present in fork; if broken, `-j` output lands in diagnostics | +| Interface fallback to `eth0` if route detection fails | `ss` listener check in setup surfaces misconfigured BGP bind addresses early | +| `allowDefault=1` in `network.local.conf` accepts a default route from mesh | No default route is configured in the test network, so no routing loop occurs | + +--- + +## Triggers + +| Event | Condition | +|---|---| +| Pull request to `main` | Only if `deploy/**` or the workflow file itself changed | +| Push to `stage` | Only if `deploy/**` changed | +| Manual (`workflow_dispatch`) | Always | + +Concurrency is set to cancel in-progress runs for the same ref, so pushing +multiple commits in quick succession does not queue up redundant runs. diff --git a/.github/workflows/integration-test.yml b/.github/workflows/integration-test.yml new file mode 100644 index 0000000..268478c --- /dev/null +++ b/.github/workflows/integration-test.yml @@ -0,0 +1,545 @@ +name: Integration Test - VPN Mesh Connectivity + +# Tests the full stack end-to-end: +# 1. ZeroTier controller creates a private mesh network +# 2. BIRD border router establishes BGP session with mock ISP +# 3. Mesh range (44.30.127.0/24) is exported via BGP +# 4. Two ZeroTier client nodes join the mesh and receive IPs +# 5. Nodes can ping each other over the ZeroTier overlay +# +# Networking strategy: +# - Controller + BIRD services run with --network host (share host namespace) +# - ZT client nodes run on a bridge network (own namespace, no zt* name conflict) +# - Host gets secondary IPs for BGP: 172.30.0.1 (ISP), 172.30.0.100 (border) + +on: + pull_request: + branches: [main] + paths: + - "deploy/**" + - ".github/workflows/integration-test.yml" + push: + branches: [stage] + paths: + - "deploy/**" + workflow_dispatch: + +concurrency: + group: integration-test-${{ github.ref }} + cancel-in-progress: true + +permissions: + contents: read + +jobs: + mesh-integration-test: + name: Mesh VPN Integration Test + runs-on: ubuntu-latest + timeout-minutes: 25 + + env: + MESH_RANGE: "44.30.127.0/24" + ISP_IP: "172.30.0.1" + BORDER_IP: "172.30.0.100" + ISP_AS: "65001" + BORDER_AS: "65000" + + steps: + # ── SETUP ─────────────────────────────────────────────────────────────── + + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Load kernel modules and configure host networking + run: | + set -Eeuo pipefail + + # TUN module is required by ZeroTier + sudo modprobe tun + + # Detect the default egress interface (works on EC2, GCP, Azure, etc.) + DEFAULT_IFACE=$(ip -o route get 8.8.8.8 | awk '{print $5}' | head -1) + DEFAULT_IFACE="${DEFAULT_IFACE:-eth0}" + echo "Default interface: $DEFAULT_IFACE" + echo "DEFAULT_IFACE=$DEFAULT_IFACE" >> "$GITHUB_ENV" + + # Add secondary IPs so BIRD ISP (172.30.0.1) and border router + # (172.30.0.100) can peer over the same physical interface + sudo ip addr add 172.30.0.1/24 dev "$DEFAULT_IFACE" label "${DEFAULT_IFACE}:isp" + sudo ip addr add 172.30.0.100/24 dev "$DEFAULT_IFACE" label "${DEFAULT_IFACE}:border" + + # Isolated Docker bridge for ZT client nodes + # (each client gets its own network namespace → no zt* name conflicts) + docker network create \ + --driver bridge \ + --subnet 172.31.0.0/24 \ + mesh-test-net + + echo "=== Host network setup ===" + ip addr show "$DEFAULT_IFACE" + docker network inspect mesh-test-net --format '{{.IPAM.Config}}' + + - name: Build BIRD border router image + run: docker build -t 44mesh-bird-border:ci deploy/bird-border/ + + - name: Build mock ISP image + run: docker build -t 44mesh-bird-isp:ci deploy/rpi-isp/ + + - name: Pull ZeroTier image + run: docker pull buzondefede/44mesh-zerotier:latest + + # ── ZEROTIER CONTROLLER ───────────────────────────────────────────────── + + - name: Start ZeroTier controller + run: | + set -Eeuo pipefail + # Runs on host network: its zt* interface will appear in the host + # namespace and will be visible to bird-border later. + # SYS_ADMIN is NOT required; NET_ADMIN + /dev/net/tun are sufficient. + docker run -d \ + --name zt-controller \ + --network host \ + --cap-add NET_ADMIN \ + --device /dev/net/tun \ + -v zt-ctrl-data:/var/lib/zerotier-one \ + -v "$(pwd)/deploy/zerotier/controller-entrypoint.sh:/usr/local/bin/controller-entrypoint.sh:ro" \ + --entrypoint /bin/bash \ + buzondefede/44mesh-zerotier:latest \ + /usr/local/bin/controller-entrypoint.sh + + - name: Wait for ZeroTier controller to come online + run: | + set -Eeuo pipefail + echo "Waiting for controller (up to 90s)..." + timeout 90 bash -c ' + until docker exec zt-controller zerotier-cli -j info 2>/dev/null \ + | jq -e ".online == true" > /dev/null 2>&1; do + sleep 3 + done + ' + docker exec zt-controller zerotier-cli info + + - name: Create ZeroTier mesh network + run: | + set -Eeuo pipefail + CTRL_ID=$(docker exec zt-controller zerotier-cli -j info | jq -r '.address') + API_SECRET=$(docker exec zt-controller cat /var/lib/zerotier-one/authtoken.secret) + + echo "Controller node ID: $CTRL_ID" + + # Network ID = + 6 hex chars. Using ______ as + # the suffix lets the controller expand it to a valid unique ID. + NETWORK=$(curl -sf -X POST \ + -H "X-ZT1-Auth: ${API_SECRET}" \ + -H "Content-Type: application/json" \ + -d "{ + \"name\": \"ci-test-mesh\", + \"private\": true, + \"v4AssignMode\": {\"zt\": true}, + \"ipAssignmentPools\": [{ + \"ipRangeStart\": \"44.30.127.10\", + \"ipRangeEnd\": \"44.30.127.50\" + }], + \"routes\": [{\"target\": \"44.30.127.0/24\"}] + }" \ + "http://localhost:9993/controller/network/${CTRL_ID}______") + + NETWORK_ID=$(echo "$NETWORK" | jq -r '.id') + [ -n "$NETWORK_ID" ] || { echo "✗ FAIL: empty network ID from controller API"; exit 1; } + echo "Network created: $NETWORK_ID" + + echo "ZT_NETWORK_ID=$NETWORK_ID" >> "$GITHUB_ENV" + echo "ZT_API_SECRET=$API_SECRET" >> "$GITHUB_ENV" + echo "ZT_CTRL_ID=$CTRL_ID" >> "$GITHUB_ENV" + + - name: Controller joins its own network and receives a static mesh IP + run: | + set -Eeuo pipefail + + # Pre-seed per-network config BEFORE joining so ZeroTier accepts + # globally-routable IPs (44.x.x.x). Without allowGlobal=1 the daemon + # silently refuses to assign non-RFC1918 addresses to the zt* interface. + docker exec zt-controller bash -c " + mkdir -p /var/lib/zerotier-one/networks.d + printf 'allowManaged=1\nallowGlobal=1\nallowDefault=0\nallowDNS=0\n' \ + > /var/lib/zerotier-one/networks.d/${ZT_NETWORK_ID}.local.conf + " + + docker exec zt-controller zerotier-cli join "$ZT_NETWORK_ID" + + # Poll until the member record appears in the controller, then authorize + timeout 30 bash -c ' + until curl -sf \ + -H "X-ZT1-Auth: $ZT_API_SECRET" \ + "http://localhost:9993/controller/network/$ZT_NETWORK_ID/member/$ZT_CTRL_ID" \ + | jq -e ".id" > /dev/null 2>&1; do + sleep 2 + done + ' + + curl -sf -X POST \ + -H "X-ZT1-Auth: $ZT_API_SECRET" \ + -H "Content-Type: application/json" \ + -d '{"authorized": true, "ipAssignments": ["44.30.127.1"]}' \ + "http://localhost:9993/controller/network/${ZT_NETWORK_ID}/member/${ZT_CTRL_ID}" + + echo "Controller authorized with IP 44.30.127.1" + + - name: Verify host zt* interface has mesh IP before starting BIRD + run: | + set -Eeuo pipefail + echo "Waiting for zt* interface on host with 44.30.127.1/24 (up to 90s)..." + timeout 90 bash -c ' + until ip addr show | grep -q "44.30.127.1"; do sleep 3; done + ' + ZT_IFACE=$(ip -o link show | awk -F": " "{print \$2}" | grep "^zt" | head -1) + echo "ZT interface: $ZT_IFACE" + ip addr show "$ZT_IFACE" + + # ── BGP SERVICES ──────────────────────────────────────────────────────── + + - name: Start BIRD border router + run: | + set -Eeuo pipefail + docker run -d \ + --name bird-border \ + --network host \ + --cap-add NET_ADMIN \ + -e BORDER_ROUTER_AS=65000 \ + -e BORDER_ROUTER_IP=172.30.0.100 \ + -e ISP_AS=65001 \ + -e ISP_IP=172.30.0.1 \ + -e MESH_ADDRESS_RANGE=44.30.127.0/24 \ + -e BGP_HOLD_TIME=30 \ + -e BGP_KEEPALIVE_TIME=10 \ + -e MAX_WAIT=30 \ + -v "$(pwd)/deploy/bird-border/bird.conf.template:/etc/bird/bird.conf.template:ro" \ + 44mesh-bird-border:ci + + - name: Start mock ISP (BIRD) + run: | + set -Eeuo pipefail + docker run -d \ + --name bird-isp \ + --network host \ + --cap-add NET_ADMIN \ + -e ISP_AS=65001 \ + -e ISP_IP=172.30.0.1 \ + -e BORDER_ROUTER_AS=65000 \ + -e BORDER_ROUTER_IP=172.30.0.100 \ + -e BGP_NETWORK_RANGE=172.30.0.0/24 \ + -e MESH_ADDRESS_RANGE=44.30.127.0/24 \ + -e TEST_PREFIX_1=192.0.2.0/24 \ + -e TEST_PREFIX_2=198.51.100.0/24 \ + -e TEST_PREFIX_3=203.0.113.0/24 \ + -e BGP_HOLD_TIME=30 \ + -e BGP_KEEPALIVE_TIME=10 \ + -v "$(pwd)/deploy/rpi-isp/bird.conf.template:/etc/bird/bird.conf.template:ro" \ + 44mesh-bird-isp:ci + + - name: Wait for BIRD containers and BGP listeners + run: | + set -Eeuo pipefail + + echo "Waiting for BIRD containers to start and BGP to listen (up to 90s)..." + elapsed=0 + while [ $elapsed -lt 90 ]; do + border="$(docker inspect -f '{{.State.Status}}' bird-border 2>/dev/null || echo missing)" + isp="$(docker inspect -f '{{.State.Status}}' bird-isp 2>/dev/null || echo missing)" + + # Fail fast if either container has already crashed + if [ "$border" = "exited" ] || [ "$isp" = "exited" ]; then + echo "✗ Container crashed — border=$border isp=$isp" + echo "" + echo "=== bird-border logs ===" + docker logs bird-border 2>&1 || true + echo "" + echo "=== bird-isp logs ===" + docker logs bird-isp 2>&1 || true + exit 1 + fi + + # Check if both TCP/179 listeners are up + if ss -nltp 'sport = :179' | grep -q "172.30.0.100" && \ + ss -nltp 'sport = :179' | grep -q "172.30.0.1"; then + echo "✓ Both BGP listeners are up" + ss -nltp 'sport = :179' + exit 0 + fi + + echo " ${elapsed}s — border=$border isp=$isp listeners=$(ss -nltp 'sport = :179' | grep -c ':179' || echo 0)" + sleep 3 + elapsed=$((elapsed + 3)) + done + + echo "✗ Timeout waiting for BGP listeners after ${elapsed}s" + docker ps -a --filter "name=bird-" + echo "=== bird-border logs ===" + docker logs bird-border 2>&1 || true + echo "=== bird-isp logs ===" + docker logs bird-isp 2>&1 || true + ss -nltp 'sport = :179' || true + exit 1 + + # ── TEST 1: BGP ────────────────────────────────────────────────────────── + + - name: "TEST 1: BGP session established" + run: | + set -Eeuo pipefail + echo "Waiting for BGP session between border router and ISP (up to 90s)..." + timeout 90 bash -c ' + until docker exec bird-border birdc show protocols 2>/dev/null | grep -q Established; do + echo " $(docker exec bird-border birdc show protocols 2>/dev/null | grep -i bgp || echo "not ready")" + sleep 5 + done + ' + echo "=== BGP protocols ===" + docker exec bird-border birdc show protocols + echo "✓ BGP session established" + + # ── TEST 2: Route export ───────────────────────────────────────────────── + + - name: "TEST 2: Mesh route exported via BGP" + run: | + set -Eeuo pipefail + + # Wait up to 60s for route propagation before asserting + echo "Waiting for 44.30.127.0/24 to appear in ISP routing table (up to 60s)..." + timeout 60 bash -c ' + until docker exec bird-isp birdc show route 2>/dev/null | grep -q "44.30.127.0/24"; do + sleep 5 + done + ' || true + + echo "=== Routing table (border router) ===" + docker exec bird-border birdc show route + echo "" + echo "=== ISP received routes ===" + docker exec bird-isp birdc show route + + if docker exec bird-isp birdc show route | grep -q "44.30.127.0/24"; then + echo "✓ Mesh route 44.30.127.0/24 received by ISP" + else + echo "✗ FAIL: Mesh route not found in ISP routing table" + exit 1 + fi + + # ── ZT CLIENT NODES ────────────────────────────────────────────────────── + # Clients run on a bridge network (mesh-test-net). Each gets its own Linux + # network namespace so each ZT daemon creates a separate zt* interface with + # no name collision. Outbound UDP from the bridge is NAT-ted through the + # host, enabling the ZT daemons to reach the controller and each other. + + - name: Start ZeroTier mesh node 1 + run: | + set -Eeuo pipefail + docker run -d \ + --name zt-node1 \ + --network mesh-test-net \ + --cap-add NET_ADMIN \ + --device /dev/net/tun \ + -v zt-node1-data:/var/lib/zerotier-one \ + -v "$(pwd)/deploy/zerotier/entrypoint.sh:/entrypoint.sh:ro" \ + -v "$(pwd)/deploy/zerotier/local.conf:/var/lib/zerotier-one/local.conf:ro" \ + -v "$(pwd)/deploy/zerotier/network.local.conf:/tmp/network.local.conf:ro" \ + --entrypoint /entrypoint.sh \ + buzondefede/44mesh-zerotier:latest \ + "$ZT_NETWORK_ID" + + - name: Start ZeroTier mesh node 2 + run: | + set -Eeuo pipefail + docker run -d \ + --name zt-node2 \ + --network mesh-test-net \ + --cap-add NET_ADMIN \ + --device /dev/net/tun \ + -v zt-node2-data:/var/lib/zerotier-one \ + -v "$(pwd)/deploy/zerotier/entrypoint.sh:/entrypoint.sh:ro" \ + -v "$(pwd)/deploy/zerotier/local.conf:/var/lib/zerotier-one/local.conf:ro" \ + -v "$(pwd)/deploy/zerotier/network.local.conf:/tmp/network.local.conf:ro" \ + --entrypoint /entrypoint.sh \ + buzondefede/44mesh-zerotier:latest \ + "$ZT_NETWORK_ID" + + - name: Wait for client nodes to appear in controller and authorize them + run: | + set -Eeuo pipefail + echo "Waiting for 2 client nodes to reach the controller (up to 120s)..." + timeout 120 bash -c ' + while true; do + COUNT=$(curl -sf \ + -H "X-ZT1-Auth: $ZT_API_SECRET" \ + "http://localhost:9993/controller/network/$ZT_NETWORK_ID/member" \ + | jq "keys | length") + echo " Members seen by controller: $COUNT (need ≥3: controller + 2 clients)" + [ "$COUNT" -ge 3 ] && break + sleep 5 + done + ' + + # Authorize every member that is not the controller + MEMBERS=$(curl -sf \ + -H "X-ZT1-Auth: $ZT_API_SECRET" \ + "http://localhost:9993/controller/network/$ZT_NETWORK_ID/member" \ + | jq -r 'keys[]') + + for MEMBER in $MEMBERS; do + if [ "$MEMBER" != "$ZT_CTRL_ID" ]; then + echo "Authorizing member: $MEMBER" + curl -sf -X POST \ + -H "X-ZT1-Auth: $ZT_API_SECRET" \ + -H "Content-Type: application/json" \ + -d '{"authorized": true}' \ + "http://localhost:9993/controller/network/$ZT_NETWORK_ID/member/$MEMBER" + fi + done + + # Short settle window: give clients time to fetch updated config + echo "Settling (10s) after authorization..." + sleep 10 + echo "All client nodes authorized" + + # ── TEST 3: Mesh node join ──────────────────────────────────────────────── + + - name: "TEST 3: ZeroTier nodes receive mesh IPs" + run: | + set -Eeuo pipefail + + echo "Waiting for node1 to reach OK status (up to 180s)..." + timeout 180 bash -c ' + until docker exec zt-node1 zerotier-cli -j listnetworks 2>/dev/null \ + | jq -e ".[0].status == \"OK\"" > /dev/null 2>&1; do + echo " node1: $(docker exec zt-node1 zerotier-cli listnetworks 2>/dev/null | tail -1)" + sleep 5 + done + ' + + echo "Waiting for node2 to reach OK status (up to 180s)..." + timeout 180 bash -c ' + until docker exec zt-node2 zerotier-cli -j listnetworks 2>/dev/null \ + | jq -e ".[0].status == \"OK\"" > /dev/null 2>&1; do + echo " node2: $(docker exec zt-node2 zerotier-cli listnetworks 2>/dev/null | tail -1)" + sleep 5 + done + ' + + # Extract IPs from JSON and validate they are within 44.30.127.0/24 + NODE1_IP=$(docker exec zt-node1 \ + zerotier-cli -j listnetworks | \ + jq -r --arg nw "$ZT_NETWORK_ID" \ + '.[] | select(.id == $nw) | .assignedAddresses[] | select(test("^44\\.30\\.127\\.")) | split("/")[0]' \ + | head -1) + + NODE2_IP=$(docker exec zt-node2 \ + zerotier-cli -j listnetworks | \ + jq -r --arg nw "$ZT_NETWORK_ID" \ + '.[] | select(.id == $nw) | .assignedAddresses[] | select(test("^44\\.30\\.127\\.")) | split("/")[0]' \ + | head -1) + + echo "Node 1 mesh IP: $NODE1_IP" + echo "Node 2 mesh IP: $NODE2_IP" + echo "NODE1_IP=$NODE1_IP" >> "$GITHUB_ENV" + echo "NODE2_IP=$NODE2_IP" >> "$GITHUB_ENV" + + [ -n "$NODE1_IP" ] || { echo "✗ FAIL: node1 has no mesh IP in 44.30.127.0/24"; exit 1; } + [ -n "$NODE2_IP" ] || { echo "✗ FAIL: node2 has no mesh IP in 44.30.127.0/24"; exit 1; } + echo "✓ Both nodes received mesh IPs in 44.30.127.0/24" + + # ── TEST 4: Mesh ping ───────────────────────────────────────────────────── + + - name: "TEST 4: Node-to-node ping over ZeroTier mesh" + run: | + set -Eeuo pipefail + + echo "Installing ping in ZT containers..." + docker exec zt-node1 sh -c "apt-get update -qq && apt-get install -y -qq iputils-ping >/dev/null 2>&1" + docker exec zt-node2 sh -c "apt-get update -qq && apt-get install -y -qq iputils-ping >/dev/null 2>&1" + + echo "=== Ping: node1 ($NODE1_IP) → node2 ($NODE2_IP) ===" + docker exec zt-node1 ping -c 5 -W 3 "$NODE2_IP" || { + echo "✗ FAIL: node1 cannot ping node2" + exit 1 + } + + echo "" + echo "=== Ping: node2 ($NODE2_IP) → node1 ($NODE1_IP) ===" + docker exec zt-node2 ping -c 5 -W 3 "$NODE1_IP" || { + echo "✗ FAIL: node2 cannot ping node1" + exit 1 + } + + echo "✓ Bidirectional mesh connectivity verified" + + # ── DIAGNOSTICS ────────────────────────────────────────────────────────── + + - name: Collect diagnostics on failure + if: failure() + run: | + echo "::group::Container logs" + for c in zt-controller bird-border bird-isp zt-node1 zt-node2; do + echo "--- $c ---" + docker logs "$c" 2>/dev/null | tail -50 || true + done + echo "::endgroup::" + + echo "::group::Host network state" + ip addr + ip link + ip route + ss -nltp 'sport = :179' || true + echo "::endgroup::" + + echo "::group::BIRD state" + docker exec bird-border birdc show protocols 2>/dev/null || true + docker exec bird-border birdc show route 2>/dev/null || true + docker exec bird-isp birdc show protocols 2>/dev/null || true + echo "::endgroup::" + + echo "::group::ZeroTier controller members" + curl -s \ + -H "X-ZT1-Auth: ${ZT_API_SECRET:-}" \ + "http://localhost:9993/controller/network/${ZT_NETWORK_ID:-}/member" \ + 2>/dev/null | jq . || true + echo "::endgroup::" + + echo "::group::ZeroTier node status" + docker exec zt-node1 zerotier-cli -j listnetworks 2>/dev/null | jq . || true + docker exec zt-node2 zerotier-cli -j listnetworks 2>/dev/null | jq . || true + docker exec zt-node1 zerotier-cli -j peers 2>/dev/null | jq . || true + docker exec zt-node2 zerotier-cli -j peers 2>/dev/null | jq . || true + docker exec zt-node1 ip addr 2>/dev/null || true + docker exec zt-node2 ip addr 2>/dev/null || true + echo "::endgroup::" + + # ── SUMMARY ────────────────────────────────────────────────────────────── + + - name: Write job summary + if: always() + run: | + STATUS="${{ job.status }}" + ICON="$( [ "$STATUS" = success ] && echo '✅' || echo '❌' )" + { + echo "## Integration Test — $ICON ${STATUS^}" + echo "" + echo "**Branch:** \`${{ github.head_ref || github.ref_name }}\`" + echo "**Commit:** \`${{ github.sha }}\`" + echo "" + echo "### Scenarios" + echo "| # | Scenario | Validates |" + echo "|---|----------|-----------|" + echo "| 1 | BGP session established | Border ↔ ISP eBGP peering |" + echo "| 2 | Mesh route exported | 44.30.127.0/24 announced to ISP |" + echo "| 3 | ZT nodes receive IPs | Controller assigns from pool |" + echo "| 4 | Node-to-node ping | Overlay dataplane works |" + echo "" + echo "### Topology" + echo '```' + echo " bird-isp (AS 65001 · 172.30.0.1)" + echo " │ eBGP" + echo " bird-border (AS 65000 · 172.30.0.100)" + echo " │ sees zt* (44.30.127.1)" + echo " zt-controller ────── zt-node1 (44.30.127.1x)" + echo " └─── zt-node2 (44.30.127.1x)" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" diff --git a/deploy/bird-border/bird.conf.template b/deploy/bird-border/bird.conf.template index 3b7666f..f86aba9 100644 --- a/deploy/bird-border/bird.conf.template +++ b/deploy/bird-border/bird.conf.template @@ -3,7 +3,7 @@ router id ${BORDER_ROUTER_IP}; -log syslog all; +log stderr all; protocol device { scan time 10; @@ -16,7 +16,6 @@ protocol kernel { }; } -# Direct routes - learn ZeroTier interface protocol direct { ipv4; interface "zt*"; @@ -27,6 +26,8 @@ protocol bgp isp { description "ISP AS ${ISP_AS}"; local ${BORDER_ROUTER_IP} as ${BORDER_ROUTER_AS}; neighbor ${ISP_IP} as ${ISP_AS}; + multihop; + strict bind yes; ipv4 { # Accept all routes from peer (works for both iBGP and eBGP) diff --git a/deploy/rpi-isp/bird.conf.template b/deploy/rpi-isp/bird.conf.template index 9c3f8b7..31d4e32 100644 --- a/deploy/rpi-isp/bird.conf.template +++ b/deploy/rpi-isp/bird.conf.template @@ -3,7 +3,7 @@ router id ${ISP_IP}; -log syslog all; +log stderr all; protocol device { scan time 10; @@ -29,6 +29,8 @@ protocol bgp border_router { description "Border Router AS ${BORDER_ROUTER_AS}"; local ${ISP_IP} as ${ISP_AS}; neighbor ${BORDER_ROUTER_IP} as ${BORDER_ROUTER_AS}; + multihop; + strict bind yes; ipv4 { import filter {