Expand gateway portal peer lifecycle and reinforce healthchecks

docs/spec-alpha-support.md

@@ -28,9 +28,9 @@ Feature states:
 | `pki.rotation_policy` | `experimental` | `{enabled: true, default_interval_hours: 24, default_warning_days: 30, cert_type_overrides: {}}` | `netengine.handlers.phase_pki`, `netengine.workers.pki_cert_rotation_worker`, `netengine.api.routes` | Wired from the spec into worker registration and live-reloaded from runtime state; policy shape and cert-type semantics may change during alpha. |
 | `gateway_portal.real_internet.mode` | `experimental` | `isolated` | `netengine.handlers.gateway_handler`, `netengine.handlers.gateway_portal_handler`, `netengine.spec.loader` | Isolated, shadowed, mirrored, and exposed nftables policies are wired; real-host integration remains alpha. |
 | `gateway_portal.real_internet.service_mirrors` | `experimental` | `[]` | `netengine.handlers.gateway_handler`, `netengine.handlers.gateway_portal_handler`, `netengine.spec.loader` | Mirrored-mode allowlists and operator output are wired; `in_world_service` must be an IPv4 address until automatic DNS aliasing for real hostnames lands. |
-| `gateway_portal.real_internet.upstream_resolver_enabled` | `unsupported` | `false` | `netengine.spec.loader` | Upstream resolver forwarding is not implemented. |
-| `gateway_portal.cross_world.mode` | `unsupported` | `none` | `netengine.spec.loader` | Cross-world federation is not implemented. |
-| `gateway_portal.cross_world.peers` | `unsupported` | `[]` | `netengine.spec.loader` | Cross-world peer provisioning is not implemented. |
+| `gateway_portal.real_internet.upstream_resolver_enabled` | `experimental` | `false` | `netengine.handlers.gateway_portal_handler`, `netengine.spec.loader` | CoreDNS upstream forwarding stubs are appended and reloaded; duplicate-stub reconciliation remains alpha. |
+| `gateway_portal.cross_world.mode` | `experimental` | `none` | `netengine.handlers.gateway_handler`, `netengine.handlers.gateway_portal_handler`, `netengine.spec.loader` | PEERED/FEDERATED lifecycle hooks install peer routing, DNS stubs, trust anchors, runtime artifacts, rollback, and healthchecks; interoperability remains alpha. |
+| `gateway_portal.cross_world.peers` | `experimental` | `[]` | `netengine.handlers.gateway_handler`, `netengine.handlers.gateway_portal_handler`, `netengine.spec.loader` | Peer add/update/remove, trust-anchor rotation, and routing reapply are implemented with mocked two-world DNS coverage; live multi-host federation is not yet guaranteed. |
 | `ands.profiles.*.dynamic_ip` | `unsupported` | profile default | `netengine.spec.loader` | Dynamic IP allocation in AND profiles is not implemented. |
 | `ands.profiles.*.reverse_dns` | `unsupported` | profile default | `netengine.spec.loader` | Reverse DNS delegation from AND profiles is not implemented. |
 | `ands.profiles.*.bgp` | `unsupported` | profile default | `netengine.spec.loader` | BGP profile configuration is not implemented. |

docs/support-matrix.md

@@ -28,9 +28,9 @@ Feature states:
 | `pki.rotation_policy` | `experimental` | `{enabled: true, default_interval_hours: 24, default_warning_days: 30, cert_type_overrides: {}}` | `netengine.handlers.phase_pki`, `netengine.workers.pki_cert_rotation_worker`, `netengine.api.routes` | Wired from the spec into worker registration and live-reloaded from runtime state; policy shape and cert-type semantics may change during alpha. |
 | `gateway_portal.real_internet.mode` | `experimental` | `isolated` | `netengine.handlers.gateway_handler`, `netengine.handlers.gateway_portal_handler`, `netengine.spec.loader` | Isolated, shadowed, mirrored, and exposed nftables policies are wired; real-host integration remains alpha. |
 | `gateway_portal.real_internet.service_mirrors` | `experimental` | `[]` | `netengine.handlers.gateway_handler`, `netengine.handlers.gateway_portal_handler`, `netengine.spec.loader` | Mirrored-mode allowlists and operator output are wired; `in_world_service` must be an IPv4 address until automatic DNS aliasing for real hostnames lands. |
-| `gateway_portal.real_internet.upstream_resolver_enabled` | `unsupported` | `false` | `netengine.spec.loader` | Upstream resolver forwarding is not implemented. |
-| `gateway_portal.cross_world.mode` | `unsupported` | `none` | `netengine.spec.loader` | Cross-world federation is not implemented. |
-| `gateway_portal.cross_world.peers` | `unsupported` | `[]` | `netengine.spec.loader` | Cross-world peer provisioning is not implemented. |
+| `gateway_portal.real_internet.upstream_resolver_enabled` | `experimental` | `false` | `netengine.handlers.gateway_portal_handler`, `netengine.spec.loader` | CoreDNS upstream forwarding stubs are appended and reloaded; duplicate-stub reconciliation remains alpha. |
+| `gateway_portal.cross_world.mode` | `experimental` | `none` | `netengine.handlers.gateway_handler`, `netengine.handlers.gateway_portal_handler`, `netengine.spec.loader` | PEERED/FEDERATED lifecycle hooks install peer routing, DNS stubs, trust anchors, runtime artifacts, rollback, and healthchecks; interoperability remains alpha. |
+| `gateway_portal.cross_world.peers` | `experimental` | `[]` | `netengine.handlers.gateway_handler`, `netengine.handlers.gateway_portal_handler`, `netengine.spec.loader` | Peer add/update/remove, trust-anchor rotation, and routing reapply are implemented with mocked two-world DNS coverage; live multi-host federation is not yet guaranteed. |
 | `ands.profiles.*.dynamic_ip` | `unsupported` | profile default | `netengine.spec.loader` | Dynamic IP allocation in AND profiles is not implemented. |
 | `ands.profiles.*.reverse_dns` | `unsupported` | profile default | `netengine.spec.loader` | Reverse DNS delegation from AND profiles is not implemented. |
 | `ands.profiles.*.bgp` | `unsupported` | profile default | `netengine.spec.loader` | BGP profile configuration is not implemented. |

netengine/handlers/gateway_handler.py

@@ -1,7 +1,7 @@
 import ipaddress
 import os
 import tempfile
-from typing import TYPE_CHECKING, Any, Optional
+from typing import TYPE_CHECKING, Any
 
 from netengine.errors import GatewayError
 from netengine.gateways.base import BaseGatewayHandler
@@ -15,6 +15,79 @@ def __init__(self, docker: Any) -> None:
         self.docker = docker
         self.gateway_container = "netengine_gateway"
 
+    async def gateway_reachable(self) -> bool:
+        """Return True when the gateway container accepts exec commands."""
+        try:
+            exit_code, _ = await self.docker.exec_command(
+                self.gateway_container, ["true"]
+            )
+            return exit_code == 0
+        except Exception:
+            return False
+
+    async def nft_table_exists(self, family: str, table_name: str) -> bool:
+        """Return True when an nftables table exists in the gateway container."""
+        exit_code, _ = await self.docker.exec_command(
+            self.gateway_container, ["nft", "list", "table", family, table_name]
+        )
+        return exit_code == 0
+
+    async def path_exists(self, path: str) -> bool:
+        """Return True when *path* exists in the gateway container."""
+        exit_code, _ = await self.docker.exec_command(
+            self.gateway_container, ["test", "-e", path]
+        )
+        return exit_code == 0
+
+    async def peer_endpoint_reachable(
+        self, endpoint: str, timeout_seconds: int = 2
+    ) -> bool | None:
+        """Best-effort TCP reachability probe for host:port endpoints.
+
+        Returns None when the endpoint has no explicit port or the container lacks
+        a probe utility; callers can treat that as unknown rather than unhealthy.
+        """
+        host, sep, port = endpoint.rpartition(":")
+        if not sep or not host or not port.isdigit():
+            return None
+        probes = (
+            ["nc", "-z", "-w", str(timeout_seconds), host, port],
+            [
+                "sh",
+                "-c",
+                f"timeout {timeout_seconds} bash -c '</dev/tcp/{host}/{port}'",
+            ],
+        )
+        unsupported = False
+        for cmd in probes:
+            exit_code, output = await self.docker.exec_command(
+                self.gateway_container, cmd
+            )
+            if exit_code == 0:
+                return True
+            unsupported = (
+                unsupported
+                or "not found" in output.lower()
+                or "no such file" in output.lower()
+            )
+        return None if unsupported else False
+
+    async def reapply_peer_routing(self, peers: list[dict[str, Any]]) -> None:
+        """Re-load persisted peer nftables rule files after a gateway restart."""
+        for peer in peers:
+            peer_name = peer.get("name")
+            rules_path = (
+                peer.get("routing_rules_path")
+                or f"/etc/nftables/rules/peer_{peer_name}.nft"
+            )
+            exit_code, output = await self.docker.exec_command(
+                self.gateway_container, ["nft", "-f", rules_path]
+            )
+            if exit_code != 0:
+                raise GatewayError(
+                    f"Failed to reapply peer routing for {peer_name}: {output}"
+                )
+
     async def generate_rules(self, and_name: str, profile: str, cidr: str) -> str:
         """Generate nftables ruleset for the given AND profile."""
         if profile == "residential":
@@ -95,22 +168,28 @@ async def apply_rules(self, and_name: str, rules: str) -> None:
             f.write(rules)
             tmp_path = f.name
         try:
-            await self.docker.copy_to_container(self.gateway_container, tmp_path, dest_path)
+            await self.docker.copy_to_container(
+                self.gateway_container, tmp_path, dest_path
+            )
         finally:
             os.unlink(tmp_path)
 
         cmd = ["nft", "-f", dest_path]
         exit_code, output = await self.docker.exec_command(self.gateway_container, cmd)
         if exit_code != 0:
-            raise GatewayError(f"Failed to apply nftables rules for {and_name}: {output}")
+            raise GatewayError(
+                f"Failed to apply nftables rules for {and_name}: {output}"
+            )
 
     async def remove_rules(self, and_name: str) -> None:
         """Delete the nftables table for this AND."""
         cmd = ["nft", "delete", "table", "ip", f"netengine_{and_name}"]
         exit_code, output = await self.docker.exec_command(self.gateway_container, cmd)
         # Table-not-found is acceptable on teardown
         if exit_code != 0 and "No such table" not in output:
-            raise GatewayError(f"Failed to remove nftables table for {and_name}: {output}")
+            raise GatewayError(
+                f"Failed to remove nftables table for {and_name}: {output}"
+            )
         cmd = ["rm", "-f", f"/etc/nftables/rules/{and_name}.nft"]
         await self.docker.exec_command(self.gateway_container, cmd)
 
@@ -145,7 +224,9 @@ async def apply_internet_policy(self, config: "RealInternetConfig") -> None:
             f.write(rules)
             tmp_path = f.name
         try:
-            await self.docker.copy_to_container(self.gateway_container, tmp_path, dest_path)
+            await self.docker.copy_to_container(
+                self.gateway_container, tmp_path, dest_path
+            )
         except Exception as exc:
             os.unlink(tmp_path)
             raise GatewayError(
@@ -157,7 +238,9 @@ async def apply_internet_policy(self, config: "RealInternetConfig") -> None:
             self.gateway_container, ["nft", "-f", dest_path]
         )
         if exit_code != 0:
-            raise GatewayError(f"Failed to apply internet policy ({config.mode.value}): {output}")
+            raise GatewayError(
+                f"Failed to apply internet policy ({config.mode.value}): {output}"
+            )
 
     async def remove_internet_policy(self) -> None:
         """Remove internet policy rules (reverts to isolated state)."""
@@ -179,7 +262,9 @@ def _internet_rules(self, config: "RealInternetConfig") -> str:
         dispatch = {
             GatewayRealInternetMode.ISOLATED: self._isolated_internet_rules,
             GatewayRealInternetMode.SHADOWED: self._shadowed_internet_rules,
-            GatewayRealInternetMode.MIRRORED: lambda: self._mirrored_internet_rules(config),
+            GatewayRealInternetMode.MIRRORED: lambda: self._mirrored_internet_rules(
+                config
+            ),
             GatewayRealInternetMode.EXPOSED: self._exposed_internet_rules,
         }
         return dispatch[config.mode]()
@@ -300,7 +385,9 @@ async def apply_peer_routing(self, peer_name: str, peer_endpoint_ip: str) -> Non
             f.write(rules)
             tmp_path = f.name
         try:
-            await self.docker.copy_to_container(self.gateway_container, tmp_path, dest_path)
+            await self.docker.copy_to_container(
+                self.gateway_container, tmp_path, dest_path
+            )
         except Exception as exc:
             os.unlink(tmp_path)
             raise GatewayError(
@@ -312,7 +399,9 @@ async def apply_peer_routing(self, peer_name: str, peer_endpoint_ip: str) -> Non
             self.gateway_container, ["nft", "-f", dest_path]
         )
         if exit_code != 0:
-            raise GatewayError(f"Failed to apply peer routing for {peer_name}: {output}")
+            raise GatewayError(
+                f"Failed to apply peer routing for {peer_name}: {output}"
+            )
 
     async def remove_peer_routing(self, peer_name: str) -> None:
         """Remove routing rules for a cross-world peer."""
@@ -321,7 +410,9 @@ async def remove_peer_routing(self, peer_name: str) -> None:
             ["nft", "delete", "table", "inet", f"netengine_peer_{peer_name}"],
         )
         if exit_code != 0 and "No such table" not in output:
-            raise GatewayError(f"Failed to remove peer routing for {peer_name}: {output}")
+            raise GatewayError(
+                f"Failed to remove peer routing for {peer_name}: {output}"
+            )
         await self.docker.exec_command(
             self.gateway_container,
             ["rm", "-f", f"/etc/nftables/rules/peer_{peer_name}.nft"],
@@ -348,7 +439,9 @@ async def setup_dhcp(self, and_name: str, cidr: str, gateway_ip: str) -> None:
             f.write(conf)
             tmp_path = f.name
         try:
-            await self.docker.copy_to_container(self.gateway_container, tmp_path, conf_path)
+            await self.docker.copy_to_container(
+                self.gateway_container, tmp_path, conf_path
+            )
         finally:
             os.unlink(tmp_path)
 
@@ -369,13 +462,17 @@ async def remove_dhcp(self, and_name: str) -> None:
         await self.docker.exec_command(
             self.gateway_container, ["rm", "-f", f"/etc/dnsmasq.d/{and_name}.conf"]
         )
-        await self.docker.exec_command(self.gateway_container, ["pkill", "-SIGHUP", "dnsmasq"])
+        await self.docker.exec_command(
+            self.gateway_container, ["pkill", "-SIGHUP", "dnsmasq"]
+        )
 
     # ─────────────────────────────────────────────
     # BGP speaker (bgp profile feature)
     # ─────────────────────────────────────────────
 
-    async def setup_bgp(self, and_name: str, cidr: str, gateway_ip: str, bgp_mode: str) -> None:
+    async def setup_bgp(
+        self, and_name: str, cidr: str, gateway_ip: str, bgp_mode: str
+    ) -> None:
         """Provision a Bird2 BGP speaker sidecar container for the AND."""
         bird_conf = self._bird_conf(and_name, cidr, gateway_ip)
         conf_path = f"/etc/bird/bird_{and_name}.conf"