feat(system-tests): support API boundary node playnets on the local backend

[basvandijk]

with_api_boundary_nodes_playnet gives API boundary nodes a domain name and a certificate for it that the nodes trust. Both halves came from Farm, so every system-test using it — and therefore every system-test with a cloud engine subnet, which requires it — was pinned to backend = "farm".

This stands up an equivalent locally.

DNS in the local backend

The group's dnsmasq becomes its DNS server. It already ran on the group bridge as an RA/DHCPv4 daemon with --port=0; dropping that and adding --no-resolv --no-hosts makes it a hermetic resolver — with no upstream left to forward to, anything it cannot answer is REFUSED. It answers from two sources: an --addn-hosts file, written by the new LocalBackend::add_dns_record and re-read on SIGHUP, and a --synth-domain that mirrors the public nip.io wildcard service (so canister_http_socks_test needs no DNS-specific code and stays identical on both backends).

GuestOS has no name-server knob and boots with IPv6AcceptRA=no, so rather than reconfiguring the guests, create_group assigns the four addresses GuestOS is hard-coded to query to the bridge. Inside the backend's own network namespace those addresses are free and no query can escape, so every node gets a working resolver without touching IC-OS. Those addresses are now a single IPV6_NAME_SERVERS constant, kept in sync with the networkd contents by a unit test.

A local playnet

InternetComputer::setup_api_bn_local_playnet issues an ephemeral CA plus a leaf covering the API boundary nodes' domains, registers those domains with the group's dnsmasq, and stores the material in a new LocalApiBoundaryNodesPlaynet attribute — deliberately separate from Playnet, which the IC gateway VM also reads and writes. bootstrap serves the leaf from ic-boundary through the existing ic_boundary_tls_cert mechanism, so that pipeline is unchanged.

The one production change

nns_delegation_manager built its root store from the compiled-in webpki_roots only, with no config, env var, feature or cfg hook — no test-issued certificate could ever satisfy it. It now also honours extra_api_boundary_node_trust_anchors_pem, a new dev-only GuestOSDevSettings field (config version 1.15.0 → 1.16.0). Production nodes get None and keep the public roots as their only trust anchors.

This is the part that needs a careful look from the delegation-manager owners.

Tests enabled

backend = "farm" dropped from canister_http_socks_test, cloud_engine_canister_sig_test, xnet_cloud_engine_isolation_test, nns_delegation_branch_nns_version_test and delete_subnet_test. The cpus of canister_http_socks_test was understated — its comment omitted the four cloud engine nodes.

Two tests stay on Farm for unrelated reasons, now recorded accurately in place of the old comment:

• nns_delegation_mainnet_nns_version_test boots the mainnet GuestOS, whose replica predates the new field. It can be enabled once that has rolled out.
• firewall_correctness_test asserts that port 8080 is closed between particular pairs of nodes, but on the local backend ic-prep always adds 8080 to the fd00::/8 rule the driver needs to reach the nodes at all — and the nodes themselves live in fd00::/8. Enabling it means narrowing that rule to the driver's own addresses, which changes shared local-backend behaviour for every local test and is left as a follow-up.

Verification

Run on the local backend, all passing: cloud_engine_canister_sig_test_local, xnet_cloud_engine_isolation_test_local, nns_delegation_branch_nns_version_test_local, delete_subnet_test_local. Their logs show a cloud engine node resolving apibn-0.ic.net through the group's dnsmasq, completing a TLS handshake against the ephemeral CA, and fetching its NNS delegation.

canister_http_socks_test_local is enabled but was not run: at 11 node VMs plus a UVM it needs ~48 GiB, more than the machine I verified on had.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR enables API boundary node “playnet” functionality (domain + trusted TLS) on the local system-test backend by adding a hermetic in-group DNS resolver and issuing an ephemeral CA/cert for API BNs, allowing several previously Farm-only tests (notably those requiring cloud-engine subnets) to run locally. It also introduces a production-facing change to allow nns_delegation_manager to optionally trust additional API BN roots via config (intended for system tests).

Changes:

• Add local-backend DNS + per-group API BN domain/TLS issuance to support API BN playnets without Farm.
• Plumb a new dev-only GuestOS setting/config field for extra API BN trust anchors, and use it in nns_delegation_manager.
• Enable multiple system tests on the local backend and update firewall test logic to account for pre-seeded global firewall rules.

Reviewed changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
rs/tests/nns/BUILD.bazel	Enables delete_subnet_test off Farm by removing the Farm-only backend constraint.
rs/tests/networking/firewall/firewall_correctness_test.rs	Reads existing global firewall rules before proposing updates (local backend isn’t empty).
rs/tests/networking/firewall/BUILD.bazel	Keeps firewall correctness test on Farm, updating rationale for why local backend differs.
rs/tests/networking/canister_http_socks_test.rs	Clarifies why a hostname (nip.io-style) is required for SOCKS outcalls.
rs/tests/networking/BUILD.bazel	Enables canister_http_socks_test locally and corrects CPU sizing/commentary; documents remaining Farm-only test.
rs/tests/message_routing/xnet/BUILD.bazel	Enables xnet_cloud_engine_isolation_test off Farm by removing backend pin.
rs/tests/driver/src/util.rs	Adds template variable for extra_api_boundary_node_trust_anchors_pem.
rs/tests/driver/src/driver/test_env_api.rs	Adds TopologySnapshot::firewall_rules helper for reading registry firewall state.
rs/tests/driver/src/driver/local_backend.rs	Expands dnsmasq role to RA/DHCPv4/DNS; adds local DNS record registration and nip.io synthesis.
rs/tests/driver/src/driver/ic.rs	Implements local playnet setup: assign local domains, issue ephemeral CA/leaf, register DNS records.
rs/tests/driver/src/driver/bootstrap.rs	Plumbs local playnet TLS material + extra trust anchors into GuestOS config generation.
rs/tests/driver/Cargo.toml	Adds dependency on network crate to share GuestOS DNS server constants.
rs/tests/driver/BUILD.bazel	Adds Bazel dep on //rs/ic_os/networking/network.
rs/tests/crypto/BUILD.bazel	Enables cloud_engine_canister_sig_test off Farm by removing backend pin.
rs/orchestrator/src/firewall.rs	Updates config rendering template inputs for the new trust-anchor field in tests.
rs/ic_os/networking/network/src/systemd.rs	Introduces IPV6_NAME_SERVERS constant + unit test to keep it in sync with networkd contents.
rs/ic_os/networking/network/BUILD.bazel	Adjusts crate visibility to allow system tests to depend on network.
rs/ic_os/config/types/src/lib.rs	Bumps config version and adds dev-only extra_api_boundary_node_trust_anchors_pem field.
rs/ic_os/config/types/compatibility_tests/fixtures/hostos_v1.16.0.json	Adds v1.16.0 fixture covering the new dev setting.
rs/ic_os/config/types/compatibility_tests/fixtures/guestos_v1.16.0.json	Adds v1.16.0 fixture covering the new dev setting.
rs/ic_os/config/tool/templates/ic.json5.template	Adds new http_handler config field to rendered IC config.
rs/ic_os/config/tool/src/guestos/generate_ic_config.rs	JSON-encodes PEM into template var and wires it into config rendering.
rs/http_endpoints/nns_delegation_manager/src/nns_delegation_manager.rs	Extends trust store with optional extra roots from config when contacting API BNs.
rs/config/src/http_handler.rs	Adds new optional config field with default None.
Cargo.lock	Records the new network dependency in the lockfile.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 25 out of 26 changed files in this pull request and generated no new comments.

Suppressed comments (1)

rs/tests/networking/canister_http_socks_test.rs:162

• This cross-reference still uses the old method name, but this PR renames it to LocalBackend::start_dnsmasq; as written, readers cannot find the referenced implementation.

    // `LocalBackend::start_ra_daemon`), so no external DNS is involved there.

[basvandijk]

Picking up the suppressed comment from the last Copilot review (canister_http_socks_test.rs:162): correct, that cross-reference pointed at LocalBackend::start_ra_daemon, which this PR renames to start_dnsmasq.

Fixed in 6c287ca, along with three comments in local_backend.rs that still called it "the RA daemon" — it now serves RA, DHCPv4 and DNS, which is the whole reason for the rename. No remaining references to the old names.

[github-actions]

This pull request changes code owned by the Governance team. Therefore, make sure that
you have considered the following (for Governance-owned code):

1. Update unreleased_changelog.md (if there are behavior changes, even if they are
   non-breaking).

2. Are there BREAKING changes?

3. Is a data migration needed?

4. Security review?

How to Satisfy This Automatic Review

1. Go to the bottom of the pull request page.

2. Look for where it says this bot is requesting changes.

3. Click the three dots to the right.

4. Select "Dismiss review".

5. In the text entry box, respond to each of the numbered items in the previous
   section, declare one of the following:

• Done.

• $REASON_WHY_NO_NEED. E.g. for unreleased_changelog.md, "No
  canister behavior changes.", or for item 2, "Existing APIs
  behave as before.".

Brief Guide to "Externally Visible" Changes

"Externally visible behavior change" is very often due to some NEW canister API.

Changes to EXISTING APIs are more likely to be "breaking".

If these changes are breaking, make sure that clients know how to migrate, how to
maintain their continuity of operations.

If your changes are behind a feature flag, then, do NOT add entrie(s) to
unreleased_changelog.md in this PR! But rather, add entrie(s) later, in the PR
that enables these changes in production.

Reference(s)

For a more comprehensive checklist, see here.

GOVERNANCE_CHECKLIST_REMINDER_DEDUP

[zeropath-ai]

✅ No security or compliance issues detected. Reviewed everything up to 3d1e1e0.

Security Overview

• 🔎 Scanned files: 24 changed file(s)
• 🔗 Scan Link: https://zeropath.com/app/repositories/6992424c-495e-4621-b441-efd6821a4be6?scanId=c7d5fb73-3f3d-434c-9a7a-8bb35106adac&codeScanTypes=PrScan&tab=issues

Detected Code Changes

Change Type	Relevant files
Enhancement	► rs/config/src/http_handler.rs     Add extra_api_boundary_node_trust_anchors_pem to Config struct and Default implementation ► rs/http_endpoints/nns_delegation_manager/src/nns_delegation_manager.rs     Use extra_api_boundary_node_trust_anchors_pem to build API boundary node root store ► rs/ic_os/config/tool/src/guestos/generate_ic_config.rs     Encode extra_api_boundary_node_trust_anchors_pem as JSON for IC config template ► rs/ic_os/config/tool/templates/ic.json5.template     Expose extra_api_boundary_node_trust_anchors_pem field in http_handler config ► rs/ic_os/config/types/compatibility_tests/fixtures/guestos_v1.16.0.json     Add extra_api_boundary_node_trust_anchors_pem to guestos_dev_settings (null) ► rs/ic_os/config/types/compatibility_tests/fixtures/hostos_v1.16.0.json     Add extra_api_boundary_node_trust_anchors_pem to hostos_dev_settings (null)
Enhancement	► rs/ic_os/config/types/src/lib.rs     Update CONFIG_VERSION to 1.16.0 ► rs/ic_os/config/types/src/lib.rs     Add extra_api_boundary_node_trust_anchors_pem field to GuestOSDevSettings with documentation ► rs/ic_os/networking/network/BUILD.bazel     Add visibility for network crate ► rs/ic_os/networking/network/src/systemd.rs     Define IPV6_NAME_SERVERS constant and related contents, plus tests for contents
Enhancement	► rs/orchestrator/src/firewall.rs     Update test template to include extra_api_boundary_node_trust_anchors_pem in IC config template
Enhancement	► rs/tests/driver/BUILD.bazel     Add dependency on networking/network for tests ► rs/tests/driver/Cargo.toml     Add network = { path = "../../ic_os/networking/network" }
Enhancement	► rs/tests/driver/src/driver/bootstrap.rs     Adjust API boundary node TLS handling to accommodate local playnet trust anchors; pass config to connect() ► rs/tests/driver/src/driver/ic.rs     Import LocalApiBoundaryNodesPlaynet and adjust related code ► rs/tests/driver/src/driver/ic.rs     Introduce LocalApiBoundaryNodesPlaynet struct and related environment attribute; implement local playnet setup ► rs/tests/driver/src/driver/local_backend.rs     Add IPV6 name-server handling and DNS server integration; extend DNSMASQ configuration to support name-server DNS records ► rs/tests/driver/src/util.rs     Update test fixture to include extra_api_boundary_node_trust_anchors_pem in config

[zeropath-ai]

✅ No security or compliance issues detected. Reviewed everything up to 3d1e1e0.

Security Overview

• 🔎 Scanned files: 24 changed file(s)
• 🔗 Scan Link: https://zeropath.com/app/repositories/f5e9d2bb-29a4-4b64-902a-391d65fd8b2d?scanId=d2a736a1-f373-4944-ad63-188536c2e293&codeScanTypes=PrScan&tab=issues

Detected Code Changes

Change Type	Relevant files
Enhancement	► rs/config/src/http_handler.rs     Add extra_api_boundary_node_trust_anchors_pem to Config struct and Default implementation ► rs/http_endpoints/nns_delegation_manager/src/nns_delegation_manager.rs     Use extra_api_boundary_node_trust_anchors_pem to build API boundary node root store ► rs/ic_os/config/tool/src/guestos/generate_ic_config.rs     Encode extra_api_boundary_node_trust_anchors_pem as JSON for IC config template ► rs/ic_os/config/tool/templates/ic.json5.template     Expose extra_api_boundary_node_trust_anchors_pem field in http_handler config ► rs/ic_os/config/types/compatibility_tests/fixtures/guestos_v1.16.0.json     Add extra_api_boundary_node_trust_anchors_pem to guestos_dev_settings (null) ► rs/ic_os/config/types/compatibility_tests/fixtures/hostos_v1.16.0.json     Add extra_api_boundary_node_trust_anchors_pem to hostos_dev_settings (null)
Enhancement	► rs/ic_os/config/types/src/lib.rs     Update CONFIG_VERSION to 1.16.0 ► rs/ic_os/config/types/src/lib.rs     Add extra_api_boundary_node_trust_anchors_pem field to GuestOSDevSettings with documentation ► rs/ic_os/networking/network/BUILD.bazel     Add visibility for network crate ► rs/ic_os/networking/network/src/systemd.rs     Define IPV6_NAME_SERVERS constant and related contents, plus tests for contents
Enhancement	► rs/orchestrator/src/firewall.rs     Update test template to include extra_api_boundary_node_trust_anchors_pem in IC config template
Enhancement	► rs/tests/driver/BUILD.bazel     Add dependency on networking/network for tests ► rs/tests/driver/Cargo.toml     Add network = { path = "../../ic_os/networking/network" }
Enhancement	► rs/tests/driver/src/driver/bootstrap.rs     Adjust API boundary node TLS handling to accommodate local playnet trust anchors; pass config to connect() ► rs/tests/driver/src/driver/ic.rs     Import LocalApiBoundaryNodesPlaynet and adjust related code ► rs/tests/driver/src/driver/ic.rs     Introduce LocalApiBoundaryNodesPlaynet struct and related environment attribute; implement local playnet setup ► rs/tests/driver/src/driver/local_backend.rs     Add IPV6 name-server handling and DNS server integration; extend DNSMASQ configuration to support name-server DNS records ► rs/tests/driver/src/util.rs     Update test fixture to include extra_api_boundary_node_trust_anchors_pem in config

No canister changes.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 24 changed files in this pull request and generated no new comments.