Skip to content

feat(dns): add native PowerDNS secondary replication - #1667

Draft
djdomi wants to merge 6 commits into
current_devfrom
1164_atomic_check_rebased
Draft

feat(dns): add native PowerDNS secondary replication#1667
djdomi wants to merge 6 commits into
current_devfrom
1164_atomic_check_rebased

Conversation

@djdomi

@djdomi djdomi commented Aug 24, 2026

Copy link
Copy Markdown
Member

Summary

  • Add native PowerDNS primary/secondary replication for the local dns-standard and dns-ssl pair so dns-standard is the single local writer and dns-ssl receives TSIG-protected AXFR/NOTIFY updates.
  • Route Kea D2 DDNS writes only to the primary DNS service and disable local NATS record writes on secondary DNS nodes with NATS_RECORD_WRITES=0.
  • Extend remote secondary registration/setup so generated secondary deployments receive the shared DDNS TSIG key and the native AXFR primary endpoint.
  • Update the DNS/DHCP/UI architecture docs and stale current-state notes for the native replication model.

Refs #1164

Impact

  • Reduces DNS drift risk between primary and secondary PowerDNS instances by making one node authoritative for local DDNS/admin writes.
  • Keeps secondaries as PowerDNS-native replicas instead of independent local writers.
  • Preserves the existing NATS registration/event path while preventing secondary-local record mutations.

Validation

  • Local quick gates:
    • git diff --check
    • bash -n services/dns/entrypoint.sh services/dhcp/entrypoint.sh setup.sh tests/bats/helpers/dns-ddns-tsig-helpers.sh tests/bats/dns_ddns_tsig_revoke.bats
    • cargo fmt --manifest-path services/dns/nats-subscriber/Cargo.toml -- --check
    • cargo fmt --manifest-path services/ui/Cargo.toml -- --check
  • Build-tools container on the LXC validation host, cloned from pushed commit 2757daa:
    • bash scripts/tracked/check-file-headers.sh
    • bash scripts/untracked/check-workflow-line-limit.sh
    • timeout --kill-after=30 300 bash scripts/untracked/check-review-chronology-comments.sh
    • cargo fmt --manifest-path services/dns/nats-subscriber/Cargo.toml -- --check
    • cargo fmt --manifest-path services/ui/Cargo.toml -- --check
    • find . \( -name "*.sh" -o -name "*.bats" \) ... | xargs shellcheck --severity=warning
    • actionlint .github/workflows/*.yml
    • bats tests/bats/dhcp_kea_config_generation.bats tests/bats/setup_secondary_docker_compose.bats tests/bats/dns_ddns_tsig_revoke.bats (46/46 passed)
    • cargo test --quiet --manifest-path services/dns/nats-subscriber/Cargo.toml record_write_mode_disables_common_false_spellings
    • cargo test --quiet --manifest-path services/ui/Cargo.toml register_response_serializes_image_tag_for_secondary_setup
    • docker compose config --quiet with CI validation env for deploy/prod, deploy/quickstart, deploy/full-setup, and deploy/secondary
  • Temporary LXC validation directories were verified cleaned up (final-temp-clean).

Risk

  • A full live multi-container AXFR/NOTIFY soak test was not run in this pass.
  • Remote secondaries that are not listed in DNS_XFR_NOTIFY_TARGETS rely on PowerDNS refresh polling for convergence.
  • The package audit found Alpine 3.24 currently provides PowerDNS Authoritative 5.0.7-r0 while upstream stable was observed at 5.1.4; this PR does not change the PowerDNS package version.

Follow-up

  • Run a live AXFR/NOTIFY/refresh convergence test that includes blocked-NOTIFY recovery and Kea DDNS SOA serial behavior.
  • Decide separately whether PowerDNS Authoritative should be bumped once the base image/package source impact is reviewed.
  • Add a delivery/serial verification mechanism for any remaining NATS DNS consumers if they continue to exist after native replication is fully adopted.

AGENTS.md Compliance Notes

  • Existing repo mechanisms were searched and reused before adding behavior: services/dns/entrypoint.sh, configure_ddns_tsig, cmd_secondary, and the existing Bats helper-extraction pattern.
  • No new source/config files were added for this implementation.
  • GitHub-visible text in this PR is intentionally written in English.

@github-actions github-actions Bot added documentation Improvements or additions to documentation rust Pull requests that update rust code docker Pull requests that update docker code kea Component: ISC Kea DHCP and control API integration. pdns Component: PowerDNS authoritative DNS and API integration. admin-ui Component: Admin UI control plane. dns Component: DNS routing, authoritative DNS, and resolver behavior. dhcp Component: DHCP service behavior and configuration. setup Component: setup.sh install, update, migration, and first-user experience. labels Aug 24, 2026
@djdomi djdomi added the enhancement New feature or request label Aug 24, 2026
@djdomi djdomi added this to the LanCache-NG Roadmap milestone Aug 24, 2026
@github-actions github-actions Bot added github_actions Pull requests that update GitHub Actions code ci Continuous integration and runner workflow changes labels Aug 24, 2026
djdomi added 3 commits August 24, 2026 16:07
What: three real bugs found while reproducing the failing UI/NATS/DNS
  integration simulation on lancache-229: the validation compose fixture's
  DDNS_TSIG_KEY was a plain string, not valid base64 (pdnsutil's TSIG
  import needs real key material); the primary never set allow-axfr-ips,
  so PowerDNS's loopback-only default REFUSED every secondary's AXFR
  regardless of a valid TSIG; and the simulation only gave dns-ssl 10s to
  see a change, shorter than PDNS_XFR_CYCLE_INTERVAL's own 15s default.
Why: confirmed empirically (2026-08-24, lancache-229, manual compose
  reproduction) that fixing the TSIG format alone let the zone actually
  transfer, just past the test's 10s window; allow-axfr-ips derives from
  DNS_XFR_NOTIFY_TARGETS via the existing dns_xfr_primary_endpoint()
  resolver, matching PDNS_ALLOW_NOTIFY_FROM's own pattern.
From: Issue #1164
@github-actions github-actions Bot added the tooling Developer or CI tooling work label Aug 24, 2026
@djdomi

djdomi commented Aug 24, 2026

Copy link
Copy Markdown
Member Author

Reproduced the failing UI/NATS/DNS integration simulation locally on lancache-229 (manual docker compose run against the actual published pr-1667-sha-... image, TSIG/AXFR traced with dig/pdnsutil/pdns_control directly). Found and fixed 3 real, confirmed bugs, pushed in 4e3ece7:

  1. deploy/full-setup/docker-compose.yml's DDNS_TSIG_KEY=validation-ddns-key was a plain string, not valid base64. pdnsutil import-tsig-key ... hmac-sha256 $DDNS_TSIG_KEY needs real base64 key material for hmac-sha256; PowerDNS logged Unable to Base-64 decode TSIG key 'lancache-ddns-key' ... can not AXFR for every zone on dns-ssl. Production setup.sh generates this correctly via base64_32 -- only this test fixture had the bad placeholder.
  2. allow-axfr-ips was never set on the primary. PowerDNS's own default is loopback-only (127.0.0.0/8,::1); a valid TSIG signature does not bypass this ACL. Added PDNS_ALLOW_AXFR_IPS, resolved from DNS_XFR_NOTIFY_TARGETS via the existing dns_xfr_primary_endpoint() helper (mirrors PDNS_ALLOW_NOTIFY_FROM's own pattern on the secondary side).
  3. The simulation's verify_record_resolves only gave dns-ssl a 10s budget, shorter than PDNS_XFR_CYCLE_INTERVAL's own 15s default -- confirmed the zone genuinely does transfer once TSIG is fixed, just past the test's original window. Bumped dns-ssl's check to 25 attempts.

Still open, not resolved by the above: with all three fixes in place (TSIG valid, allow-axfr-ips correctly listing the secondary's IP, zone kind MASTER/SLAVE both confirmed correct, SOA-EDIT-API confirmed incrementing the primary's serial), a manual dig ... AXFR +tcp from dns-ssl against dns-standard still gets status: REFUSED -- reproduced even with allow-axfr-ips=0.0.0.0/0,::/0 (wide open) and TSIG-ALLOW-AXFR metadata cleared entirely, so neither the IP ACL nor TSIG is the actual gate for this REFUSED. pdns_control retrieve lan accepts the request but the secondary's SOA serial never advances either. No AXFR-related log line appears on either side even at loglevel=8. I wasn't able to isolate this fourth cause within a reasonable time budget -- it needs either PowerDNS 5.4.5-specific AXFR-authorization knowledge I don't have, or a packet-level capture of the refused exchange. Flagging rather than guessing further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

admin-ui Component: Admin UI control plane. ci Continuous integration and runner workflow changes dhcp Component: DHCP service behavior and configuration. dns Component: DNS routing, authoritative DNS, and resolver behavior. docker Pull requests that update docker code documentation Improvements or additions to documentation enhancement New feature or request github_actions Pull requests that update GitHub Actions code kea Component: ISC Kea DHCP and control API integration. pdns Component: PowerDNS authoritative DNS and API integration. rust Pull requests that update rust code setup Component: setup.sh install, update, migration, and first-user experience. tooling Developer or CI tooling work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant