feat: primary mode — AXFR zone transfers with automatic NOTIFY and TSIG

README.md

@@ -37,6 +37,11 @@ the fourth development release, considered alpha quality. Any testing is appreci
     * e.g. in an `SRV` entry: `20 5 _ server1`, the port will be searched for in default values, the name `server1` will be appended with the zone name
     * same entry in JSON5 syntax: `{priority: 20, weight: 5, target: "server1"}` (this is longer but clearer)
 * [`ALIAS`](https://doc.powerdns.com/authoritative/guides/alias.html) support
+* [Primary (master) mode with AXFR zone transfer](#primary-mode-axfr-zone-transfer)
+    * every zone is served to PowerDNS as `MASTER`, so secondaries can `AXFR` it (the `list` remote-backend method)
+    * automatic `NOTIFY` on zone changes (via `getUpdatedMasters` / `setNotified`) in any run mode (the notified serial is persisted in ETCD)
+    * AXFR ACL by IP (`allow-axfr-ips` / `ALLOW-AXFR-FROM` metadata) and/or [TSIG key](doc/ETCD-structure.md#tsig-keys) (`TSIG-ALLOW-AXFR` metadata)
+    * pre-signed DNSSEC zones are transferred as-is
 * [Multi-level defaults and options](doc/ETCD-structure.md#defaults-and-options), overridable
 * [Domain metadata](https://doc.powerdns.com/authoritative/domainmetadata.html)
     * can also be read and modified with the command line tool `pdnsutil` (`pdnssec` in v3.4)
@@ -87,8 +92,10 @@ the fourth development release, considered alpha quality. Any testing is appreci
 * Redirecting (or duplicating) log output to something else than stderr
 
 ### Overview over the support of optional [PDNS features in a remote backend][pdns-remote]:
-* Primary and (Auto)Secondary: no
-  * AXFR support: not yet
+* Primary (master): yes — see [Primary mode (AXFR zone transfer)](#primary-mode-axfr-zone-transfer)
+  * AXFR support: yes (`list` method), with IP and/or TSIG ACL — requires PowerDNS 4.0+ (the legacy 3.4 remote-backend protocol does not support AXFR-out)
+  * automatic NOTIFY: yes, in any run mode (the notified serial is persisted in ETCD)
+* (Auto)Secondary: no
 * DNSSEC: pre-signed yes, live-signing not yet (planned feature)
   * Metadata: yes
 * Search (web API): not yet (planned feature)
@@ -195,6 +202,100 @@ Because there is no 'initialize' call, the version of a connecting PowerDNS must
 if it differs from the default PDNS version. The default PDNS version can be changed via the `-pdns-version` option (see below).
 Other parameters could be set the same way, the URL path replaces the 'initialize' call for the HTTP connector.
 
+### Primary mode (AXFR zone transfer)
+
+pdns-etcd3 reports every zone it holds to PowerDNS as a [primary (master)][pdns-modes],
+so PowerDNS can answer outgoing zone transfers (AXFR) and send `NOTIFY` to your secondaries.
+On the backend side this is implemented by the remote-backend methods `getDomainInfo` / `getAllDomains`
+(report `kind=MASTER`, an integer `id` and a `notified_serial`), `list` (serves the full zone for AXFR),
+`getUpdatedMasters` / `getUpdatedPrimaries` + `setNotified` (drive automatic NOTIFY),
+and `getTSIGKey` / `getTSIGKeys` (TSIG verification/signing). There is nothing to enable in the backend itself —
+just configure PowerDNS and (optionally) the per-zone [metadata](doc/ETCD-structure.md#primary--axfr) below.
+Note: TSIG-secured transfers additionally require `remote-dnssec=yes` in PowerDNS (see [TSIG](#tsig)).
+
+The authoritative on-ETCD layout for everything mentioned here is in the ETCD structure document:
+[Primary / AXFR](doc/ETCD-structure.md#primary--axfr) and [TSIG keys](doc/ETCD-structure.md#tsig-keys).
+
+[pdns-modes]: https://doc.powerdns.com/authoritative/modes-of-operation.html
+
+#### PowerDNS configuration
+
+Enable primary operation in the PowerDNS configuration (in addition to the `remote-connection-string` from the run-mode
+sections above):
+```text
+# PowerDNS >= 4.5
+primary=yes
+# PowerDNS < 4.5 use the old spelling instead:
+#master=yes
+```
+For AXFR, PowerDNS notifies the zone's `NS` records plus any [`also-notify`][pdns-also-notify] targets
+(per-zone via the `ALSO-NOTIFY` metadata). IP-based AXFR access can be restricted with the PowerDNS
+[`allow-axfr-ips`][pdns-allow-axfr-ips] setting (global) and/or the per-zone `ALLOW-AXFR-FROM` metadata.
+
+[pdns-also-notify]: https://doc.powerdns.com/authoritative/settings.html#also-notify
+[pdns-allow-axfr-ips]: https://doc.powerdns.com/authoritative/settings.html#allow-axfr-ips
+
+#### Run mode
+
+Both plain AXFR serving (a secondary pulling the zone) and automatic `NOTIFY` on zone changes work in
+**any** run mode (pipe or standalone). PowerDNS detects a changed zone by comparing the serial to the last
+*notified* serial, which pdns-etcd3 persists in ETCD under a global `-notified-/<id>` entry (kept outside any
+zone's prefix so that recording it never bumps a zone's own serial — which would otherwise cause a NOTIFY
+feedback loop). Because that state lives in ETCD it is shared across processes, so it also works in pipe mode,
+where PowerDNS spawns a fresh short-lived process per request thread.
+
+#### Pointing an external secondary
+
+Configure the secondary (BIND, Knot, PowerDNS, …) to transfer the zone *from your PowerDNS server* (not from ETCD or
+pdns-etcd3 directly). Make sure the transfer is permitted: either by IP (`allow-axfr-ips` / `ALLOW-AXFR-FROM`) or by
+TSIG (see below), or both. Verify with `dig` (see [Verifying](#verifying) below) before relying on the secondary.
+
+#### TSIG
+
+**Required PowerDNS setting:** enable `remote-dnssec=yes`. PowerDNS's remote backend gates the DNSSEC/TSIG
+methods (including `getTSIGKey`) behind the backend's `dnssec` flag — without `remote-dnssec=yes` PowerDNS never
+queries the backend for the TSIG key and denies the signed transfer with `NOTAUTH`. (pe3 manages no DNSSEC keys, so
+it answers `getDomainKeys` with an empty set; pre-signed zones still work via the `PRESIGNED` metadata.)
+
+TSIG keys are stored as a *global* pseudo-entry in ETCD (not under any zone), read on demand by `getTSIGKey` /
+`getTSIGKeys`:
+```text
+<prefix>-tsig-/<keyname>   →   "<algorithm> <base64-secret>"
+# e.g.
+<prefix>-tsig-/axfrkey.    →   "hmac-sha256 <base64-secret>"
+```
+Then authorize the key for a zone's AXFR with the per-zone `TSIG-ALLOW-AXFR` metadata (a list of allowed key names,
+e.g. via `pdnsutil set-meta <zone> TSIG-ALLOW-AXFR <keyname>`). Each value names one key, which must match a
+`-tsig-/<keyname>` entry.
+
+**Trailing-dot caveat:** `getTSIGKey` does an *exact-match* lookup in ETCD with no name canonicalization, so the key
+must be stored under the exact name PowerDNS requests. Whether that name carries a trailing `.` (FQDN) depends on
+PowerDNS; to be safe, store the secret under both spellings — `<keyname>` and `<keyname>.` — so the lookup matches
+either way.
+
+See [TSIG keys](doc/ETCD-structure.md#tsig-keys) in the ETCD structure document for the authoritative key layout.
+
+#### Pre-signed DNSSEC over AXFR
+
+A [pre-signed DNSSEC zone](doc/ETCD-structure.md#pre-signed-dnssec) is transferred as-is: store the signed records
+(`RRSIG`, `NSEC`/`NSEC3`, `DNSKEY`, …) in ETCD like any other record, set `PRESIGNED=1` as metadata on the zone, and
+pin the served serial with the [`X-PE3-FIXED-SERIAL`](doc/ETCD-structure.md#reserved-x-pe3--keys) metadata so it matches
+the serial the signer baked into `RRSIG(SOA)` (otherwise validating resolvers reject the answer). Delegation `NS` records
+and glue are emitted non-authoritative. See the [Pre-signed DNSSEC zones](#features) feature note and the
+[ETCD structure section](doc/ETCD-structure.md#pre-signed-dnssec) for details.
+
+#### Verifying
+
+Trigger a transfer from the PowerDNS host to confirm it works:
+```shell
+# plain AXFR (allowed by allow-axfr-ips / ALLOW-AXFR-FROM)
+dig AXFR example.com @<pdns-host>
+
+# TSIG-protected AXFR
+dig -y hmac-sha256:<keyname>:<base64-secret> AXFR example.com @<pdns-host>
+```
+A successful transfer prints the full zone (starting and ending with the `SOA` record).
+
 ### Parameters
 
 All parameter keys must be given exactly as denoted here (no case modifications). The ETCD related parameters in standalone mode

doc/ETCD-structure.md

@@ -224,7 +224,7 @@ from above: 1, 2 (only added entries), 4, 6, 8 and 9.
 
 #### Current version
 
-The current data version is `0.2.0` and is described in this document.
+The current data version is `0.2.1` and is described in this document.
 
 ### Defaults and options
 
@@ -313,6 +313,39 @@ Metadata keys starting with `X-PE3-` are reserved for use by this backend. They
 Every zone can be locked for transactions. These entries have a prefix of `<domain>/-lock-` and are handled automatically,
 there is no need to create or delete them manually. They are not part of the automatic SOA serial determination.
 
+## TSIG keys
+
+[TSIG](https://doc.powerdns.com/authoritative/tsig.html) keys are stored as a *global* pseudo-entry, not under any zone:
+
+* Key: `-tsig-/<keyname>` (a single key, no `<domain>` prefix; `<keyname>` is the literal TSIG key name).
+* Value: `"<algorithm> <base64-secret>"` — the algorithm name and the base64-encoded shared secret, separated by whitespace.
+  * Example: `-tsig-/axfrkey.` → `hmac-sha256 <base64-secret>`
+
+These entries are read on demand by the `getTSIGKey` / `getTSIGKeys` remote-backend methods (which PowerDNS calls when it needs to verify or sign a TSIG-protected message, e.g. for [AXFR](#primary--axfr)). They are **never cached in the data tree** (so they are not part of any zone reload) and **never affect any zone serial**. Malformed values (not of the `<algorithm> <base64-secret>` form) are logged and skipped.
+
+**PowerDNS must be configured with `remote-dnssec=yes`** for any of this to take effect: the remote backend gates `getTSIGKey`/`getTSIGKeys` (and the other DNSSEC methods) behind its `dnssec` flag, and otherwise never queries the backend for the key (the signed transfer is refused with `NOTAUTH`). pe3 manages no DNSSEC keys, so it answers `getDomainKeys` with an empty set.
+
+The `<keyname>` is whatever PowerDNS sends; it may or may not carry a trailing `.` (FQDN). To be safe, store the key under both spellings (`<name>` and `<name>.`) so the lookup matches regardless of canonicalization.
+
+## Primary / AXFR
+
+pdns-etcd3 acts as a PowerDNS [primary (master)](https://doc.powerdns.com/authoritative/modes-of-operation.html) — every zone is reported with kind `MASTER`. Outgoing zone transfers (AXFR-OUT) and `NOTIFY` are driven entirely by the existing [metadata](#metadata) passthrough plus the [TSIG keys](#tsig-keys) above; there are no new on-etcd key shapes beyond `-tsig-`.
+
+The relevant per-zone metadata keys (stored as ordinary metadata, `<zone>/-metadata-/<KEY>#<id>`, value passed verbatim to PowerDNS) are:
+
+* `TSIG-ALLOW-AXFR` — list of TSIG key names allowed to request AXFR. Each value is one key name; the name must match a [`-tsig-/<keyname>`](#tsig-keys) entry. Use one entry per allowed key (different `#<id>` per value).
+* `ALLOW-AXFR-FROM` — list of IP addresses / networks allowed to request AXFR without TSIG.
+* `ALSO-NOTIFY` — list of extra `ip[:port]` targets to send `NOTIFY` to (in addition to the zone's `NS` records).
+* `PRESIGNED` — marks a [pre-signed DNSSEC](#pre-signed-dnssec) zone (`PRESIGNED=1`); PowerDNS then serves the stored `RRSIG`/`NSEC`/`DNSKEY` records as-is.
+
+Automatic `NOTIFY` on zone changes relies on tracking the last *notified* serial per zone (PowerDNS compares it to the current serial to decide whether to notify). pdns-etcd3 persists this value in ETCD under a **global pseudo-entry keyed by domain id**:
+
+```text
+<prefix>-notified-/<id>   →   "<serial>"
+```
+
+It is written by `setNotified` and read by `getUpdatedMasters`/`getDomainInfo`/`getAllDomains`. The `<id>` is the `domain_id` PowerDNS uses — a deterministic 31-bit hash of the zone name (stable across processes). The entry lives **outside any zone's prefix**, so recording it never enters a zone's `zoneRev()`/serial (which would otherwise trigger a NOTIFY feedback loop); like the `-tsig-` keys it is read/written on demand and is **never** part of a zone reload. Because the state is in ETCD (not process memory), automatic `NOTIFY` works in **any run mode — pipe as well as standalone**.
+
 ## Pre-signed DNSSEC
 
 pdns-etcd3 supports DNSSEC currently only in the [pre-signed (front-signing)][rfc4035-presigned] model: an external signer (`ldns-signzone`, `dnssec-signzone`, OpenDNSSEC, …) produces the signed records and pushes them into ETCD under the same key layout as ordinary records. The backend itself does not sign anything yet; [online signing](https://doc.powerdns.com/authoritative/dnssec/modes-of-operation.html) is on the [Planned](../README.md#planned) list.
@@ -590,6 +623,11 @@ The `TXT` record is not parsed, when being written in a plain string syntax.
 The changelog lists every change which led to a data version increase (major or minor).
 One can use it to check their data - whether an adjustment is needed for a new program version which has a new data version.
 
+### 0.2.1
+* added global TSIG key pseudo-entry `-tsig-/<keyname>` → `"<algorithm> <base64-secret>"` (for AXFR-OUT)
+* added global notified-serial pseudo-entry `-notified-/<id>` → `"<serial>"` (drives automatic `NOTIFY` in any run mode, incl. pipe)
+* documented [primary / AXFR](#primary--axfr) operation: per-zone metadata `TSIG-ALLOW-AXFR`, `ALLOW-AXFR-FROM`, `ALSO-NOTIFY`, `PRESIGNED` (all via the existing metadata passthrough)
+
 ### 0.2.0
 * allow JSON5 syntax
 * allow YAML syntax for objects

src/common_test.go

@@ -407,3 +407,13 @@ func getenvT(key, def string) string {
 	}
 	return v
 }
+
+// testClient returns a minimal *pdnsClient suitable for unit tests. RootLog is
+// initialized at package load (logging.go), so client.Logf works without further
+// setup; only ID and PdnsVersion are needed by the request handlers under test.
+func testClient(_ *testing.T) *pdnsClient {
+	return &pdnsClient{
+		ID:          pipeClientID{},
+		PdnsVersion: 4,
+	}
+}

src/const.go

@@ -54,6 +54,8 @@ const (
 	optionsKey       = "-options-"
 	metadataKey      = "-metadata-"
 	lockKey          = "-lock-"
+	tsigKey          = "-tsig-"
+	notifiedKey      = "-notified-"
 	keySeparator     = "/"
 	labelPrefix      = "+"
 	idSeparator      = "#"
@@ -66,6 +68,9 @@ const (
 	MetaFixedSerial   = "X-PE3-FIXED-SERIAL"
 )
 
+// kindMaster is the PowerDNS domain kind reported for every zone (we are always primary).
+const kindMaster = "MASTER"
+
 type ipMetaT map[int]struct {
 	totalOctets int
 	partOctets  int
@@ -77,7 +82,7 @@ var (
 	nameRegex  = regexp.MustCompile(`^([a-z_0-9]|[a-z_0-9][a-z_0-9-]*[a-z_0-9]|\*)([./])`)
 	entryRegex = regexp.MustCompile(`^(-[a-z]+-)(?:$|/|#)`)
 	valsRegex  = regexp.MustCompile(`^([A-Z][A-Z0-9]*)?(?:#([^@#]*))?$`)
-	metaRegex  = regexp.MustCompile(`^([A-Z][A-Z0-9-]*)#([^@#]*)$`)
+	metaRegex  = regexp.MustCompile(`^([A-Z][A-Z0-9-]*)(?:#([^@#]*))?$`)
 	ipMeta     = ipMetaT{
 		4: {4, 1, `.`},
 		6: {16, 2, `:`},

src/data.go

@@ -27,7 +27,7 @@ import (
 
 var (
 	// update this when changing data structure (only major/minor, patch is always 0). also change it in docs and in build workflow!
-	dataVersion = VersionType{IsDevelopment: true, Major: 2, Minor: 0}
+	dataVersion = VersionType{IsDevelopment: true, Major: 2, Minor: 1}
 )
 
 type recordType struct {
@@ -278,15 +278,32 @@ func (dn *dataNode) zonesCount() int {
 }
 
 type domainInfo struct {
-	Zone   string `json:"zone"`
-	Serial int64  `json:"serial"`
+	ID             int64  `json:"id"`
+	Zone           string `json:"zone"`
+	Serial         int64  `json:"serial"`
+	NotifiedSerial int64  `json:"notified_serial"`
+	Kind           string `json:"kind"`
 }
 
 func (dn *dataNode) allDomains(result []domainInfo) []domainInfo {
-	if _, ok := dn.records["SOA"][""]; ok {
-		zone, serial := dn.getQname(), dn.zoneRev()
+	// RLock this node while reading its records/children: zone reloads rebuild them
+	// under a write lock. Each recursive call self-locks the child, and we keep this
+	// node RLocked until the child loop finishes, so the walked path is locked
+	// top-down (parent-before-child, matching getChild's order) — no deadlock vs the
+	// reload WLock and no concurrent-map access.
+	dn.RLock(false)
+	defer dn.RUnlock(false)
+	if dn.hasSOA() {
+		zone := dn.getQname()
+		serial := int64(soaWireSerial(dn))
 		dn.Logf(3)("allDomains: found zone %q", zone)("serial", serial)
-		result = append(result, domainInfo{zone, serial})
+		result = append(result, domainInfo{
+			ID:     domainID(zone),
+			Zone:   zone,
+			Serial: serial,
+			Kind:   kindMaster,
+			// NotifiedSerial is filled in by the request handler from the persisted etcd state.
+		})
 	}
 	for _, child := range dn.children {
 		result = child.allDomains(result)
@@ -358,8 +375,10 @@ func parseEntryKey(key string) (name Name, entryType entryType, qtype, id string
 				switch qtype {
 				case "A", "AAAA", "ALIAS", "CNAME", "DNAME", "MX", "NS", "PTR", "SOA": // TODO add others, even not-supported ones?
 					for _, lname := range name {
-						if strings.ContainsRune(lname.name, '_') {
-							err = fmt.Errorf("records for hostnames may not have underscores: %q", lname.name)
+						// a single leading underscore is allowed (RFC 8552 underscored
+						// node names, e.g. _domainkey for delegated DKIM CNAMEs)
+						if strings.ContainsRune(strings.TrimPrefix(lname.name, "_"), '_') {
+							err = fmt.Errorf("records for hostnames may not have underscores (except a single leading one): %q", lname.name)
 							return
 						}
 					}
@@ -376,6 +395,14 @@ func parseEntryKey(key string) (name Name, entryType entryType, qtype, id string
 	case lockEntry:
 		id = key
 		return
+	case tsigEntry:
+		// the remainder after "-tsig-/" is the key name (may contain dots)
+		id = key
+		return
+	case notifiedEntry:
+		// the remainder after "-notified-/" is the domain id
+		id = key
+		return
 	default:
 		err = fmt.Errorf("unhandled entry type: %q", entryType)
 		return
@@ -465,6 +492,13 @@ ITEMS:
 			debug3("ignoring lock entry")(item.Key)
 			continue ITEMS
 		}
+		if entryType == tsigEntry || entryType == notifiedEntry {
+			// TSIG keys and notified-serial markers are global, read on demand, never stored
+			// in the data tree, and must not influence any zone serial → skip before the
+			// maxRev update below.
+			debug3("ignoring %s entry", entryType)(item.Key)
+			continue ITEMS
+		}
 		// check if the entry belongs to this domain
 		if name.len() < depth {
 			continue ITEMS
@@ -503,8 +537,12 @@ ITEMS:
 			vals[qtype][id] = valueType{item.Key, content, itemVersion}
 			debug3values("stored %v for %s", entryType, target)(content)
 		case metadataEntry:
+			// store on itemData (the node the entry belongs to), NOT on dn (the reload
+			// receiver): a full/parent reload has dn above the zone, so using dn would put
+			// a sub-zone's metadata on the wrong node — breaking soaSerial (FIXED-SERIAL),
+			// PRESIGNED detection and TSIG-ALLOW-AXFR for freshly-created zones.
 			value := string(item.Value)
-			dn.metadata[qtype] = append(dn.metadata[qtype], value)
+			itemData.metadata[qtype] = append(itemData.metadata[qtype], value)
 			debug3values("stored %v for %s", entryType, target)(value)
 		default:
 			dn.Errorf()("unhandled entry type")(entryType)