Summary
Connection#connect (lib/datadog/statsd/udp_connection.rb) resolves the agent host via Addrinfo.udp(host, port).afamily, which performs a dual-stack (AF_UNSPEC) getaddrinfo lookup — i.e. it queries both A and AAAA regardless of which family is actually needed. This was introduced in #280 (merged in v5.6.1) to support IPv6 agents, and the PR description already flagged the tradeoff:
This will additionally work for DNS hostnames, but it will add a bit of time penalty due to the double-DNS lookup.
In a Kubernetes environment with a search domain list and ndots configured (the default Pod DNS setup — see Kubernetes DNS docs), this "bit of time penalty" becomes a genuine query-volume amplifier when the agent host is IPv4-only, which is the common case (e.g. the Datadog Cluster Agent's Kubernetes Service typically has no AAAA record at all).
Root cause
ndots is documented to try the absolute (fully-qualified) form of a name first, falling back to search-domain-suffixed candidates only if that lookup fails. We found that a NODATA response (NOERROR, zero records) for one of the two dual-stack query types gets treated the same as an outright failure, re-triggering the search-domain fallback that's meant for genuine resolution failure (NXDOMAIN).
Confirmed live against three cases, each isolated with a distinct/never-queried hostname where relevant, capturing the actual DNS queries issued (CoreDNS query log):
| Hostname |
Absolute A |
Absolute AAAA |
Search-domain fallback fires? |
A real dual-stack host (one.one.one.one, both A and AAAA exist) |
success |
success |
no — 2 queries total, both absolute, no search-domain candidates generated |
An IPv4-only Kubernetes Service (<name>.<namespace>.svc.cluster.local, our case: the Datadog Agent's own dogstatsd Service) |
success |
NODATA (NOERROR, 0 records) |
yes — 8 queries: all 4 name variants (absolute + 3 search-suffixed, one per configured search domain) queried as both A and AAAA |
| A nonexistent test hostname |
NXDOMAIN |
NXDOMAIN |
yes (expected/documented behavior) |
So ndots isn't malfunctioning — the interaction is between its NXDOMAIN-triggers-fallback semantics and a dual-stack lookup where one address family legitimately has no records. Since the agent host is IPv4-only, every single resolution attempt against it hits this path, generating 4x the DNS query volume of a single-family lookup (and unlike a single-family lookup, half of every pair (the AAAA half) can structurally never succeed).
Impact
In our cluster, this specific hostname alone accounted for ~77% of all NXDOMAIN responses observed on a single CoreDNS replica in a 90-second capture window, and cluster-wide NXDOMAIN volume sustained ~160K–188K/minute. Not all of that is attributable to this gem specifically (other resolvers/services contribute too), but the dual-stack behavior here measurably doubles the query cost versus a single-family lookup for every dogstatsd client instantiation against a hostname (vs. IP literal).
Suggested mitigations (any of these would help; not asking for all)
- Document the interaction with Kubernetes
ndots/search domains explicitly, so users can proactively lower ndots or use a fully-qualified (trailing-dot) hostname to avoid it.
- Offer a config option to force a single address family (e.g.
family: :INET) when the caller knows the agent is IPv4-only, skipping the dual-stack lookup entirely.
- Cache/reuse the resolved
Addrinfo across reconnects more aggressively (if not already done) to reduce how often the amplified lookup recurs.
Environment
dogstatsd-ruby 5.6.1- Kubernetes (EKS), Pod DNS with
ndots:2 explicitly configured (also reproduced conceptually at the cluster default ndots:5)
- Ruby 3.4.9
Happy to provide more detail (exact resolv.conf, raw CoreDNS log excerpts) if useful. Also noticed #287 (open, same version, Sidekiq integration silently losing metrics on EKS after upgrading to 5.6.1) — I can't confirm it shares this root cause, but the version/platform overlap seemed worth mentioning there too.
Summary
Connection#connect(lib/datadog/statsd/udp_connection.rb) resolves the agent host viaAddrinfo.udp(host, port).afamily, which performs a dual-stack (AF_UNSPEC)getaddrinfolookup — i.e. it queries both A and AAAA regardless of which family is actually needed. This was introduced in #280 (merged in v5.6.1) to support IPv6 agents, and the PR description already flagged the tradeoff:In a Kubernetes environment with a
searchdomain list andndotsconfigured (the default Pod DNS setup — see Kubernetes DNS docs), this "bit of time penalty" becomes a genuine query-volume amplifier when the agent host is IPv4-only, which is the common case (e.g. the Datadog Cluster Agent's Kubernetes Service typically has no AAAA record at all).Root cause
ndotsis documented to try the absolute (fully-qualified) form of a name first, falling back to search-domain-suffixed candidates only if that lookup fails. We found that a NODATA response (NOERROR, zero records) for one of the two dual-stack query types gets treated the same as an outright failure, re-triggering the search-domain fallback that's meant for genuine resolution failure (NXDOMAIN).Confirmed live against three cases, each isolated with a distinct/never-queried hostname where relevant, capturing the actual DNS queries issued (CoreDNS query log):
one.one.one.one, both A and AAAA exist)<name>.<namespace>.svc.cluster.local, our case: the Datadog Agent's own dogstatsd Service)So
ndotsisn't malfunctioning — the interaction is between its NXDOMAIN-triggers-fallback semantics and a dual-stack lookup where one address family legitimately has no records. Since the agent host is IPv4-only, every single resolution attempt against it hits this path, generating 4x the DNS query volume of a single-family lookup (and unlike a single-family lookup, half of every pair (the AAAA half) can structurally never succeed).Impact
In our cluster, this specific hostname alone accounted for ~77% of all NXDOMAIN responses observed on a single CoreDNS replica in a 90-second capture window, and cluster-wide NXDOMAIN volume sustained ~160K–188K/minute. Not all of that is attributable to this gem specifically (other resolvers/services contribute too), but the dual-stack behavior here measurably doubles the query cost versus a single-family lookup for every dogstatsd client instantiation against a hostname (vs. IP literal).
Suggested mitigations (any of these would help; not asking for all)
ndots/search domains explicitly, so users can proactively lowerndotsor use a fully-qualified (trailing-dot) hostname to avoid it.family: :INET) when the caller knows the agent is IPv4-only, skipping the dual-stack lookup entirely.Addrinfoacross reconnects more aggressively (if not already done) to reduce how often the amplified lookup recurs.Environment
dogstatsd-ruby5.6.1ndots:2explicitly configured (also reproduced conceptually at the cluster defaultndots:5)Happy to provide more detail (exact
resolv.conf, raw CoreDNS log excerpts) if useful. Also noticed #287 (open, same version, Sidekiq integration silently losing metrics on EKS after upgrading to 5.6.1) — I can't confirm it shares this root cause, but the version/platform overlap seemed worth mentioning there too.