Mobile clients do not seem to route NetBird Custom Zones through the NetBird DNS resolver

[pzimbrod]

I‘m having problems on mobile devices getting my DNS zones to resolve. I’ve tried to give a comprehensive overview of the problem and my current insight below. I also read the DNS troubleshooting section in the netbird docs. Unfortunately, my problem still persists. I’d greatly appreciate some guidance on what the root cause might be and/or if this is a configuration problem or rather something else entirely.

Environment

I am running a self-hosted NetBird setup.

• NetBird mobile app: 0.3.3
• NetBird management: 0.74.6
• Self-hosted Management, Signal and Relay
• Clients connect directly as peers. I am not using Network Routes.
• Internal DNS is provided through a NetBird Custom Zone.
• No NetBird Nameserver Groups were configured initially.
• NetBird's internal DNS resolver is reachable at 100.64.0.1 in the examples below.
• NetBird mobile apps were simply installed and used as intended from the respective app stores. No further manual configuration was done other than logging on to my self hosted instance.

The problem occurs on both an iPhone and an Android device when connected to the same hotel WiFi network.

The NetBird tunnel itself works correctly. I can access services directly through their NetBird IPs, and HTTPS connections work when using the correct SNI.

Problem

DNS resolution through the operating system resolver fails for names that should be handled by the NetBird DNS resolver.

I‘ve used the a-Shell iOS app below for below console output. Note that the domains below with the .internal suffix are just some generic ones replacing my real, private domains.

For example:

python3 -c "import socket; print(socket.getaddrinfo('service.example.internal', 443, proto=socket.IPPROTO_TCP))"

socket.gaierror: [Errno 8] nodename nor servname provided, or not known

The NetBird peer DNS name fails in exactly the same way:

python3 -c "import socket; print(socket.getaddrinfo('peer01.netbird.example', 443, proto=socket.IPPROTO_TCP))"

socket.gaierror: [Errno 8] nodename nor servname provided, or not known

This is particularly interesting because the latter name is entirely independent of my public DNS configuration.

At the same time, explicitly querying the NetBird DNS resolver works:

dig service.example.internal

;; ->>HEADER<<- opcode: QUERY, status: NOERROR
;; ANSWER SECTION:
service.example.internal. 300 IN A 100.64.0.2

;; SERVER: 100.64.0.1#53(100.64.0.1)

The same is true for other internal names.

The system resolver itself is working. For example:

python3 -c "import socket; print(socket.getaddrinfo('example.com', 443, proto=socket.IPPROTO_TCP))"

returns the expected public IPv4/IPv6 addresses.

Additional diagnostic result

I initially had no NetBird Primary Nameserver configured.

Based on the discussion around mobile DNS routing, I subsequently added Cloudflare (1.1.1.1) as a NetBird Primary Nameserver.

This did not resolve the problem.

After adding the Primary Nameserver, the following still fails:

python3 -c "import socket; print(socket.getaddrinfo('service.example.internal', 443, proto=socket.IPPROTO_TCP))"

socket.gaierror: [Errno 8] nodename nor servname provided, or not known

and:

python3 -c "import socket; print(socket.getaddrinfo('peer01.netbird.example', 443, proto=socket.IPPROTO_TCP))"

socket.gaierror: [Errno 8] nodename nor servname provided, or not known

Interestingly, getaddrinfo() does resolve a hostname that is not part of the NetBird Custom Zone:

python3 -c "import socket; print(socket.getaddrinfo('auth.example.com', 443, proto=socket.IPPROTO_TCP))"

[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('203.0.113.10', 443))]

while a direct query to the NetBird resolver returns the internal address:

dig auth.example.com

;; ->>HEADER<<- opcode: QUERY, status: NOERROR

;; ANSWER SECTION:
auth.example.com. 300 IN A 100.64.0.3

;; SERVER: 100.64.0.1#53(100.64.0.1)

The addresses in these examples are intentionally anonymized.

This suggests to me that the system resolver is using the normal/public DNS path for auth.example.com, while it is not routing the NetBird-specific DNS names through the NetBird resolver.

What appears to work

The following parts have been verified to work:

• NetBird connection is established.
• Peer connection is P2P.
• Overlay connectivity works.
• TCP and TLS connections over the NetBird IPs work.
• Caddy correctly handles the expected SNI.
• The NetBird DNS resolver itself returns the correct internal records.
• The system resolver can resolve ordinary public hostnames.
• The issue affects both the Custom Zone and the NetBird peer domain.

For example, this works:

curl -vk --resolve service.example.internal:443:100.64.0.2 https://service.example.internal

and returns the expected service page.

Resolver comparison

The important distinction seems to be between explicitly querying the NetBird resolver and using the operating system resolver.

Query	Result
dig @100.64.0.1 service.example.internal	Correct internal address
getaddrinfo(service.example.internal)	Fails
getaddrinfo(peer01.netbird.example)	Fails
getaddrinfo(example.com)	Works
getaddrinfo(auth.example.com)	Works, but returns the public address
dig auth.example.com via NetBird resolver	Returns internal address
Adding Cloudflare as Primary Nameserver	No change

The resolver itself and all internal addresses in this issue are anonymized examples.

Note on Reproducibility

The issue has so far only been reproducible on this particular hotel WiFi.

The network uses private RFC1918 addresses and appears to have a captive portal.

However, the NetBird connection itself is healthy and the DNS resolver is reachable from the NetBird environment.

I cannot currently verify whether the same problem still occurs on another network, so I do not want to claim that the hotel network is the root cause.

However, I have been running this setup from my home WiFi, other private WiFi networks and via mobile data without any issues and for quite some time. That’s why this symptom appears very puzzling to me.

Questions

Could this be related to how the NetBird mobile clients configure DNS scopes / match domains through the platform VPN APIs?

In particular, I would like to understand:

1. How does NetBird 0.3.3 configure the mobile OS DNS resolver for Custom Zones?
2. Should a Custom Zone automatically become a DNS match domain on iOS and Android?
3. Is a Primary Nameserver required for Custom Zones on iOS/Android?
4. Is there a known limitation or bug where Custom Zones work with explicit queries to the NetBird DNS resolver, but are not selected by the OS resolver used by applications?
5. Could the fact that this happens on both iOS and Android point to an issue in the common NetBird DNS configuration rather than in either platform's DNS implementation?

The most useful comparison seems to be:

dig -> NetBird resolver -> correct internal address

getaddrinfo() -> OS resolver -> public DNS / failure

The NetBird DNS server itself appears to be working correctly. The problem seems to be getting the mobile OS resolver to actually use it for the appropriate domains.

Debug bundle

Debug bundle key:

51145cfa42d00e6df84bae2a62b31f25243be6795fd075f621133b885382236b/b64213e8-56ba-4cc1-9ab9-78447c3f04a7

I hope I haven’t overlooked anything obvious. Please let me know if I can provide you with any more background on this issue.

Thanks in advance
Patrick