security: SSRF gate resolves the host httpx connects to; allowlist matches its spellings

[fduple]

What this changes

check_url now resolves the hostname httpx will connect to (httpx.URL(url).raw_host) instead of urlparse's spelling of it. allow_private_hosts matches hostnames across spellings of the same name and rejects entries that can never match, and check_url raises only SafeHTTPError, so one bad URL no longer fails a whole batch.

Why

Three groups of defects in the SSRF gate from #53.

Lookup name. check_url resolved urlparse's hostname, and socket.getaddrinfo encodes a non-ASCII name with the stdlib idna codec (IDNA 2003), while httpx connects to the IDNA 2008 name. The two differ for the deviation characters ß, ς, ZWJ and ZWNJ: for http://straße.attacker.example/ the gate looked up strasse.attacker.example and httpx connects to xn--strae-oqa.attacker.example, two separate DNS names. Whoever controls the zone can point the first at a public address and the second at a private one, and the gate passes the URL. That is the precondition the module docstring already accepts for DNS rebinding (control of the zone), but nothing depends on timing.

The gate now resolves raw_host, the ASCII name httpx puts on the wire. The stdlib codec passes an ASCII name through unchanged, so the gate and the connection look up the same string. A URL httpx cannot parse or encode (an invalid IDNA label, or a lone surrogate, which is what a non-UTF-8 byte in a command-line argument becomes) is refused as unparseable. Every other gate path (safe_get hops, the browser lane's entry and final-URL checks, the batch fetch, check_oa_url) goes through check_url.

Hostname spellings. Allowlist entries were compared to the URL host as lowercased strings, so a trailing root dot (mirror.internal. vs mirror.internal, the same host whenever the resolver looks the name up as absolute) or unicode vs punycode (münchen.example vs xn--mnchen-3ya.example) missed the entry, and the user's own mirror was refused. An entry with a stray % (my%host), or one httpx refuses as a URL host (a fullwidth letter, a name not in NFC, a soft hyphen, a zero-width space), was accepted silently and could never match.

Both sides now go through one normalization: lowercase, a non-ASCII name through idna.encode, then strip the trailing dot. idna.encode is the same IDNA 2008 encoder httpx applies to a URL host, now a declared dependency (unpinned, as httpx declares it). The stdlib codec is not used because it maps straße to strasse, a separately registrable name. An entry idna.encode rejects, or a % entry that is not an address, raises SafeHTTPError naming the entry.

Exception containment. The batch fetch skips a refused URL by catching SafeHTTPError, so anything else out of check_url fails the whole batch. check_url("http://[::1/x") raised ValueError from urlparse, and a 300-character label raised UnicodeError from getaddrinfo. Both now become SafeHTTPError.

Side effect: for a resolver OSError other than gaierror, check_oa_url's refusal reason now reads DNS lookup failed for ... instead of DNS resolution failed: ...; test_rejects_unresolvable_host asserts the new text.

How it was verified

• 26 new cases in tests/test_web/test_safe_http.py and one batch test in tests/test_web/test_fetch_many_gate.py; no new test sends a DNS query.
• With the safe_http.py change reverted, 22 tests fail; routing the normalization through the stdlib codec fails the case with entry straße.example and URL strasse.example.
• A resolver trace shows getaddrinfo("straße.attacker.example") asking for strasse.attacker.example; a Playwright Chromium 153 build asks for the same IDNA 2008 names as httpx for all four deviation characters.
• pytest tests/: 1746 passed on 3.11, 3.12, 3.13 and 3.14. ruff check src/ tests/ is clean.
• Limits: Windows was not run, and Chromium was checked on one build only.

---

Scope check — the review is much faster when the diff contains only the change being proposed:

• No unrelated reformatting, whitespace, or prose rewriting outside the lines this change needs
• CHANGELOG.md touched only to add an entry under [Unreleased]
• No edits to user-facing output strings, docstrings, or test fixtures unrelated to this fix
• If you ran a formatter or an automated cleanup pass, its results are in a separate PR