httpx.URL's percent encoding may be too strict

[luizcdc]

Hi, folks.

I work on a project that uses aiohttp and httpx interchangeably, as we've recently started migrating from aiohttp to httpx. I've found a bug this week which only happens when using httpx, which is as follows:

After sending a request (with redirects enabled) that returns a 302 Redirect, the Location header returns as:

Location: '?codg_juizado=8116;;16�%20Juizado%20Esp.C�vel%20e%20das%20Rel.Consumo%20da%20Capital'

On aiohttp and on every actual browser I've tested (Chrome, Firefox, Edge), the request redirects to '?codg_juizado=8116;;16%C3%AF%C2%BF%C2%BD%20Juizado%20Esp.C%C3%AF%C2%BF%C2%BDvel%20e%20das%20Rel.Consumo%20da%20Capital'

Meanwhile, on httpx the already url-encoded '%20' is re-encoded to %2520:

'?codg_juizado=8116;;16%C3%AF%C2%BF%C2%BD%2520Juizado%2520Esp.C%C3%AF%C2%BF%C2%BDvel%2520e%2520das%2520Rel.Consumo%2520da%2520Capital'

This breaks the redirect and the request fails.

aiohttp's behavior is explained by its use of yarl.URL. Yarl uses a callable yarl._quoting_py._Quoter to percent-encode urls. Its _requote parameter is set to True by default. _requote=True establishes that if a '%' is followed by two 0-9A-F (hexadecimal) chars, it should be left as is.

I couldn't establish which library's behavior follows the standard the closest, but aiohttp behaves exactly like the browsers and it actually succeds on reaching the webpage.

I apologize but I wasn't able to provide a reproducible test case, as the page on which this happened requires authentication.

I'd love to get some opinions on this. Maybe we could have a "requote" parameter on httpx.URL's constructor?

[jhominal]

I have an issue that is also related to the way that % characters get, in some cases, automatically re-encoded to %25:

• I have a URL stored as an str object, that I wish to process;
• That URL contains a query, with a query parameter that is itself an URL containing query parameters:
  • http://host/path?sub_url=https://other?a%3Db%26c%3Dd
  • In order to make the specification of that str possible, the query parameters from the URL need to be encoded;
  • However, as slashes are not required to be encoded, they were left as regular / characters;
• httpx double encodes the already existing % characters, transforming the URL to http://host/path?sub_url=https:%2F%2Fother?a%253Db%2526c%253Dd - and the server finds itself processing a doubly-encoded URL parameter.

I am not really bothered by the fact that slashes get URL-encoded (unlike https://github.com/encode/httpx/issues/2864)

In short, I do not think that the % character in my context should be URL encoded.

Here is the unit test that shows my issue:

import httpx

def test_httpx_url():
    url = "http://host/path?sub_url=https://other?a%3Db%26c%3Dd"

    httpx_url = httpx.URL(url)

    assert httpx_url.params.get("sub_url") == "https://other?a=b&c=d"

[thetanz-geoff]

Hi @luizcdc, @jhominal. This issue is originally raised in this discussion: #2856

Recommend you downgrade to httpx 0.24.1 temporarily until the team responds to our threads.

[luizcdc]

@thetanz-geoff Thanks for replying and for the tip.

I've tested 0.24.1 and, although the issue is similar to mine, it's not the same. Downgrading to 0.24.1 only changes that '/' is considered a safe character. The result of httpx.URL('?abc6�%20Juizado%20') is identical between 0.24.1 and 0.25.0.

> httpx.URL('?abc6�%20Juizado%20')
URL('?abc6%C3%AF%C2%BF%C2%BD%2520Juizado%2520')

[ttys0dev]

I'm seeing this issue with servers that treat %2F differently from /.

For example if I have a URL with the following query parameter type=application/pdf it gets incorrectly encoded to type=application%2Fpdf which causes the server to throw an error.