Apparent auto-escaping of query params that contain "/" causes double-escaping

[thetanz-geoff]

Hi team.

As of release 0.25.0, we have found that some blob storage SAS URLs which carry their key in the query-string are returning 403 exceptions.

This has been tracked down to the presence of the forward slash (/) within the key. The key being invalid leads to an authorization failure.

We can also reproduce this example with simpler URLs. Let's Google, first for the term [ham and eggs], and then for [ham / eggs].

        test_url = "https://www.google.com/search?q=ham%20and%20eggs"
        response = client.get(test_url, timeout=60.0)
        response.raise_for_status()
        
        test_url = "https://www.google.com/search?q=ham%20/%20eggs"
        response = client.get(test_url, timeout=60.0)
        response.raise_for_status()

Here's our output:

[2023-09-19T17:16:40.546Z] HTTP Request: GET https://www.google.com/search?q=ham%20and%20eggs "HTTP/1.1 200 OK"
[2023-09-19T17:16:41.300Z] HTTP Request: GET https://www.google.com/search?q=ham%2520%2F%2520eggs "HTTP/1.1 200 OK"

Note how the first request leaves our original URL encoding in place. The second request, noticing the / in the query string, decides to re-encode the parameter and munges %20 to %2520.

The result is that I am accessing a URL other than that intended, and the value I'm passing in my query is mutated by httpx.

httpx 0.24.1 and earlier - all the way back at least to 0.13.3 - get it right:

[2023-09-19T17:20:36.986Z] HTTP Request: GET https://www.google.com/search?q=ham%20and%20eggs "HTTP/1.1 200 OK"
[2023-09-19T17:20:37.682Z] HTTP Request: GET https://www.google.com/search?q=ham%20/%20eggs "HTTP/1.1 200 OK"

[craigholm]

I ran into this same problem with Internet Archive URLs. In this case, I'm not even supplying the query parameters; they're just part of a redirect.

Testing with https://archive.org/download/msdos_XERIX2_shareware/XERIX2.zip/XERIX2%2FXERIX2.DOC

With 0.25.0:

INFO [2023-10-01 08:26:35] httpx - HTTP Request: HEAD https://ia804701.us.archive.org/view_archive.php?archive=%2F13%2Fitems%2Fmsdos_XERIX2_shareware%2FXERIX2.zip&file=XERIX2%252FXERIX2.DOC "HTTP/1.1 503 Service Unavailable"

With 0.24.1:

INFO [2023-10-01 08:28:51] httpx - HTTP Request: HEAD https://ia804701.us.archive.org/view_archive.php?archive=/13/items/msdos_XERIX2_shareware/XERIX2.zip&file=XERIX2%2FXERIX2.DOC "HTTP/1.1 200 OK"

[DankTechnologies]

This change in 0.25.0 cauesd a regression with AWS S3 presigned uploads and downloads. Those URLs have lengthy query params for tokens and signatures, and the double-escaping leads to 400 responses with the error The provided token is malformed or otherwise invalid.