Did 0.24 break params?

[marban]

Haven't gotten into the details but by upgrading to 0.24 and using a regular dict as GET params that contains e.g. an url like {"q": "http://example.com"} will somehow break the request to, in this case, Twitter's API. If if urllib.parse.quote_plus the "http://example.com", everything works fine (The slash is the culprit).
If I urlencode the whole dict manually before passing it to httpx and call the r.url after the request the / is again not encoded, although it previously was.
https://api.twitter.com/1.1/search/tweets.json?q=url%3Ahttp%3A//google.com

I assume it has to do with

• Query parameter encoding switches from using + for spaces and %2F for forward slash, to instead using %20 for spaces and treating forward slash as a safe, unescaped character. This differs from requests, but is in line with browser behavior in Chrome, Safari, and Firefox. Both options are RFC valid. (Use '%20' for encoding spaces in query parameters. #2543)

[rslinckx]

I posted an issue here: https://github.com/encode/httpx/issues/2670

[lovelydinosaur]

So... yes our treatment of / characters in query parameters has changed, and is now in line with browser behaviour.
For example if you enter the following into a browser URL bar...

https://example.com/1.1/search/tweets.json?q=url%3Ahttp%3A//google.com

You'll see that the // part remains unescaped.

[marban]

Yes but it will, at least, break a call to Twitters API.

Via requests:
https://api.twitter.com/1.1/search/tweets.json?q=url%3Ahttps%3A%2F%2Fedition.cnn.com&lang=en

-> Works

Via httpx:
https://api.twitter.com/1.1/search/tweets.json?q=url%3Ahttps%3A//edition.cnn.com&lang=en
-> {'errors': [{'code': 32, 'message': 'Could not authenticate you.'}]}

[BjoernPetersen]

Just want to chime in here with a comment I made on a related closed issue already:

I have a project in which I want to request a signed URL to an image (from the OpenAI API) using httpx. Some signature query params contain forward slashes, and the server evidently doesn't accept it if those are encoded as %2F.

I want to note that the URL works in Firefox, Chrome, and using requests.get(image_url), but httpx's encoding breaks it.

It might be true that encoding the slashes is technically "more correct", but that doesn't really help if the server just won't accept it.

[marban]

So even if I handle the %2F for calling Twitter myself, it still won't work since now the % obviously gets encoded as well.

[marban]

The simple answer is: Twitter needs the forward slash encoded.

If I run a GET against their API with params={"q":"url=http://example.com"}, they throw an error (It works with requests as mentioned).
A workaround of doing params={"q":"url=http:%2F%2Fexample.com"} won't work either because you're now double-encoding the %.

[lovelydinosaur]

Okay, I think the upshot of this is... twitter's decoding behaviour isn't correct here, but also evidently we should switch our behaviour for / encoding in query parameters here to be more gentle.
If you'd like to escalate this into an issue I can help guide us through to a resolution.

Thanks for your patience with this.

[elupus]

I think this ended up trying to fix the wrong thing. The problem was httpx handling of double encoding.

Secondly, the "working" url above actually has a query of "url:...", not an "url=....", so the params tests here would be wrong, and it would make sense for that to fail.

The twitter docs explicitly state the q parameter should be URL encoded. So it likely should be pre-encoded. That said we should not double up escaping.

I've made a fixup here: #2980 that should work for both the twitter case (if you need to pre-encode the slashes, it allows it) but it does not by default encode all slashes.

[lovelydinosaur]

Resolved in #2990. (Both the double encoding, and the / behaviour)