AssertionError

[gyula-lakatos]

Hi everyone,

I got an assertion error out of nowhere that killed our crawler. Does anyone know what could be the problem?

Traceback (most recent call last):
  File "/...py", line 81, in <module>
    trio_asyncio.run(main)
  File "/root/.local/share/virtualenvs/crawler-Rh48lQiD/lib/python3.11/site-packages/trio_asyncio/_loop.py", line 446, in run
    return trio.run(_run_task, proc, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/virtualenvs/crawler-Rh48lQiD/lib/python3.11/site-packages/trio/_core/_run.py", line 2093, in run
    raise runner.main_task_outcome.error
  File "/root/.local/share/virtualenvs/crawler-Rh48lQiD/lib/python3.11/site-packages/trio_asyncio/_loop.py", line 444, in _run_task
    return await proc(*args)
           ^^^^^^^^^^^^^^^^^
  File "/crawler/....py", line 61, in main
    async with trio.open_nursery() as nursery:
  File "/root/.local/share/virtualenvs/crawler-Rh48lQiD/lib/python3.11/site-packages/trio/_core/_run.py", line 881, in __aexit__
    raise combined_error_from_nursery
  File "/crawler/web/url_visitor.py", line 42, in visit_url
    async for text in response.aiter_text(chunk_size=8192):
  File "/root/.local/share/virtualenvs/crawler-Rh48lQiD/lib/python3.11/site-packages/httpx/_models.py", line 949, in aiter_text
    text_content = decoder.decode(byte_content)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/virtualenvs/crawler-Rh48lQiD/lib/python3.11/site-packages/httpx/_decoders.py", line 252, in decode
    return self.decoder.decode(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/encodings/base64_codec.py", line 34, in decode
    assert self.errors == 'strict'
           ^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

I'm not at all sure why those assertions are there in the code of cpython and looking it up on GitHub did not help at all.

Did the website really sent back base64 as an encoding? O.o

Unfortunately, I was unable to catch the website that caused the error.

[T-256]

Did the website really sent back base64 as an encoding? O.o

It seems yes, and TextDecoder is set to "replace" to all encodings:

httpx/httpx/_decoders.py

Line 249 in 3ba5fe0

self.decoder = codecs.getincrementaldecoder(encoding)(errors="replace")

[T-256]

I also think there should be errors="strict" for encodings like base64 (are there any need it too?)

For temporary solution I suggest use aiter_bytes instead of aiter_text, then manually decode content.

[lovelydinosaur]

I mean is it logical to support every type of encoding that a webpage can return? Quite strange and risky decision in my book, but I could live with it.

@gyula-lakatos Interesting. What set of encodings would you expect to be supported?

Currently we're just doing an open lookup from Python's codec registry. The issue here is being caused because "base64" isn't a text encoding. The codec API doesn't appear to have a good way to restrict lookups to text codecs only.

We could explicitly restrict our set of supported encodings to those listed in Python docs.

Or... something else?

---

Edit

• I assume? these are the set of charset encodings that the Chromium browser supports.
• Also useful background... https://encoding.spec.whatwg.org/

[gyula-lakatos]

@tomchristie Restricting the supported encodings to what Python supports is a great idea. I'm pretty sure that there are obscure usecases where strange encodings might be used, but by default accepting everything is not restrictive enough.

I'm mostly asking this for security reasons. If one of the encoders has any security problems, then a malicious actor can craft a website that's once hit by httpx can cause some kind of havoc. I know the attack vector is small, but it can be a problem in certain contexts (ie. web crawling). There are quite a lot of codecs in Python that are strange and has nothing to do with text encoding.

[lovelydinosaur]

I've escalated this to https://github.com/encode/httpx/issues/2892 - would you like to issue a pull request resolving it?

[gyula-lakatos]

Thank you! We are very busy with hitting an end of the year release date, so at the moment I can't create a PR for this, but early next year should be fine. It's quite far away, but unfortunately this is the best I can do at the moment.

[lovelydinosaur]

PR... #2896