Concurrent requests take forever to be executed

[novitae]

Hello, I am currently having a big issue with httpx. When sending many requests at the same time, they all literally freeze. I am on Mac m2, latest version, and a friend of mine, on Mac m3, latest version, has no issues. Here is my screenshot:

Here is his:

I am running it in a poetry virtual environment, and uninstalled reinstalled it many times, on the latest version. I don't understand at all where can this come from.

It only happens for concurrent requests, and the more there are, the more the "freezing time" lasts, or even, at a point, timeouts.

Here is my traceback, if it is useful:

---------------------------------------------------------------------------
ConnectTimeout                            Traceback (most recent call last)
File ~/Library/Caches/pypoetry/virtualenvs/project-ImmZ4SqW-py3.11/lib/python3.11/site-packages/httpx/_transports/default.py:69, in map_httpcore_exceptions()
     68 try:
---> 69     yield
     70 except Exception as exc:

File ~/Library/Caches/pypoetry/virtualenvs/project-ImmZ4SqW-py3.11/lib/python3.11/site-packages/httpx/_transports/default.py:373, in AsyncHTTPTransport.handle_async_request(self, request)
    372 with map_httpcore_exceptions():
--> 373     resp = await self._pool.handle_async_request(req)
    375 assert isinstance(resp.stream, typing.AsyncIterable)

File ~/Library/Caches/pypoetry/virtualenvs/project-ImmZ4SqW-py3.11/lib/python3.11/site-packages/httpcore/_async/connection_pool.py:216, in AsyncConnectionPool.handle_async_request(self, request)
    215     await self._close_connections(closing)
--> 216     raise exc from None
    218 # Return the response. Note that in this case we still have to manage
    219 # the point at which the response is closed.

File ~/Library/Caches/pypoetry/virtualenvs/project-ImmZ4SqW-py3.11/lib/python3.11/site-packages/httpcore/_async/connection_pool.py:196, in AsyncConnectionPool.handle_async_request(self, request)
    194 try:
    195     # Send the request on the assigned connection.
--> 196     response = await connection.handle_async_request(
    197         pool_request.request
    198     )
    199 except ConnectionNotAvailable:
    200     # In some cases a connection may initially be available to
    201     # handle a request, but then become unavailable.
    202     #
    203     # In this case we clear the connection and try again.

File ~/Library/Caches/pypoetry/virtualenvs/project-ImmZ4SqW-py3.11/lib/python3.11/site-packages/httpcore/_async/connection.py:99, in AsyncHTTPConnection.handle_async_request(self, request)
     98     self._connect_failed = True
---> 99     raise exc
    101 return await self._connection.handle_async_request(request)

File ~/Library/Caches/pypoetry/virtualenvs/project-ImmZ4SqW-py3.11/lib/python3.11/site-packages/httpcore/_async/connection.py:76, in AsyncHTTPConnection.handle_async_request(self, request)
     75 if self._connection is None:
---> 76     stream = await self._connect(request)
     78     ssl_object = stream.get_extra_info("ssl_object")

File ~/Library/Caches/pypoetry/virtualenvs/project-ImmZ4SqW-py3.11/lib/python3.11/site-packages/httpcore/_async/connection.py:122, in AsyncHTTPConnection._connect(self, request)
    121 async with Trace("connect_tcp", logger, request, kwargs) as trace:
--> 122     stream = await self._network_backend.connect_tcp(**kwargs)
    123     trace.return_value = stream

File ~/Library/Caches/pypoetry/virtualenvs/project-ImmZ4SqW-py3.11/lib/python3.11/site-packages/httpcore/_backends/auto.py:30, in AutoBackend.connect_tcp(self, host, port, timeout, local_address, socket_options)
     29 await self._init_backend()
---> 30 return await self._backend.connect_tcp(
     31     host,
     32     port,
     33     timeout=timeout,
     34     local_address=local_address,
     35     socket_options=socket_options,
     36 )

File ~/Library/Caches/pypoetry/virtualenvs/project-ImmZ4SqW-py3.11/lib/python3.11/site-packages/httpcore/_backends/anyio.py:112, in AnyIOBackend.connect_tcp(self, host, port, timeout, local_address, socket_options)
    107 exc_map = {
    108     TimeoutError: ConnectTimeout,
    109     OSError: ConnectError,
    110     anyio.BrokenResourceError: ConnectError,
    111 }
--> 112 with map_exceptions(exc_map):
    113     with anyio.fail_after(timeout):

File /opt/homebrew/Cellar/python@3.11/3.11.7_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py:158, in _GeneratorContextManager.__exit__(self, typ, value, traceback)
    157 try:
--> 158     self.gen.throw(typ, value, traceback)
    159 except StopIteration as exc:
    160     # Suppress StopIteration *unless* it's the same exception that
    161     # was passed to throw().  This prevents a StopIteration
    162     # raised inside the "with" statement from being suppressed.

File ~/Library/Caches/pypoetry/virtualenvs/project-ImmZ4SqW-py3.11/lib/python3.11/site-packages/httpcore/_exceptions.py:14, in map_exceptions(map)
     13     if isinstance(exc, from_exc):
---> 14         raise to_exc(exc) from exc
     15 raise

ConnectTimeout: 

The above exception was the direct cause of the following exception:

ConnectTimeout                            Traceback (most recent call last)
Cell In[1], line 11
      8 import time
     10 t1 = time.time()
---> 11 await asyncio.gather(*[test() for _ in range(100)])
     12 print('time to launch 100 httpx clients with a request', time.time() - t1)

Cell In[1], line 6
      4 async def test():
      5     async with httpx.AsyncClient() as client:
----> 6         await client.get('http://ident.me')

File ~/Library/Caches/pypoetry/virtualenvs/project-ImmZ4SqW-py3.11/lib/python3.11/site-packages/httpx/_client.py:1801, in AsyncClient.get(self, url, params, headers, cookies, auth, follow_redirects, timeout, extensions)
   1784 async def get(
   1785     self,
   1786     url: URLTypes,
   (...)
   1794     extensions: RequestExtensions | None = None,
   1795 ) -> Response:
   1796     """
   1797     Send a `GET` request.
   1798 
   1799     **Parameters**: See `httpx.request`.
   1800     """
-> 1801     return await self.request(
   1802         "GET",
   1803         url,
   1804         params=params,
   1805         headers=headers,
   1806         cookies=cookies,
   1807         auth=auth,
   1808         follow_redirects=follow_redirects,
   1809         timeout=timeout,
   1810         extensions=extensions,
   1811     )

File ~/Library/Caches/pypoetry/virtualenvs/project-ImmZ4SqW-py3.11/lib/python3.11/site-packages/httpx/_client.py:1574, in AsyncClient.request(self, method, url, content, data, files, json, params, headers, cookies, auth, follow_redirects, timeout, extensions)
   1559     warnings.warn(message, DeprecationWarning)
   1561 request = self.build_request(
   1562     method=method,
   1563     url=url,
   (...)
   1572     extensions=extensions,
   1573 )
-> 1574 return await self.send(request, auth=auth, follow_redirects=follow_redirects)

File ~/Library/Caches/pypoetry/virtualenvs/project-ImmZ4SqW-py3.11/lib/python3.11/site-packages/httpx/_client.py:1661, in AsyncClient.send(self, request, stream, auth, follow_redirects)
   1653 follow_redirects = (
   1654     self.follow_redirects
   1655     if isinstance(follow_redirects, UseClientDefault)
   1656     else follow_redirects
   1657 )
   1659 auth = self._build_request_auth(request, auth)
-> 1661 response = await self._send_handling_auth(
   1662     request,
   1663     auth=auth,
   1664     follow_redirects=follow_redirects,
   1665     history=[],
   1666 )
   1667 try:
   1668     if not stream:

File ~/Library/Caches/pypoetry/virtualenvs/project-ImmZ4SqW-py3.11/lib/python3.11/site-packages/httpx/_client.py:1689, in AsyncClient._send_handling_auth(self, request, auth, follow_redirects, history)
   1686 request = await auth_flow.__anext__()
   1688 while True:
-> 1689     response = await self._send_handling_redirects(
   1690         request,
   1691         follow_redirects=follow_redirects,
   1692         history=history,
   1693     )
   1694     try:
   1695         try:

File ~/Library/Caches/pypoetry/virtualenvs/project-ImmZ4SqW-py3.11/lib/python3.11/site-packages/httpx/_client.py:1726, in AsyncClient._send_handling_redirects(self, request, follow_redirects, history)
   1723 for hook in self._event_hooks["request"]:
   1724     await hook(request)
-> 1726 response = await self._send_single_request(request)
   1727 try:
   1728     for hook in self._event_hooks["response"]:

File ~/Library/Caches/pypoetry/virtualenvs/project-ImmZ4SqW-py3.11/lib/python3.11/site-packages/httpx/_client.py:1763, in AsyncClient._send_single_request(self, request)
   1758     raise RuntimeError(
   1759         "Attempted to send an sync request with an AsyncClient instance."
   1760     )
   1762 with request_context(request=request):
-> 1763     response = await transport.handle_async_request(request)
   1765 assert isinstance(response.stream, AsyncByteStream)
   1766 response.request = request

File ~/Library/Caches/pypoetry/virtualenvs/project-ImmZ4SqW-py3.11/lib/python3.11/site-packages/httpx/_transports/default.py:372, in AsyncHTTPTransport.handle_async_request(self, request)
    358 assert isinstance(request.stream, AsyncByteStream)
    360 req = httpcore.Request(
    361     method=request.method,
    362     url=httpcore.URL(
   (...)
    370     extensions=request.extensions,
    371 )
--> 372 with map_httpcore_exceptions():
    373     resp = await self._pool.handle_async_request(req)
    375 assert isinstance(resp.stream, typing.AsyncIterable)

File /opt/homebrew/Cellar/python@3.11/3.11.7_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py:158, in _GeneratorContextManager.__exit__(self, typ, value, traceback)
    156     value = typ()
    157 try:
--> 158     self.gen.throw(typ, value, traceback)
    159 except StopIteration as exc:
    160     # Suppress StopIteration *unless* it's the same exception that
    161     # was passed to throw().  This prevents a StopIteration
    162     # raised inside the "with" statement from being suppressed.
    163     return exc is not value

File ~/Library/Caches/pypoetry/virtualenvs/project-ImmZ4SqW-py3.11/lib/python3.11/site-packages/httpx/_transports/default.py:86, in map_httpcore_exceptions()
     83     raise
     85 message = str(exc)
---> 86 raise mapped_exc(message) from exc

ConnectTimeout: 

[zanieb]

Why are you creating 100 clients and sending a single request with each? Instead you should use a single client to send hundreds of requests.

It sounds like you're running into a simple network saturation, since you're creating each client separately each request is sent on a new connection instead of sharing an existing connection. As each client creates a new connection, you eventually run into a timeout error since establishing new connections takes time and usually there's an upper bound on the number of concurrent connections you can have.

[novitae]

I see ... But why does it work for my friend ? And why does it work with aiohttp or curlcffi ?

[zanieb]

Perhaps they have a different network or connection limits 🤷‍♀️ I'm not sure about those either, maybe they release connections more aggressively or bound concurrency another way?

My primary point is that you're severely misusing the library so we're unlikely to spend much time investigating the root cause.

[novitae]

Unfortunately the way you told me didn't change anything for httpx ...

[zanieb]

I'm pretty sure that does not automatically create a session for you in httpx, I think you want something like this:

async with httpx.AsyncClient() as client:
    await asyncio.gather(*[client.get...])

[novitae]

Damn, it was that, thanks a lot

[zanieb]

No problem! Sorry about the confusion.