conda-libmamba-solver should be able to use http/2 to fetch shards.
The most established http/2 library for Python, httpx, was chosen for a prototype. The prototype was used to measure the difference between http/2 (httpx) and http 1.1 (requests). The "pipelined" shard traversal algorithm in conda-libmamba-solver 25.11.0 decouples shard traversal from the concurrency strategy (asyncio or threads) used to fetch shards over the network, but the prototype can't use authenticated repositories or other features of conda's requests-based network stack like support for telemetry, file:// and s3:// URL schemes, etc.
Sharded repodata saves time
On a 10Mbps connection (e.g. hotel wifi), a conda-forge user spends about 58s to transfer 69.05MiB of noarch+linux-64 repodata.json.zst, which has to be downloaded again when any part of repodata is updated. With sharded repodata the same “hotel wifi” user transfers a 1.34MiB shards index plus 0.86MiB for 154 shards (install django, celery) taking less than 10s; individual shards remain valid in cache until that particular package is changed (i.e. receives a new release).
Sharded repodata benefits greatly from http/2
When there are many shards, the latency to retrieve each shard can become as important or more important than the bandwidth. With http/1 and requests, we open 10 parallel connections that we try to re-use to fetch additional shards, but with http/2 we can send a much larger number of requests over a single connection. This allows us to begin waiting for individual shards without waiting for other requests to finish, reducing the total time spent.
------------------------------------------------------------ benchmark: 24 tests ------------------------------------------------------------
Name (time in ms) Mean Min Max
---------------------------------------------------------------------------------------------------------------------------------------------
test_traversal_algorithm_benchmarks[python-httpx-cold] 449.6007 (63.30) 449.6007 (63.30) 449.6007 (63.30)
test_traversal_algorithm_benchmarks[python-httpx-warm] 7.1022 (1.0) 7.1022 (1.0) 7.1022 (1.0)
test_traversal_algorithm_benchmarks[python-pipelined-cold] 729.4001 (102.70) 729.4001 (102.70) 729.4001 (102.70)
test_traversal_algorithm_benchmarks[python-pipelined-warm] 7.1593 (1.01) 7.1593 (1.01) 7.1593 (1.01)
test_traversal_algorithm_benchmarks[scientific_computing-httpx-cold] 1,164.8993 (164.02) 1,164.8993 (164.02) 1,164.8993 (164.02)
test_traversal_algorithm_benchmarks[scientific_computing-pipelined-cold] 1,766.5735 (248.74) 1,766.5735 (248.74) 1,766.5735 (248.74)
test_traversal_algorithm_benchmarks[vaex-httpx-cold] 2,413.0382 (339.76) 2,413.0382 (339.76) 2,413.0382 (339.76)
test_traversal_algorithm_benchmarks[vaex-pipelined-cold] 6,573.8091 (925.60) 6,573.8091 (925.60) 6,573.8091 (925.60)
---------------------------------------------------------------------------------------------------------------------------------------------
http/2 saves 4.16 seconds over http/1.1 for a complicated solve
Shard traversal can be more constrained by latency than bandwidth. This benchmark, run on a machine with ~380Mbps download speed, shows that http/2 is 1.6 times faster in the “python” scenario and 2.75 times faster, or 4.16 seconds faster in the “vaex” scenario which requires many shards.
For the "warm" case the "pipelined" and "httpx" strategies are the same, and the network portion does not do any work. Only some "warm" results are included.
Proposed implementation
The conda-httpx project is meant to expose conda's networking features to httpx by calling conda's existing requests-based authentication handlers with adapter classes providing requests APIs to authentication handlers, copying any headers set by the authenticator into the httpx request. conda code would opt-in to using httpx.Client or httpx.AsyncClient by calling get_httpx_client() or get_httpx_async_client() functions without a url argument. Unlike conda get_session(url), these clients would transparently authenticate with any URL known to conda. A gradual migration to httpx based clients would be possible, instead of immediately replacing all requests code with a hopefully-compatible alternative.
As in the prototype, the networking thread can run an asyncio loop for httpx.AsyncClient and does not affect the rest of the shard traversal algorithm. Although httpx supports "sync" and "async" clients, asyncio is needed to launch many requests over the same http/2 TCP connection for shards.
conda-libmamba-solver should be able to use http/2 to fetch shards.
The most established http/2 library for Python, httpx, was chosen for a prototype. The prototype was used to measure the difference between http/2 (httpx) and http 1.1 (requests). The "pipelined" shard traversal algorithm in
conda-libmamba-solver 25.11.0decouples shard traversal from the concurrency strategy (asyncio or threads) used to fetch shards over the network, but the prototype can't use authenticated repositories or other features of conda's requests-based network stack like support for telemetry, file:// and s3:// URL schemes, etc.Sharded repodata saves time
On a 10Mbps connection (e.g. hotel wifi), a conda-forge user spends about 58s to transfer 69.05MiB of noarch+linux-64
repodata.json.zst, which has to be downloaded again when any part of repodata is updated. With sharded repodata the same “hotel wifi” user transfers a 1.34MiB shards index plus 0.86MiB for 154 shards (install django, celery) taking less than 10s; individual shards remain valid in cache until that particular package is changed (i.e. receives a new release).Sharded repodata benefits greatly from http/2
When there are many shards, the latency to retrieve each shard can become as important or more important than the bandwidth. With http/1 and
requests, we open 10 parallel connections that we try to re-use to fetch additional shards, but with http/2 we can send a much larger number of requests over a single connection. This allows us to begin waiting for individual shards without waiting for other requests to finish, reducing the total time spent.http/2 saves 4.16 seconds over http/1.1 for a complicated solve
Shard traversal can be more constrained by latency than bandwidth. This benchmark, run on a machine with ~380Mbps download speed, shows that http/2 is 1.6 times faster in the “python” scenario and 2.75 times faster, or 4.16 seconds faster in the “vaex” scenario which requires many shards.
For the "warm" case the "pipelined" and "httpx" strategies are the same, and the network portion does not do any work. Only some "warm" results are included.
Proposed implementation
The conda-httpx project is meant to expose conda's networking features to httpx by calling conda's existing requests-based authentication handlers with adapter classes providing
requestsAPIs to authentication handlers, copying any headers set by the authenticator into the httpx request. conda code would opt-in to usinghttpx.Clientorhttpx.AsyncClientby callingget_httpx_client()orget_httpx_async_client()functions without aurlargument. Unlike condaget_session(url), these clients would transparently authenticate with any URL known to conda. A gradual migration tohttpxbased clients would be possible, instead of immediately replacing allrequestscode with a hopefully-compatible alternative.As in the prototype, the networking thread can run an asyncio loop for
httpx.AsyncClientand does not affect the rest of the shard traversal algorithm. Although httpx supports "sync" and "async" clients, asyncio is needed to launch many requests over the same http/2 TCP connection for shards.