ENH: Support Sharded Repodata (CEP 16)

[adrinjalali]

In an IRL conversation with @wolfv and others he mentioned the infra for avoiding the download of the massive package metadata when running micromamba commands is there, but I can't find a way to use it. Is there a way already to do that with a micromamba command?

To be specific, this is what I want to avoid:

$ mmamba install jupyter
conda-forge/noarch                                  16.3MB @   7.9MB/s  2.1s
conda-forge/linux-64                                37.6MB @   9.8MB/s  3.9s

...

[wolfv]

Sadly, no. It's only implemented in rattler / pixi for now. There you can point to https://fast.prefix.dev/conda-forge to get a speedy version.

[adrinjalali]

Is this completed? Or it's a "wontfix" kinda thing?

[Hind-M]

So at the moment it's not implemented in mamba/micromamba.
It could be at some point, but not in the short/medium term unfortunately.
We can also mark it as a feature request.

[jjerphan]

For some context, conda/ceps#75 formalizes sharded repodata, an improved indexing solution for conda channels.

It has been accepted, but it needs to be implemented by forges (including conda-forge), and by packages managers (such as conda and mamba/micromamba).

[jjerphan]

conda/conda#14060 is the main requirement which is being implemented by conda/conda-index#161.

[jjerphan]

I would prioritize it, since UX-wise this is what cause latency and fatigue for users.

[baszalmstra]

Good to mention that the conda-forge and bioconda mirrors (but any channel really) at https://prefix.dev/ fully support shared repodata. So I think an implementation is not blocked on server support.

[jjerphan]

Yes, mamba can implement the support for the sharded repodata now without waiting for conda-lock to implement it.

[jjerphan]

I am not sure about libsolv being usable with the sharded repodata since libsolv has to load all the metadata of packages upfront AFAIK.