Support repodata v3 draft for monolithic, sharded repodata including .whl by dholth · Pull Request #265 · conda/conda-index

dholth · 2026-03-11T22:12:00Z

Description

Ability to output wheel metadata to monolithic and sharded repodata.

Does not create the wheel metadata; a different program must have already inserted it into our database before repodata generation runs.

Per the pattern in https://github.com/conda-incubator/conda-pypi-test or the draft repodata v3 at conda/ceps#146

Checklist - did you ...

Add a file to the news directory (using the template) for the next release's release notes?
Add / update necessary tests?
Add / update outdated documentation?

dholth · 2026-03-12T19:00:08Z

Will update to jump to the v3 key containing per-format subsections.

ForgottenProgramme · 2026-03-17T12:36:28Z

+                continue

-        return new_repodata_packages, new_repodata_conda_packages
+            if v3:


this means if v3 is true, older sections are ignored, and new_packages will be empty:

new_packages = { "packages": {}, "packages.conda": {}, "packages.whl": {}, }

Is that intended?

If v3 is True, then we put all packages into the new_v3_packages dict and "packages", "packages.conda" will be empty. This is simpler than the "migrated-since" in the draft CEP.
I think the "new_" prefix is confusing here since it's just "all of them"; this name might be left over from old-conda-index which didn't pull all packages from cache every time.

That doesn't follow the spirit of the CEP, though. The point is to be backwards compatible, so this will effective leave incompatible clients behind.

This was written to follow experimental wheel work and to defer the spirit of the CEP.

ForgottenProgramme · 2026-03-17T15:17:49Z

+                    if section_and_key is None:
+                        log.warning("%s has unsupported package extension", path)
+                        continue
+                    key, v3_path = section_and_key


the name key is used to refer to the second element of the returned tuple here. It's a bit confusing.
We should change it to: section, key = section_and_key.

ForgottenProgramme · 2026-03-17T15:22:07Z

+                        if section_and_key is None:
+                            log.warning("%s has unsupported package extension", path)
+                            continue
+                        key, v3_path = section_and_key


Created #272 to resolve these

ForgottenProgramme · 2026-03-17T15:53:58Z

The main objective of this PR is to add a top level key called "v3" to repodata.json.
Under this key, there could be three sections (named after three supported package extensions).
Under each of these package extensions names are inturn package names.
Under each package name will be package metadata
So we get something like:

"v3": {
    "tar.bz2": {
      "package_a": { "version": "1.2.3", ... }
    },
    "conda": {
      "package_b": { "version": "2.3.4", ... }
    },
    "whl": {
      "package_c": { "version": "3.4.5", ... }
    }

I have added a few comments but the PR looks good to me.

dholth · 2026-03-24T14:52:27Z

+                    }
+                }
+                if v3
+                else {"packages": {}, "packages.conda": {}}


shard should always include the packages, packages.conda keys

dholth · 2026-03-24T14:53:18Z

+                        }
+                    }
+                    if v3
+                    else {"packages": {}, "packages.conda": {}}


It would be nice if the boring "constructing the big dict" could be moved to the abstract base class instead of duplicated per database.

yes, agreed. There is a lot of duplicate code. We can either write a function to create these dictionaries in this PR or do that in a subsequent PR.
I am okay with addressing this in the next PR.

jaimergp · 2026-03-28T20:32:57Z

    show_default=True,
 )
+@click.option(
+    "--repodata-v3/--no-repodata-v3",


Eventually, there'll be a v4, v5... should we instead have a non-boolean flag like --max-repodata-version or something?

Are repodata versions something that are sequential, or are they more like CPU flags where we may use any set of them?

Hey @dholth, they are sequential and we would probably keep all previous versions as well for backwards compatibility (although maybe there is a deprecation time eventually?)

jaimergp · 2026-03-28T20:37:56Z

There's no repodata v3 yet, so I'm not sure what this is trying to do (I guess adding support for the wheels preview). In that case, maybe it should leave conda packages alone for now and simply populate whl. Also there are a lot of type hint changes here that are unrelated and pollute the PR, so ideally those are proposed separately.

Co-authored-by: jaimergp <jaimergp@users.noreply.github.com>

dholth · 2026-03-30T16:45:39Z

There's no repodata v3 yet, so I'm not sure what this is trying to do (I guess adding support for the wheels preview). In that case, maybe it should leave conda packages alone for now and simply populate whl. Also there are a lot of type hint changes here that are unrelated and pollute the PR, so ideally those are proposed separately.

Yes this is only intended to support the wheels preview.

ryanskeith

Overall, I am not seeing any huge red flags about going forward this way. I don't think anyone is a fan of having version 2's of code but it is a necessary evil at times to get things working well.

ryanskeith · 2026-04-03T16:31:45Z

-        for name, shard in cache.indexed_shards():
-            shard_data = compressor.compress(sqlitecache.packb_typed(shard))
-            shard_hash = hashlib.sha256(shard_data).digest()
+        v3_data = {


Is the concern processing time? If so, could this be an elective post processing function?

ryanskeith · 2026-04-03T18:14:22Z

-        for name, shard in cache.indexed_shards():
-            shard_data = compressor.compress(sqlitecache.packb_typed(shard))
-            shard_hash = hashlib.sha256(shard_data).digest()
+        v3_data = {


Ah, I see. It is part of the spec. I am not sure what the intent of this spec provides.

Mention conda-pypi

dholth · 2026-04-07T13:38:22Z

-            if fn in repodata[key]:
-                repodata[key][fn]["revoked"] = True
-                repodata[key][fn]["depends"].append("package_has_been_revoked")
+            records = repodata.get(key, {})


repodata patch doesn't work with wheel or v3

Co-authored-by: Dan Yeaw <dan@yeaw.me>

dholth · 2026-04-07T19:04:04Z

+        package_sections = {
+            ".tar.bz2": "packages",
+            ".conda": "packages.conda",
+            ".whl": "packages.whl",


A little obsolete since there isn't a "packages.whl" in current specs but we also use it to populate IndexedPackages / IndexedShard dataclasses.

danyeaw

Hey @dholth, this looks great - thanks for this! I made a couple of additional suggestions.

Co-authored-by: Dan Yeaw <dan@yeaw.me>

* cep-21 run exports in shards * update typing

dholth added 2 commits March 11, 2026 17:38

update typing

af7baaa

ability to generate "packages.whl" section

dce47ed

dholth requested a review from danyeaw March 11, 2026 22:12

dholth self-assigned this Mar 11, 2026

conda-bot added this to 🔎 Review Mar 11, 2026

github-project-automation Bot moved this to 🆕 New in 🔎 Review Mar 11, 2026

conda-bot added the cla-signed [bot] added once the contributor has signed the CLA label Mar 11, 2026

dholth changed the title ~~Support "packages.whl" section in API~~ Support outputing .whl metadata for monolithic and sharded repodata Mar 12, 2026

dholth added 2 commits March 12, 2026 16:42

ignore some files for ruff; remove bare except: in conftest

4f19f29

draft repodata v3 implementation

d0ae208

dholth changed the title ~~Support outputing .whl metadata for monolithic and sharded repodata~~ Support repodata v3 draft for monolithic, sharded repodata including .whl Mar 12, 2026

dholth moved this from 🆕 New to 🏗️ In Progress in 🔎 Review Mar 12, 2026

ForgottenProgramme reviewed Mar 17, 2026

View reviewed changes

Comment thread conda_index/index/__init__.py

ForgottenProgramme reviewed Mar 17, 2026

View reviewed changes

ForgottenProgramme mentioned this pull request Mar 17, 2026

Implement WheelIndex for pure Python wheel indexing #221

Closed

2 tasks

danyeaw mentioned this pull request Mar 17, 2026

Extract and convert wheel metadata from local .whl files during indexing #270

Closed

dholth commented Mar 24, 2026

View reviewed changes

Comment thread conda_index/index/sqlitecache.py

dholth commented Mar 24, 2026

View reviewed changes

fix naming issue (#272)

5a12faf

jaimergp reviewed Mar 28, 2026

View reviewed changes

Comment thread conda_index/cli/__init__.py

jaimergp reviewed Mar 28, 2026

View reviewed changes

Comment thread conda_index/index/cache.py Outdated

Update conda_index/cli/__init__.py

a083728

Co-authored-by: jaimergp <jaimergp@users.noreply.github.com>

dholth added 5 commits April 2, 2026 14:11

remove unused import

8f37fcd

tidy v3 logic and add typing

5cfe9d7

remove unused import

58bd3f2

no .whl for indexed_shards()

d148cb4

fix test

160196e

ryanskeith reviewed Apr 3, 2026

View reviewed changes

dholth added 2 commits April 6, 2026 16:21

Update news

1f0b44d

Mention conda-pypi

show repodata-with-wheels API

136b07c

dholth commented Apr 7, 2026

View reviewed changes

Fix key format for wheel records

9c98ee6

danyeaw reviewed Apr 7, 2026

View reviewed changes

Comment thread conda_index/index/cache.py Outdated

dholth commented Apr 7, 2026

View reviewed changes

Comment thread conda_index/index/__init__.py

dholth and others added 3 commits April 7, 2026 14:50

Update conda_index/index/__init__.py with comment

e0d9c18

wrap lines

ea7a884

Update conda_index/index/cache.py

e2ca1e8

Co-authored-by: Dan Yeaw <dan@yeaw.me>

dholth commented Apr 7, 2026

View reviewed changes

danyeaw reviewed Apr 7, 2026

View reviewed changes

Comment thread conda_index/postgres/cache.py Outdated

danyeaw reviewed Apr 7, 2026

View reviewed changes

Comment thread conda_index/postgres/cache.py Outdated

danyeaw approved these changes Apr 7, 2026

View reviewed changes

github-project-automation Bot moved this from 🏗️ In Progress to ✅ Approved in 🔎 Review Apr 7, 2026

dholth and others added 5 commits April 7, 2026 15:18

Apply suggestion from @danyeaw

cc28205

Co-authored-by: Dan Yeaw <dan@yeaw.me>

Apply suggestion from @danyeaw

40d0054

Co-authored-by: Dan Yeaw <dan@yeaw.me>

package_section_for_path() replaces endswith()

722d6c5

compare whl section from output

1db53cd

Support CEP 21 run exports in shards (#275)

f51a29f

* cep-21 run exports in shards * update typing

dholth merged commit 228fd6a into main Apr 8, 2026
23 checks passed

dholth deleted the 262-wheel branch April 8, 2026 15:47

github-project-automation Bot moved this from ✅ Approved to 🏁 Done in 🔎 Review Apr 8, 2026

soapy1 mentioned this pull request Apr 14, 2026

Cache pypi metadata to conda index cache conda/conda-pypi#306

Merged

3 tasks

soapy1 mentioned this pull request Apr 22, 2026

Use conda-pypi to generate repodata with conda-index conda-incubator/conda-pypi-test#25

Merged

Uh oh!

Conversation

dholth commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist - did you ...

Uh oh!

dholth commented Mar 12, 2026

Uh oh!

Uh oh!

ForgottenProgramme Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ForgottenProgramme commented Mar 17, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dholth Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jaimergp commented Mar 28, 2026

Uh oh!

dholth commented Mar 30, 2026

Uh oh!

ryanskeith left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

danyeaw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

dholth commented Mar 11, 2026 •

edited

Loading

ForgottenProgramme Mar 17, 2026 •

edited

Loading

dholth Mar 24, 2026 •

edited

Loading