CEP XXXX: A backwards-compatible repodata update strategy for repodata `v3` by jaimergp · Pull Request #146 · conda/ceps

jaimergp · 2026-01-12T21:19:08Z

Checklist for submitter

I am submitting a new CEP: A backwards-compatible repodata update strategy for repodata v3.
- I am using the CEP template by creating a copy cep-0000.md named cep-XXXX.md in the root level.
I am submitting modifications to CEP XX.
Something else: (add your description here).

h-vetinari

I'm not sure if "migrated" / migrated_at is the best naming. Aside from a question of what purpose the timestamp serves (see below), I think we should probably rename this to something simpler like since:. If we have to choose a verb, I'd rather we take something like "provide" rather than "migrate"; the latter is IMO not a good description for providing multiple repodata revisions simultaneously for an extended period of time.

For argument's sake, I think the below would be pretty self-explanatory

    "repodata_revisions": [
      {
        "revision": 3,
        "since": 1768249989751,     // 2026-01-12 20:33 UTC
        "dropped": 2234563200000,   // 2040-10-23 00:00 UTC
      },
      {
        "revision": 4,
        "since": 1842998400000,     // 2028-05-27 00:00 UTC
      },
      {
        "revision": 5,
        "since": 2053641600000,     // 2035-01-29 00:00 UTC
      }
    ]

Co-authored-by: H. Vetinari <h-vetinari@users.noreply.github.com>

…elds

jaimergp · 2026-03-18T16:39:00Z

@conda/steering-council, I'd like to start the RFC period for this CEP. It will be open until March 30th, EOD, AoE. Thank you!

jaimergp · 2026-03-28T18:59:53Z

Looks like we have no further comments, nice 😎

chenghlee · 2026-03-30T14:49:10Z

+* `oldest: int | None`: Optional. The timestamp (in milliseconds) of the oldest record published in this revision. If set to `None` or missing, the timestamp information is not available.
+* `newest: int | None`: Optional. The timestamp (in milliseconds) of the newest record published in this revision. If set to `None` or missing, the timestamp information is not available.


We could be a little clearer and say: "The Unix time in milliseconds" (a la CEP 34).

I'm unsure about tracking these numbers and the counts and propagating them back from individual shards generation to the shards index which has the info section. If we imagine a shards-only repodata, then the client would have to download all shards to double check package counts and timestamps.
Allowing conda-index to say "just pull all newer timestamps than X to the v3 section" would be the sort of implementation that I would be interested in, where conda-index would stay simple.
As a client, a boolean "client should be upgraded because there's repodata we don't understand?" is the most interesting information.

I agree with @dholth here.

I made these values local to the file only. See this comment too.

chenghlee · 2026-03-30T15:07:55Z

+
+A new top-level field identified by the syntax `v{revision}` (where `revision` comes from `info.repodata_revisions[*].revision`) MUST map to a dictionary whose schema is presented in the relevant CEP.
+
+The CEP MUST also specify how to identify whether a given record belongs in the newer version, or can be added to the previous ones (e.g. a new field extending CEP 20's `info/index.json`)


Does "The CEP" here mean this CEP, or later "implementation" CEPs making repodata revisions under this CEP?

This is hopefully clearer now, PTAL 🙏

chenghlee · 2026-03-30T15:51:11Z

+
+A repodata revision introduces backwards incompatible features in a way that does not disrupt existing metadata. To do so, a new CEP MUST be proposed, following these guidelines below.
+
+A new item MUST be added to the `info.repodata_revisions` array, that MUST list the revisions found in the repodata file as dictionaries with the following schema:


Is there a reason to make this an array rather than a dict-of-dicts, with vN as the key and the other fields specified below as sub-dicts; e.g.,

"info": { ... "repodata_revisions": { "v3": { "n_packages": 42, "oldest": ...., "newest": .... }, "v4": { "n_packages": 42, "oldest": ...., "newest": .... }, } ... }

A big issue I can see with repodata_revisions an array is that it (by itself) imposes no uniqueness constraint on revision, which leads us to have to answer, "what should tools do if they see multiple {"revision": 3, ...} in the array?"

See e65a5c5

chenghlee · 2026-03-30T15:55:01Z

+A new item MUST be added to the `info.repodata_revisions` array, that MUST list the revisions found in the repodata file as dictionaries with the following schema:
+
+* `revision: int`: Required. The integer identifying the revision.
+* `n_packages: int`: Required. The number of packages available in this revision.


Since this is an required element, we should specify if tools MUST do anything particular with this data (e.g., verify the number of packages in the vN field), or is this field only exists for tools to use as they see fit (e.g., making package count verification optional).

maybe we should make it optional? I think it's purely informational

See e65a5c5

chenghlee · 2026-03-30T15:58:17Z

+* `oldest: int | None`: Optional. The timestamp (in milliseconds) of the oldest record published in this revision. If set to `None` or missing, the timestamp information is not available.
+* `newest: int | None`: Optional. The timestamp (in milliseconds) of the newest record published in this revision. If set to `None` or missing, the timestamp information is not available.


Also, not sure if we need to explicitly say these here, but:

I assume timestamp ranges for various repodata revisions are allowed to overlap? E.g., the existing (v1/v2) packages.conda section can continue to publish packages, independent of any v3 date range.

If newest and oldest are both present and not None, newest must be greater than or equal to oldest.

makes sense

I think that's implied by the commonly understood meaning of "oldest" and "newest". Still, I reworded this part to ensure that it matches actual metadata in the file.

Follow the CEP draft (conda/ceps#146) into the in-memory representation: the top-level info.repodata_revisions (and the sharded index equivalent) is now an IndexMap keyed by RepodataRevision, mirroring the dictionary structure that encodes revision uniqueness instead of a Vec that allowed duplicates and redundantly repeated the revision in each element. - Add the RepodataRevisions map alias and a RepodataRevisionMetadata value struct; RepodataRevisionInfo is kept as the flattened (revision + metadata) unit used for indexer input and reporter messages, with from_metadata/ metadata helpers to convert between the two. - Update the gateway accessors (returning &RepodataRevisions, with a shared empty-map helper) and the indexer to produce the map. The wire format is unchanged from the previous commit (a vN-keyed dict).

Update the top-level info.repodata_revisions (and the sharded index equivalent) to match the latest CEP draft (conda/ceps#146), which describes it as a dictionary keyed by revision identifier (e.g. "v3") instead of a list. The in-memory representation is a revision-keyed map so a revision cannot appear twice. The Python indexing bindings accept the same dictionary shape for repodata_revisions, with timestamps given as datetime values for consistency with the rest of the API. Includes a regression test that reads the dictionary form from a repodata file in the test data and asserts it parses correctly. https://claude.ai/code/session_01ModY1fX2YqL6ULMbJtfqWa

…e` field, rework specification schema explainer, restrict which type of MatchSpec is allowed

jaimergp · 2026-06-23T17:01:55Z

+    - The MatchSpec strings mentioned in the fields `depends`, `constrains` and the lists of strings within `extra_depends` groups:
+      - MUST set the `name` field to an exact string (no globbing allowed).
+      - MAY set the fields: `version`, `build`, `build_number`, CEP 43's `when`, CEP 44 `extra`, CEP 45 `flags`.
+      - MUST NOT set any other fields.
+      - MUST be represented with the `name` + square-brackets form (e.g. `name[version="1.2.*",build_number=0]`)


Reviewers, please pay attention to this part, since it restricts which MatchSpec strings can be used.

Right now with v3 repodata we are generating with conda-index, version doesn't appear in square brackets. I understand that this CEP as worded would require pytest[version=">=2.7.2"] instead of pytest>=2.7.2. Is that a correct understanding and can you help me understand the motivation to make the syntax more restrictive? Thanks!

Also curious about this!

Non-square brackets syntax can be very ambiguous (especially when build strings are also added). I want to move away from that syntax in stored repodata (and leave it as syntactic sugar for user input only). Users can still add whatever in the recipes (that bit is not specified here), but repodata creation tools (indexing) must canonicalize the representation in the repodata for clarity.

Is that a good enough reason?

cc @baszalmstra @wolfv

danyeaw · 2026-07-14T18:18:37Z

+    - The MatchSpec strings mentioned in the fields `depends`, `constrains` and the lists of strings within `extra_depends` groups:
+      - MUST set the `name` field to an exact string (no globbing allowed).
+      - MAY set the fields: `version`, `build`, `build_number`, CEP 43's `when`, CEP 44 `extra`, CEP 45 `flags`.
+      - MUST NOT set any other fields.
+      - MUST be represented with the `name` + square-brackets form (e.g. `name[version="1.2.*",build_number=0]`)


Right now with v3 repodata we are generating with conda-index, version doesn't appear in square brackets. I understand that this CEP as worded would require pytest[version=">=2.7.2"] instead of pytest>=2.7.2. Is that a correct understanding and can you help me understand the motivation to make the syntax more restrictive? Thanks!

Co-authored-by: Dan Yeaw <dan@yeaw.me>

jaimergp · 2026-08-05T16:29:21Z

@conda/steering-council, the vote for this CEP will start soon, like in 24-48h. If you have any comments, this is your last chance. Thanks!

jaimergp · 2026-08-06T14:25:21Z

wolfv · 2026-08-06T14:30:19Z

Thanks to everyone who chimed in! Really really excited to see this moving forward!

CEP: A backwards-compatible repodata update strategy

39b2868

jaimergp mentioned this pull request Jan 12, 2026

CEP XXXX: Conditional dependencies, extras and flags #111

Closed

16 tasks

Add link to PR

1cb27f0

danyeaw mentioned this pull request Feb 18, 2026

Updated repodata v3 support conda/conda-pypi#247

Closed

2 tasks

baszalmstra reviewed Feb 19, 2026

View reviewed changes

Comment thread cep-XXXX.md Outdated

baszalmstra mentioned this pull request Feb 20, 2026

refactor: experimental repodata v3 conda/rattler#2093

Merged

7 tasks

jezdez mentioned this pull request Mar 5, 2026

CEP 47: Index timestamp in package record metadata #154

Merged

12 tasks

jaimergp commented Mar 6, 2026

View reviewed changes

Comment thread cep-XXXX.md Outdated

jaimergp added 3 commits March 6, 2026 17:16

Merge branch 'main' of github.com:conda/ceps into repodata-updates

db25fb7

Delegate retrofitting to implementing CEPs, remove key

0f34bfe

bump line length

2a869f1

h-vetinari reviewed Mar 6, 2026

View reviewed changes

Comment thread cep-XXXX.md Outdated

Comment thread cep-XXXX.md Outdated

Comment thread cep-XXXX.md Outdated

Comment thread cep-XXXX.md Outdated

Comment thread cep-XXXX.md Outdated

jaimergp and others added 3 commits March 9, 2026 18:03

Move Rationale _after_ Specification

8d38d9c

Typos

0949a7e

Co-authored-by: H. Vetinari <h-vetinari@users.noreply.github.com>

Extend abstract

c58d212

Co-authored-by: H. Vetinari <h-vetinari@users.noreply.github.com>

jaimergp force-pushed the repodata-updates branch from 5090659 to c58d212 Compare March 9, 2026 17:05

pre-commit

3759625

dholth reviewed Mar 11, 2026

View reviewed changes

Comment thread cep-XXXX.md

danyeaw mentioned this pull request Mar 12, 2026

Update repodata to v3 syntax conda/conda-pypi#273

Merged

4 tasks

dholth mentioned this pull request Mar 12, 2026

Support repodata v3 draft for monolithic, sharded repodata including .whl conda/conda-index#265

Merged

3 tasks

jaimergp mentioned this pull request Mar 13, 2026

Status Report (2026-03-13) Quansight-Labs/conda-ecosystem-sta-mgmt#121

Closed

danyeaw mentioned this pull request Mar 16, 2026

py-rattler 0.23.1 AnacondaRecipes/py-rattler-feedstock#3

Merged

Drop migrated_at and implement generic timestamps and n_packages fi…

c05c090

…elds

jaimergp mentioned this pull request Mar 23, 2026

Status Report (2026-03-20) Quansight-Labs/conda-ecosystem-sta-mgmt#122

Closed

jezdez mentioned this pull request Mar 30, 2026

Epic: PyPI wheel support in cx-wasm (repodata v3 / conda-pypi) jezdez/conda-express#3

Closed

10 tasks

chenghlee reviewed Mar 30, 2026

View reviewed changes

soapy1 mentioned this pull request Apr 22, 2026

Use conda-pypi to generate repodata with conda-index conda-incubator/conda-pypi-test#25

Merged

wolfv mentioned this pull request Apr 24, 2026

feat: add repodata revisions as proposed in CEP conda/rattler#2379

Merged

7 tasks

jaimergp mentioned this pull request May 29, 2026

Status Report (2026-05-29) Quansight-Labs/conda-ecosystem-sta-mgmt#135

Closed

danyeaw reviewed Jun 3, 2026

View reviewed changes

Comment thread cep-XXXX.md

baszalmstra mentioned this pull request Jun 4, 2026

fix: (de)serialize info.repodata_revisions as dictionary conda/rattler#2485

Merged

7 tasks

jaimergp added 3 commits June 8, 2026 14:43

Small precision

da6f17a

Merge branch 'main' of github.com:conda/ceps into pr/jaimergp/146

f1ceca9

Update references to PRs that are now CEPs, remove url, add `messag…

c094e29

…e` field, rework specification schema explainer, restrict which type of MatchSpec is allowed

jaimergp commented Jun 23, 2026

View reviewed changes

Comment thread cep-XXXX.md Outdated

jaimergp mentioned this pull request Jun 30, 2026

Status Report (2026-06-26) Quansight-Labs/conda-ecosystem-sta-mgmt#140

Closed

danyeaw reviewed Jul 14, 2026

View reviewed changes

jaimergp mentioned this pull request Jul 15, 2026

CEP 44: Use underscore-separated field names #181

Merged

8 tasks

danyeaw mentioned this pull request Jul 15, 2026

Ensure .conda info/index.json uses extra_depends conda/conda-pypi#453

Closed

wolfv and others added 4 commits July 22, 2026 09:53

Use extras instead of extra

19c18df

Co-authored-by: Dan Yeaw <dan@yeaw.me>

Merge branch 'main' of github.com:conda/ceps into repodata-updates

1df9d5c

Incorporate latest feedback

dbb5527

Cleanup

5d52ddb

jaimergp mentioned this pull request Jul 26, 2026

Status Report (2026-07-24) Quansight-Labs/conda-ecosystem-sta-mgmt#147

Closed

Merge branch 'main' into repodata-updates

a2293a3

jaimergp added vote Voting following governance policy and removed rfc Request for comments period to wrap up the discussion before starting a vote. labels Aug 7, 2026

jonashaag mentioned this pull request Aug 7, 2026

Convert to rattler-build recipe; make ghostscript an extra (optional dependency group) conda-forge/imagemagick-feedstock#385

Draft

5 tasks

jaimergp mentioned this pull request Aug 7, 2026

Status Report (2026-08-07) Quansight-Labs/conda-ecosystem-sta-mgmt#149

Open

This was referenced Aug 7, 2026

Update repodata_revisions to be a dict instead of a list conda/conda-index#340

Merged

CEP 47: Emit indexed_timestamp and oldest/newest conda/conda-index#341

Open

		* `oldest: int \| None`: Optional. The timestamp (in milliseconds) of the oldest record published in this revision. If set to `None` or missing, the timestamp information is not available.
		* `newest: int \| None`: Optional. The timestamp (in milliseconds) of the newest record published in this revision. If set to `None` or missing, the timestamp information is not available.


		A new top-level field identified by the syntax `v{revision}` (where `revision` comes from `info.repodata_revisions[*].revision`) MUST map to a dictionary whose schema is presented in the relevant CEP.

		The CEP MUST also specify how to identify whether a given record belongs in the newer version, or can be added to the previous ones (e.g. a new field extending CEP 20's `info/index.json`)


		A repodata revision introduces backwards incompatible features in a way that does not disrupt existing metadata. To do so, a new CEP MUST be proposed, following these guidelines below.

		A new item MUST be added to the `info.repodata_revisions` array, that MUST list the revisions found in the repodata file as dictionaries with the following schema:

Uh oh!

Conversation

jaimergp commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist for submitter

Uh oh!

Uh oh!

Uh oh!

h-vetinari left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jaimergp commented Mar 18, 2026

Uh oh!

jaimergp commented Mar 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dholth Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaimergp Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaimergp commented Aug 5, 2026

Uh oh!

jaimergp commented Aug 6, 2026 • edited by jezdez Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wolfv commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

jaimergp commented Jan 12, 2026 •

edited

Loading

dholth Mar 30, 2026 •

edited

Loading

jaimergp Jul 26, 2026 •

edited

Loading

jaimergp commented Aug 6, 2026 •

edited by jezdez

Loading