From 39b28682521d28a5e51e080db8615661dde30635 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Mon, 12 Jan 2026 22:18:06 +0100 Subject: [PATCH 01/19] CEP: A backwards-compatible repodata update strategy --- cep-XXXX.md | 161 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 161 insertions(+) create mode 100644 cep-XXXX.md diff --git a/cep-XXXX.md b/cep-XXXX.md new file mode 100644 index 00000000..85ddd749 --- /dev/null +++ b/cep-XXXX.md @@ -0,0 +1,161 @@ +# CEP XXXX - A backwards-compatible repodata update strategy + + + + + + + + + + + + +
Title A backwards-compatible repodata update strategy
Status Draft
Author(s) + Jaime Rodríguez-Guerra <jaime.rogue@gmail.com>, + Bas Zalmstra <bas@prefix.dev>, + Wolf Vollprecht <w.vollprecht@gmail.com> +
Created Jan 12, 2026
Updated Jan 12, 2026
Discussion ...
Implementation N/A
Requires N/A
+ +## Abstract + +This document addresses the challenges of updating the specification of `repodata.json` files (and its sharded derivatives) by proposing a strategy based on new top-level fields instead of `repodata_version` bumps. + +## Motivation + +`repodata.json` files are central to the conda ecosystem. They are the main source of packaging metadata and inform solvers about the catalog of available packages and their dependency constraints. As such, innovation efforts often refrain from modifying it, and the format itself has seen very few changes over its lifetime. However, a few ongoing efforts will inevitably result in `repodata.json` modifications (conditional dependencies, optional dependency groups, non-conda dependencies, etc). + +The main problem is the introduction of backwards incompatible changes. The obvious solution is to bump the `repodata_version` field (like it was done with [CEP 15](./cep-0015.md)). However, this is not desirable for existing channels, since it immediately prevents non-compatible clients from interacting with the channel. Since most clients would update via a new version available in the channel, it creates a chichen-and-egg problem that would significantly delay the introduction of new features and hinder adoption. + +There must be a strategy to introduce backwards incompatible changes without breaking existing channels. This CEP centralizes the discussion for the update strategy and consolidates that feedback into a concrete proposal. + +## Rationale + +### Version metadata in `info` + +Adding a new field in the `info` dictionary is backwards compatible, and can be used by clients to parse the necessary keys directly without having to traverse the whole dictionary. The migration timestamp is useful for client messaging, like "the client is not recent enough to see all records in this channel, please update to ensure you can obtain access to all packages". They are not added as a top-level field to stop polluting the global namespace. + +### Using top-level fields for new metadata schemas + +Adding new fields is backwards compatible and does not break older clients, which will simply ignore those and continue operating as usual. + +### Not retrofitting for older versions + +The solver would provide different solutions depending on the supported versions, which would make behavior surprising and difficult to debug. For example, let's say that optional dependency groups are implemented in `v3`, which requires adding a new `extras` key to the package metadata. `scipy` starts using this in its release `1.20`. As such, this is release is exposed under the `v3` top-level field. + +Shortly after a new package `using-scipy` is published, and it has `depends: ["scipy>=1.17"]`. `using-scipy` does not use any `v3` metadata features. If it was published under `packages.conda`, different clients would provide different solutions: older clients with no `v3` support would provide `scipy==1.19`, while newer clients would give `scipy==1.20`. This is confusing and difficult to debug. + +Instead, we propose that once a channel starts using the newer repodata revision, all packages are published under the new top-level key. This way, `using-scipy` would simply not exist for older clients, preventing version mismatches in the solution. Using the `info.repodata_revisions` metadata, clients would be able to even provide hints about the possibility of new records being available once client updates are performed. + +Another example is the introduction of new archive formats (e.g. `.conda-v3`) in a new repodata revision, for which conda clients would need to implement support. Once the format starts being used, all new packages should be published under the new key so solver asymmetries are minimized. + +### Freezing `repodata_version` to `1` + +Bumps in this number should only result in backwards incompatible changes that would anyway prevent a channel from operating completely. While `repodata_version: 2` exists (as per CEP 15), its implementations are not sufficiently old to guarantee that the majority of existing conda clients would support it: + +- `rattler` supports it [v0.9.0](https://github.com/conda/rattler/blob/main/CHANGELOG.md#090---2023-09-22) (released on 2023-09-22), which means that `pixi` supports it since [v0.4.0](https://github.com/prefix-dev/pixi/blob/d8d2d8a3e8e1ce99707885aa1437e3768614456b/Cargo.toml#L38) (released on 2023-09-22 too). +- `conda` only supports it as of [v24.5.0](https://github.com/conda/conda/blob/main/CHANGELOG.md#2450-2024-05-08) (released on 2024-05-08) +- `mamba` started supporting it in [v2.0](https://github.com/mamba-org/mamba/blob/main/CHANGELOG.md#20240925) (released on 2024-09-25). + +Hence, we suggest to stick to `repodata_version: 1` and _only_ use `repodata_version: 2` when a new channel needs a global `base_url` for all the entries in the `packages` and `packages.conda` fields. + +## Specification + +A repodata revision introduces backwards incompatible features in a way that does not disrupt existing metadata. To do so, a new CEP MUST be proposed, following these guidelines below. + +A new item will be added to the `info.repodata_revisions` array, that lists the revisions found in the repodata file as dictionaries with the following schema: + +- `revision: int`: The integer identifying the revision. +- `key: str`: The top-level key that contains the records for this revision. It MUST match the regex `[a-z0-9\._]+`. It MUST NOT match any of the existing top-level fields (`repodata_version`, `info`, `packages`, `packages.conda`, `removed`). +- `migrated_at: int`: The timestamp (in milliseconds) that signals when the channel started indexing packages in the new revision. + +A new top-level field, with the identifier as specified in `info.repodata_revisions[*].key`. MUST map to a dictionary whose schema is presented in the relevant CEP. + +Once a channel starts publishing for a new revision, all packages published after that moment SHOULD be included only in the new revision field, even if some of them may be technically backwards compatible because their record metadata does not use any new features. + +The `repodata_version` MUST be `1`. + +## Examples + +A hypothetical new repodata revision `3` would need to present the following `info.repodata_revisions` entry, accompanied by this sample top-level `v3` field aggregating some of the proposed CEP ideas (at the time of writing, Jan 2026): + +```js +{ + "repodata_version": 1, + "info": { + "subdir": "noarch", + "repodata_revisions": [ + { + "revision": 3, + "key": "v3", + "migrated_at": 1768249989751, // 2026-01-12 20:33 UTC + } + ] + }, + "packages": { + "example-1.0.0-0.tar.bz2": { + "build": "0", + "build_number": 0, + "depends": [], + "md5": "82ecc40f09b9c44483e6b70cad2545d7", + "name": "example", + "noarch": "generic", + "sha256": "eb65e866067865793b981c2ba74485f75bef441842b5998badc4ec66717685c7", + "size": 1234, + "subdir": "noarch", + "timestamp": 1689209309623, + "version": "1.0.0" + } + }, + "packages.conda": { + "example-1.0.0-0.conda": { + "build": "0", + "build_number": 0, + "depends": [], + "md5": "82ecc40f09b9c44483e6b70cad2545d7", + "name": "example", + "noarch": "generic", + "sha256": "eb65e866067865793b981c2ba74485f75bef441842b5998badc4ec66717685c7", + "size": 1234, + "subdir": "noarch", + "timestamp": 1689209309623, + "version": "1.0.0" + } + }, + "v3": { + "tar.bz2": {}, + "conda": { + "example-1.0.0-0": { // key does not have the extension anymore + "build": "0", + "build_number": 0, + "depends": [ + "package[version=3,build_number=0,if=__unix]" // bracket syntax, w/ conditional + ], + "md5": "82ecc40f09b9c44483e6b70cad2545d7", + "name": "example", + "noarch": "generic", + "sha256": "eb65e866067865793b981c2ba74485f75bef441842b5998badc4ec66717685c7", + "size": 1234, + "subdir": "noarch", + "timestamp": 1768249989851, + "version": "1.0.0", + "new_field": "CRITICAL" // new field + } + } + } +} +``` + +## Rejected ideas + +### Encoding version information in the filename + +One alternative would be to create new `repodata.json` filenames (e.g. `repodata.v4.json`) for each new incompatible bump. However, this was rejected by the authors as it introduces complexity in other areas: + +- It would require more HTTP calls to retrieve the latest version served by the channel. +- It would introduce duplication across `repodata.json` versions and their shards. +- Indexing tools would need to maintain the different versions of `repodata.json` in sync. + +## Copyright + +All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/). From 1cb27f044c97451254786009a608b46bd2c55c8d Mon Sep 17 00:00:00 2001 From: jaimergp Date: Mon, 12 Jan 2026 22:19:51 +0100 Subject: [PATCH 02/19] Add link to PR --- cep-XXXX.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 85ddd749..edbdbd2b 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -12,7 +12,7 @@ Created Jan 12, 2026 Updated Jan 12, 2026 - Discussion ... + Discussion https://github.com/conda/ceps/pull/146 Implementation N/A Requires N/A From 0f34bfe2fa8c18ad15946be9f6b6494b75d77c86 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Fri, 6 Mar 2026 17:31:01 +0100 Subject: [PATCH 03/19] Delegate retrofitting to implementing CEPs, remove `key` --- cep-XXXX.md | 26 +++++++------------------- 1 file changed, 7 insertions(+), 19 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index edbdbd2b..357c4a8b 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -11,7 +11,7 @@ Created Jan 12, 2026 - Updated Jan 12, 2026 + Updated Mar 6, 2026 Discussion https://github.com/conda/ceps/pull/146 Implementation N/A Requires N/A @@ -23,7 +23,7 @@ This document addresses the challenges of updating the specification of `repodat ## Motivation -`repodata.json` files are central to the conda ecosystem. They are the main source of packaging metadata and inform solvers about the catalog of available packages and their dependency constraints. As such, innovation efforts often refrain from modifying it, and the format itself has seen very few changes over its lifetime. However, a few ongoing efforts will inevitably result in `repodata.json` modifications (conditional dependencies, optional dependency groups, non-conda dependencies, etc). +`repodata.json` files are central to the conda ecosystem. They are the main source of packaging metadata and inform solvers about the catalog of available packages and their dependency constraints. As such, innovation work often refrains from modifying it, and the format itself has seen very few changes over its lifetime. However, a few ongoing efforts will inevitably result in `repodata.json` modifications (conditional dependencies, optional dependency groups, non-conda dependencies, etc). The main problem is the introduction of backwards incompatible changes. The obvious solution is to bump the `repodata_version` field (like it was done with [CEP 15](./cep-0015.md)). However, this is not desirable for existing channels, since it immediately prevents non-compatible clients from interacting with the channel. Since most clients would update via a new version available in the channel, it creates a chichen-and-egg problem that would significantly delay the introduction of new features and hinder adoption. @@ -33,22 +33,12 @@ There must be a strategy to introduce backwards incompatible changes without bre ### Version metadata in `info` -Adding a new field in the `info` dictionary is backwards compatible, and can be used by clients to parse the necessary keys directly without having to traverse the whole dictionary. The migration timestamp is useful for client messaging, like "the client is not recent enough to see all records in this channel, please update to ensure you can obtain access to all packages". They are not added as a top-level field to stop polluting the global namespace. +Adding a new field in the `info` dictionary is backwards compatible, and can be used by clients to parse the necessary keys directly without having to traverse the whole document. The migration timestamp is useful for client messaging, like "the client is not recent enough to see all records in this channel, please update to ensure you can obtain access to all packages". They are not added as a top-level field to stop polluting the global namespace. ### Using top-level fields for new metadata schemas Adding new fields is backwards compatible and does not break older clients, which will simply ignore those and continue operating as usual. -### Not retrofitting for older versions - -The solver would provide different solutions depending on the supported versions, which would make behavior surprising and difficult to debug. For example, let's say that optional dependency groups are implemented in `v3`, which requires adding a new `extras` key to the package metadata. `scipy` starts using this in its release `1.20`. As such, this is release is exposed under the `v3` top-level field. - -Shortly after a new package `using-scipy` is published, and it has `depends: ["scipy>=1.17"]`. `using-scipy` does not use any `v3` metadata features. If it was published under `packages.conda`, different clients would provide different solutions: older clients with no `v3` support would provide `scipy==1.19`, while newer clients would give `scipy==1.20`. This is confusing and difficult to debug. - -Instead, we propose that once a channel starts using the newer repodata revision, all packages are published under the new top-level key. This way, `using-scipy` would simply not exist for older clients, preventing version mismatches in the solution. Using the `info.repodata_revisions` metadata, clients would be able to even provide hints about the possibility of new records being available once client updates are performed. - -Another example is the introduction of new archive formats (e.g. `.conda-v3`) in a new repodata revision, for which conda clients would need to implement support. Once the format starts being used, all new packages should be published under the new key so solver asymmetries are minimized. - ### Freezing `repodata_version` to `1` Bumps in this number should only result in backwards incompatible changes that would anyway prevent a channel from operating completely. While `repodata_version: 2` exists (as per CEP 15), its implementations are not sufficiently old to guarantee that the majority of existing conda clients would support it: @@ -66,12 +56,11 @@ A repodata revision introduces backwards incompatible features in a way that doe A new item will be added to the `info.repodata_revisions` array, that lists the revisions found in the repodata file as dictionaries with the following schema: - `revision: int`: The integer identifying the revision. -- `key: str`: The top-level key that contains the records for this revision. It MUST match the regex `[a-z0-9\._]+`. It MUST NOT match any of the existing top-level fields (`repodata_version`, `info`, `packages`, `packages.conda`, `removed`). -- `migrated_at: int`: The timestamp (in milliseconds) that signals when the channel started indexing packages in the new revision. +- `migrated_at: int`: The timestamp (in milliseconds) that signals when the channel started indexing packages in the new revision; i.e. the timestamp of the first record published in the new metadata. -A new top-level field, with the identifier as specified in `info.repodata_revisions[*].key`. MUST map to a dictionary whose schema is presented in the relevant CEP. +A new top-level field identified by the syntax `v{revision}` (where `revision` comes from `info.repodata_revisions[*].revision`) MUST map to a dictionary whose schema is presented in the relevant CEP. -Once a channel starts publishing for a new revision, all packages published after that moment SHOULD be included only in the new revision field, even if some of them may be technically backwards compatible because their record metadata does not use any new features. +The CEP MUST also specify how to identify whether a given record belongs in the newer version, or can be added to the previous ones (e.g. a new field extending CEP 20's `info/index.json`) The `repodata_version` MUST be `1`. @@ -87,7 +76,6 @@ A hypothetical new repodata revision `3` would need to present the following `in "repodata_revisions": [ { "revision": 3, - "key": "v3", "migrated_at": 1768249989751, // 2026-01-12 20:33 UTC } ] @@ -129,7 +117,7 @@ A hypothetical new repodata revision `3` would need to present the following `in "build": "0", "build_number": 0, "depends": [ - "package[version=3,build_number=0,if=__unix]" // bracket syntax, w/ conditional + "package[version=3,build_number=0,when=__unix]" // bracket syntax, w/ conditional ], "md5": "82ecc40f09b9c44483e6b70cad2545d7", "name": "example", From 2a869f1401e0f1820845905f9bc070d072ece227 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Fri, 6 Mar 2026 17:32:07 +0100 Subject: [PATCH 04/19] bump line length --- .markdownlint.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.markdownlint.yaml b/.markdownlint.yaml index 69f97754..3fd4e9ca 100644 --- a/.markdownlint.yaml +++ b/.markdownlint.yaml @@ -19,4 +19,4 @@ MD013: # Number of characters # we have set this to be very high right now due to the linter not # ignoring the limit for inline html and tables - line_length: 500 + line_length: 600 From 8d38d9ca6084a97bd193391d08b896f2cc07dcdb Mon Sep 17 00:00:00 2001 From: jaimergp Date: Mon, 9 Mar 2026 18:03:16 +0100 Subject: [PATCH 05/19] Move Rationale _after_ Specification --- cep-XXXX.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 357c4a8b..1fe15b2c 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -29,6 +29,21 @@ The main problem is the introduction of backwards incompatible changes. The obvi There must be a strategy to introduce backwards incompatible changes without breaking existing channels. This CEP centralizes the discussion for the update strategy and consolidates that feedback into a concrete proposal. +## Specification + +A repodata revision introduces backwards incompatible features in a way that does not disrupt existing metadata. To do so, a new CEP MUST be proposed, following these guidelines below. + +A new item MUST be added to the `info.repodata_revisions` array, that MUST list the revisions found in the repodata file as dictionaries with the following schema: + +- `revision: int`: The integer identifying the revision. +- `migrated_at: int`: The timestamp (in milliseconds) that signals when the channel started indexing packages in the new revision; i.e. the timestamp of the first record published in the new metadata. + +A new top-level field identified by the syntax `v{revision}` (where `revision` comes from `info.repodata_revisions[*].revision`) MUST map to a dictionary whose schema is presented in the relevant CEP. + +The CEP MUST also specify how to identify whether a given record belongs in the newer version, or can be added to the previous ones (e.g. a new field extending CEP 20's `info/index.json`) + +The `repodata_version` MUST be `1` or, if [CEP 15](./cep-0015.md) applies, `2`. + ## Rationale ### Version metadata in `info` @@ -49,21 +64,6 @@ Bumps in this number should only result in backwards incompatible changes that w Hence, we suggest to stick to `repodata_version: 1` and _only_ use `repodata_version: 2` when a new channel needs a global `base_url` for all the entries in the `packages` and `packages.conda` fields. -## Specification - -A repodata revision introduces backwards incompatible features in a way that does not disrupt existing metadata. To do so, a new CEP MUST be proposed, following these guidelines below. - -A new item will be added to the `info.repodata_revisions` array, that lists the revisions found in the repodata file as dictionaries with the following schema: - -- `revision: int`: The integer identifying the revision. -- `migrated_at: int`: The timestamp (in milliseconds) that signals when the channel started indexing packages in the new revision; i.e. the timestamp of the first record published in the new metadata. - -A new top-level field identified by the syntax `v{revision}` (where `revision` comes from `info.repodata_revisions[*].revision`) MUST map to a dictionary whose schema is presented in the relevant CEP. - -The CEP MUST also specify how to identify whether a given record belongs in the newer version, or can be added to the previous ones (e.g. a new field extending CEP 20's `info/index.json`) - -The `repodata_version` MUST be `1`. - ## Examples A hypothetical new repodata revision `3` would need to present the following `info.repodata_revisions` entry, accompanied by this sample top-level `v3` field aggregating some of the proposed CEP ideas (at the time of writing, Jan 2026): From 0949a7e0da831b5905ad3f1b8eea0434174ab239 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Mon, 9 Mar 2026 18:04:08 +0100 Subject: [PATCH 06/19] Typos Co-authored-by: H. Vetinari --- cep-XXXX.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 1fe15b2c..da71b58c 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -25,7 +25,7 @@ This document addresses the challenges of updating the specification of `repodat `repodata.json` files are central to the conda ecosystem. They are the main source of packaging metadata and inform solvers about the catalog of available packages and their dependency constraints. As such, innovation work often refrains from modifying it, and the format itself has seen very few changes over its lifetime. However, a few ongoing efforts will inevitably result in `repodata.json` modifications (conditional dependencies, optional dependency groups, non-conda dependencies, etc). -The main problem is the introduction of backwards incompatible changes. The obvious solution is to bump the `repodata_version` field (like it was done with [CEP 15](./cep-0015.md)). However, this is not desirable for existing channels, since it immediately prevents non-compatible clients from interacting with the channel. Since most clients would update via a new version available in the channel, it creates a chichen-and-egg problem that would significantly delay the introduction of new features and hinder adoption. +The main problem is the introduction of backwards incompatible changes. The obvious solution is to bump the `repodata_version` field (like it was done with [CEP 15](./cep-0015.md)). However, this is not desirable for existing channels, since it immediately prevents non-compatible clients from interacting with the channel. Since most clients would update via a new version available in the channel, it creates a chicken-and-egg problem that would significantly delay the introduction of new features and hinder adoption. There must be a strategy to introduce backwards incompatible changes without breaking existing channels. This CEP centralizes the discussion for the update strategy and consolidates that feedback into a concrete proposal. @@ -58,7 +58,7 @@ Adding new fields is backwards compatible and does not break older clients, whic Bumps in this number should only result in backwards incompatible changes that would anyway prevent a channel from operating completely. While `repodata_version: 2` exists (as per CEP 15), its implementations are not sufficiently old to guarantee that the majority of existing conda clients would support it: -- `rattler` supports it [v0.9.0](https://github.com/conda/rattler/blob/main/CHANGELOG.md#090---2023-09-22) (released on 2023-09-22), which means that `pixi` supports it since [v0.4.0](https://github.com/prefix-dev/pixi/blob/d8d2d8a3e8e1ce99707885aa1437e3768614456b/Cargo.toml#L38) (released on 2023-09-22 too). +- `rattler` supports it since [v0.9.0](https://github.com/conda/rattler/blob/main/CHANGELOG.md#090---2023-09-22) (released on 2023-09-22), which means that `pixi` supports it since [v0.4.0](https://github.com/prefix-dev/pixi/blob/d8d2d8a3e8e1ce99707885aa1437e3768614456b/Cargo.toml#L38) (released on 2023-09-22 too). - `conda` only supports it as of [v24.5.0](https://github.com/conda/conda/blob/main/CHANGELOG.md#2450-2024-05-08) (released on 2024-05-08) - `mamba` started supporting it in [v2.0](https://github.com/mamba-org/mamba/blob/main/CHANGELOG.md#20240925) (released on 2024-09-25). From c58d212cc9067b50c23c6ffe7b6e8b46d7f6c583 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Mon, 9 Mar 2026 18:05:03 +0100 Subject: [PATCH 07/19] Extend abstract Co-authored-by: H. Vetinari --- cep-XXXX.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index da71b58c..2b7fc3aa 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -11,7 +11,7 @@ Created Jan 12, 2026 - Updated Mar 6, 2026 + Updated Mar 9, 2026 Discussion https://github.com/conda/ceps/pull/146 Implementation N/A Requires N/A @@ -19,7 +19,13 @@ ## Abstract -This document addresses the challenges of updating the specification of `repodata.json` files (and its sharded derivatives) by proposing a strategy based on new top-level fields instead of `repodata_version` bumps. +This document addresses the challenges of updating the specification of `repodata.json` files (and its sharded derivatives) by proposing: + +* to provide a way to extend repodata with new variants, _without_ incrementing `repodata_version` +* to encode new versions of the repodata in parallel with existing one, behind a `vN` key where `N` is the repodata revision +* to indicate the latest available repodata revision as part of `info` +* to provide a timestamp indicating the point in time when the index started providing the most recent repodata revision ("migration timestamp") +* to use the migration timestamp for client messaging, e.g. when an installer is too old to see repodata of a given revision. ## Motivation From 37596258c1fb266d4f2c3d9febac8cb6148275fd Mon Sep 17 00:00:00 2001 From: jaimergp Date: Mon, 9 Mar 2026 18:05:30 +0100 Subject: [PATCH 08/19] pre-commit --- cep-XXXX.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 2b7fc3aa..d0f8ab92 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -41,8 +41,8 @@ A repodata revision introduces backwards incompatible features in a way that doe A new item MUST be added to the `info.repodata_revisions` array, that MUST list the revisions found in the repodata file as dictionaries with the following schema: -- `revision: int`: The integer identifying the revision. -- `migrated_at: int`: The timestamp (in milliseconds) that signals when the channel started indexing packages in the new revision; i.e. the timestamp of the first record published in the new metadata. +* `revision: int`: The integer identifying the revision. +* `migrated_at: int`: The timestamp (in milliseconds) that signals when the channel started indexing packages in the new revision; i.e. the timestamp of the first record published in the new metadata. A new top-level field identified by the syntax `v{revision}` (where `revision` comes from `info.repodata_revisions[*].revision`) MUST map to a dictionary whose schema is presented in the relevant CEP. @@ -64,9 +64,9 @@ Adding new fields is backwards compatible and does not break older clients, whic Bumps in this number should only result in backwards incompatible changes that would anyway prevent a channel from operating completely. While `repodata_version: 2` exists (as per CEP 15), its implementations are not sufficiently old to guarantee that the majority of existing conda clients would support it: -- `rattler` supports it since [v0.9.0](https://github.com/conda/rattler/blob/main/CHANGELOG.md#090---2023-09-22) (released on 2023-09-22), which means that `pixi` supports it since [v0.4.0](https://github.com/prefix-dev/pixi/blob/d8d2d8a3e8e1ce99707885aa1437e3768614456b/Cargo.toml#L38) (released on 2023-09-22 too). -- `conda` only supports it as of [v24.5.0](https://github.com/conda/conda/blob/main/CHANGELOG.md#2450-2024-05-08) (released on 2024-05-08) -- `mamba` started supporting it in [v2.0](https://github.com/mamba-org/mamba/blob/main/CHANGELOG.md#20240925) (released on 2024-09-25). +* `rattler` supports it since [v0.9.0](https://github.com/conda/rattler/blob/main/CHANGELOG.md#090---2023-09-22) (released on 2023-09-22), which means that `pixi` supports it since [v0.4.0](https://github.com/prefix-dev/pixi/blob/d8d2d8a3e8e1ce99707885aa1437e3768614456b/Cargo.toml#L38) (released on 2023-09-22 too). +* `conda` only supports it as of [v24.5.0](https://github.com/conda/conda/blob/main/CHANGELOG.md#2450-2024-05-08) (released on 2024-05-08) +* `mamba` started supporting it in [v2.0](https://github.com/mamba-org/mamba/blob/main/CHANGELOG.md#20240925) (released on 2024-09-25). Hence, we suggest to stick to `repodata_version: 1` and _only_ use `repodata_version: 2` when a new channel needs a global `base_url` for all the entries in the `packages` and `packages.conda` fields. @@ -146,9 +146,9 @@ A hypothetical new repodata revision `3` would need to present the following `in One alternative would be to create new `repodata.json` filenames (e.g. `repodata.v4.json`) for each new incompatible bump. However, this was rejected by the authors as it introduces complexity in other areas: -- It would require more HTTP calls to retrieve the latest version served by the channel. -- It would introduce duplication across `repodata.json` versions and their shards. -- Indexing tools would need to maintain the different versions of `repodata.json` in sync. +* It would require more HTTP calls to retrieve the latest version served by the channel. +* It would introduce duplication across `repodata.json` versions and their shards. +* Indexing tools would need to maintain the different versions of `repodata.json` in sync. ## Copyright From c05c090a4aa0cabedca9f0bdb83cbacf709e5a5d Mon Sep 17 00:00:00 2001 From: jaimergp Date: Wed, 18 Mar 2026 17:36:59 +0100 Subject: [PATCH 09/19] Drop `migrated_at` and implement generic timestamps and n_packages fields --- cep-XXXX.md | 52 ++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 36 insertions(+), 16 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index d0f8ab92..1a06e71d 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -11,7 +11,7 @@ Created Jan 12, 2026 - Updated Mar 9, 2026 + Updated Mar 18, 2026 Discussion https://github.com/conda/ceps/pull/146 Implementation N/A Requires N/A @@ -23,9 +23,8 @@ This document addresses the challenges of updating the specification of `repodat * to provide a way to extend repodata with new variants, _without_ incrementing `repodata_version` * to encode new versions of the repodata in parallel with existing one, behind a `vN` key where `N` is the repodata revision -* to indicate the latest available repodata revision as part of `info` -* to provide a timestamp indicating the point in time when the index started providing the most recent repodata revision ("migration timestamp") -* to use the migration timestamp for client messaging, e.g. when an installer is too old to see repodata of a given revision. +* to indicate the latest available repodata revision as part of the `info` top-level dictionary +* to provide useful metadata for end-user error reporting in case of lack of support in the client (oldest and newest timestamps, number of packages available) ## Motivation @@ -41,8 +40,10 @@ A repodata revision introduces backwards incompatible features in a way that doe A new item MUST be added to the `info.repodata_revisions` array, that MUST list the revisions found in the repodata file as dictionaries with the following schema: -* `revision: int`: The integer identifying the revision. -* `migrated_at: int`: The timestamp (in milliseconds) that signals when the channel started indexing packages in the new revision; i.e. the timestamp of the first record published in the new metadata. +* `revision: int`: Required. The integer identifying the revision. +* `n_packages: int`: Required. The number of packages available in this revision. +* `oldest: int | None`: Optional. The timestamp (in milliseconds) of the oldest record published in this revision. If set to `None` or missing, the timestamp information is not available. +* `newest: int | None`: Optional. The timestamp (in milliseconds) of the newest record published in this revision. If set to `None` or missing, the timestamp information is not available. A new top-level field identified by the syntax `v{revision}` (where `revision` comes from `info.repodata_revisions[*].revision`) MUST map to a dictionary whose schema is presented in the relevant CEP. @@ -54,7 +55,7 @@ The `repodata_version` MUST be `1` or, if [CEP 15](./cep-0015.md) applies, `2`. ### Version metadata in `info` -Adding a new field in the `info` dictionary is backwards compatible, and can be used by clients to parse the necessary keys directly without having to traverse the whole document. The migration timestamp is useful for client messaging, like "the client is not recent enough to see all records in this channel, please update to ensure you can obtain access to all packages". They are not added as a top-level field to stop polluting the global namespace. +Adding a new field in the `info` dictionary is backwards compatible, and can be used by clients to parse the necessary keys directly without having to traverse the whole document. The `oldest`, `newest`, and `n_packages` fields are useful for client messaging, like "the client is not recent enough to see all records in this channel, please update to ensure you can obtain access to {n_packages} additional packages published between {oldest} and {newest}". They are not added as a top-level field to stop polluting the global namespace. ### Using top-level fields for new metadata schemas @@ -82,7 +83,9 @@ A hypothetical new repodata revision `3` would need to present the following `in "repodata_revisions": [ { "revision": 3, - "migrated_at": 1768249989751, // 2026-01-12 20:33 UTC + "n_packages": 2, + "oldest": 1768249989851, + "newest": 1773851561010, } ] }, @@ -102,28 +105,29 @@ A hypothetical new repodata revision `3` would need to present the following `in } }, "packages.conda": { - "example-1.0.0-0.conda": { + "package-1.0.0-0.conda": { "build": "0", "build_number": 0, "depends": [], - "md5": "82ecc40f09b9c44483e6b70cad2545d7", - "name": "example", + "md5": "4483e6b70c82ecc40f09b9c4ad2545d7", + "name": "package", "noarch": "generic", - "sha256": "eb65e866067865793b981c2ba74485f75bef441842b5998badc4ec66717685c7", + "sha256": "4485f75bef441842b59eb65e866067865793b981c2ba798badc4ec66717685c7", "size": 1234, "subdir": "noarch", - "timestamp": 1689209309623, + "timestamp": 1689209359623, "version": "1.0.0" } }, "v3": { + // This is a hypothetical v3, not a real proposal "tar.bz2": {}, "conda": { - "example-1.0.0-0": { // key does not have the extension anymore + "example-2.0.0-0": { // key does not have the extension anymore "build": "0", "build_number": 0, "depends": [ - "package[version=3,build_number=0,when=__unix]" // bracket syntax, w/ conditional + "package[version=1,build_number=0,when=__unix]" // bracket syntax, w/ conditional ], "md5": "82ecc40f09b9c44483e6b70cad2545d7", "name": "example", @@ -132,7 +136,23 @@ A hypothetical new repodata revision `3` would need to present the following `in "size": 1234, "subdir": "noarch", "timestamp": 1768249989851, - "version": "1.0.0", + "version": "2.0.0", + "new_field": "CRITICAL" // new field + }, + "example-3.0.0-0": { // key does not have the extension anymore + "build": "0", + "build_number": 0, + "depends": [ + "package[version=3,build_number=0,when=__unix]" // bracket syntax, w/ conditional + ], + "md5": "6b70cad2545d782ecc40f09b9c44483e", + "name": "example", + "noarch": "generic", + "sha256": "74485f75bef441842b5998badc4ec66717685c7eb65e866067865793b981c2ba", + "size": 2345, + "subdir": "noarch", + "timestamp": 1773851561010, + "version": "3.0.0", "new_field": "CRITICAL" // new field } } From 51ecdb481cce04bb94b995fbc81b45d8be5f4ff3 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Tue, 5 May 2026 23:46:47 +0200 Subject: [PATCH 10/19] Do propose v3 --- cep-XXXX.md | 70 +++++++++++++++++++++++++++++------------------------ 1 file changed, 39 insertions(+), 31 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 1a06e71d..52b2ed46 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -1,7 +1,7 @@ -# CEP XXXX - A backwards-compatible repodata update strategy +# CEP XXXX - A backwards-compatible update strategy for repodata `v3` - + - + - +
Title A backwards-compatible repodata update strategy
Title A backwards-compatible update strategy for repodata v3
Status Draft
Author(s) @@ -11,20 +11,19 @@
Created Jan 12, 2026
Updated Mar 18, 2026
Updated May 5, 2026
Discussion https://github.com/conda/ceps/pull/146
Implementation N/A
Requires N/A
Requires https://github.com/conda/ceps/pull/164, https://github.com/conda/ceps/pull/165, https://github.com/conda/ceps/pull/166, https://github.com/conda/ceps/pull/151
-## Abstract +> The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119][RFC2119] when, and only when, they appear in all capitals, as shown here. + +[RFC2119]: https://datatracker.ietf.org/doc/html/rfc2119 -This document addresses the challenges of updating the specification of `repodata.json` files (and its sharded derivatives) by proposing: +## Abstract -* to provide a way to extend repodata with new variants, _without_ incrementing `repodata_version` -* to encode new versions of the repodata in parallel with existing one, behind a `vN` key where `N` is the repodata revision -* to indicate the latest available repodata revision as part of the `info` top-level dictionary -* to provide useful metadata for end-user error reporting in case of lack of support in the client (oldest and newest timestamps, number of packages available) +This document proposes a set of updates to `repodata.json` files and its sharded derivatives to include the improvements introduced in CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#166](https://github.com/conda/ceps/pull/166), and [#151](https://github.com/conda/ceps/pull/151). To do so in a backwards compatible manner, it also proposes a revision system as a way to extend repodata with new variants, _without_ incrementing `repodata_version`. ## Motivation @@ -32,22 +31,36 @@ This document addresses the challenges of updating the specification of `repodat The main problem is the introduction of backwards incompatible changes. The obvious solution is to bump the `repodata_version` field (like it was done with [CEP 15](./cep-0015.md)). However, this is not desirable for existing channels, since it immediately prevents non-compatible clients from interacting with the channel. Since most clients would update via a new version available in the channel, it creates a chicken-and-egg problem that would significantly delay the introduction of new features and hinder adoption. -There must be a strategy to introduce backwards incompatible changes without breaking existing channels. This CEP centralizes the discussion for the update strategy and consolidates that feedback into a concrete proposal. +There must be a strategy to introduce backwards incompatible changes without breaking existing channels. This CEP centralizes the discussion for the update strategy and consolidates that feedback into a concrete proposal for `v3`. + +The `v3` update includes breaking changes in CEP CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#164](https://github.com/conda/ceps/pull/166), and [#151](https://github.com/conda/ceps/pull/151), that wouldn't otherwise reach existing channels without disrupting the user experience for outdated clients. ## Specification -A repodata revision introduces backwards incompatible features in a way that does not disrupt existing metadata. To do so, a new CEP MUST be proposed, following these guidelines below. +This CEP introduces two new keys: + +- A top-level `v3` key +- A `repodata_revisions` key under the top-level `info` dictionary -A new item MUST be added to the `info.repodata_revisions` array, that MUST list the revisions found in the repodata file as dictionaries with the following schema: +### The `v3` top-level key -* `revision: int`: Required. The integer identifying the revision. -* `n_packages: int`: Required. The number of packages available in this revision. -* `oldest: int | None`: Optional. The timestamp (in milliseconds) of the oldest record published in this revision. If set to `None` or missing, the timestamp information is not available. -* `newest: int | None`: Optional. The timestamp (in milliseconds) of the newest record published in this revision. If set to `None` or missing, the timestamp information is not available. +This key MUST map to a dictionary where: -A new top-level field identified by the syntax `v{revision}` (where `revision` comes from `info.repodata_revisions[*].revision`) MUST map to a dictionary whose schema is presented in the relevant CEP. +- Each key SHOULD represent the file extension (without the leading period) of the included artifacts. These are usually `tar.bz2` and `conda`. +- Each value MUST be a dictionary where: + - Each key MUST be the artifact filename without its extension + - Each value MUST be a valid CEP 36 "package record metadata" dictionary, optionally extended with CEP PR [#151](https://github.com/conda/ceps/pull/151)'s `url` key and/or the keys introduced in `index.json`'s `schema_version: 3` update (CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#164](https://github.com/conda/ceps/pull/166)). -The CEP MUST also specify how to identify whether a given record belongs in the newer version, or can be added to the previous ones (e.g. a new field extending CEP 20's `info/index.json`) +### The `info.repodata_revisions` key + +A repodata _revision_ introduces backwards incompatible features in a way that does not disrupt existing metadata. + +The new `info.repodata_revisions` key maps to an array that MUST list the revisions found in the repodata file or shard as dictionaries with the following schema: + +- `revision: int`: Required. The integer identifying the revision. It MUST be 3 or larger. It MUST be used in the corresponding file top-level key, with syntax `v{revision}`. For the `v3` update proposed in this CEP, the value is `3`. +- `n_packages: int`: Required. The number of packages available in this revision in the current `repodata.json` or shard. +- `oldest: int | None`: Optional. The timestamp (in milliseconds) of the oldest record published in this revision in the current `repodata.json` or shard. If set to `None` or missing, the timestamp information is not available. +- `newest: int | None`: Optional. The timestamp (in milliseconds) of the newest record published in this revision in the current `repodata.json` or shard. If set to `None` or missing, the timestamp information is not available. The `repodata_version` MUST be `1` or, if [CEP 15](./cep-0015.md) applies, `2`. @@ -65,15 +78,13 @@ Adding new fields is backwards compatible and does not break older clients, whic Bumps in this number should only result in backwards incompatible changes that would anyway prevent a channel from operating completely. While `repodata_version: 2` exists (as per CEP 15), its implementations are not sufficiently old to guarantee that the majority of existing conda clients would support it: -* `rattler` supports it since [v0.9.0](https://github.com/conda/rattler/blob/main/CHANGELOG.md#090---2023-09-22) (released on 2023-09-22), which means that `pixi` supports it since [v0.4.0](https://github.com/prefix-dev/pixi/blob/d8d2d8a3e8e1ce99707885aa1437e3768614456b/Cargo.toml#L38) (released on 2023-09-22 too). -* `conda` only supports it as of [v24.5.0](https://github.com/conda/conda/blob/main/CHANGELOG.md#2450-2024-05-08) (released on 2024-05-08) -* `mamba` started supporting it in [v2.0](https://github.com/mamba-org/mamba/blob/main/CHANGELOG.md#20240925) (released on 2024-09-25). +- `rattler` supports it since [v0.9.0](https://github.com/conda/rattler/blob/main/CHANGELOG.md#090---2023-09-22) (released on 2023-09-22), which means that `pixi` supports it since [v0.4.0](https://github.com/prefix-dev/pixi/blob/d8d2d8a3e8e1ce99707885aa1437e3768614456b/Cargo.toml#L38) (released on 2023-09-22 too). +- `conda` only supports it as of [v24.5.0](https://github.com/conda/conda/blob/main/CHANGELOG.md#2450-2024-05-08) (released on 2024-05-08) +- `mamba` started supporting it in [v2.0](https://github.com/mamba-org/mamba/blob/main/CHANGELOG.md#20240925) (released on 2024-09-25). Hence, we suggest to stick to `repodata_version: 1` and _only_ use `repodata_version: 2` when a new channel needs a global `base_url` for all the entries in the `packages` and `packages.conda` fields. -## Examples - -A hypothetical new repodata revision `3` would need to present the following `info.repodata_revisions` entry, accompanied by this sample top-level `v3` field aggregating some of the proposed CEP ideas (at the time of writing, Jan 2026): +## Example ```js { @@ -120,7 +131,6 @@ A hypothetical new repodata revision `3` would need to present the following `in } }, "v3": { - // This is a hypothetical v3, not a real proposal "tar.bz2": {}, "conda": { "example-2.0.0-0": { // key does not have the extension anymore @@ -137,7 +147,6 @@ A hypothetical new repodata revision `3` would need to present the following `in "subdir": "noarch", "timestamp": 1768249989851, "version": "2.0.0", - "new_field": "CRITICAL" // new field }, "example-3.0.0-0": { // key does not have the extension anymore "build": "0", @@ -153,7 +162,6 @@ A hypothetical new repodata revision `3` would need to present the following `in "subdir": "noarch", "timestamp": 1773851561010, "version": "3.0.0", - "new_field": "CRITICAL" // new field } } } @@ -166,9 +174,9 @@ A hypothetical new repodata revision `3` would need to present the following `in One alternative would be to create new `repodata.json` filenames (e.g. `repodata.v4.json`) for each new incompatible bump. However, this was rejected by the authors as it introduces complexity in other areas: -* It would require more HTTP calls to retrieve the latest version served by the channel. -* It would introduce duplication across `repodata.json` versions and their shards. -* Indexing tools would need to maintain the different versions of `repodata.json` in sync. +- It would require more HTTP calls to retrieve the latest version served by the channel. +- It would introduce duplication across `repodata.json` versions and their shards. +- Indexing tools would need to maintain the different versions of `repodata.json` in sync. ## Copyright From e65a5c5903829332c08e71f106420a85bb8640fa Mon Sep 17 00:00:00 2001 From: jaimergp Date: Wed, 6 May 2026 00:04:30 +0200 Subject: [PATCH 11/19] Make info.repodata_revisions a dict --- cep-XXXX.md | 38 ++++++++++++++++++-------------------- 1 file changed, 18 insertions(+), 20 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 52b2ed46..2d0cad5f 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -11,7 +11,7 @@ Created Jan 12, 2026 - Updated May 5, 2026 + Updated May 6, 2026 Discussion https://github.com/conda/ceps/pull/146 Implementation N/A Requires https://github.com/conda/ceps/pull/164, https://github.com/conda/ceps/pull/165, https://github.com/conda/ceps/pull/166, https://github.com/conda/ceps/pull/151 @@ -31,7 +31,7 @@ This document proposes a set of updates to `repodata.json` files and its sharded The main problem is the introduction of backwards incompatible changes. The obvious solution is to bump the `repodata_version` field (like it was done with [CEP 15](./cep-0015.md)). However, this is not desirable for existing channels, since it immediately prevents non-compatible clients from interacting with the channel. Since most clients would update via a new version available in the channel, it creates a chicken-and-egg problem that would significantly delay the introduction of new features and hinder adoption. -There must be a strategy to introduce backwards incompatible changes without breaking existing channels. This CEP centralizes the discussion for the update strategy and consolidates that feedback into a concrete proposal for `v3`. +There must be a strategy to introduce backwards incompatible changes without breaking existing channels. This CEP centralizes the discussion for the update strategy by introducing the concept of _revisions_ and consolidates that feedback into a concrete proposal for `v3`. The `v3` update includes breaking changes in CEP CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#164](https://github.com/conda/ceps/pull/166), and [#151](https://github.com/conda/ceps/pull/151), that wouldn't otherwise reach existing channels without disrupting the user experience for outdated clients. @@ -39,8 +39,20 @@ The `v3` update includes breaking changes in CEP CEP PRs [#164](https://github.c This CEP introduces two new keys: -- A top-level `v3` key - A `repodata_revisions` key under the top-level `info` dictionary +- A top-level `v3` key + +### The `info.repodata_revisions` key + +This key MUST map to a dictionary where: + +- Each key MUST correspond to a newly introduced top-level key with syntax `vN`, where `N` MUST be `3` or a larger integer. +- Each value MUST be a dictionary with the following optional key-value pairs. Additional keys SHOULD be ignored. + - `n_packages: int | None`: If present and not `None`, it MUST match the sum of all the resulting records found under the `vN` key in the current repodata file or shard. + - `oldest: int | None`: If present and not `None`, a timestamp (in milliseconds) that MUST match the oldest record published in this revision in the current repodata file or shard. + - `newest: int | None`: If present and not `None`, a timestamp (in milliseconds) that MUST match the newest record published in this revision in the current repodata file or shard. + +The `info.repodata_version` value MUST be `1` or, if [CEP 15](./cep-0015.md) applies, `2`. ### The `v3` top-level key @@ -51,19 +63,6 @@ This key MUST map to a dictionary where: - Each key MUST be the artifact filename without its extension - Each value MUST be a valid CEP 36 "package record metadata" dictionary, optionally extended with CEP PR [#151](https://github.com/conda/ceps/pull/151)'s `url` key and/or the keys introduced in `index.json`'s `schema_version: 3` update (CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#164](https://github.com/conda/ceps/pull/166)). -### The `info.repodata_revisions` key - -A repodata _revision_ introduces backwards incompatible features in a way that does not disrupt existing metadata. - -The new `info.repodata_revisions` key maps to an array that MUST list the revisions found in the repodata file or shard as dictionaries with the following schema: - -- `revision: int`: Required. The integer identifying the revision. It MUST be 3 or larger. It MUST be used in the corresponding file top-level key, with syntax `v{revision}`. For the `v3` update proposed in this CEP, the value is `3`. -- `n_packages: int`: Required. The number of packages available in this revision in the current `repodata.json` or shard. -- `oldest: int | None`: Optional. The timestamp (in milliseconds) of the oldest record published in this revision in the current `repodata.json` or shard. If set to `None` or missing, the timestamp information is not available. -- `newest: int | None`: Optional. The timestamp (in milliseconds) of the newest record published in this revision in the current `repodata.json` or shard. If set to `None` or missing, the timestamp information is not available. - -The `repodata_version` MUST be `1` or, if [CEP 15](./cep-0015.md) applies, `2`. - ## Rationale ### Version metadata in `info` @@ -91,14 +90,13 @@ Hence, we suggest to stick to `repodata_version: 1` and _only_ use `repodata_ver "repodata_version": 1, "info": { "subdir": "noarch", - "repodata_revisions": [ - { - "revision": 3, + "repodata_revisions": { + "v3": { "n_packages": 2, "oldest": 1768249989851, "newest": 1773851561010, } - ] + } }, "packages": { "example-1.0.0-0.tar.bz2": { From 0763db0e8cf3f3a855ec94dc5e3ee24416a38c53 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Wed, 6 May 2026 21:49:00 +0200 Subject: [PATCH 12/19] Wording --- cep-XXXX.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 2d0cad5f..40d416c6 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -11,7 +11,7 @@ Created Jan 12, 2026 - Updated May 6, 2026 + Updated May 7, 2026 Discussion https://github.com/conda/ceps/pull/146 Implementation N/A Requires https://github.com/conda/ceps/pull/164, https://github.com/conda/ceps/pull/165, https://github.com/conda/ceps/pull/166, https://github.com/conda/ceps/pull/151 @@ -23,17 +23,17 @@ ## Abstract -This document proposes a set of updates to `repodata.json` files and its sharded derivatives to include the improvements introduced in CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#166](https://github.com/conda/ceps/pull/166), and [#151](https://github.com/conda/ceps/pull/151). To do so in a backwards compatible manner, it also proposes a revision system as a way to extend repodata with new variants, _without_ incrementing `repodata_version`. +This document proposes a set of updates to `repodata.json` files and its sharded derivatives to include the improvements introduced in CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#166](https://github.com/conda/ceps/pull/166), and [#151](https://github.com/conda/ceps/pull/151). To do so in a backwards compatible manner, it also proposes a revision system as a way to extend repodata with new additions, _without_ incrementing `repodata_version`. ## Motivation -`repodata.json` files are central to the conda ecosystem. They are the main source of packaging metadata and inform solvers about the catalog of available packages and their dependency constraints. As such, innovation work often refrains from modifying it, and the format itself has seen very few changes over its lifetime. However, a few ongoing efforts will inevitably result in `repodata.json` modifications (conditional dependencies, optional dependency groups, non-conda dependencies, etc). +`repodata.json` files are central to the conda ecosystem. They are the main source of packaging metadata and inform solvers about the catalog of available packages and their dependency constraints. As such, innovation work often refrains from modifying it, and the format itself has seen very few changes over its lifetime. However, a few ongoing as (of the first half of 2026) efforts will inevitably result in `repodata.json` modifications (conditional dependencies, optional dependency groups, non-conda dependencies, etc). The main problem is the introduction of backwards incompatible changes. The obvious solution is to bump the `repodata_version` field (like it was done with [CEP 15](./cep-0015.md)). However, this is not desirable for existing channels, since it immediately prevents non-compatible clients from interacting with the channel. Since most clients would update via a new version available in the channel, it creates a chicken-and-egg problem that would significantly delay the introduction of new features and hinder adoption. There must be a strategy to introduce backwards incompatible changes without breaking existing channels. This CEP centralizes the discussion for the update strategy by introducing the concept of _revisions_ and consolidates that feedback into a concrete proposal for `v3`. -The `v3` update includes breaking changes in CEP CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#164](https://github.com/conda/ceps/pull/166), and [#151](https://github.com/conda/ceps/pull/151), that wouldn't otherwise reach existing channels without disrupting the user experience for outdated clients. +The `v3` update includes breaking changes in CEP CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#166](https://github.com/conda/ceps/pull/166), and [#151](https://github.com/conda/ceps/pull/151), that wouldn't otherwise reach existing channels without disrupting the user experience for outdated clients. ## Specification @@ -56,12 +56,14 @@ The `info.repodata_version` value MUST be `1` or, if [CEP 15](./cep-0015.md) app ### The `v3` top-level key -This key MUST map to a dictionary where: +This key MUST map to a dictionary of type `dict[str, dict]`. + +More precisely, each key MUST be a non-empty string. The key SHOULD represent the file extension (without the leading period) of the included artifacts (usually `tar.bz2` and `conda`). + +Each value MUST be a dictionary of type `dict[str, dict]` where: -- Each key SHOULD represent the file extension (without the leading period) of the included artifacts. These are usually `tar.bz2` and `conda`. -- Each value MUST be a dictionary where: - - Each key MUST be the artifact filename without its extension - - Each value MUST be a valid CEP 36 "package record metadata" dictionary, optionally extended with CEP PR [#151](https://github.com/conda/ceps/pull/151)'s `url` key and/or the keys introduced in `index.json`'s `schema_version: 3` update (CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#164](https://github.com/conda/ceps/pull/166)). +- Each key MUST be a non-empty string representing the artifact filename without its extension. +- Each value MUST be a valid CEP 36 "package record metadata" dictionary, optionally extended with CEP PR [#151](https://github.com/conda/ceps/pull/151)'s `url` key and/or the keys introduced in `index.json`'s `schema_version: 3` update (CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#166](https://github.com/conda/ceps/pull/166)). ## Rationale From 2b37cd2164bf55f8e80aad61b94cd4c4f347b86c Mon Sep 17 00:00:00 2001 From: jaimergp Date: Wed, 6 May 2026 21:59:20 +0200 Subject: [PATCH 13/19] Add more rejected ideas --- cep-XXXX.md | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+) diff --git a/cep-XXXX.md b/cep-XXXX.md index 40d416c6..2a89eb0f 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -178,6 +178,83 @@ One alternative would be to create new `repodata.json` filenames (e.g. `repodata - It would introduce duplication across `repodata.json` versions and their shards. - Indexing tools would need to maintain the different versions of `repodata.json` in sync. +### Nested `name`-`version`-`build_string` dictionaries + +A review comment proposed subdividing the dictionary of filenames to records in nested dictionaries. In other words, instead of listing the artifact `package-version-build.conda` as: + +```js +{ + ..., + "v3": { + "conda": { + "example-2.0.0-0": { + ... // record dictionary + } + } + } +} +``` + +... expose it as: + +```js +{ + ..., + "v3": { + "conda": { // extension + "example": { // name + "2.0.0": // version + "0": { // build string + ... // record dictionary + } + } + } + } + } +} +``` + +This would simplify parsing of records, but would also complicate the enumeration of existing records. + +### Expose `MatchSpec` entries in records as dictionaries + +MatchSpec parsing in `depends` and other record fields has a non-negligible cost that may add up over thousands of records. Instead of keeping these fields as `list[str]` where each string is a `MatchSpec` expression, it was suggested to expose each entry as a dictionary of `MatchSpec` fields. + +For example, for this input `index.json`: + +```js +{ + "build": "0", + "build_number": 0, + "depends": [ + "package[version=3,build_number=0,when=__unix]" + ], + "name": "example", + "noarch": "generic", + "subdir": "noarch", + "version": "3.0.0", +} +``` + +... `depends` would adopt this form: + +```js +{ + ..., + "depends": [ + { + "name": "package", + "version": "=3", + "build_number": 0, + "when": "__unix" + } + ], + ... +} +``` + +This would involve bigger changes in the implementations, delaying the adoption of `v3`. + ## Copyright All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/). From 968a49d44016f421e78d421f2b6082efd65ac441 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Wed, 6 May 2026 22:16:37 +0200 Subject: [PATCH 14/19] Bring in `indexed_timestamp` too --- cep-XXXX.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 2a89eb0f..511f812e 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -14,7 +14,7 @@ Updated May 7, 2026 Discussion https://github.com/conda/ceps/pull/146 Implementation N/A - Requires https://github.com/conda/ceps/pull/164, https://github.com/conda/ceps/pull/165, https://github.com/conda/ceps/pull/166, https://github.com/conda/ceps/pull/151 + Requires https://github.com/conda/ceps/pull/164, https://github.com/conda/ceps/pull/165, https://github.com/conda/ceps/pull/166, https://github.com/conda/ceps/pull/151, https://github.com/conda/ceps/pull/154 > The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119][RFC2119] when, and only when, they appear in all capitals, as shown here. @@ -23,7 +23,7 @@ ## Abstract -This document proposes a set of updates to `repodata.json` files and its sharded derivatives to include the improvements introduced in CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#166](https://github.com/conda/ceps/pull/166), and [#151](https://github.com/conda/ceps/pull/151). To do so in a backwards compatible manner, it also proposes a revision system as a way to extend repodata with new additions, _without_ incrementing `repodata_version`. +This document proposes a set of updates to `repodata.json` files and its sharded derivatives to include the improvements introduced in CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#166](https://github.com/conda/ceps/pull/166), [#154](https://github.com/conda/ceps/pull/154), and [#151](https://github.com/conda/ceps/pull/151). To do so in a backwards compatible manner, it also proposes a revision system as a way to extend repodata with new additions, _without_ incrementing `repodata_version`. ## Motivation @@ -33,7 +33,7 @@ The main problem is the introduction of backwards incompatible changes. The obvi There must be a strategy to introduce backwards incompatible changes without breaking existing channels. This CEP centralizes the discussion for the update strategy by introducing the concept of _revisions_ and consolidates that feedback into a concrete proposal for `v3`. -The `v3` update includes breaking changes in CEP CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#166](https://github.com/conda/ceps/pull/166), and [#151](https://github.com/conda/ceps/pull/151), that wouldn't otherwise reach existing channels without disrupting the user experience for outdated clients. +The `v3` update includes breaking changes in CEP CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#166](https://github.com/conda/ceps/pull/166), [#154](https://github.com/conda/ceps/pull/154), and [#151](https://github.com/conda/ceps/pull/151), that wouldn't otherwise reach existing channels without disrupting the user experience for outdated clients. ## Specification @@ -49,8 +49,8 @@ This key MUST map to a dictionary where: - Each key MUST correspond to a newly introduced top-level key with syntax `vN`, where `N` MUST be `3` or a larger integer. - Each value MUST be a dictionary with the following optional key-value pairs. Additional keys SHOULD be ignored. - `n_packages: int | None`: If present and not `None`, it MUST match the sum of all the resulting records found under the `vN` key in the current repodata file or shard. - - `oldest: int | None`: If present and not `None`, a timestamp (in milliseconds) that MUST match the oldest record published in this revision in the current repodata file or shard. - - `newest: int | None`: If present and not `None`, a timestamp (in milliseconds) that MUST match the newest record published in this revision in the current repodata file or shard. + - `oldest: int | None`: If present and not `None`, a timestamp (in milliseconds) that MUST match the `indexed_timestamp` field of the oldest record published in this revision in the current repodata file or shard. + - `newest: int | None`: If present and not `None`, a timestamp (in milliseconds) that MUST match the `indexed_timestamp` field of the newest record published in this revision in the current repodata file or shard. The `info.repodata_version` value MUST be `1` or, if [CEP 15](./cep-0015.md) applies, `2`. @@ -63,7 +63,7 @@ More precisely, each key MUST be a non-empty string. The key SHOULD represent th Each value MUST be a dictionary of type `dict[str, dict]` where: - Each key MUST be a non-empty string representing the artifact filename without its extension. -- Each value MUST be a valid CEP 36 "package record metadata" dictionary, optionally extended with CEP PR [#151](https://github.com/conda/ceps/pull/151)'s `url` key and/or the keys introduced in `index.json`'s `schema_version: 3` update (CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#166](https://github.com/conda/ceps/pull/166)). +- Each value MUST be a valid CEP 36 "package record metadata" dictionary, optionally extended with the fields introduced by CEP PRs [#151](https://github.com/conda/ceps/pull/151), [#154](https://github.com/conda/ceps/pull/154), and [#165](https://github.com/conda/ceps/pull/165). The relevant `MatchSpec` strings MAY include changes introduced by `index.json`'s `schema_version: 3` (PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), and [#166](https://github.com/conda/ceps/pull/166)). ## Rationale @@ -145,7 +145,8 @@ Hence, we suggest to stick to `repodata_version: 1` and _only_ use `repodata_ver "sha256": "eb65e866067865793b981c2ba74485f75bef441842b5998badc4ec66717685c7", "size": 1234, "subdir": "noarch", - "timestamp": 1768249989851, + "timestamp": 1768249940850, + "indexed_at": 1768249989851, "version": "2.0.0", }, "example-3.0.0-0": { // key does not have the extension anymore @@ -160,7 +161,8 @@ Hence, we suggest to stick to `repodata_version: 1` and _only_ use `repodata_ver "sha256": "74485f75bef441842b5998badc4ec66717685c7eb65e866067865793b981c2ba", "size": 2345, "subdir": "noarch", - "timestamp": 1773851561010, + "timestamp": 1773851540030, + "indexed_at": 1773851561010, "version": "3.0.0", } } @@ -216,7 +218,7 @@ A review comment proposed subdividing the dictionary of filenames to records in This would simplify parsing of records, but would also complicate the enumeration of existing records. -### Expose `MatchSpec` entries in records as dictionaries +### Expose `MatchSpec` entries in records as dictionaries MatchSpec parsing in `depends` and other record fields has a non-negligible cost that may add up over thousands of records. Instead of keeping these fields as `list[str]` where each string is a `MatchSpec` expression, it was suggested to expose each entry as a dictionary of `MatchSpec` fields. From da6f17a8b8d6b6aedbb4eaec60f7ce1fc905e698 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Mon, 8 Jun 2026 14:43:46 +0200 Subject: [PATCH 15/19] Small precision --- cep-XXXX.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 511f812e..f8f068d9 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -23,7 +23,7 @@ ## Abstract -This document proposes a set of updates to `repodata.json` files and its sharded derivatives to include the improvements introduced in CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#166](https://github.com/conda/ceps/pull/166), [#154](https://github.com/conda/ceps/pull/154), and [#151](https://github.com/conda/ceps/pull/151). To do so in a backwards compatible manner, it also proposes a revision system as a way to extend repodata with new additions, _without_ incrementing `repodata_version`. +This document proposes a set of updates to `repodata.json` files and its derivatives (sharded, subsets) to include the improvements introduced in CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#166](https://github.com/conda/ceps/pull/166), [#154](https://github.com/conda/ceps/pull/154), and [#151](https://github.com/conda/ceps/pull/151). To do so in a backwards compatible manner, it also proposes a revision system as a way to extend repodata with new additions, _without_ incrementing `repodata_version`. ## Motivation From c094e29693cb10f6429e829cd4b5eb28659e0aca Mon Sep 17 00:00:00 2001 From: jaimergp Date: Tue, 23 Jun 2026 12:59:53 -0400 Subject: [PATCH 16/19] Update references to PRs that are now CEPs, remove `url`, add `message` field, rework specification schema explainer, restrict which type of MatchSpec is allowed --- cep-XXXX.md | 57 ++++++++++++++++++++++++----------------------------- 1 file changed, 26 insertions(+), 31 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index f8f068d9..8c64d9f6 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -11,10 +11,10 @@ Created Jan 12, 2026 - Updated May 7, 2026 + Updated Jun 21, 2026 Discussion https://github.com/conda/ceps/pull/146 Implementation N/A - Requires https://github.com/conda/ceps/pull/164, https://github.com/conda/ceps/pull/165, https://github.com/conda/ceps/pull/166, https://github.com/conda/ceps/pull/151, https://github.com/conda/ceps/pull/154 + Requires https://github.com/conda/ceps/pull/151 > The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119][RFC2119] when, and only when, they appear in all capitals, as shown here. @@ -23,17 +23,17 @@ ## Abstract -This document proposes a set of updates to `repodata.json` files and its derivatives (sharded, subsets) to include the improvements introduced in CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#166](https://github.com/conda/ceps/pull/166), [#154](https://github.com/conda/ceps/pull/154), and [#151](https://github.com/conda/ceps/pull/151). To do so in a backwards compatible manner, it also proposes a revision system as a way to extend repodata with new additions, _without_ incrementing `repodata_version`. +This document proposes a set of updates to `repodata.json` files and its derivatives (sharded, subsets) to include the improvements introduced in CEP [43](./cep-0043.md), [44](./cep-0044.md), and [45](./cep-0045.md), as well as PR [#151](https://github.com/conda/ceps/pull/151). To do so in a backwards compatible manner, it also proposes a revision system as a way to extend repodata with new additions, _without_ incrementing `repodata_version`. ## Motivation -`repodata.json` files are central to the conda ecosystem. They are the main source of packaging metadata and inform solvers about the catalog of available packages and their dependency constraints. As such, innovation work often refrains from modifying it, and the format itself has seen very few changes over its lifetime. However, a few ongoing as (of the first half of 2026) efforts will inevitably result in `repodata.json` modifications (conditional dependencies, optional dependency groups, non-conda dependencies, etc). +`repodata.json` files are central to the conda ecosystem. They are the main source of packaging metadata and inform solvers about the catalog of available packages and their dependency constraints. As such, innovation work often refrains from modifying it, and the format itself has seen very few changes over its lifetime. However, the adoption of a few ongoing (as of the first half of 2026) efforts will inevitably result in `repodata.json` modifications (conditional dependencies, optional dependency groups, non-conda dependencies, etc). The main problem is the introduction of backwards incompatible changes. The obvious solution is to bump the `repodata_version` field (like it was done with [CEP 15](./cep-0015.md)). However, this is not desirable for existing channels, since it immediately prevents non-compatible clients from interacting with the channel. Since most clients would update via a new version available in the channel, it creates a chicken-and-egg problem that would significantly delay the introduction of new features and hinder adoption. There must be a strategy to introduce backwards incompatible changes without breaking existing channels. This CEP centralizes the discussion for the update strategy by introducing the concept of _revisions_ and consolidates that feedback into a concrete proposal for `v3`. -The `v3` update includes breaking changes in CEP CEP PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), [#166](https://github.com/conda/ceps/pull/166), [#154](https://github.com/conda/ceps/pull/154), and [#151](https://github.com/conda/ceps/pull/151), that wouldn't otherwise reach existing channels without disrupting the user experience for outdated clients. +The `v3` update includes breaking changes in CEPs [43](./cep-0043.md), [44](./cep-0044.md), and [45](./cep-0045.md), and PR [#151](https://github.com/conda/ceps/pull/151), that wouldn't otherwise reach existing channels without disrupting the user experience for outdated clients. ## Specification @@ -48,22 +48,28 @@ This key MUST map to a dictionary where: - Each key MUST correspond to a newly introduced top-level key with syntax `vN`, where `N` MUST be `3` or a larger integer. - Each value MUST be a dictionary with the following optional key-value pairs. Additional keys SHOULD be ignored. + - `message: str | None`: If present and not `None`, a free-form string to be set by channel operators at their convenience. Its length measured in Unicode characters MUST NOT exceed 8192. - `n_packages: int | None`: If present and not `None`, it MUST match the sum of all the resulting records found under the `vN` key in the current repodata file or shard. - - `oldest: int | None`: If present and not `None`, a timestamp (in milliseconds) that MUST match the `indexed_timestamp` field of the oldest record published in this revision in the current repodata file or shard. - `newest: int | None`: If present and not `None`, a timestamp (in milliseconds) that MUST match the `indexed_timestamp` field of the newest record published in this revision in the current repodata file or shard. + - `oldest: int | None`: If present and not `None`, a timestamp (in milliseconds) that MUST match the `indexed_timestamp` field of the oldest record published in this revision in the current repodata file or shard. The `info.repodata_version` value MUST be `1` or, if [CEP 15](./cep-0015.md) applies, `2`. ### The `v3` top-level key -This key MUST map to a dictionary of type `dict[str, dict]`. - -More precisely, each key MUST be a non-empty string. The key SHOULD represent the file extension (without the leading period) of the included artifacts (usually `tar.bz2` and `conda`). +This key MUST map to a dictionary of type `dict[str, dict]`: -Each value MUST be a dictionary of type `dict[str, dict]` where: - -- Each key MUST be a non-empty string representing the artifact filename without its extension. -- Each value MUST be a valid CEP 36 "package record metadata" dictionary, optionally extended with the fields introduced by CEP PRs [#151](https://github.com/conda/ceps/pull/151), [#154](https://github.com/conda/ceps/pull/154), and [#165](https://github.com/conda/ceps/pull/165). The relevant `MatchSpec` strings MAY include changes introduced by `index.json`'s `schema_version: 3` (PRs [#164](https://github.com/conda/ceps/pull/164), [#165](https://github.com/conda/ceps/pull/165), and [#166](https://github.com/conda/ceps/pull/166)). +- Each key MUST be a non-empty string. The key SHOULD represent the file extension (without the leading period) of the included artifacts (usually `tar.bz2` and `conda`). +- Each value MUST be a dictionary of type `dict[str, dict]` where: + - Each subkey MUST be a non-empty string representing the artifact filename without its extension. + - Each subvalue MUST be a valid [CEP 36](./cep-0036.md) "package record metadata" dictionary, including these changes: + - The `indexed_timestamp` field introduced by CEP PR [#154](https://github.com/conda/ceps/pull/154) SHOULD be set. + - The `extra_depends` field introduced by [44](./cep-0045.md) MAY be present. + - The MatchSpec strings mentioned in the fields `depends`, `constrains` and the lists of strings within `extra_depends` groups: + - MUST set the `name` field to an exact string (no globbing allowed). + - MAY set the fields: `version`, `build`, `build_number`, CEP 43's `when`, CEP 44 `extra`, CEP 45 `flags`. + - MUST NOT set any other fields. + - MUST be represented with the `name` + square-brackets form (e.g. `name[version="1.2.*",build_number=0]`) ## Rationale @@ -133,28 +139,17 @@ Hence, we suggest to stick to `repodata_version: 1` and _only_ use `repodata_ver "v3": { "tar.bz2": {}, "conda": { - "example-2.0.0-0": { // key does not have the extension anymore - "build": "0", - "build_number": 0, - "depends": [ - "package[version=1,build_number=0,when=__unix]" // bracket syntax, w/ conditional - ], - "md5": "82ecc40f09b9c44483e6b70cad2545d7", - "name": "example", - "noarch": "generic", - "sha256": "eb65e866067865793b981c2ba74485f75bef441842b5998badc4ec66717685c7", - "size": 1234, - "subdir": "noarch", - "timestamp": 1768249940850, - "indexed_at": 1768249989851, - "version": "2.0.0", - }, "example-3.0.0-0": { // key does not have the extension anymore "build": "0", "build_number": 0, "depends": [ - "package[version=3,build_number=0,when=__unix]" // bracket syntax, w/ conditional + "package[version=2,build_number=0,when=__unix]" // bracket syntax, w/ conditional ], + "extras_depends": { // NEW + "test": [ + "test-dependency" + ] + }, "md5": "6b70cad2545d782ecc40f09b9c44483e", "name": "example", "noarch": "generic", @@ -162,7 +157,7 @@ Hence, we suggest to stick to `repodata_version: 1` and _only_ use `repodata_ver "size": 2345, "subdir": "noarch", "timestamp": 1773851540030, - "indexed_at": 1773851561010, + "indexed_timestamp": 1773851561010, // NEW "version": "3.0.0", } } From 19c18df73c58d31617ad9797efcbd3413a4940de Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Wed, 22 Jul 2026 09:53:22 +0200 Subject: [PATCH 17/19] Use `extras` instead of `extra` Co-authored-by: Dan Yeaw --- cep-XXXX.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 8c64d9f6..889395dc 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -67,7 +67,7 @@ This key MUST map to a dictionary of type `dict[str, dict]`: - The `extra_depends` field introduced by [44](./cep-0045.md) MAY be present. - The MatchSpec strings mentioned in the fields `depends`, `constrains` and the lists of strings within `extra_depends` groups: - MUST set the `name` field to an exact string (no globbing allowed). - - MAY set the fields: `version`, `build`, `build_number`, CEP 43's `when`, CEP 44 `extra`, CEP 45 `flags`. + - MAY set the fields: `version`, `build`, `build_number`, CEP 43 `when`, CEP 44 `extras`, CEP 45 `flags`. - MUST NOT set any other fields. - MUST be represented with the `name` + square-brackets form (e.g. `name[version="1.2.*",build_number=0]`) From dbb552736d60a90f54f18d177c6a6a1e3c03574e Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sun, 26 Jul 2026 12:38:58 +0200 Subject: [PATCH 18/19] Incorporate latest feedback --- cep-XXXX.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 889395dc..ce4ad523 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -11,10 +11,9 @@ Created Jan 12, 2026 - Updated Jun 21, 2026 + Updated Jul 26, 2026 Discussion https://github.com/conda/ceps/pull/146 Implementation N/A - Requires https://github.com/conda/ceps/pull/151 > The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119][RFC2119] when, and only when, they appear in all capitals, as shown here. @@ -23,17 +22,17 @@ ## Abstract -This document proposes a set of updates to `repodata.json` files and its derivatives (sharded, subsets) to include the improvements introduced in CEP [43](./cep-0043.md), [44](./cep-0044.md), and [45](./cep-0045.md), as well as PR [#151](https://github.com/conda/ceps/pull/151). To do so in a backwards compatible manner, it also proposes a revision system as a way to extend repodata with new additions, _without_ incrementing `repodata_version`. +This document proposes a set of updates to `repodata.json` files and its derivatives (sharded, subsets) to include the improvements introduced in CEP [43](./cep-0043.md), [44](./cep-0044.md), [45](./cep-0045.md), and [47](./cep-0047.md). To do so in a backwards compatible manner, it also proposes a revision system as a way to extend repodata with new additions, _without_ incrementing `repodata_version`. ## Motivation -`repodata.json` files are central to the conda ecosystem. They are the main source of packaging metadata and inform solvers about the catalog of available packages and their dependency constraints. As such, innovation work often refrains from modifying it, and the format itself has seen very few changes over its lifetime. However, the adoption of a few ongoing (as of the first half of 2026) efforts will inevitably result in `repodata.json` modifications (conditional dependencies, optional dependency groups, non-conda dependencies, etc). +`repodata.json` files are central to the conda ecosystem. They are the main source of packaging metadata and inform solvers about the catalog of available packages and their dependency constraints. As such, innovation work often refrains from modifying it, and the format itself has seen very few changes over its lifetime. However, the adoption of CEPs 43, 44, 45 and 47 will inevitably result in `repodata.json` modifications (conditional dependencies, optional dependency groups, non-conda dependencies, etc). The main problem is the introduction of backwards incompatible changes. The obvious solution is to bump the `repodata_version` field (like it was done with [CEP 15](./cep-0015.md)). However, this is not desirable for existing channels, since it immediately prevents non-compatible clients from interacting with the channel. Since most clients would update via a new version available in the channel, it creates a chicken-and-egg problem that would significantly delay the introduction of new features and hinder adoption. There must be a strategy to introduce backwards incompatible changes without breaking existing channels. This CEP centralizes the discussion for the update strategy by introducing the concept of _revisions_ and consolidates that feedback into a concrete proposal for `v3`. -The `v3` update includes breaking changes in CEPs [43](./cep-0043.md), [44](./cep-0044.md), and [45](./cep-0045.md), and PR [#151](https://github.com/conda/ceps/pull/151), that wouldn't otherwise reach existing channels without disrupting the user experience for outdated clients. +The `v3` update includes breaking changes in CEPs [43](./cep-0043.md), [44](./cep-0044.md), [45](./cep-0045.md), and [47](./cep-0047.md) that wouldn't otherwise reach existing channels without disrupting the user experience for outdated clients. ## Specification @@ -48,7 +47,7 @@ This key MUST map to a dictionary where: - Each key MUST correspond to a newly introduced top-level key with syntax `vN`, where `N` MUST be `3` or a larger integer. - Each value MUST be a dictionary with the following optional key-value pairs. Additional keys SHOULD be ignored. - - `message: str | None`: If present and not `None`, a free-form string to be set by channel operators at their convenience. Its length measured in Unicode characters MUST NOT exceed 8192. + - `message: str | None`: If present and not `None`, a free-form string to be set by channel operators at their convenience. Its length MUST NOT exceed 8192 bytes. - `n_packages: int | None`: If present and not `None`, it MUST match the sum of all the resulting records found under the `vN` key in the current repodata file or shard. - `newest: int | None`: If present and not `None`, a timestamp (in milliseconds) that MUST match the `indexed_timestamp` field of the newest record published in this revision in the current repodata file or shard. - `oldest: int | None`: If present and not `None`, a timestamp (in milliseconds) that MUST match the `indexed_timestamp` field of the oldest record published in this revision in the current repodata file or shard. @@ -63,13 +62,14 @@ This key MUST map to a dictionary of type `dict[str, dict]`: - Each value MUST be a dictionary of type `dict[str, dict]` where: - Each subkey MUST be a non-empty string representing the artifact filename without its extension. - Each subvalue MUST be a valid [CEP 36](./cep-0036.md) "package record metadata" dictionary, including these changes: - - The `indexed_timestamp` field introduced by CEP PR [#154](https://github.com/conda/ceps/pull/154) SHOULD be set. + - The `indexed_timestamp` field introduced by [CEP 47](./cep-0047.md) SHOULD be set. - The `extra_depends` field introduced by [44](./cep-0045.md) MAY be present. - The MatchSpec strings mentioned in the fields `depends`, `constrains` and the lists of strings within `extra_depends` groups: - MUST set the `name` field to an exact string (no globbing allowed). - MAY set the fields: `version`, `build`, `build_number`, CEP 43 `when`, CEP 44 `extras`, CEP 45 `flags`. - MUST NOT set any other fields. - - MUST be represented with the `name` + square-brackets form (e.g. `name[version="1.2.*",build_number=0]`) + - If only `name` is set, the MatchSpec MUST be the bare name string (e.g. `pip`). Empty brackets (e.g. `pip[]`) MUST NOT be used. + - If any field other than `name` is set, the MatchSpec MUST use the `name` + square-brackets form (e.g. `name[version="1.2.*",build_number=0]`). ## Rationale From 5d52ddbd11050a628b2b4b695d2e40047515af77 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sun, 26 Jul 2026 12:46:27 +0200 Subject: [PATCH 19/19] Cleanup --- cep-XXXX.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index ce4ad523..f3f06f25 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -22,7 +22,7 @@ ## Abstract -This document proposes a set of updates to `repodata.json` files and its derivatives (sharded, subsets) to include the improvements introduced in CEP [43](./cep-0043.md), [44](./cep-0044.md), [45](./cep-0045.md), and [47](./cep-0047.md). To do so in a backwards compatible manner, it also proposes a revision system as a way to extend repodata with new additions, _without_ incrementing `repodata_version`. +This document proposes a set of updates to `repodata.json` files and its derivatives (sharded, subsets) to include the improvements introduced in CEP [43](./cep-0043.md), [44](./cep-0044.md), [45](./cep-0045.md), and [47](./cep-0047.md). To do so in a backwards compatible manner, it also proposes a revision system as a way to extend repodata with new additions, _without_ incrementing `repodata_version`. ## Motivation @@ -38,8 +38,8 @@ The `v3` update includes breaking changes in CEPs [43](./cep-0043.md), [44](./ce This CEP introduces two new keys: -- A `repodata_revisions` key under the top-level `info` dictionary -- A top-level `v3` key +- A `repodata_revisions` key under the top-level `info` dictionary. +- A top-level `v3` key. ### The `info.repodata_revisions` key @@ -47,7 +47,7 @@ This key MUST map to a dictionary where: - Each key MUST correspond to a newly introduced top-level key with syntax `vN`, where `N` MUST be `3` or a larger integer. - Each value MUST be a dictionary with the following optional key-value pairs. Additional keys SHOULD be ignored. - - `message: str | None`: If present and not `None`, a free-form string to be set by channel operators at their convenience. Its length MUST NOT exceed 8192 bytes. + - `message: str | None`: If present and not `None`, a free-form string to be set by channel operators to provide contextual information (e.g., announcements, deprecation notices, or a change in client requirements). Its length MUST NOT exceed 8192 bytes. - `n_packages: int | None`: If present and not `None`, it MUST match the sum of all the resulting records found under the `vN` key in the current repodata file or shard. - `newest: int | None`: If present and not `None`, a timestamp (in milliseconds) that MUST match the `indexed_timestamp` field of the newest record published in this revision in the current repodata file or shard. - `oldest: int | None`: If present and not `None`, a timestamp (in milliseconds) that MUST match the `indexed_timestamp` field of the oldest record published in this revision in the current repodata file or shard. @@ -61,9 +61,9 @@ This key MUST map to a dictionary of type `dict[str, dict]`: - Each key MUST be a non-empty string. The key SHOULD represent the file extension (without the leading period) of the included artifacts (usually `tar.bz2` and `conda`). - Each value MUST be a dictionary of type `dict[str, dict]` where: - Each subkey MUST be a non-empty string representing the artifact filename without its extension. - - Each subvalue MUST be a valid [CEP 36](./cep-0036.md) "package record metadata" dictionary, including these changes: + - Each subvalue MUST be a valid [CEP 36](./cep-0036.md) "package record metadata" dictionary, including these changes: - The `indexed_timestamp` field introduced by [CEP 47](./cep-0047.md) SHOULD be set. - - The `extra_depends` field introduced by [44](./cep-0045.md) MAY be present. + - The `extra_depends` field introduced by [44](./cep-0044.md) MAY be present. - The MatchSpec strings mentioned in the fields `depends`, `constrains` and the lists of strings within `extra_depends` groups: - MUST set the `name` field to an exact string (no globbing allowed). - MAY set the fields: `version`, `build`, `build_number`, CEP 43 `when`, CEP 44 `extras`, CEP 45 `flags`. @@ -89,7 +89,7 @@ Bumps in this number should only result in backwards incompatible changes that w - `conda` only supports it as of [v24.5.0](https://github.com/conda/conda/blob/main/CHANGELOG.md#2450-2024-05-08) (released on 2024-05-08) - `mamba` started supporting it in [v2.0](https://github.com/mamba-org/mamba/blob/main/CHANGELOG.md#20240925) (released on 2024-09-25). -Hence, we suggest to stick to `repodata_version: 1` and _only_ use `repodata_version: 2` when a new channel needs a global `base_url` for all the entries in the `packages` and `packages.conda` fields. +Hence, we recommend sticking to `repodata_version: 1` and only using `repodata_version: 2` when a new channel needs a global `base_url` for all the entries in the `packages` and `packages.conda` fields. ## Example @@ -102,7 +102,7 @@ Hence, we suggest to stick to `repodata_version: 1` and _only_ use `repodata_ver "v3": { "n_packages": 2, "oldest": 1768249989851, - "newest": 1773851561010, + "newest": 1773851561010 } } }, @@ -145,7 +145,7 @@ Hence, we suggest to stick to `repodata_version: 1` and _only_ use `repodata_ver "depends": [ "package[version=2,build_number=0,when=__unix]" // bracket syntax, w/ conditional ], - "extras_depends": { // NEW + "extra_depends": { // NEW "test": [ "test-dependency" ] @@ -158,7 +158,7 @@ Hence, we suggest to stick to `repodata_version: 1` and _only_ use `repodata_ver "subdir": "noarch", "timestamp": 1773851540030, "indexed_timestamp": 1773851561010, // NEW - "version": "3.0.0", + "version": "3.0.0" } } } @@ -229,7 +229,7 @@ For example, for this input `index.json`: "name": "example", "noarch": "generic", "subdir": "noarch", - "version": "3.0.0", + "version": "3.0.0" } ```