CEP XXXX: Conditional dependencies, extras and flags

cep-XXX1.md

@@ -0,0 +1,136 @@
+# CEP XXXX - Conditional dependencies
+
+<table>
+<tr><td> Title </td><td> Conditional dependencies </td>
+<tr><td> Status </td><td> Draft </td></tr>
+<tr><td> Author(s) </td><td>
+  Wolf Vollprecht &lt;w.vollprecht@gmail.com&gt;,
+  Bas Zalmstra &lt;bas@prefix.dev&gt;,
+  Jaime Rodríguez-Guerra &lt;jaime.rogue@gmail.com&gt;
+</td></tr>
+<tr><td> Created </td><td> Feb 5, 2025</td></tr>
+<tr><td> Updated </td><td> Apr 27, 2026</td></tr>
+<tr><td> Discussion </td><td> https://github.com/conda/ceps/pull/111 </td></tr>
+<tr><td> Implementation </td><td> NA </td></tr>
+<tr><td> Requires </td><td> https://github.com/conda/ceps/pull/146 </td></tr>
+</table>
+
+## Abstract
+
+This CEP proposes the introduction of conditional dependencies in the conda ecosystem. The work comprises two changes: the addition of a new keyword argument, `when`, to the `MatchSpec` syntax, and its evaluation at solve time. The value of the `when` argument is always a logical expression of one or more MatchSpec queries targeting the solver context.
+
+## Motivation
+
+Conditional dependencies are often requested in the conda ecosystem. After all, an equivalent feature exists in Python packaging in the form of "environment markers" (introduced in [PEP 508](https://peps.python.org/pep-0508/)), so it's a reasonable expectation. There are even traces of planned support of it in `conda.models.match_spec` (see commit [`ab33436`](https://github.com/conda/conda/pull/7606/changes/ab33436591cd2f8ce9c792a50a412d1d02c75504)) in the context of ["pip interoperability"](https://github.com/conda/conda/issues/7053).
+
+The feature never landed though, and its absence in the conda ecosystem has forced packagers to (inefficiently) translate this expressivity into a set of different package variants which only differ in their dependency metadata. For example, consider the following runtime requirements of a pure Python package:
+
+```toml
+# ...
+requires_python = ">=3.8"
+requirements = [
+  "typing-extensions; python_version < '3.9'",
+  "pywin32; sys_platform == 'win32'"
+  "requests",
+]
+```
+
+They can only be converted into conda packages by making a potentially `noarch: python` package (because it's pure Python) a non-noarch package distributed as as many as 25 platforms (five supported Python versions, times five platforms). This way, the conditional dependency is resolved (in a way) at build time by rendering the dependencies into the granular artifacts.
+For example, the `typing-extensions` dependency would only be present in the variants built for Python 3.8, and we would only find `pywin32` in the packages published to the `win-*` subdirs.
+
+The adoption of conditional dependencies in conda would allow packagers to express the same complexity in a single `noarch: python` package, because these conditions would be evaluated at "solve time". Providing these capabilities requires new `MatchSpec` syntax and a bump in the repodata version to allow the new syntax in the `depends` field.
+
+## Rationale
+
+The syntax chosen utilizes the same key-value syntax already present in the brackets form of the `MatchSpec` language. This makes it easier to implement and less surprising to novel users. The `when` keyword is chosen to avoid confusion with the `if`/`then`/`else` syntax in [CEP 13](./cep-0013.md) `recipe.yaml` files.
+
+Classic `MatchSpec` queries have traditionally matched fields present in candidate package records. For example, `python>=3.10` would match records with package name `python` and version greater or equal than `3.10`. The `when` field is a bit different in that sense, since it doesn't target a `when` field in the target records.
+Instead, its value is a condition that will match a context comprised of (potentially) other records. This is a deviation for the `MatchSpec` design but we still feel that the ergonomics offered by this syntax are worth the extra complexity.
+
+## Specification
+
+This CEP extends [CEP 29](./cep-0029.md) with a new keyword, `when`.
+
+### Syntax
+
+A conditional dependency is defined as a `MatchSpec` string that features a `when` keyword, the value of which MUST be a string that encodes a logical expression of one or more `MatchSpec` queries. The logical expression follows a Python-like syntax: `MatchSpec` strings MAY be joined with operators `and` (logical AND) and `or` (logical OR), and grouped within parentheses `()` for precedence overrides.
+
+The inner `MatchSpec` queries inside `when` MUST be expressed in their square brackets syntax, with the exception of simple `name` and `version` queries that MAY be expressed as `{name}{operator}{version}` (no space separators). These inner `MatchSpec` queries MUST NOT feature their own `when` field.
+
+The `when` value MUST be quoted to avoid parsing ambiguities.
+
+### Evaluation
+
+A conditional dependency MUST only be taken into account when the `when` condition evaluates to true. Otherwise, it MUST be ignored by the solver.
+
+A condition MUST evaluate to true when the logical concatenation of the `MatchSpec` queries is truthy. Each `MatchSpec` expression would evaluate to true if:
+
+- It matches a package record potentially present in the proposed solution.
+- It matches a virtual package present in the target system.
+- For `run_exports` only: it matches a field injected in the evaluation context.
+
+### Impact in `info/*.json`
+
+This is a backwards incompatible change. To guarantee backwards compatibility, the CEP 34 `info/index.json` `schema_version` field MUST be bumped to `3`. When present in `run_exports.json`, its `schema_version` value MUST be bumped to `2`.
+
+## Examples
+
+Conditional dependencies can be used in the same places a regular `MatchSpec` expression is used.
+
+As CLI input:
+
+```shell
+conda create -n env-with-conditions python 'numpy>=2[when="python>=3.10"]'
+```
+
+In an input file:
+
+```yaml
+name: env-with-conditions
+channels: [conda-forge]
+dependencies:
+- python
+- numpy>=2[when="python>=3.10"]
+```
+
+```toml
+[dependencies]
+python = "*"
+numpy = { version=">=2", when="python>=3.10" }
+```
+
+In a recipe file:
+
+```yaml:
+# ...
+requirements:
+  run:
+    - python
+    - numpy>=2[when="python>=3.10"]
+```
+
+## Backwards compatibility
+
+Adding a new field to the `MatchSpec` keywords is a backwards incompatible change. For example, in `conda`:
+
+```shell
+$ conda create -d "python[when=__unix]"
+InvalidMatchSpec: Invalid spec 'python[when=python]': Invalid spec 'python[when=python]': Cannot match on field(s): {'when'}
+```
+
+When used as part of the runtime requirements of a newly built package, the conditional dependency must be included in the `depends` field of the resulting repodata record. Since this new syntax is backwards incompatible with older clients, the resulting `repodata.json` documents (or a derivative, like the sharded forms) MUST NOT include records of packages whose `info/index.json` features a value of `schema_version` equals to or greater than 3.
+
+## Rejected ideas
+
+- `...; if ...` syntax, and other backwards compatible proposals, are rejected because the condition would be ignored by solvers which do not implement this CEP, resulting in invalid solutions.
+
+## References
+
+- <https://github.com/conda/conda/issues/2984>
+- <https://github.com/conda/conda/issues/7438>
+- <https://github.com/conda/conda/issues/7439>
+- <https://github.com/conda/conda/issues/5699>
+
+## Copyright
+
+All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/).

cep-XXX2.md

@@ -0,0 +1,104 @@
+# CEP XXXX - Optional dependency groups
+
+<table>
+<tr><td> Title </td><td> Optional dependency groups </td>
+<tr><td> Status </td><td> Draft </td></tr>
+<tr><td> Author(s) </td><td>
+  Wolf Vollprecht &lt;w.vollprecht@gmail.com&gt;,
+  Bas Zalmstra &lt;bas@prefix.dev&gt;,
+  Jaime Rodríguez-Guerra &lt;jaime.rogue@gmail.com&gt;
+</td></tr>
+<tr><td> Created </td><td> Feb 5, 2025</td></tr>
+<tr><td> Updated </td><td> Apr 27, 2026</td></tr>
+<tr><td> Discussion </td><td> https://github.com/conda/ceps/pull/111 </td></tr>
+<tr><td> Implementation </td><td> NA </td></tr>
+<tr><td> Requires </td><td> https://github.com/conda/ceps/pull/146 </td></tr>
+</table>
+
+## Abstract
+
+This CEP proposes the introduction of optional dependency groups in conda packaging. Optional dependency groups (or "extras") allow packagers to specify additional requirements behind a named set. This is implemented by adding a new repodata record field, `extras`, as a dictionary that maps group names to a list of dependencies. Extras can be selected via the corresponding `MatchSpec` keyword matching one or more group names.
+
+## Motivation
+
+Optional dependency groups are popular in Python packaging, initially introduced as part of `setuptools` via `extras_require`, and formalized years later in PEP 508.
+
+The need in conda packaging can be also identified by analyzing certain patterns in the ecosystem:
+
+- The `{package}-with-{group-name}` pattern can be found in projects like `ray` or `arrow`.
+- The `{package}` + `{package}-base` pattern was introduced for `matplotlib`.
+- Some projects choose to add all extras in the base package to avoid the complexity of having to add more outputs to their recipe.
+
+Instead of overloading the repodata documents with more package names (which causes additional solver strain and works against optimization efforts like sharded repodata), authors propose to extend the repodata records with the additional dependency metadata, thus reducing unnecessary duplication.
+
+## Rationale
+
+The chosen keyword in CEP 29 `MatchSpec`, keywords is `extras` given its popularity in Python packaging. For CEP 34 `info/index.json` records is `extra-depends` to avoid ambiguity with a hypothetical `extra-constraints`.
+
+## Specification
+
+The `info/index.json` dictionary of each conda artifact MUST allow a new field, `extra-depends`, the value of which MUST be a dictionary that maps the group names (expressed as a string that MUST match the regex `[a-z0-9_.-+]{1,64}`) to a list of `MatchSpec` strings encoding the optional dependencies. Since this is backwards incompatible behavior-wise, the `schema_version` value MUST be bumped to `3`.
+
+The `extras` field MUST also be exposed in recipe formats v1 and above, under the `requirements.extras` section (sibling to `build`, `host`, `run`, etc), that renders to the same schema (`dict[str, list[MatchSpec]]`).
+
+This new field must be selectable by `MatchSpec` syntax using the `extras` keyword inside square brackets, the value of which must be a string or a list of string targeting group names. If an optional dependency group is matched by the `extras` field in `MatchSpec`, its list of dependencies MUST be unioned with the regular `depends` field and resolved accordingly.
+
+## Examples
+
+A repodata record with `extra-depends`:
+
+```js
+{
+  // ...,
+  "packages.conda": {
+    // ...,
+    "example-1.0-0.conda": {
+      // ...,
+      "depends": [
+        "main-dependency"
+      ]
+      "extra-depends": {
+        "group-name": [
+          "extra-dependency>=2",
+          "another-dependency>=1"
+        ]
+      }
+    }
+  }
+}
+```
+
+A `MatchSpec` query targeting the above record:
+
+```shell
+example[extras="group-name"]
+```
+
+A recipe defining an output named `example` with a `group-name` extras:
+
+```yaml
+# ...
+requirements:
+  ...
+  run:
+    - main-dependency
+  extras:
+    group-name:
+      - extra-dependency >=2
+      - another-dependency >=1
+# ...
+```
+
+## Rejected ideas
+
+PEP 735 "optional dependency groups" are not covered in this CEP. Those are not part of the installation metadata, but this CEP proposes groups that are indeed part of it.
+
+## References
+
+- <https://packaging.python.org/en/latest/specifications/dependency-specifiers/#extras>
+- [conda/conda#1696: Optional dependencies for conda packages](https://github.com/conda/conda/issues/1696), [conda/conda#7502: Optional groups of dependencies](https://github.com/conda/conda/issues/7502)
+- <https://github.com/conda/rattler/pull/1542>
+
+## Copyright
+
+All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/).

cep-XXX3.md

@@ -0,0 +1,100 @@
+# CEP XXXX - Simplified variant selection
+
+<table>
+<tr><td> Title </td><td> Simplified variant selection </td>
+<tr><td> Status </td><td> Draft </td></tr>
+<tr><td> Author(s) </td><td>
+  Wolf Vollprecht &lt;w.vollprecht@gmail.com&gt;,
+  Bas Zalmstra &lt;bas@prefix.dev&gt;,
+  Jaime Rodríguez-Guerra &lt;jaime.rogue@gmail.com&gt;
+</td></tr>
+<tr><td> Created </td><td> Feb 5, 2025</td></tr>
+<tr><td> Updated </td><td> Apr 27, 2026</td></tr>
+<tr><td> Discussion </td><td> https://github.com/conda/ceps/pull/111 </td></tr>
+<tr><td> Implementation </td><td> NA </td></tr>
+<tr><td> Requires </td><td> https://github.com/conda/ceps/pull/146 </td></tr>
+</table>
+
+## Abstract
+
+This CEP proposes a simplified mechanism to select package variants without relying on partial build string globbing. A package carries a list of plain string flags; a solver constraint that names a flag excludes any package that does not carry it. This mechanism involves adding a new `flags` repodata record field along with the corresponding `MatchSpec` extensions.
+
+## Motivation
+
+Within a channel and subdir, a conda package for a given project release may have different builds or artifacts. These are used to either correct deficiencies in the build process, or to provide different build alternatives (e.g. linking to diverse library backends). These artifacts can be distinguished by their build string, which can be matched with glob strings.
+
+In principle, the solver is responsible for choosing the right variant for a given package version. However, the user may also want to force a particular variant to satisfy their needs. The solution so far has been to plant a special substring in the build string so it can be selected by the corresponding glob string. For example, given a package with a CPU and a GPU variant (`package-1.0-h123abc_cpu_0.conda` and `package-1.0-h453cbd_gpu_0.conda`), the GPU variant can be chosen by asking for `package=*=*gpu*`.
+
+The problem with this approach is that it doesn't scale well for more than one "flag" per build string. What if a package needs to distinguish among more than one feature? That is, not just GPU vs CPU, but also BLAS backend, MPI or licensing?
+The glob strings are not expressive enough for a single spec, so several ones need to be supplied (e.g. `package=*=*gpu*`, `package=*=*mkl*`, `package=*=*mpich*` and `package=*=*nogpl*`), resulting in complicated lookahead regexes that balloon in computational complexity.
+
+The answer to this problem is to provide a specific field that is designed to provide such selection capabilities without the expressivity and complexity problems observed above: `flags`.
+
+## Rationale
+
+The chosen keyword is `flags` which makes sense as a "compile time flag". This feature is mainly relevant for compiled packages with different compile time feature selection, making `flags` a matching name.
+
+## Specification
+
+### Repodata record syntax
+
+The `info/index.json` file of each conda artifact MUST support two new fields, `flags`.
+
+The value of the `flags` field MUST be a list of non-empty strings matching the regex `^[a-z0-9_]+(:[a-z0-9_]+)?$`. We allow a _single_ `:` for `key:value` semantics.
+
+Subsequently, the CEP 34 `info/index.json`'s `schema_version` value MUST be bumped to `3`.
+
+In recipes, these values MUST be supported by a `flags` key in the `build` section of each output (i.e. sibling to `number` and `track_features`) with a list of strings as the value; i.e. `build.flags = [str]`.
+
+### MatchSpec syntax changes
+
+Values in the `flags` field MUST be matchable by the corresponding keyword in `MatchSpec`, placed in the square brackets section. Its value MUST be a string or list of strings. Each entry MUST match the regex `^[a-z0-9_\*]+(:[a-z0-9_\*]+)?$`. The `*` character is a glob operator, with the same meaning as in CEP 29 "String Matching".
+
+Flag matching is intentionally simple: a package is excluded from consideration if it does not contain every flag listed in the `flags` constraint. A flag either matches the string (exactly or globbed) or the package is filtered out.
+
+### `index.json` and `repodata_record` changes
+
+The records gain a new `flags`:
+
+```json
+{
+  "name": "foobar",
+  "version": "1.2.3",
+  ...,
+  "flags": ["cuda", "release", "blas:mkl"],
+}
+```
+
+### Changes to the `recipe.yaml` file
+
+```yaml
+build:
+  string: ...
+  flags: ["cuda", "blas:mkl", "release"]
+```
+
+## Examples
+
+In practice, this would look like the following from a user perspective:
+
+```shell
+conda install 'pytorch[version=">=3.1", flags=["cuda", "blas:*"]]'
+```
+
+Any package that does not carry both the `cuda` and a flag starting with `blas:` is excluded from the candidate set. Among the remaining candidates the solver applies its normal version and build-number and variant order preference sorting.
+
+## Rejected ideas
+
+This proposal may remind the readers of the old `features` properties in the first iterations of conda packaging. This is not a reimplementation.
+
+## Future plans
+
+Several extensions are deferred to future revisions:
+
+- Richer flag matching: numeric values, key-value items, and comparison operators (`>`, `<`) for numeric matches.
+- Additional operators: `?` for "match if present" and `!` to exclude a flag.
+- Variant prioritization: a `variant_priority` field to disambiguate among many matching variants. We could not agree on the sort order, so this is deferred for further discussion.
+
+## Copyright
+
+All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/).