From abfb21f4ec10680bf4d97f29b379fc8f7ea64700 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Fri, 29 Aug 2025 15:10:16 +1100 Subject: [PATCH 01/39] first draft of CEP for new export machinery --- cep-XXXX.md | 216 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 216 insertions(+) create mode 100644 cep-XXXX.md diff --git a/cep-XXXX.md b/cep-XXXX.md new file mode 100644 index 00000000..3e341d49 --- /dev/null +++ b/cep-XXXX.md @@ -0,0 +1,216 @@ +# CEP XXXX - Improving dependency export infrastructure + + + + + + + + +
Title Improving dependency export infrastructure
Status Draft
Author(s) Axel Obermeier <h.vetinari@gmx.com>
Created Aug 29, 2025
Discussion https://github.com/conda/ceps/pull/tbd
Implementation N/A
+ +## Abstract + +This CEP proposes to overhaul the way that packages "export" some required dependency +or constraint, in a way that considers both the source environment during the build phase +(one of `build:`, `host:` and `run:`) as well as the target environment where the respective +dependency or constraint should be applied. This has a wide variety of important uses which +are so far difficult or impossible to express, in addition to increasing clarity. + +This document builds upon [CEP 13](cep-0013.md) and [CEP 14](cep-0014.md), which define what +is known as the "v1" recipe format. + +## History + +From the beginning, one of the big advantages of `conda` over `pip` has been the ability to +track dependencies sufficiently well in order to be able to share artefacts between packages, +rather than vendoring dependent libraries for every consumer. + +In contrast to static or header-only libraries, a shared library `foo` occurs not just during build +time of `bar` (the headers for `# include ` to work and `libfoo.so` for linkage to succeed), +but need to be present as well in the final environment for `bar` as well. Aside from corner-cases, +this runtime dependency *always* follows, and it is therefore natural to ask the build system to do +this work for us. + +This is what the original [design discussion](https://github.com/conda/conda-build/issues/1142) started +out from, which was first implemented first as [`pin_downstream`](https://github.com/conda/conda-build/commit/e344bbae369658ca7e2defab8a3960d8570fbf8a) +and soon after renamed to [`run_exports`](https://github.com/conda/conda-build/commit/d90aa3135cc81a5db28e8160b521f11d27083453). +The original [documentation](https://github.com/conda/conda-docs/pull/414) for this feature provides +additional context. + +This yielded a feature that at the time of conda-build 3.0.0 (CB3) +[looked like](https://github.com/conda/conda-build/blob/3.0.0/tests/test-recipes/variants/10_runtimes/uses_run_exports/meta.yaml): + +```yaml +build: + run_exports: + - foo {{ pin_compatible('foo') }} + +requirements: + build: + - foo +``` + +It's worth noting that the introduction of "variants" and "run_exports" happened during the same CB3 timeframe +which also saw the introduction of the separation into `build:`/`host:`/`run:` environments (necessary to correctly +handle more complex scenarios like cross-compilation). As well as can be gleaned from the history of the repository, +it appears that the design for run-exports did not originally take this new tripartite environment separation into +account. + +This was [rectified](https://github.com/conda/conda-build/commit/f7133b61c75ba1b5c82b5fb20729a37d05ae28a5) +by introducing a separation into strong and weak run-exports, where the scenario described above is default, in +which a `host:` dependency exports a dependency to `run:`, whereas "strong" exports are necessary for a `build:` +dependency to have the same effect (in some way, the jump between `build:` and `run:` is farther than the one +between `host:` and `run:`, and hence needs a bigger push). + +However, this doesn't cover the full gamut of scenarios how packages may influence or constrain each other +between environments, and opens further questions along the lines of: +> If `qux` has a strong run-export, and appears in a `run:` environment of another package, + does the export trigger or not? + +Later still, the constraint variants `weak_constrains` and `strong_constrains` were +[added](https://github.com/conda/conda-build/pull/4125). + +The v1 recipe format moved the `run_exports:` key from the `build:` section to the `requirements:` of the +respective output, but otherwise did not change the semantics of this feature. + +## Motivation + +One key motivation for this proposal is that even with the weak/strong distinction, run-exports are not +powerful enough to handle relevant scenarios that are a natural consequence of the separation into +`build:` / `host:` / `run:` environments. + +Abstractly speaking, there are cases where one wants to express relations between environments, e.g. +from `build:` to `host:` (see https://github.com/conda/ceps/issues/77). In many ways the urgency of this +need was reduced by the fact that it could be passably emulated by using strong run-exports; while this +would "over-export" things into the `run:` environment, this is harmless in many cases. + +However, there are cases where that is not so, in particular, C++ and Fortran modules (as of 2025) are +not portable between compilers and need to be consumed by the same compiler that produced them. + +Assuming we have a package `foo-devel` containing Fortran modules, to which we would like to attach a +`_fortran_modules_abi =*=compiler_flavour*` constraint that ensures that they can only be consumed by +the approprate compiler. The problem in this case is that + +```yaml + - name: i-consume-fortran-modules + requirements: + build: + - {{ stdlib("c") }} + - {{ compiler("fortran") }} + host: + - foo-devel +``` + +there's no way to make the "wrong" fortran compiler conflict with `foo-devel`, because we explicitly +do not want a strong run-export from the general-purpose `{{ compiler("fortran") }}` to enforce a specific +compiler ABI in `run:` (making the package unusable with other compilers unnecessarily). + +The solution in this case would be to add an export to `{{ compiler("fortran") }}` that injects +`_fortran_modules_abi =*=compiler_flavour*` *only* into the `host:` environment; this would impose the +right constraints (i.e. conflict if ABI between the compiler and the contrainst attached to `foo-devel` +doesn't match), while avoiding too-tight constraints at runtime. The situation is explained/discussed +in more detail in https://github.com/conda-forge/conda-forge.github.io/issues/2525. + +## Design + +We begin with the following observations based on the above: + +- There is a need for a flexible mechanism to do cross-environment dependency injection in the conda ecosystem. +- There are cases where the "run" in `run_exports:` is not appropriate (e.g. `build:` to `host:`), because + no aspect of the export in question involves something happening at "run"time. +- Just as important as where we're exporting *to* is where we're exporting *from*. Leaving this implicit + gets exponentially more complicated the more export-flavours there are. +- Already the existing strong run-exports can be subject to this confusion (e.g. "does a strong run-export + trigger in a host environment?"), [even](https://github.com/conda/ceps/issues/77#issuecomment-3187310320) + by very experienced contributors. +- It would be nice to cover constraints as well as dependencies with the same pattern. +- There are other use-cases ([example](https://github.com/conda-forge/ctng-compiler-activation-feedstock/blob/e2bdf15eb170008eda386056a900ce93e0f9cb16/recipe/meta.yaml#L147-L150)) + which have so far been under-served by the existing run-export infrastructure. +- The v1 recipe format unified all requirement-related topics (including run-exports) under `requirements:`. + +Based on this, we propose the following pattern: + +``` +requirements: + build: + - [...] + host: + - [...] + run: + - [...] + # relying on the surrounding "requirements" key for context + exports: + host_to_run: # matches weak run-export + - a_shared_library + build_to_host: # "host-export" + - a_build_constraint =*=*foo + build_to_run: # produces same effect as strong run-export when used together with build_to_host + - a_compiler_runtime + host_to_constraints: # matches weak_constrains + - a_run_constraint + build_to_constraints: # matches strong_constrains + - a_run_constraint + host_to_host: # see below + - a_transitive_dependency + build_to_build: # see below + - a_transitive_dependency +``` +As indicated by the comments, `host_to_run:` matches the existing weak run-export. If taken together with +`build_to_run:` this produces the same effect of a strong run-export. Similarly for `host_to_constraints:` +and `build_to_constraints:`. The other keys introduce new functionality, which is explained below. + +Before explaining the transitive case, we note though that this design has the advantage that it's +immediately clear from the key pattern `_to_` under which conditions a given export +triggers (i.e. the package carrying the export finds itself in an environment matching ``), +and what it influences (i.e. the export gets added to ``). This avoids a lot of mental arithmetic +(and ideally, implementation complexity) in keeping track of which export triggers when. + +The case for `build_to_host:` was already made in the Motivation section, though there are other cases +beyond C++/Fortran modules where the ability to constrain interactions between compilers and host variants +is desirable (e.g. openmp, openmpi, etc.). + +The most surprising additions might be `host_to_host:` and `build_to_build:`. It would be natural to ask +why whatever is being exported in such a manner could not be a direct (run-)dependency of the package. +The answer is that there may be transitive dependencies *at compilation time* that we do not want +consumers to inherit at runtime (similar to the situation with the Fortan modules ABI). + +An example of this is if a library `foo` depends on the headers of another library `bar` at compile-time, +but we do not want packages built atop of `foo` to carry along those `bar` headers (because at that point +they're not needed anymore; the concern is about a compile-time quantity, which only concerns either +`build:` and/or `host:`). + +Additionally, to keep the `_to_` pattern, we rename +```yaml +requirements: + constraints: # renamed from run_constraints + - [...] +``` +because constraints only make sense when that package gets installed somewhere in any case, so the +"run_" is superfluous (aside from being inconsistent with the proposed schema). On top of that, the +"run_" is also confusing, because if a package with `run_constraints:` gets installed into a `host:` +or `build:` environment, those constraints will still take effect. + +Likewise we rename `ignore_run_exports` +```yaml + requirements: + ignore_exports: # renamed from ignore_run_exports + from_package: + - zlib + by_name: + - libzlib +``` +which should ignore exports from any matching export (whether by source or by name of the export), +regardless of where the export comes from. + +## Specification + +TODO! + +### Open questions + +- if we change the keys, do we need to bump the recipe version to v2? +- do we want to specify that implementations need to automatically map `run_exports:` to the corresponding newer keys? +- do we need repodata changes to represent the new, more granular, export metadata? +- how does this interact with sharded repodata (c.f. [CEP 21](cep-0021.md))? +- do we need to specify that installers need to be updated to not choke on unknown keys? From e8853831116e35c3e0b7725574f1d831e91c590b Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Fri, 29 Aug 2025 04:16:30 +0000 Subject: [PATCH 02/39] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- cep-XXXX.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 3e341d49..14d43fc1 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -81,7 +81,7 @@ powerful enough to handle relevant scenarios that are a natural consequence of t `build:` / `host:` / `run:` environments. Abstractly speaking, there are cases where one wants to express relations between environments, e.g. -from `build:` to `host:` (see https://github.com/conda/ceps/issues/77). In many ways the urgency of this +from `build:` to `host:` (see ). In many ways the urgency of this need was reduced by the fact that it could be passably emulated by using strong run-exports; while this would "over-export" things into the `run:` environment, this is harmless in many cases. @@ -90,7 +90,7 @@ not portable between compilers and need to be consumed by the same compiler that Assuming we have a package `foo-devel` containing Fortran modules, to which we would like to attach a `_fortran_modules_abi =*=compiler_flavour*` constraint that ensures that they can only be consumed by -the approprate compiler. The problem in this case is that +the appropriate compiler. The problem in this case is that ```yaml - name: i-consume-fortran-modules @@ -110,7 +110,7 @@ The solution in this case would be to add an export to `{{ compiler("fortran") } `_fortran_modules_abi =*=compiler_flavour*` *only* into the `host:` environment; this would impose the right constraints (i.e. conflict if ABI between the compiler and the contrainst attached to `foo-devel` doesn't match), while avoiding too-tight constraints at runtime. The situation is explained/discussed -in more detail in https://github.com/conda-forge/conda-forge.github.io/issues/2525. +in more detail in . ## Design @@ -156,6 +156,7 @@ requirements: build_to_build: # see below - a_transitive_dependency ``` + As indicated by the comments, `host_to_run:` matches the existing weak run-export. If taken together with `build_to_run:` this produces the same effect of a strong run-export. Similarly for `host_to_constraints:` and `build_to_constraints:`. The other keys introduce new functionality, which is explained below. @@ -173,7 +174,7 @@ is desirable (e.g. openmp, openmpi, etc.). The most surprising additions might be `host_to_host:` and `build_to_build:`. It would be natural to ask why whatever is being exported in such a manner could not be a direct (run-)dependency of the package. The answer is that there may be transitive dependencies *at compilation time* that we do not want -consumers to inherit at runtime (similar to the situation with the Fortan modules ABI). +consumers to inherit at runtime (similar to the situation with the Fortran modules ABI). An example of this is if a library `foo` depends on the headers of another library `bar` at compile-time, but we do not want packages built atop of `foo` to carry along those `bar` headers (because at that point @@ -181,17 +182,20 @@ they're not needed anymore; the concern is about a compile-time quantity, which `build:` and/or `host:`). Additionally, to keep the `_to_` pattern, we rename + ```yaml requirements: constraints: # renamed from run_constraints - [...] ``` + because constraints only make sense when that package gets installed somewhere in any case, so the "run_" is superfluous (aside from being inconsistent with the proposed schema). On top of that, the "run_" is also confusing, because if a package with `run_constraints:` gets installed into a `host:` or `build:` environment, those constraints will still take effect. Likewise we rename `ignore_run_exports` + ```yaml requirements: ignore_exports: # renamed from ignore_run_exports @@ -200,6 +204,7 @@ Likewise we rename `ignore_run_exports` by_name: - libzlib ``` + which should ignore exports from any matching export (whether by source or by name of the export), regardless of where the export comes from. From 73e6004d2b0908e9a6c57be81649226ae10fddf6 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Fri, 29 Aug 2025 15:18:15 +1100 Subject: [PATCH 03/39] add missing language --- cep-XXXX.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 14d43fc1..44760355 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -131,7 +131,7 @@ We begin with the following observations based on the above: Based on this, we propose the following pattern: -``` +```yaml requirements: build: - [...] From dd093ece8329258e5940b24bc8c44a3bd5d4d695 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sat, 30 Aug 2025 07:55:12 +1100 Subject: [PATCH 04/39] explain lack of `run_to_run:` --- cep-XXXX.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 44760355..753caea9 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -5,7 +5,7 @@ Status Draft Author(s) Axel Obermeier <h.vetinari@gmx.com> Created Aug 29, 2025 - Discussion https://github.com/conda/ceps/pull/tbd + Discussion https://github.com/conda/ceps/pull/129 Implementation N/A @@ -181,11 +181,15 @@ but we do not want packages built atop of `foo` to carry along those `bar` heade they're not needed anymore; the concern is about a compile-time quantity, which only concerns either `build:` and/or `host:`). +This is also why no `run_to_run:` key is proposed here -- dependency exports address constraints arising +from compilation. At runtime, when the build process is long past, the situation simplifies back to +the question whether another package is a dependency or not. + Additionally, to keep the `_to_` pattern, we rename ```yaml requirements: - constraints: # renamed from run_constraints + constraints: # changed from run_constraints - [...] ``` From 28dd02c04c91ac939787e729ee658533d892c00e Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sat, 30 Aug 2025 08:04:00 +1100 Subject: [PATCH 05/39] clarify ignore_exports; minor fixes --- cep-XXXX.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 753caea9..d3ca1132 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -90,7 +90,7 @@ not portable between compilers and need to be consumed by the same compiler that Assuming we have a package `foo-devel` containing Fortran modules, to which we would like to attach a `_fortran_modules_abi =*=compiler_flavour*` constraint that ensures that they can only be consumed by -the appropriate compiler. The problem in this case is that +the appropriate compiler. The problem in this case is that with a recipe like ```yaml - name: i-consume-fortran-modules @@ -108,7 +108,7 @@ compiler ABI in `run:` (making the package unusable with other compilers unneces The solution in this case would be to add an export to `{{ compiler("fortran") }}` that injects `_fortran_modules_abi =*=compiler_flavour*` *only* into the `host:` environment; this would impose the -right constraints (i.e. conflict if ABI between the compiler and the contrainst attached to `foo-devel` +right constraints (i.e. conflict if ABI between the compiler and the constraint attached to `foo-devel` doesn't match), while avoiding too-tight constraints at runtime. The situation is explained/discussed in more detail in . @@ -118,7 +118,7 @@ We begin with the following observations based on the above: - There is a need for a flexible mechanism to do cross-environment dependency injection in the conda ecosystem. - There are cases where the "run" in `run_exports:` is not appropriate (e.g. `build:` to `host:`), because - no aspect of the export in question involves something happening at "run"time. + no aspect of the export in question involves something happening related to the `run:` environment or runtime. - Just as important as where we're exporting *to* is where we're exporting *from*. Leaving this implicit gets exponentially more complicated the more export-flavours there are. - Already the existing strong run-exports can be subject to this confusion (e.g. "does a strong run-export @@ -209,8 +209,8 @@ Likewise we rename `ignore_run_exports` - libzlib ``` -which should ignore exports from any matching export (whether by source or by name of the export), -regardless of where the export comes from. +which should ignore any exports into `run:` or `constraints:` matching the conditions (whether +by source or by name of the export), regardless of where the export comes from. ## Specification From 6797e5ba363381e024445a83b80661222a444626 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sat, 30 Aug 2025 13:15:28 +1100 Subject: [PATCH 06/39] improve motivation section, add subheaders --- cep-XXXX.md | 43 ++++++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 17 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index d3ca1132..6662e4b8 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -28,7 +28,7 @@ rather than vendoring dependent libraries for every consumer. In contrast to static or header-only libraries, a shared library `foo` occurs not just during build time of `bar` (the headers for `# include ` to work and `libfoo.so` for linkage to succeed), -but need to be present as well in the final environment for `bar` as well. Aside from corner-cases, +but needs to be present in the final environment for `bar` as well. Aside from corner-cases, this runtime dependency *always* follows, and it is therefore natural to ask the build system to do this work for us. @@ -72,7 +72,8 @@ Later still, the constraint variants `weak_constrains` and `strong_constrains` w [added](https://github.com/conda/conda-build/pull/4125). The v1 recipe format moved the `run_exports:` key from the `build:` section to the `requirements:` of the -respective output, but otherwise did not change the semantics of this feature. +respective output, but otherwise did not change the semantics of this feature, aside from renaming +`run_constrained` to `run_constraints` and `{weak,strong}_constrains` to `{weak,strong}_constraints`. ## Motivation @@ -80,17 +81,19 @@ One key motivation for this proposal is that even with the weak/strong distincti powerful enough to handle relevant scenarios that are a natural consequence of the separation into `build:` / `host:` / `run:` environments. -Abstractly speaking, there are cases where one wants to express relations between environments, e.g. -from `build:` to `host:` (see ). In many ways the urgency of this -need was reduced by the fact that it could be passably emulated by using strong run-exports; while this -would "over-export" things into the `run:` environment, this is harmless in many cases. +Abstractly speaking, this separation introduces more cases where one wants to express relations between +environments, e.g. from `build:` to `host:` (see ). For this case +in particular, the urgency was reduced by the fact that it could be passably emulated by using strong +run-exports; while this would "over-export" things into the `run:` environment, this is harmless in many cases. -However, there are cases where that is not so, in particular, C++ and Fortran modules (as of 2025) are -not portable between compilers and need to be consumed by the same compiler that produced them. +However, there are cases where that is not so, and the relevant constraints cannot currently be expressed +(resp. where abuse of the strong run-export would require all consumers to ignore the run-exports, which +is not feasible at scale, and would be a constant tripping hazard). -Assuming we have a package `foo-devel` containing Fortran modules, to which we would like to attach a -`_fortran_modules_abi =*=compiler_flavour*` constraint that ensures that they can only be consumed by -the appropriate compiler. The problem in this case is that with a recipe like +For example, C++ and Fortran modules (as of 2025) are not portable between compilers and need to be consumed +by the same compiler that produced them. Assuming we have a package `foo-devel` containing Fortran modules, +to which we would like to attach a `_fortran_modules_abi =*=compiler_flavour*` constraint that ensures that +it can only be combined with the appropriate compiler. The problem in this case is that with a recipe like ```yaml - name: i-consume-fortran-modules @@ -147,9 +150,9 @@ requirements: - a_build_constraint =*=*foo build_to_run: # produces same effect as strong run-export when used together with build_to_host - a_compiler_runtime - host_to_constraints: # matches weak_constrains + host_to_constraints: # matches weak_constrains (v0) / weak_constraints (v1) - a_run_constraint - build_to_constraints: # matches strong_constrains + build_to_constraints: # matches strong_constrains (v0) / strong_constraints (v1) - a_run_constraint host_to_host: # see below - a_transitive_dependency @@ -171,20 +174,26 @@ The case for `build_to_host:` was already made in the Motivation section, though beyond C++/Fortran modules where the ability to constrain interactions between compilers and host variants is desirable (e.g. openmp, openmpi, etc.). +### Transitive compilation requirements + The most surprising additions might be `host_to_host:` and `build_to_build:`. It would be natural to ask why whatever is being exported in such a manner could not be a direct (run-)dependency of the package. The answer is that there may be transitive dependencies *at compilation time* that we do not want consumers to inherit at runtime (similar to the situation with the Fortran modules ABI). An example of this is if a library `foo` depends on the headers of another library `bar` at compile-time, -but we do not want packages built atop of `foo` to carry along those `bar` headers (because at that point -they're not needed anymore; the concern is about a compile-time quantity, which only concerns either -`build:` and/or `host:`). +but we do not want packages built atop of `foo` to carry along those `bar` headers or related version +constraints (because at that point they're not needed anymore; the concern is about a compile-time +quantity, which only concerns either `build:` and/or `host:`). + +### Omitted combinations -This is also why no `run_to_run:` key is proposed here -- dependency exports address constraints arising +The above is also why no `run_to_run:` key is proposed here -- dependency exports address constraints arising from compilation. At runtime, when the build process is long past, the situation simplifies back to the question whether another package is a dependency or not. +### Other modifications + Additionally, to keep the `_to_` pattern, we rename ```yaml From 638aa0d25a00a14ee40336692a4f2fa09628849e Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sat, 30 Aug 2025 13:19:25 +1100 Subject: [PATCH 07/39] explain absence of `host_to_build:` --- cep-XXXX.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/cep-XXXX.md b/cep-XXXX.md index 6662e4b8..28c9311c 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -192,6 +192,13 @@ The above is also why no `run_to_run:` key is proposed here -- dependency export from compilation. At runtime, when the build process is long past, the situation simplifies back to the question whether another package is a dependency or not. +Furthermore, one could ask about a possible `host_to_build:` key. While this would arguably be an even better +fit for the C++/Fortran modules ABI issue described above, the reason this proposal refrains from suggesting +such a key is to limit implementation complexity, by having an implicit order of environment resolution from +`build:` to `host:` to `run:`. Allowing both `host_to_build:` as well as `build_to_*:` would complicate this +process unnecessarily, and we believe the relevant use-cases are fully expressible using `build_to_host:` +together with relevant constraints (such as `_fortran_modules_abi`). + ### Other modifications Additionally, to keep the `_to_` pattern, we rename From 0064162f0a26669d236e1f93ea8d271ec64330d8 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sat, 30 Aug 2025 13:44:14 +1100 Subject: [PATCH 08/39] sort omitted combinations to the back, add summary --- cep-XXXX.md | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 28c9311c..0d64b4bc 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -121,7 +121,7 @@ We begin with the following observations based on the above: - There is a need for a flexible mechanism to do cross-environment dependency injection in the conda ecosystem. - There are cases where the "run" in `run_exports:` is not appropriate (e.g. `build:` to `host:`), because - no aspect of the export in question involves something happening related to the `run:` environment or runtime. + no aspect of the export in question involves something related to the `run:` environment or runtime. - Just as important as where we're exporting *to* is where we're exporting *from*. Leaving this implicit gets exponentially more complicated the more export-flavours there are. - Already the existing strong run-exports can be subject to this confusion (e.g. "does a strong run-export @@ -186,19 +186,6 @@ but we do not want packages built atop of `foo` to carry along those `bar` heade constraints (because at that point they're not needed anymore; the concern is about a compile-time quantity, which only concerns either `build:` and/or `host:`). -### Omitted combinations - -The above is also why no `run_to_run:` key is proposed here -- dependency exports address constraints arising -from compilation. At runtime, when the build process is long past, the situation simplifies back to -the question whether another package is a dependency or not. - -Furthermore, one could ask about a possible `host_to_build:` key. While this would arguably be an even better -fit for the C++/Fortran modules ABI issue described above, the reason this proposal refrains from suggesting -such a key is to limit implementation complexity, by having an implicit order of environment resolution from -`build:` to `host:` to `run:`. Allowing both `host_to_build:` as well as `build_to_*:` would complicate this -process unnecessarily, and we believe the relevant use-cases are fully expressible using `build_to_host:` -together with relevant constraints (such as `_fortran_modules_abi`). - ### Other modifications Additionally, to keep the `_to_` pattern, we rename @@ -226,7 +213,23 @@ Likewise we rename `ignore_run_exports` ``` which should ignore any exports into `run:` or `constraints:` matching the conditions (whether -by source or by name of the export), regardless of where the export comes from. +by originating package or by name of the export), regardless of which environment it comes from. + +### Omitted combinations + +In all cases, dependency exports address constraints or interactions arising from compilation. At runtime, +when the build process is long past, the situation simplifies back to the question whether another package +is a dependency or not, which is why no `run_to_run:` key is proposed here. + +Furthermore, one could ask about a possible `host_to_build:` key. While this would arguably be an even better +fit for the C++/Fortran modules ABI issue described above, the reason this proposal refrains from suggesting +such a key is to limit implementation complexity, by having an implicit order of environment resolution from +`build:` to `host:` to `run:`. Allowing both `host_to_build:` as well as `build_to_*:` would complicate this +process unnecessarily, and we believe the relevant use-cases are fully expressible using `build_to_host:` +together with constraints (such as `_fortran_modules_abi`) attached to packages that appear in `host:`. + +Summing up, `build:` can export to all others (i.e. `build:`, `host:`, `run:`, `constraints:`), `host:` can +export to everything but `build:`, while nothing can be exported from either `run:` or `constraints:`. ## Specification From 8a147e2b62973f94a0f4aff5e9919606ff222b6b Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sat, 30 Aug 2025 14:51:09 +1100 Subject: [PATCH 09/39] cover noarch exports --- cep-XXXX.md | 43 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 36 insertions(+), 7 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 0d64b4bc..4bdddd3d 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -75,6 +75,24 @@ The v1 recipe format moved the `run_exports:` key from the `build:` section to t respective output, but otherwise did not change the semantics of this feature, aside from renaming `run_constrained` to `run_constraints` and `{weak,strong}_constrains` to `{weak,strong}_constraints`. +### Run-exports vs. noarch + +In 2020, a `noarch:` type of run-exports got [added](https://github.com/conda/conda-build/pull/3868); the +design for this seems to come only from comments in that PR and boils down to: +> Ray Donnelly: Do we not really want to use a different run_export type here? Dropping all but the package name? + +> Isuru Fernando: I don't understand. Are you suggesting a different run_exports scheme ('noarch' in addition to +> 'weak', 'strong') that would be applied if the package is a dependency of a `noarch` package? + +> Ray Donnelly: I am. + +This is a relatively little-used feature, c.f. this approximate [search](https://github.com/search?q=org%3Aconda-forge+%2F%28%3Fs%29%5Csrun_exports%3A.*noarch%3A%2F+path%3Arecipe%2F*.yaml&type=code) +(note: many false positives, but should be exhaustive), though crucially including core packages like +python and R, which are of course key use-cases for building noarch consumers on top. + +Perhaps most notably, since the linked PR, regular run-exports (either weak or strong) do not apply to +`noarch: {generic, python}` packages anymore, as thereafter, those received their own special export type. + ## Motivation One key motivation for this proposal is that even with the weak/strong distinction, run-exports are not @@ -144,20 +162,23 @@ requirements: - [...] # relying on the surrounding "requirements" key for context exports: - host_to_run: # matches weak run-export + host_to_run: # matches `weak:` run-export - a_shared_library build_to_host: # "host-export" - a_build_constraint =*=*foo - build_to_run: # produces same effect as strong run-export when used together with build_to_host + build_to_run: # produces same effect as `strong:` run-export when used together with build_to_host - a_compiler_runtime - host_to_constraints: # matches weak_constrains (v0) / weak_constraints (v1) + host_to_constraints: # matches `weak_constrains:` (v0) / `weak_constraints:` (v1) - a_run_constraint - build_to_constraints: # matches strong_constrains (v0) / strong_constraints (v1) + build_to_constraints: # matches `strong_constrains:` (v0) / `strong_constraints:` (v1) - a_run_constraint host_to_host: # see below - a_transitive_dependency build_to_build: # see below - a_transitive_dependency + # all the above _do not_ apply when building `noarch: generic` or `noarch: python` packages + noarch_to_run: # matches `noarch:` run-export; does apply to noarch packages + - a_dependency_exported_when_consumer_is_noarch ``` As indicated by the comments, `host_to_run:` matches the existing weak run-export. If taken together with @@ -174,6 +195,12 @@ The case for `build_to_host:` was already made in the Motivation section, though beyond C++/Fortran modules where the ability to constrain interactions between compilers and host variants is desirable (e.g. openmp, openmpi, etc.). +The `noarch_to_run:` breaks from the pattern of using a `` that is an existing type of environment. +Given how existing run-exports do not apply to noarch packages at all, and how important use-cases like +python require this functionality, we cannot remove it just for the sake of foolish consistency, and this +seems like the most natural way to incorporate it. The overall rule can still be summarized as "exports do +not apply when building noarch packages, unless the export is of type `noarch_to_run:`." + ### Transitive compilation requirements The most surprising additions might be `host_to_host:` and `build_to_build:`. It would be natural to ask @@ -188,7 +215,8 @@ quantity, which only concerns either `build:` and/or `host:`). ### Other modifications -Additionally, to keep the `_to_` pattern, we rename +Additionally, to keep the `_to_` pattern (aside from +the special case for `noarch`), we rename ```yaml requirements: @@ -205,7 +233,7 @@ Likewise we rename `ignore_run_exports` ```yaml requirements: - ignore_exports: # renamed from ignore_run_exports + ignore_exports: # changed from ignore_run_exports from_package: - zlib by_name: @@ -229,7 +257,8 @@ process unnecessarily, and we believe the relevant use-cases are fully expressib together with constraints (such as `_fortran_modules_abi`) attached to packages that appear in `host:`. Summing up, `build:` can export to all others (i.e. `build:`, `host:`, `run:`, `constraints:`), `host:` can -export to everything but `build:`, while nothing can be exported from either `run:` or `constraints:`. +export to everything but `build:`, while nothing can be exported from either `run:` or `constraints:`. None +of these exports apply when building noarch packages, which have a separate `noarch_to_run:` export type. ## Specification From 87245efebae77c0f04b4a3d6847cb2b34041eccf Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sat, 30 Aug 2025 16:11:56 +1100 Subject: [PATCH 10/39] first cut at dealing with metadata --- cep-XXXX.md | 151 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 151 insertions(+) diff --git a/cep-XXXX.md b/cep-XXXX.md index 4bdddd3d..785df2fb 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -260,8 +260,159 @@ Summing up, `build:` can export to all others (i.e. `build:`, `host:`, `run:`, ` export to everything but `build:`, while nothing can be exported from either `run:` or `constraints:`. None of these exports apply when building noarch packages, which have a separate `noarch_to_run:` export type. +## Impacts on package and channel metadata + +### Background + +Both [CEP 12](cep-0012.md) and [CEP 21](cep-0021.md) worked in the area of specifying how run-exports are +represented in package and channel metadata. + +To the best of our knowledge, the format of `run_exports.json` on a per-package level has not been formalized, +though, unsurprisingly, it is [simply](https://github.com/conda/conda-build/blob/25.7.0/conda_build/build.py#L1378-L1387) +a JSON-extract of the relevant `run_exports:` portion of the rendered recipe. +CB3 originally introduced this as `run_exports.yaml`, which got [switched](https://github.com/conda/conda-build/commit/1347f3df264c57d79ab078f88fae2d8862a58d9f) +to JSON (by default) in 2018. In practice, it is fair to assume that only `run_exports.json` files exist nowadays. + +CEP 12 introduced a channel-level `run_exports.json` which provides the information in aggregated format, allowing +extraction of run-export metadata (e.g. for conda-forge's bot infrastructure) at scale without having to download +every individual package first. This effort refrained from touching `repodata.json`, among other reasons because +> It would require extending the `repodata` schema, currently not formally standardized. + +Indeed, inspection of package-level `repodata_record.json` of contemporary (mid-2025) packages shows that run-exports +do not appear in the regular package-metadata. + +CEP 21 (building on top of [CEP 16](cep-0016.md)) added channel-level run-export information, though in contrast +to CEP 12, added this to the physically shareded but logically unified repodata. + +### Transition plan + +It's easy to map the new export structure to the respective metadata, the complexity lies in providing a smooth +transition for the ecosystem across various versions of tools that build or consume packages and metadata. + +The suggested transition approach looks as follows. The intention is to avoid having to introduce a repodata v2, +but if such an [effort](https://github.com/conda/ceps/pull/111) should come to fruition, the below should certainly +be simplified. + +* Package-level: + * Add another `exports.json` next to `repodata_record.json` and `run_exports.json`, to be preferred by tools + which know how to handle it. + * Populate `run_exports.json` with "compatible" metadata +* Channel-level: + * Add another `exports.json` to the monolithic channel metadata, to be preferred by tools who which how to handle it. + * Add the structure of run-exports within sharded metadata, as the same argument to footprint applies as in CEP 21, + i.e. the data is highly compressible and will not have more than ~5% impact. Long-term, the existing `run_exports:` + information should be removed, freeing up the additional space again. + +The reason to add separate files is that this provides the easiest compatibility story: tools which are aware of this +CEP can prefer `exports.json` and equivalents, whereas older versions of these tools continue to work unchanged. + +### Compatibility mapping back to `run_exports.json` + +To smooth the transition, even tools that are aware of this CEP should still populate `run_exports.json` etc., to +avoid breaking behaviour changes in older versions during the transition. For setting the values, we propose a +conservative approach, in the sense that we default to strong exports in case of doubt: + +* replace keys that have a 1:1 equivalent + * `host_to_run:` --> `weak:` + * `host_to_constraints:` --> `weak_constrains:` + * `build_to_constraints:` -- > `strong_constrains:` + * `noarch_to_run:` --> `noarch:` +* use strong run-export in case of doubt + * `build_to_host:` --> `strong:` + * `build_to_run:` --> `strong:` +* do not map unknown keys `host_to_host:` & `build_to_build:` + ## Specification +### Recipes, Parsing, Package Building + +TODO! + +### Package and Channel Metadata + +On package-level, if an output has any `exports:` specified, build tools MUST produce an `exports.json` in the +root of the artefact (next to `index.json` etc.), and populate the following schema with the exports as specified +in the rendered recipe. If the output has no (or empty) `exports:`, populating `exports.json` MAY be omitted. +If not, it MUST still conform to the schema below, with empty lists as values for the respective export keys. + +```json +"exports": { + "build_to_build": [ + "string", + ], + "build_to_constraints": [ + "string", + ], + "build_to_host": [ + "string", + ], + "build_to_run": [ + "string", + ], + "host_to_constraints": [ + "string", + ], + "host_to_host": [ + "string", + ], + "host_to_run": [ + "string", + ], + "noarch_to_run": [ + "string", + ] +} +``` + +On channel-level the `exports.json` file MUST be populated when indexing the channel, in the same way +as for `run_exports.json`: + +``` +{ + "info": { + "platform": "string", + "arch": "string", + "subdir": "string", + "version": 0 + }, + "packages": { + "package-version-build.conda": { # or package-version-build.tar.bz + "exports": { + "build_to_build": [ + "string", + ], + "build_to_constraints": [ + "string", + ], + "build_to_host": [ + "string", + ], + "build_to_run": [ + "string", + ], + "host_to_constraints": [ + "string", + ], + "host_to_host": [ + "string", + ], + "host_to_run": [ + "string", + ], + "noarch_to_run": [ + "string", + ] + } + } + }, +} +``` + +Within the sharded repodata, indexers MUST add an `exports:` key and populate it with the respective +package-level metadata. + +### Patching + TODO! ### Open questions From 7d0786527620050dd0687887fa184c1a0e94bb05 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Sat, 30 Aug 2025 05:12:50 +0000 Subject: [PATCH 11/39] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- cep-XXXX.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 785df2fb..395e0386 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -293,13 +293,13 @@ The suggested transition approach looks as follows. The intention is to avoid ha but if such an [effort](https://github.com/conda/ceps/pull/111) should come to fruition, the below should certainly be simplified. -* Package-level: - * Add another `exports.json` next to `repodata_record.json` and `run_exports.json`, to be preferred by tools +- Package-level: + - Add another `exports.json` next to `repodata_record.json` and `run_exports.json`, to be preferred by tools which know how to handle it. - * Populate `run_exports.json` with "compatible" metadata -* Channel-level: - * Add another `exports.json` to the monolithic channel metadata, to be preferred by tools who which how to handle it. - * Add the structure of run-exports within sharded metadata, as the same argument to footprint applies as in CEP 21, + - Populate `run_exports.json` with "compatible" metadata +- Channel-level: + - Add another `exports.json` to the monolithic channel metadata, to be preferred by tools who which how to handle it. + - Add the structure of run-exports within sharded metadata, as the same argument to footprint applies as in CEP 21, i.e. the data is highly compressible and will not have more than ~5% impact. Long-term, the existing `run_exports:` information should be removed, freeing up the additional space again. @@ -312,15 +312,15 @@ To smooth the transition, even tools that are aware of this CEP should still pop avoid breaking behaviour changes in older versions during the transition. For setting the values, we propose a conservative approach, in the sense that we default to strong exports in case of doubt: -* replace keys that have a 1:1 equivalent - * `host_to_run:` --> `weak:` - * `host_to_constraints:` --> `weak_constrains:` - * `build_to_constraints:` -- > `strong_constrains:` - * `noarch_to_run:` --> `noarch:` -* use strong run-export in case of doubt - * `build_to_host:` --> `strong:` - * `build_to_run:` --> `strong:` -* do not map unknown keys `host_to_host:` & `build_to_build:` +- replace keys that have a 1:1 equivalent + - `host_to_run:` --> `weak:` + - `host_to_constraints:` --> `weak_constrains:` + - `build_to_constraints:` -- > `strong_constrains:` + - `noarch_to_run:` --> `noarch:` +- use strong run-export in case of doubt + - `build_to_host:` --> `strong:` + - `build_to_run:` --> `strong:` +- do not map unknown keys `host_to_host:` & `build_to_build:` ## Specification From e4a5041f7515c1422b72838fdc96de384de5941f Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sat, 30 Aug 2025 16:14:58 +1100 Subject: [PATCH 12/39] make linter happy --- cep-XXXX.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 395e0386..fde7d863 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -79,11 +79,11 @@ respective output, but otherwise did not change the semantics of this feature, a In 2020, a `noarch:` type of run-exports got [added](https://github.com/conda/conda-build/pull/3868); the design for this seems to come only from comments in that PR and boils down to: -> Ray Donnelly: Do we not really want to use a different run_export type here? Dropping all but the package name? - -> Isuru Fernando: I don't understand. Are you suggesting a different run_exports scheme ('noarch' in addition to -> 'weak', 'strong') that would be applied if the package is a dependency of a `noarch` package? - +> > > Ray Donnelly: Do we not really want to use a different run_export type here? Dropping all but the package name? +> > +> > Isuru Fernando: I don't understand. Are you suggesting a different run_exports scheme ('noarch' in addition to +> > 'weak', 'strong') that would be applied if the package is a dependency of a `noarch` package? +> > Ray Donnelly: I am. This is a relatively little-used feature, c.f. this approximate [search](https://github.com/search?q=org%3Aconda-forge+%2F%28%3Fs%29%5Csrun_exports%3A.*noarch%3A%2F+path%3Arecipe%2F*.yaml&type=code) @@ -367,7 +367,7 @@ If not, it MUST still conform to the schema below, with empty lists as values fo On channel-level the `exports.json` file MUST be populated when indexing the channel, in the same way as for `run_exports.json`: -``` +```json { "info": { "platform": "string", From ca1c6a797d06e92d939b87229d6f8f72d6060dde Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sat, 30 Aug 2025 16:49:01 +1100 Subject: [PATCH 13/39] add mapping to specification --- cep-XXXX.md | 49 +++++++++++++++++++++++++++++++++++-------------- 1 file changed, 35 insertions(+), 14 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index fde7d863..80d1b42c 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -258,7 +258,7 @@ together with constraints (such as `_fortran_modules_abi`) attached to packages Summing up, `build:` can export to all others (i.e. `build:`, `host:`, `run:`, `constraints:`), `host:` can export to everything but `build:`, while nothing can be exported from either `run:` or `constraints:`. None -of these exports apply when building noarch packages, which have a separate `noarch_to_run:` export type. +of these exports apply when building noarch packages, which only take into account `noarch_to_run:` exports. ## Impacts on package and channel metadata @@ -282,23 +282,24 @@ Indeed, inspection of package-level `repodata_record.json` of contemporary (mid- do not appear in the regular package-metadata. CEP 21 (building on top of [CEP 16](cep-0016.md)) added channel-level run-export information, though in contrast -to CEP 12, added this to the physically shareded but logically unified repodata. +to CEP 12, added this to the physically sharded but logically unified repodata. ### Transition plan It's easy to map the new export structure to the respective metadata, the complexity lies in providing a smooth transition for the ecosystem across various versions of tools that build or consume packages and metadata. -The suggested transition approach looks as follows. The intention is to avoid having to introduce a repodata v2, -but if such an [effort](https://github.com/conda/ceps/pull/111) should come to fruition, the below should certainly -be simplified. +The approach suggested here is based on the intention to avoid having to introduce a repodata v2, but if such +an [effort](https://github.com/conda/ceps/pull/111) should come to fruition, the below could certainly be +simplified. We suggest to: - Package-level: - Add another `exports.json` next to `repodata_record.json` and `run_exports.json`, to be preferred by tools which know how to handle it. - - Populate `run_exports.json` with "compatible" metadata + - Populate `run_exports.json` with "compatible" metadata derived from `exports:` (see below). - Channel-level: - Add another `exports.json` to the monolithic channel metadata, to be preferred by tools who which how to handle it. + - Indexers should populate `run_exports.json` with "compatible" metadata derived from `exports:` (see below). - Add the structure of run-exports within sharded metadata, as the same argument to footprint applies as in CEP 21, i.e. the data is highly compressible and will not have more than ~5% impact. Long-term, the existing `run_exports:` information should be removed, freeing up the additional space again. @@ -312,15 +313,13 @@ To smooth the transition, even tools that are aware of this CEP should still pop avoid breaking behaviour changes in older versions during the transition. For setting the values, we propose a conservative approach, in the sense that we default to strong exports in case of doubt: -- replace keys that have a 1:1 equivalent +- Re-use values for keys which have a 1:1 equivalent in `run_exports:` schema: - `host_to_run:` --> `weak:` - `host_to_constraints:` --> `weak_constrains:` - `build_to_constraints:` -- > `strong_constrains:` - `noarch_to_run:` --> `noarch:` -- use strong run-export in case of doubt - - `build_to_host:` --> `strong:` - - `build_to_run:` --> `strong:` -- do not map unknown keys `host_to_host:` & `build_to_build:` +- Add `strong:` run-export in case of doubt, i.e. merge any values of `build_to_host:` & `build_to_run:` into `strong:`. +- Do not map keys that have no equivalent in `run_exports:`, i.e. omit `host_to_host:` & `build_to_build:`. ## Specification @@ -364,7 +363,24 @@ If not, it MUST still conform to the schema below, with empty lists as values fo } ``` -On channel-level the `exports.json` file MUST be populated when indexing the channel, in the same way +We define the following translation between this schema and the previous `run_exports:` schema: + +| `exports:` | `run_exports: | +|---|---| +| `build_to_build:` | IGNORED | +| `build_to_constraints:` | `strong_constrains:` | +| `build_to_host:` | `strong:` | +| `build_to_run:` | `strong:` | +| `host_to_constraints:` | `weak_constrains:` | +| `host_to_host:` | IGNORED | +| `host_to_run:` | `weak:` | +| `noarch_to_run:` | `noarch:` | + +Except for cells marked with "IGNORED", build tools MUST populate the output-level `run_exports.json` file +unchanged from the values of `exports:` in the recipe, though duplicates from the merge between `build_to_host:` +and `build_to_run:` MAY be removed. + +On channel-level, the `exports.json` file MUST be populated when indexing the channel, in the same way as for `run_exports.json`: ```json @@ -408,8 +424,13 @@ as for `run_exports.json`: } ``` -Within the sharded repodata, indexers MUST add an `exports:` key and populate it with the respective -package-level metadata. +As for the output-level metadata, indexers MUST populate the channel-level `run_exports.json` in a way that +is consistent with the output-level metadata: either calculated from `exports:` using the above compatibility +mapping, or aggregated from the output-level `run_exports.json`. + +Within the sharded repodata, indexers MUST add an `exports:` key and populate it with the respective output-level +metadata. Furthermore, indexers MUST populate the value `run_exports:` derived from the values of `exports:`, +using the schema mapping specified above. ### Patching From 2833643d5ac0ec2d3191e24e1daa1226cf64de26 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Sat, 30 Aug 2025 05:49:57 +0000 Subject: [PATCH 14/39] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- cep-XXXX.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 80d1b42c..83f92a85 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -313,7 +313,7 @@ To smooth the transition, even tools that are aware of this CEP should still pop avoid breaking behaviour changes in older versions during the transition. For setting the values, we propose a conservative approach, in the sense that we default to strong exports in case of doubt: -- Re-use values for keys which have a 1:1 equivalent in `run_exports:` schema: +- Reuse values for keys which have a 1:1 equivalent in `run_exports:` schema: - `host_to_run:` --> `weak:` - `host_to_constraints:` --> `weak_constrains:` - `build_to_constraints:` -- > `strong_constrains:` From 2fd56377250e2bcd57ca245281f1eb521e719a52 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sat, 30 Aug 2025 17:04:54 +1100 Subject: [PATCH 15/39] iterate on specification --- cep-XXXX.md | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 83f92a85..1e6a03b2 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -329,13 +329,14 @@ TODO! ### Package and Channel Metadata -On package-level, if an output has any `exports:` specified, build tools MUST produce an `exports.json` in the -root of the artefact (next to `index.json` etc.), and populate the following schema with the exports as specified -in the rendered recipe. If the output has no (or empty) `exports:`, populating `exports.json` MAY be omitted. -If not, it MUST still conform to the schema below, with empty lists as values for the respective export keys. +On output-level, if there are any non-empty `exports:` specified, build tools MUST produce an `exports.json` +in the root of the artefact (next to `index.json` etc.), and populate the values with the exports as specified +in the rendered recipe for that output. If the output has no (or empty) `exports:`, populating `exports.json` +MAY be omitted. If the file `exports.json` gets created, its content MUST be a valid JSON object according to +the schema below, where keys that have empty values MAY be omitted. ```json -"exports": { +{ "build_to_build": [ "string", ], @@ -365,7 +366,7 @@ If not, it MUST still conform to the schema below, with empty lists as values fo We define the following translation between this schema and the previous `run_exports:` schema: -| `exports:` | `run_exports: | +| `exports:` | `run_exports:` | |---|---| | `build_to_build:` | IGNORED | | `build_to_constraints:` | `strong_constrains:` | @@ -377,11 +378,12 @@ We define the following translation between this schema and the previous `run_ex | `noarch_to_run:` | `noarch:` | Except for cells marked with "IGNORED", build tools MUST populate the output-level `run_exports.json` file -unchanged from the values of `exports:` in the recipe, though duplicates from the merge between `build_to_host:` -and `build_to_run:` MAY be removed. +unchanged from the values of `exports:` in the recipe, though exact duplicates from the merge between +`build_to_host:` and `build_to_run:` MAY be removed. Values from `build_to_build:` and `host_to_host:` MUST +be ignored when populating `run_exports.json`. On channel-level, the `exports.json` file MUST be populated when indexing the channel, in the same way -as for `run_exports.json`: +as described for `run_exports.json` in CEP 12, but using the following schema. ```json { @@ -424,13 +426,13 @@ as for `run_exports.json`: } ``` -As for the output-level metadata, indexers MUST populate the channel-level `run_exports.json` in a way that -is consistent with the output-level metadata: either calculated from `exports:` using the above compatibility -mapping, or aggregated from the output-level `run_exports.json`. +Indexers MUST populate the channel-level `run_exports.json` in a way that is consistent with the output-level +metadata: either calculated from `exports:` using the above compatibility mapping, or aggregated from the +output-level `run_exports.json`. -Within the sharded repodata, indexers MUST add an `exports:` key and populate it with the respective output-level -metadata. Furthermore, indexers MUST populate the value `run_exports:` derived from the values of `exports:`, -using the schema mapping specified above. +For sharded repodata following CEP 16 & 21, indexers MUST add an `exports:` key and populate it with the +respective output-level metadata. Furthermore, indexers MUST populate the value `run_exports:` derived +from the values of `exports:`, using the schema mapping specified above. ### Patching From 24cbde159d610de324d5f1d6ba0d5b0c86ad8faf Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sat, 30 Aug 2025 17:14:45 +1100 Subject: [PATCH 16/39] deal with indexing packages that don't yet have new format --- cep-XXXX.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 1e6a03b2..b565f187 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -383,7 +383,9 @@ unchanged from the values of `exports:` in the recipe, though exact duplicates f be ignored when populating `run_exports.json`. On channel-level, the `exports.json` file MUST be populated when indexing the channel, in the same way -as described for `run_exports.json` in CEP 12, but using the following schema. +as described for `run_exports.json` in CEP 12, but using the following schema. Where artefacts do not yet +have `exports.json` metadata, the values in `exports:` MUST be populated from the respective keys in +`run_exports:` according to the above schema mapping. ```json { @@ -426,13 +428,12 @@ as described for `run_exports.json` in CEP 12, but using the following schema. } ``` -Indexers MUST populate the channel-level `run_exports.json` in a way that is consistent with the output-level -metadata: either calculated from `exports:` using the above compatibility mapping, or aggregated from the -output-level `run_exports.json`. +Indexers MUST (continue to) populate the channel-level `run_exports.json` from the output-level `run_exports.json`. -For sharded repodata following CEP 16 & 21, indexers MUST add an `exports:` key and populate it with the -respective output-level metadata. Furthermore, indexers MUST populate the value `run_exports:` derived -from the values of `exports:`, using the schema mapping specified above. +For sharded repodata following CEP 16 & 21, indexers MUST add an `exports:` key and populate it with the respective +output-level metadata. Where outputs do not yet provide `exports.json` the values of `exports:` MUST be populated +from the respective keys in `run_exports:` according to the above schema mapping. Furthermore, indexers MUST populate +the value `run_exports:` derived from output-level `run_exports.json`. ### Patching From 5e7445d0e08390a26b84c23c0bc5199f82d4c183 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sat, 30 Aug 2025 17:30:39 +1100 Subject: [PATCH 17/39] iterate --- cep-XXXX.md | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index b565f187..f8c7f288 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -91,7 +91,7 @@ This is a relatively little-used feature, c.f. this approximate [search](https:/ python and R, which are of course key use-cases for building noarch consumers on top. Perhaps most notably, since the linked PR, regular run-exports (either weak or strong) do not apply to -`noarch: {generic, python}` packages anymore, as thereafter, those received their own special export type. +`noarch: {generic, python}` packages anymore, as thereafter they had their own special export type. ## Motivation @@ -148,6 +148,7 @@ We begin with the following observations based on the above: - It would be nice to cover constraints as well as dependencies with the same pattern. - There are other use-cases ([example](https://github.com/conda-forge/ctng-compiler-activation-feedstock/blob/e2bdf15eb170008eda386056a900ce93e0f9cb16/recipe/meta.yaml#L147-L150)) which have so far been under-served by the existing run-export infrastructure. +- Run-exports are not applied when building `noarch` packages, except if the export is of type `noarch:`. - The v1 recipe format unified all requirement-related topics (including run-exports) under `requirements:`. Based on this, we propose the following pattern: @@ -177,7 +178,7 @@ requirements: build_to_build: # see below - a_transitive_dependency # all the above _do not_ apply when building `noarch: generic` or `noarch: python` packages - noarch_to_run: # matches `noarch:` run-export; does apply to noarch packages + noarch_to_run: # matches `noarch:` run-export; _does_ apply when building noarch packages - a_dependency_exported_when_consumer_is_noarch ``` @@ -187,7 +188,7 @@ and `build_to_constraints:`. The other keys introduce new functionality, which i Before explaining the transitive case, we note though that this design has the advantage that it's immediately clear from the key pattern `_to_` under which conditions a given export -triggers (i.e. the package carrying the export finds itself in an environment matching ``), +triggers (i.e. the package carrying the export finds itself in conditions matching ``), and what it influences (i.e. the export gets added to ``). This avoids a lot of mental arithmetic (and ideally, implementation complexity) in keeping track of which export triggers when. @@ -241,13 +242,13 @@ Likewise we rename `ignore_run_exports` ``` which should ignore any exports into `run:` or `constraints:` matching the conditions (whether -by originating package or by name of the export), regardless of which environment it comes from. +by originating package or by name of the export), regardless of which `*_to_run:` key it comes from. ### Omitted combinations -In all cases, dependency exports address constraints or interactions arising from compilation. At runtime, -when the build process is long past, the situation simplifies back to the question whether another package -is a dependency or not, which is why no `run_to_run:` key is proposed here. +In all cases except `noarch` packages, dependency exports address constraints or interactions arising from +compilation. At runtime, when the build process is long past, the situation simplifies back to the question +whether another package is a dependency or not, which is why no `run_to_run:` key is proposed here. Furthermore, one could ask about a possible `host_to_build:` key. While this would arguably be an even better fit for the C++/Fortran modules ABI issue described above, the reason this proposal refrains from suggesting @@ -275,7 +276,7 @@ to JSON (by default) in 2018. In practice, it is fair to assume that only `run_e CEP 12 introduced a channel-level `run_exports.json` which provides the information in aggregated format, allowing extraction of run-export metadata (e.g. for conda-forge's bot infrastructure) at scale without having to download -every individual package first. This effort refrained from touching `repodata.json`, among other reasons because +every individual package first. This effort refrained from touching `repodata.json`, among other reasons because: > It would require extending the `repodata` schema, currently not formally standardized. Indeed, inspection of package-level `repodata_record.json` of contemporary (mid-2025) packages shows that run-exports @@ -300,9 +301,10 @@ simplified. We suggest to: - Channel-level: - Add another `exports.json` to the monolithic channel metadata, to be preferred by tools who which how to handle it. - Indexers should populate `run_exports.json` with "compatible" metadata derived from `exports:` (see below). - - Add the structure of run-exports within sharded metadata, as the same argument to footprint applies as in CEP 21, - i.e. the data is highly compressible and will not have more than ~5% impact. Long-term, the existing `run_exports:` - information should be removed, freeing up the additional space again. + - Add an `exports:` key within sharded metadata without touching `run_exports:`. The same argument with respect to + the storage footprint as in CEP 21 applies, i.e. the data is highly compressible and will not have more than + ~5% impact. Long-term, the existing `run_exports:` information should be removed, freeing up the additional + space again. The reason to add separate files is that this provides the easiest compatibility story: tools which are aware of this CEP can prefer `exports.json` and equivalents, whereas older versions of these tools continue to work unchanged. @@ -379,8 +381,8 @@ We define the following translation between this schema and the previous `run_ex Except for cells marked with "IGNORED", build tools MUST populate the output-level `run_exports.json` file unchanged from the values of `exports:` in the recipe, though exact duplicates from the merge between -`build_to_host:` and `build_to_run:` MAY be removed. Values from `build_to_build:` and `host_to_host:` MUST -be ignored when populating `run_exports.json`. +`build_to_host:` and `build_to_run:` into `strong:` MAY be removed. Values from `build_to_build:` and +`host_to_host:` MUST be ignored when populating `run_exports.json`. On channel-level, the `exports.json` file MUST be populated when indexing the channel, in the same way as described for `run_exports.json` in CEP 12, but using the following schema. Where artefacts do not yet @@ -432,8 +434,8 @@ Indexers MUST (continue to) populate the channel-level `run_exports.json` from t For sharded repodata following CEP 16 & 21, indexers MUST add an `exports:` key and populate it with the respective output-level metadata. Where outputs do not yet provide `exports.json` the values of `exports:` MUST be populated -from the respective keys in `run_exports:` according to the above schema mapping. Furthermore, indexers MUST populate -the value `run_exports:` derived from output-level `run_exports.json`. +from the respective keys in `run_exports:` according to the above schema mapping. Furthermore, indexers MUST +(continue to) populate the value `run_exports:` derived from output-level `run_exports.json`. ### Patching From 7daab72c097509f408481235e3d0feefe14def2b Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sat, 30 Aug 2025 17:47:55 +1100 Subject: [PATCH 18/39] add rationale for avoiding exports without specifying type --- cep-XXXX.md | 39 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 37 insertions(+), 2 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index f8c7f288..dcfd0a67 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -242,7 +242,7 @@ Likewise we rename `ignore_run_exports` ``` which should ignore any exports into `run:` or `constraints:` matching the conditions (whether -by originating package or by name of the export), regardless of which `*_to_run:` key it comes from. +by originating package or by name of the export), regardless of which export type it comes from. ### Omitted combinations @@ -261,6 +261,39 @@ Summing up, `build:` can export to all others (i.e. `build:`, `host:`, `run:`, ` export to everything but `build:`, while nothing can be exported from either `run:` or `constraints:`. None of these exports apply when building noarch packages, which only take into account `noarch_to_run:` exports. +### No convenience shorthand + +Owing to the twists and turns of the way the feature was introduced, conda-build has allowed + +```yaml +build: + run_exports: + - libfoo +``` + +to be equivalent to + +```yaml +build: + run_exports: + weak: + - libfoo +``` + +which was introduced later (as discussed in History section). The v1 recipe format has kept this shorthand. +Even though it is likely that `host_to_run:` will represent the overwhelming majority of occurrences of exports, +we do not believe it is worth allowing a similar shortcut + +```yaml +requirements: + exports: + - libfoo # NOT PROPOSED! +``` + +For one, it complicates the schema definition and handling unnecessarily, and saving a few characters is +not worth the resulting ambiguity. Finally, using `host_to_run:` improves clarity for the recipe reader +and will naturally (we believe) lead to understanding the other export types. + ## Impacts on package and channel metadata ### Background @@ -294,7 +327,7 @@ The approach suggested here is based on the intention to avoid having to introdu an [effort](https://github.com/conda/ceps/pull/111) should come to fruition, the below could certainly be simplified. We suggest to: -- Package-level: +- Output-level: - Add another `exports.json` next to `repodata_record.json` and `run_exports.json`, to be preferred by tools which know how to handle it. - Populate `run_exports.json` with "compatible" metadata derived from `exports:` (see below). @@ -437,6 +470,8 @@ output-level metadata. Where outputs do not yet provide `exports.json` the value from the respective keys in `run_exports:` according to the above schema mapping. Furthermore, indexers MUST (continue to) populate the value `run_exports:` derived from output-level `run_exports.json`. +Tools MUST take information from `exports:` / `exports.json` (if available) over `run_exports:` / `run_exports.json`. + ### Patching TODO! From 5ac7ad4a0c9e9b00e2c742cd7fe5396ecbe8d600 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sat, 30 Aug 2025 18:16:49 +1100 Subject: [PATCH 19/39] move (remaining) open questions to top of specification section --- cep-XXXX.md | 23 ++++++++++------------- 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index dcfd0a67..fc57aada 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -282,7 +282,7 @@ build: which was introduced later (as discussed in History section). The v1 recipe format has kept this shorthand. Even though it is likely that `host_to_run:` will represent the overwhelming majority of occurrences of exports, -we do not believe it is worth allowing a similar shortcut +we do not believe it is worth allowing a similar shortcut for `exports.host_to_run:` ```yaml requirements: @@ -290,9 +290,9 @@ requirements: - libfoo # NOT PROPOSED! ``` -For one, it complicates the schema definition and handling unnecessarily, and saving a few characters is -not worth the resulting ambiguity. Finally, using `host_to_run:` improves clarity for the recipe reader -and will naturally (we believe) lead to understanding the other export types. +For one, it complicates the schema definition and handling unnecessarily, and saving a few characters is not worth +the resulting ambiguity. Finally, using `host_to_run:` improves clarity for the recipe reader and will naturally +(we believe) lead to understanding the other export types (or even run-exports as a concept in the first place). ## Impacts on package and channel metadata @@ -351,13 +351,18 @@ conservative approach, in the sense that we default to strong exports in case of - Reuse values for keys which have a 1:1 equivalent in `run_exports:` schema: - `host_to_run:` --> `weak:` - `host_to_constraints:` --> `weak_constrains:` - - `build_to_constraints:` -- > `strong_constrains:` + - `build_to_constraints:` --> `strong_constrains:` - `noarch_to_run:` --> `noarch:` - Add `strong:` run-export in case of doubt, i.e. merge any values of `build_to_host:` & `build_to_run:` into `strong:`. - Do not map keys that have no equivalent in `run_exports:`, i.e. omit `host_to_host:` & `build_to_build:`. ## Specification +### Open questions + +- if we change the keys, do we need to bump the recipe version to v2? +- do we need to update `patch_instructions_version` or other changes to patching infrastructure? + ### Recipes, Parsing, Package Building TODO! @@ -475,11 +480,3 @@ Tools MUST take information from `exports:` / `exports.json` (if available) over ### Patching TODO! - -### Open questions - -- if we change the keys, do we need to bump the recipe version to v2? -- do we want to specify that implementations need to automatically map `run_exports:` to the corresponding newer keys? -- do we need repodata changes to represent the new, more granular, export metadata? -- how does this interact with sharded repodata (c.f. [CEP 21](cep-0021.md))? -- do we need to specify that installers need to be updated to not choke on unknown keys? From f49fbb71546e10fd3faa8e23d6fb999938a5dd86 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sat, 30 Aug 2025 19:05:38 +1100 Subject: [PATCH 20/39] more motivation than "just" host-exports --- cep-XXXX.md | 41 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 37 insertions(+), 4 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index fc57aada..a7b51fe2 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -88,13 +88,15 @@ design for this seems to come only from comments in that PR and boils down to: This is a relatively little-used feature, c.f. this approximate [search](https://github.com/search?q=org%3Aconda-forge+%2F%28%3Fs%29%5Csrun_exports%3A.*noarch%3A%2F+path%3Arecipe%2F*.yaml&type=code) (note: many false positives, but should be exhaustive), though crucially including core packages like -python and R, which are of course key use-cases for building noarch consumers on top. +python and R, which are of course key use-cases for building noarch consumers atop of. Perhaps most notably, since the linked PR, regular run-exports (either weak or strong) do not apply to `noarch: {generic, python}` packages anymore, as thereafter they had their own special export type. ## Motivation +### "Host-exports" + One key motivation for this proposal is that even with the weak/strong distinction, run-exports are not powerful enough to handle relevant scenarios that are a natural consequence of the separation into `build:` / `host:` / `run:` environments. @@ -133,6 +135,32 @@ right constraints (i.e. conflict if ABI between the compiler and the constraint doesn't match), while avoiding too-tight constraints at runtime. The situation is explained/discussed in more detail in . +### Ecosystem Evolution + +Zooming out a bit, the conda ecosystem has changed dramatically over the last decade, including less +visible aspects like build infrastructure. Most platforms nowadays have a choice of multiple different +compilers (at least for C, C++, Fortran), and we need to be able to able to handle constraints arising +from this in a way that avoids inscrutable errors for unsuspecting recipe maintainers. + +On top of that, the long-in-the-making maturation of C++20 modules (which are subject to the same ABI +constraints as the Fortran case explained above), means that our infrastructure needs to be ready to +reliably handle packaging challenges in a world where these C++ modules are beginning to come into more +wide-spread use. Leaving run-exports in their current state long-term is therefore not a palatable option. + +### Teachability + +This effort started out with the intention of making minimal changes in the pursuit of effecting change +as quickly as feasible. However, owing to how organically the run-exports feature has grown over time, +a sober analysis of the situation must conclude that generalising it even further, towards cases having +not even the slightest relation to the term "run", risks creating a situation where both the concepts +and the mechanics would be very confusing. + +Run-exports are already among the most arcane and difficult-to-understand aspects of the conda ecosystem, +which makes it unsustainable to further overload them with complexity. Ideally, a feature designed from +the ground up based on all the lessons learned in the last decade would be much easier to explain and grasp, +lowering the barrier to entry into the ecosystem, and helping existing contributors navigate packaging +challenges more easily. + ## Design We begin with the following observations based on the above: @@ -199,7 +227,7 @@ is desirable (e.g. openmp, openmpi, etc.). The `noarch_to_run:` breaks from the pattern of using a `` that is an existing type of environment. Given how existing run-exports do not apply to noarch packages at all, and how important use-cases like python require this functionality, we cannot remove it just for the sake of foolish consistency, and this -seems like the most natural way to incorporate it. The overall rule can still be summarized as "exports do +seems like the most natural way to incorporate it. The overall rule can still be summarised as "exports do not apply when building noarch packages, unless the export is of type `noarch_to_run:`." ### Transitive compilation requirements @@ -301,7 +329,7 @@ the resulting ambiguity. Finally, using `host_to_run:` improves clarity for the Both [CEP 12](cep-0012.md) and [CEP 21](cep-0021.md) worked in the area of specifying how run-exports are represented in package and channel metadata. -To the best of our knowledge, the format of `run_exports.json` on a per-package level has not been formalized, +To the best of our knowledge, the format of `run_exports.json` on a per-package level has not been formalised, though, unsurprisingly, it is [simply](https://github.com/conda/conda-build/blob/25.7.0/conda_build/build.py#L1378-L1387) a JSON-extract of the relevant `run_exports:` portion of the rendered recipe. CB3 originally introduced this as `run_exports.yaml`, which got [switched](https://github.com/conda/conda-build/commit/1347f3df264c57d79ab078f88fae2d8862a58d9f) @@ -310,7 +338,7 @@ to JSON (by default) in 2018. In practice, it is fair to assume that only `run_e CEP 12 introduced a channel-level `run_exports.json` which provides the information in aggregated format, allowing extraction of run-export metadata (e.g. for conda-forge's bot infrastructure) at scale without having to download every individual package first. This effort refrained from touching `repodata.json`, among other reasons because: -> It would require extending the `repodata` schema, currently not formally standardized. +> It would require extending the `repodata` schema, currently not formally standardised. Indeed, inspection of package-level `repodata_record.json` of contemporary (mid-2025) packages shows that run-exports do not appear in the regular package-metadata. @@ -356,6 +384,11 @@ conservative approach, in the sense that we default to strong exports in case of - Add `strong:` run-export in case of doubt, i.e. merge any values of `build_to_host:` & `build_to_run:` into `strong:`. - Do not map keys that have no equivalent in `run_exports:`, i.e. omit `host_to_host:` & `build_to_build:`. +### Indexing old artefacts + +The same mapping can be inverted to populate the information necessary for `exports.json` of a channel containing a mix +of new and old artefacts, where the latter cannot be expected to have output-level `exports.json` metadata yet. + ## Specification ### Open questions From 537842f0f639787ad101b5887c50c4ba263003e8 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sun, 31 Aug 2025 09:56:09 +1100 Subject: [PATCH 21/39] make history section chronological again --- cep-XXXX.md | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index a7b51fe2..da3fd98c 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -68,13 +68,6 @@ between environments, and opens further questions along the lines of: > If `qux` has a strong run-export, and appears in a `run:` environment of another package, does the export trigger or not? -Later still, the constraint variants `weak_constrains` and `strong_constrains` were -[added](https://github.com/conda/conda-build/pull/4125). - -The v1 recipe format moved the `run_exports:` key from the `build:` section to the `requirements:` of the -respective output, but otherwise did not change the semantics of this feature, aside from renaming -`run_constrained` to `run_constraints` and `{weak,strong}_constrains` to `{weak,strong}_constraints`. - ### Run-exports vs. noarch In 2020, a `noarch:` type of run-exports got [added](https://github.com/conda/conda-build/pull/3868); the @@ -93,6 +86,18 @@ python and R, which are of course key use-cases for building noarch consumers at Perhaps most notably, since the linked PR, regular run-exports (either weak or strong) do not apply to `noarch: {generic, python}` packages anymore, as thereafter they had their own special export type. +### Extension to constraints + +Later in 2020, the constraint variants `weak_constrains` and `strong_constrains` were +[added](https://github.com/conda/conda-build/pull/4125). Since then the feature has +not seen any major changes anymore. + +### The v1 recipe format + +CEP 14 moved `run_exports:` and `ignore_run_exports:` from the `build:` section to the `requirements:` +of the respective output, but otherwise did not change the semantics of this feature, aside from renaming +`run_constrained` to `run_constraints` and `{weak,strong}_constrains` to `{weak,strong}_constraints`. + ## Motivation ### "Host-exports" @@ -386,8 +391,9 @@ conservative approach, in the sense that we default to strong exports in case of ### Indexing old artefacts -The same mapping can be inverted to populate the information necessary for `exports.json` of a channel containing a mix -of new and old artefacts, where the latter cannot be expected to have output-level `exports.json` metadata yet. +The same mapping can be inverted to populate the information necessary for `exports.json` of a channel containing +a mix of new and old artefacts, where the latter cannot be expected to consistently have output-level `exports.json` +metadata yet. ## Specification From a878e391a391ec80ac49fbd2af030cd4ecc956e0 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sun, 31 Aug 2025 10:45:04 +1100 Subject: [PATCH 22/39] edit pass --- cep-XXXX.md | 87 ++++++++++++++++++++++++++++------------------------- 1 file changed, 46 insertions(+), 41 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index da3fd98c..fc29e9cd 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -11,11 +11,12 @@ ## Abstract -This CEP proposes to overhaul the way that packages "export" some required dependency -or constraint, in a way that considers both the source environment during the build phase -(one of `build:`, `host:` and `run:`) as well as the target environment where the respective -dependency or constraint should be applied. This has a wide variety of important uses which -are so far difficult or impossible to express, in addition to increasing clarity. +This CEP proposes to overhaul the way that packages "export" some required dependency or constraint, +in a way that directly specifies the conditions (of the build phase of a dependent package) which +need to be met for the export to be triggered, as well as the target environment where the respective +dependency or constraint should be applied. This has a wide variety of important uses which are so far +difficult or impossible to express, and increases clarity of the respective recipes, as well as the +teachability of the underlying concepts. This document builds upon [CEP 13](cep-0013.md) and [CEP 14](cep-0014.md), which define what is known as the "v1" recipe format. @@ -26,13 +27,13 @@ From the beginning, one of the big advantages of `conda` over `pip` has been the track dependencies sufficiently well in order to be able to share artefacts between packages, rather than vendoring dependent libraries for every consumer. -In contrast to static or header-only libraries, a shared library `foo` occurs not just during build -time of `bar` (the headers for `# include ` to work and `libfoo.so` for linkage to succeed), +In contrast to static or header-only libraries, a shared library `foo` is not only required during the +build phase `bar` (the headers for `# include ` to work and `libfoo.so` for linkage to succeed), but needs to be present in the final environment for `bar` as well. Aside from corner-cases, -this runtime dependency *always* follows, and it is therefore natural to ask the build system to do -this work for us. +this runtime dependency *always* follows, and it is therefore natural to ask the build tool (together +with appropriate metadata) to do this work for us. -This is what the original [design discussion](https://github.com/conda/conda-build/issues/1142) started +This is what the original [design discussion](https://github.com/conda/conda-build/issues/1142) in conda-build started out from, which was first implemented first as [`pin_downstream`](https://github.com/conda/conda-build/commit/e344bbae369658ca7e2defab8a3960d8570fbf8a) and soon after renamed to [`run_exports`](https://github.com/conda/conda-build/commit/d90aa3135cc81a5db28e8160b521f11d27083453). The original [documentation](https://github.com/conda/conda-docs/pull/414) for this feature provides @@ -71,7 +72,7 @@ between environments, and opens further questions along the lines of: ### Run-exports vs. noarch In 2020, a `noarch:` type of run-exports got [added](https://github.com/conda/conda-build/pull/3868); the -design for this seems to come only from comments in that PR and boils down to: +design for this seems to primarily come from comments in that PR and boils down to: > > > Ray Donnelly: Do we not really want to use a different run_export type here? Dropping all but the package name? > > > > Isuru Fernando: I don't understand. Are you suggesting a different run_exports scheme ('noarch' in addition to @@ -81,7 +82,7 @@ design for this seems to come only from comments in that PR and boils down to: This is a relatively little-used feature, c.f. this approximate [search](https://github.com/search?q=org%3Aconda-forge+%2F%28%3Fs%29%5Csrun_exports%3A.*noarch%3A%2F+path%3Arecipe%2F*.yaml&type=code) (note: many false positives, but should be exhaustive), though crucially including core packages like -python and R, which are of course key use-cases for building noarch consumers atop of. +python and R, which are of course key use-cases for building noarch packages atop of. Perhaps most notably, since the linked PR, regular run-exports (either weak or strong) do not apply to `noarch: {generic, python}` packages anymore, as thereafter they had their own special export type. @@ -112,11 +113,11 @@ in particular, the urgency was reduced by the fact that it could be passably emu run-exports; while this would "over-export" things into the `run:` environment, this is harmless in many cases. However, there are cases where that is not so, and the relevant constraints cannot currently be expressed -(resp. where abuse of the strong run-export would require all consumers to ignore the run-exports, which -is not feasible at scale, and would be a constant tripping hazard). +(resp. where abuse of the strong run-export would require all consumers to ignore the extraneous run-exports, +which is not feasible at scale, and would be a constant tripping hazard). For example, C++ and Fortran modules (as of 2025) are not portable between compilers and need to be consumed -by the same compiler that produced them. Assuming we have a package `foo-devel` containing Fortran modules, +by the same compiler that produced them. Let's assume we have a package `foo-devel` containing Fortran modules, to which we would like to attach a `_fortran_modules_abi =*=compiler_flavour*` constraint that ensures that it can only be combined with the appropriate compiler. The problem in this case is that with a recipe like @@ -131,14 +132,16 @@ it can only be combined with the appropriate compiler. The problem in this case ``` there's no way to make the "wrong" fortran compiler conflict with `foo-devel`, because we explicitly -do not want a strong run-export from the general-purpose `{{ compiler("fortran") }}` to enforce a specific -compiler ABI in `run:` (making the package unusable with other compilers unnecessarily). +do not want a strong run-export from the general-purpose `{{ compiler("fortran") }}` (more precisely, +the underlying package `${fortran_compiler}_${target_platform}`) to enforce a specific compiler ABI in +`run:`, which would unnecessarily make the package unusable together with packages built by other +Fortran compilers. -The solution in this case would be to add an export to `{{ compiler("fortran") }}` that injects -`_fortran_modules_abi =*=compiler_flavour*` *only* into the `host:` environment; this would impose the -right constraints (i.e. conflict if ABI between the compiler and the constraint attached to `foo-devel` -doesn't match), while avoiding too-tight constraints at runtime. The situation is explained/discussed -in more detail in . +The solution in this case would be to add an export to the various `${fortran_compiler}_${target_platform}` +packages which injects `_fortran_modules_abi =*=compiler_flavour*` *only* into the `host:` environment; +this would impose the right constraints (i.e. conflict if ABI between the compiler and the constraint +attached to `foo-devel` doesn't match), while avoiding too-tight constraints at runtime. The situation is +explained/discussed in more detail in . ### Ecosystem Evolution @@ -150,7 +153,7 @@ from this in a way that avoids inscrutable errors for unsuspecting recipe mainta On top of that, the long-in-the-making maturation of C++20 modules (which are subject to the same ABI constraints as the Fortran case explained above), means that our infrastructure needs to be ready to reliably handle packaging challenges in a world where these C++ modules are beginning to come into more -wide-spread use. Leaving run-exports in their current state long-term is therefore not a palatable option. +wide-spread use. Leaving run-exports in their current state indefinitely is therefore not a palatable option. ### Teachability @@ -199,7 +202,7 @@ requirements: host_to_run: # matches `weak:` run-export - a_shared_library build_to_host: # "host-export" - - a_build_constraint =*=*foo + - a_host_constraint =*=*foo build_to_run: # produces same effect as `strong:` run-export when used together with build_to_host - a_compiler_runtime host_to_constraints: # matches `weak_constrains:` (v0) / `weak_constraints:` (v1) @@ -230,10 +233,11 @@ beyond C++/Fortran modules where the ability to constrain interactions between c is desirable (e.g. openmp, openmpi, etc.). The `noarch_to_run:` breaks from the pattern of using a `` that is an existing type of environment. -Given how existing run-exports do not apply to noarch packages at all, and how important use-cases like -python require this functionality, we cannot remove it just for the sake of foolish consistency, and this -seems like the most natural way to incorporate it. The overall rule can still be summarised as "exports do -not apply when building noarch packages, unless the export is of type `noarch_to_run:`." +Despite being used relatively rarely, it is required by important use-cases like python, and so this variant +cannot be abandoned for the sake of foolish consistency. Given how existing run-exports do not apply to +noarch packages at all, it's seems natural to consider noarch as just another condition that is either met +during the build phase of a dependent project, or not. The overall rule still remains easy to summarise as +"`exports:` do not apply when building noarch packages, unless the export is of type `noarch_to_run:`." ### Transitive compilation requirements @@ -242,10 +246,10 @@ why whatever is being exported in such a manner could not be a direct (run-)depe The answer is that there may be transitive dependencies *at compilation time* that we do not want consumers to inherit at runtime (similar to the situation with the Fortran modules ABI). -An example of this is if a library `foo` depends on the headers of another library `bar` at compile-time, -but we do not want packages built atop of `foo` to carry along those `bar` headers or related version -constraints (because at that point they're not needed anymore; the concern is about a compile-time -quantity, which only concerns either `build:` and/or `host:`). +An example of this is if a library `foo` depends on the headers of another library `bar` at compile-time +(possibly with restrictive version constraints), but we do not want packages built atop of `foo` to carry +along those `bar` headers or related constraints (because at that point they're not needed anymore; +the concern is about a compile-time quantity, which only concerns either `build:` and/or `host:`). ### Other modifications @@ -259,9 +263,10 @@ requirements: ``` because constraints only make sense when that package gets installed somewhere in any case, so the -"run_" is superfluous (aside from being inconsistent with the proposed schema). On top of that, the -"run_" is also confusing, because if a package with `run_constraints:` gets installed into a `host:` -or `build:` environment, those constraints will still take effect. +"run_" is superfluous (aside from being inconsistent with the proposed pattern for export variants). +On top of that, the "run_" can also be confusing, because the name might be misinterpreted as being +specific to the `run:` environment, when in actuality, the `run_constraints:` of a package still +take effect also when installed into a `host:` or `build:` environment. Likewise we rename `ignore_run_exports` @@ -353,7 +358,7 @@ to CEP 12, added this to the physically sharded but logically unified repodata. ### Transition plan -It's easy to map the new export structure to the respective metadata, the complexity lies in providing a smooth +It's easy to map the new export structure to the respective metadata; the complexity lies in providing a smooth transition for the ecosystem across various versions of tools that build or consume packages and metadata. The approach suggested here is based on the intention to avoid having to introduce a repodata v2, but if such @@ -366,20 +371,20 @@ simplified. We suggest to: - Populate `run_exports.json` with "compatible" metadata derived from `exports:` (see below). - Channel-level: - Add another `exports.json` to the monolithic channel metadata, to be preferred by tools who which how to handle it. - - Indexers should populate `run_exports.json` with "compatible" metadata derived from `exports:` (see below). - Add an `exports:` key within sharded metadata without touching `run_exports:`. The same argument with respect to the storage footprint as in CEP 21 applies, i.e. the data is highly compressible and will not have more than ~5% impact. Long-term, the existing `run_exports:` information should be removed, freeing up the additional space again. -The reason to add separate files is that this provides the easiest compatibility story: tools which are aware of this -CEP can prefer `exports.json` and equivalents, whereas older versions of these tools continue to work unchanged. +The reason to add separate files and keys is that this provides the easiest compatibility story: tools which are +aware of this CEP can prefer `exports.json` and equivalents, whereas older versions of these tools continue to +work unchanged. ### Compatibility mapping back to `run_exports.json` To smooth the transition, even tools that are aware of this CEP should still populate `run_exports.json` etc., to -avoid breaking behaviour changes in older versions during the transition. For setting the values, we propose a -conservative approach, in the sense that we default to strong exports in case of doubt: +avoid causing breaking behaviour changes in older versions during the transition. For setting the values, we propose +a conservative approach, in the sense that we default to strong exports in case of doubt: - Reuse values for keys which have a 1:1 equivalent in `run_exports:` schema: - `host_to_run:` --> `weak:` From 5723d2955ac764352a14c6ad360868fef14f6bb7 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Tue, 2 Sep 2025 15:08:24 +1100 Subject: [PATCH 23/39] accumulated minor edits --- cep-XXXX.md | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index fc29e9cd..5e1c787e 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -28,7 +28,7 @@ track dependencies sufficiently well in order to be able to share artefacts betw rather than vendoring dependent libraries for every consumer. In contrast to static or header-only libraries, a shared library `foo` is not only required during the -build phase `bar` (the headers for `# include ` to work and `libfoo.so` for linkage to succeed), +build phase for `bar` (the headers for `# include ` to work and `libfoo.so` for linkage to succeed), but needs to be present in the final environment for `bar` as well. Aside from corner-cases, this runtime dependency *always* follows, and it is therefore natural to ask the build tool (together with appropriate metadata) to do this work for us. @@ -91,7 +91,7 @@ Perhaps most notably, since the linked PR, regular run-exports (either weak or s Later in 2020, the constraint variants `weak_constrains` and `strong_constrains` were [added](https://github.com/conda/conda-build/pull/4125). Since then the feature has -not seen any major changes anymore. +not seen any major changes. ### The v1 recipe format @@ -134,8 +134,8 @@ it can only be combined with the appropriate compiler. The problem in this case there's no way to make the "wrong" fortran compiler conflict with `foo-devel`, because we explicitly do not want a strong run-export from the general-purpose `{{ compiler("fortran") }}` (more precisely, the underlying package `${fortran_compiler}_${target_platform}`) to enforce a specific compiler ABI in -`run:`, which would unnecessarily make the package unusable together with packages built by other -Fortran compilers. +`run:`. This would unnecessarily make the package unusable together with packages built by other Fortran +compilers, as the Fortran modules are a build-only quantity and their ABI is not relevant at runtime. The solution in this case would be to add an export to the various `${fortran_compiler}_${target_platform}` packages which injects `_fortran_modules_abi =*=compiler_flavour*` *only* into the `host:` environment; @@ -339,7 +339,7 @@ the resulting ambiguity. Finally, using `host_to_run:` improves clarity for the Both [CEP 12](cep-0012.md) and [CEP 21](cep-0021.md) worked in the area of specifying how run-exports are represented in package and channel metadata. -To the best of our knowledge, the format of `run_exports.json` on a per-package level has not been formalised, +To the best of our knowledge, the format of `run_exports.json` on a per-output level has not been formalised, though, unsurprisingly, it is [simply](https://github.com/conda/conda-build/blob/25.7.0/conda_build/build.py#L1378-L1387) a JSON-extract of the relevant `run_exports:` portion of the rendered recipe. CB3 originally introduced this as `run_exports.yaml`, which got [switched](https://github.com/conda/conda-build/commit/1347f3df264c57d79ab078f88fae2d8862a58d9f) @@ -350,8 +350,8 @@ extraction of run-export metadata (e.g. for conda-forge's bot infrastructure) at every individual package first. This effort refrained from touching `repodata.json`, among other reasons because: > It would require extending the `repodata` schema, currently not formally standardised. -Indeed, inspection of package-level `repodata_record.json` of contemporary (mid-2025) packages shows that run-exports -do not appear in the regular package-metadata. +Indeed, inspection of output-level `repodata_record.json` of contemporary (mid-2025) packages shows that run-exports +do not even appear in the regular metadata, only in `run_exports.json`. CEP 21 (building on top of [CEP 16](cep-0016.md)) added channel-level run-export information, though in contrast to CEP 12, added this to the physically sharded but logically unified repodata. @@ -370,10 +370,11 @@ simplified. We suggest to: which know how to handle it. - Populate `run_exports.json` with "compatible" metadata derived from `exports:` (see below). - Channel-level: - - Add another `exports.json` to the monolithic channel metadata, to be preferred by tools who which how to handle it. + - Add another `exports.json` to the monolithic channel metadata, to be preferred by tools who which know how + to handle it. - Add an `exports:` key within sharded metadata without touching `run_exports:`. The same argument with respect to the storage footprint as in CEP 21 applies, i.e. the data is highly compressible and will not have more than - ~5% impact. Long-term, the existing `run_exports:` information should be removed, freeing up the additional + ~5% size impact. Long-term, the existing `run_exports:` information should be removed, freeing up the additional space again. The reason to add separate files and keys is that this provides the easiest compatibility story: tools which are @@ -383,8 +384,8 @@ work unchanged. ### Compatibility mapping back to `run_exports.json` To smooth the transition, even tools that are aware of this CEP should still populate `run_exports.json` etc., to -avoid causing breaking behaviour changes in older versions during the transition. For setting the values, we propose -a conservative approach, in the sense that we default to strong exports in case of doubt: +avoid causing breaking behaviour changes in while older versions of build tools are still in use. For setting the +values, we propose a conservative approach, in the sense that we default to strong exports in case of doubt: - Reuse values for keys which have a 1:1 equivalent in `run_exports:` schema: - `host_to_run:` --> `weak:` @@ -515,9 +516,9 @@ have `exports.json` metadata, the values in `exports:` MUST be populated from th Indexers MUST (continue to) populate the channel-level `run_exports.json` from the output-level `run_exports.json`. For sharded repodata following CEP 16 & 21, indexers MUST add an `exports:` key and populate it with the respective -output-level metadata. Where outputs do not yet provide `exports.json` the values of `exports:` MUST be populated +output-level metadata. Where outputs do not yet provide `exports.json`, the values of `exports:` MUST be populated from the respective keys in `run_exports:` according to the above schema mapping. Furthermore, indexers MUST -(continue to) populate the value `run_exports:` derived from output-level `run_exports.json`. +(continue to) populate the value of `run_exports:` with the aggregation of output-level `run_exports.json`. Tools MUST take information from `exports:` / `exports.json` (if available) over `run_exports:` / `run_exports.json`. From e0c11484211983c0dab7d128fc0c85ea8abbf185 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Tue, 2 Sep 2025 15:33:48 +1100 Subject: [PATCH 24/39] distinguish between `run_exports:` v0/v1 for schema mapping in specification --- cep-XXXX.md | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 5e1c787e..dda08992 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -366,13 +366,13 @@ an [effort](https://github.com/conda/ceps/pull/111) should come to fruition, the simplified. We suggest to: - Output-level: - - Add another `exports.json` next to `repodata_record.json` and `run_exports.json`, to be preferred by tools + - Add another `exports.json` next to `repodata_record.json`, to be preferred over `run_exports.json` by tools which know how to handle it. - Populate `run_exports.json` with "compatible" metadata derived from `exports:` (see below). - Channel-level: - - Add another `exports.json` to the monolithic channel metadata, to be preferred by tools who which know how - to handle it. - - Add an `exports:` key within sharded metadata without touching `run_exports:`. The same argument with respect to + - Add a `exports.json` file to the monolithic channel metadata, to be preferred over `run_exports.json` by + tools who which know how to handle it. + - Add an `exports:` key within sharded metadata without altering `run_exports:`. The same argument with respect to the storage footprint as in CEP 21 applies, i.e. the data is highly compressible and will not have more than ~5% size impact. Long-term, the existing `run_exports:` information should be removed, freeing up the additional space again. @@ -384,7 +384,7 @@ work unchanged. ### Compatibility mapping back to `run_exports.json` To smooth the transition, even tools that are aware of this CEP should still populate `run_exports.json` etc., to -avoid causing breaking behaviour changes in while older versions of build tools are still in use. For setting the +avoid causing breaking behaviour changes for older versions of build tools that are still in use. For setting the values, we propose a conservative approach, in the sense that we default to strong exports in case of doubt: - Reuse values for keys which have a 1:1 equivalent in `run_exports:` schema: @@ -392,7 +392,7 @@ values, we propose a conservative approach, in the sense that we default to stro - `host_to_constraints:` --> `weak_constrains:` - `build_to_constraints:` --> `strong_constrains:` - `noarch_to_run:` --> `noarch:` -- Add `strong:` run-export in case of doubt, i.e. merge any values of `build_to_host:` & `build_to_run:` into `strong:`. +- Add strong run-export in case of doubt, i.e. merge any values of `build_to_host:` & `build_to_run:` into `strong:`. - Do not map keys that have no equivalent in `run_exports:`, i.e. omit `host_to_host:` & `build_to_build:`. ### Indexing old artefacts @@ -416,7 +416,7 @@ TODO! On output-level, if there are any non-empty `exports:` specified, build tools MUST produce an `exports.json` in the root of the artefact (next to `index.json` etc.), and populate the values with the exports as specified -in the rendered recipe for that output. If the output has no (or empty) `exports:`, populating `exports.json` +in the rendered recipe for that output. If the output has no (or empty) `exports:`, creation of `exports.json` MAY be omitted. If the file `exports.json` gets created, its content MUST be a valid JSON object according to the schema below, where keys that have empty values MAY be omitted. @@ -449,18 +449,18 @@ the schema below, where keys that have empty values MAY be omitted. } ``` -We define the following translation between this schema and the previous `run_exports:` schema: - -| `exports:` | `run_exports:` | -|---|---| -| `build_to_build:` | IGNORED | -| `build_to_constraints:` | `strong_constrains:` | -| `build_to_host:` | `strong:` | -| `build_to_run:` | `strong:` | -| `host_to_constraints:` | `weak_constrains:` | -| `host_to_host:` | IGNORED | -| `host_to_run:` | `weak:` | -| `noarch_to_run:` | `noarch:` | +We define the following translation between this schema and previous versions of the `run_exports:` schema: + +| `exports:` | `run_exports:` (v0) | `run_exports:` (v1) | +|---|---|--| +| `build_to_build:` | IGNORED | IGNORED | +| `build_to_constraints:` | `strong_constrains:` | `strong_constraints:` | +| `build_to_host:` | `strong:` | `strong:` | +| `build_to_run:` | `strong:` | `strong:` | +| `host_to_constraints:` | `weak_constrains:` | `weak_constraints:` | +| `host_to_host:` | IGNORED | IGNORED | +| `host_to_run:` | `weak:` | `weak:` | +| `noarch_to_run:` | `noarch:` | `noarch:` | Except for cells marked with "IGNORED", build tools MUST populate the output-level `run_exports.json` file unchanged from the values of `exports:` in the recipe, though exact duplicates from the merge between From 8eb4cdc125be4c3f14e86f7c0d1138684ec2d449 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Tue, 2 Sep 2025 16:16:01 +1100 Subject: [PATCH 25/39] allow normalization of values into `run_exports:` --- cep-XXXX.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index dda08992..3495d266 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -462,10 +462,13 @@ We define the following translation between this schema and previous versions of | `host_to_run:` | `weak:` | `weak:` | | `noarch_to_run:` | `noarch:` | `noarch:` | -Except for cells marked with "IGNORED", build tools MUST populate the output-level `run_exports.json` file -unchanged from the values of `exports:` in the recipe, though exact duplicates from the merge between -`build_to_host:` and `build_to_run:` into `strong:` MAY be removed. Values from `build_to_build:` and -`host_to_host:` MUST be ignored when populating `run_exports.json`. +Build tools MUST populate the output-level `run_exports.json` file with the payload of the `exports:` object +(for the respective output in the rendered recipe) as follows: they MUST translate keys to the v0 schema per +the table above, MUST error on illegal `PackageSelector`s (in the sense of CEP 14), MAY normalize the contained +`PackageSelector`s, MAY omit keys with empty values, and MUST omit keys marked "IGNORED" (including the +corresponding values). If both `build_to_host:` and `build_to_run:` have non-empty values for the output +in question, those values MUST be concatenated into `strong:`. For each key, exact duplicates (after +normalization of the contained `PackageSelector`s) in the corresponding value MAY be removed. On channel-level, the `exports.json` file MUST be populated when indexing the channel, in the same way as described for `run_exports.json` in CEP 12, but using the following schema. Where artefacts do not yet From 2dbf149bd1c735345da91fe9c85da16d489f34c6 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Tue, 2 Sep 2025 16:39:29 +1100 Subject: [PATCH 26/39] improve specification legalese --- cep-XXXX.md | 54 ++++++++++++++++++++++------------------------------- 1 file changed, 22 insertions(+), 32 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 3495d266..1b505d9a 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -371,7 +371,7 @@ simplified. We suggest to: - Populate `run_exports.json` with "compatible" metadata derived from `exports:` (see below). - Channel-level: - Add a `exports.json` file to the monolithic channel metadata, to be preferred over `run_exports.json` by - tools who which know how to handle it. + tools which know how to handle it. - Add an `exports:` key within sharded metadata without altering `run_exports:`. The same argument with respect to the storage footprint as in CEP 21 applies, i.e. the data is highly compressible and will not have more than ~5% size impact. Long-term, the existing `run_exports:` information should be removed, freeing up the additional @@ -417,35 +417,26 @@ TODO! On output-level, if there are any non-empty `exports:` specified, build tools MUST produce an `exports.json` in the root of the artefact (next to `index.json` etc.), and populate the values with the exports as specified in the rendered recipe for that output. If the output has no (or empty) `exports:`, creation of `exports.json` -MAY be omitted. If the file `exports.json` gets created, its content MUST be a valid JSON object according to -the schema below, where keys that have empty values MAY be omitted. +MAY be omitted. For the value of each key under `exports:`, before creating `exports.json` and in the following +order, tools: + +- MUST error on illegal `PackageSelector`s (as defined in CEP 14), +- MAY normalize the contained `PackageSelector`s (including removal of empty ones), +- MAY remove exact duplicates within the list of `PackageSelector`s, and +- MAY omit serializing a given key entirely, if its value is an empty list. + +If the file `exports.json` gets created, its content MUST be a valid JSON object according to the schema below: ```json { - "build_to_build": [ - "string", - ], - "build_to_constraints": [ - "string", - ], - "build_to_host": [ - "string", - ], - "build_to_run": [ - "string", - ], - "host_to_constraints": [ - "string", - ], - "host_to_host": [ - "string", - ], - "host_to_run": [ - "string", - ], - "noarch_to_run": [ - "string", - ] + "build_to_build": [PackageSelector], + "build_to_constraints": [PackageSelector], + "build_to_host": [PackageSelector], + "build_to_run": [PackageSelector], + "host_to_constraints": [PackageSelector], + "host_to_host": [PackageSelector], + "host_to_run": [PackageSelector], + "noarch_to_run": [PackageSelector] } ``` @@ -464,11 +455,10 @@ We define the following translation between this schema and previous versions of Build tools MUST populate the output-level `run_exports.json` file with the payload of the `exports:` object (for the respective output in the rendered recipe) as follows: they MUST translate keys to the v0 schema per -the table above, MUST error on illegal `PackageSelector`s (in the sense of CEP 14), MAY normalize the contained -`PackageSelector`s, MAY omit keys with empty values, and MUST omit keys marked "IGNORED" (including the -corresponding values). If both `build_to_host:` and `build_to_run:` have non-empty values for the output -in question, those values MUST be concatenated into `strong:`. For each key, exact duplicates (after -normalization of the contained `PackageSelector`s) in the corresponding value MAY be removed. +the table above (while leaving the corresponding values unchanged), and MUST omit keys marked "IGNORED" above +(including the corresponding values). If both `build_to_host:` and `build_to_run:` have non-empty values for +the output in question, those values MUST be concatenated into `strong:`. Tools MUST apply the same list of +normalization steps, as specified for `exports.json` above, before creating `run_exports.json`. On channel-level, the `exports.json` file MUST be populated when indexing the channel, in the same way as described for `run_exports.json` in CEP 12, but using the following schema. Where artefacts do not yet From f57690bb476e40ca5fa6cef8a438cdf47f45d870 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Wed, 3 Sep 2025 08:23:58 +1100 Subject: [PATCH 27/39] remove build_to_build & host_to_host, and document why --- cep-XXXX.md | 66 ++++++++++++++++++++++++----------------------------- 1 file changed, 30 insertions(+), 36 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 1b505d9a..71dafb16 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -208,10 +208,6 @@ requirements: host_to_constraints: # matches `weak_constrains:` (v0) / `weak_constraints:` (v1) - a_run_constraint build_to_constraints: # matches `strong_constrains:` (v0) / `strong_constraints:` (v1) - - a_run_constraint - host_to_host: # see below - - a_transitive_dependency - build_to_build: # see below - a_transitive_dependency # all the above _do not_ apply when building `noarch: generic` or `noarch: python` packages noarch_to_run: # matches `noarch:` run-export; _does_ apply when building noarch packages @@ -222,7 +218,7 @@ As indicated by the comments, `host_to_run:` matches the existing weak run-expor `build_to_run:` this produces the same effect of a strong run-export. Similarly for `host_to_constraints:` and `build_to_constraints:`. The other keys introduce new functionality, which is explained below. -Before explaining the transitive case, we note though that this design has the advantage that it's +This design has the advantage that it's immediately clear from the key pattern `_to_` under which conditions a given export triggers (i.e. the package carrying the export finds itself in conditions matching ``), and what it influences (i.e. the export gets added to ``). This avoids a lot of mental arithmetic @@ -239,18 +235,6 @@ noarch packages at all, it's seems natural to consider noarch as just another co during the build phase of a dependent project, or not. The overall rule still remains easy to summarise as "`exports:` do not apply when building noarch packages, unless the export is of type `noarch_to_run:`." -### Transitive compilation requirements - -The most surprising additions might be `host_to_host:` and `build_to_build:`. It would be natural to ask -why whatever is being exported in such a manner could not be a direct (run-)dependency of the package. -The answer is that there may be transitive dependencies *at compilation time* that we do not want -consumers to inherit at runtime (similar to the situation with the Fortran modules ABI). - -An example of this is if a library `foo` depends on the headers of another library `bar` at compile-time -(possibly with restrictive version constraints), but we do not want packages built atop of `foo` to carry -along those `bar` headers or related constraints (because at that point they're not needed anymore; -the concern is about a compile-time quantity, which only concerns either `build:` and/or `host:`). - ### Other modifications Additionally, to keep the `_to_` pattern (aside from @@ -282,11 +266,32 @@ Likewise we rename `ignore_run_exports` which should ignore any exports into `run:` or `constraints:` matching the conditions (whether by originating package or by name of the export), regardless of which export type it comes from. -### Omitted combinations +### Transitive compilation requirements -In all cases except `noarch` packages, dependency exports address constraints or interactions arising from -compilation. At runtime, when the build process is long past, the situation simplifies back to the question -whether another package is a dependency or not, which is why no `run_to_run:` key is proposed here. +A case can be made for `host_to_host:` and `build_to_build:`, despite the natural follow-up question +why whatever is being exported in such a manner could not be a direct (run-)dependency of the package. +The answer is that there may be transitive dependencies *at compilation time* that we do not want +consumers to inherit at runtime (similar to the situation with the Fortran modules ABI). + +An example of this is if a library `foo` depends on the headers of another library `bar` at compile-time +(possibly with restrictive version constraints), but we do not want packages built atop of `foo` to carry +along those `bar` headers or related constraints (because at that point they're not needed anymore; +the concern is about a compile-time quantity, which only concerns either `build:` and/or `host:`). + +While these exports were proposed in a previous iteration of this CEP, they have now been removed. +The key reason for this is that it complicates the environment resolution process in a way that is +difficult to disentangle: for a given set of dependencies, we would have to check whether they have +any `X_to_X:` exports, but to do so, we already have to resolve the environment, in order to determine +the exact set of artefacts for which we would look up the respective `exports.json`. Once the initial +solve has been made, adding further constraints might change the environment, which may go as far as +removing the artefact that was responsible for the `X_to_X:` export in the first place! + +Such use-cases are very likely better solved by conditional dependencies, which can be resolved in a +single pass, and would avoid substantial implementation complexity stemming from the unavoidable +multi-phase resolution (and attendant issues), if this were done on the same level as cross-environment +exports. + +### Other omitted combinations Furthermore, one could ask about a possible `host_to_build:` key. While this would arguably be an even better fit for the C++/Fortran modules ABI issue described above, the reason this proposal refrains from suggesting @@ -295,8 +300,8 @@ such a key is to limit implementation complexity, by having an implicit order of process unnecessarily, and we believe the relevant use-cases are fully expressible using `build_to_host:` together with constraints (such as `_fortran_modules_abi`) attached to packages that appear in `host:`. -Summing up, `build:` can export to all others (i.e. `build:`, `host:`, `run:`, `constraints:`), `host:` can -export to everything but `build:`, while nothing can be exported from either `run:` or `constraints:`. None +Summing up, `build:` can export to all others (i.e. `host:`, `run:`, `constraints:`), `host:` can export +to `run:` and `constraints:`, while nothing can be exported from either `run:` or `constraints:`. None of these exports apply when building noarch packages, which only take into account `noarch_to_run:` exports. ### No convenience shorthand @@ -393,7 +398,6 @@ values, we propose a conservative approach, in the sense that we default to stro - `build_to_constraints:` --> `strong_constrains:` - `noarch_to_run:` --> `noarch:` - Add strong run-export in case of doubt, i.e. merge any values of `build_to_host:` & `build_to_run:` into `strong:`. -- Do not map keys that have no equivalent in `run_exports:`, i.e. omit `host_to_host:` & `build_to_build:`. ### Indexing old artefacts @@ -429,12 +433,10 @@ If the file `exports.json` gets created, its content MUST be a valid JSON object ```json { - "build_to_build": [PackageSelector], "build_to_constraints": [PackageSelector], "build_to_host": [PackageSelector], "build_to_run": [PackageSelector], "host_to_constraints": [PackageSelector], - "host_to_host": [PackageSelector], "host_to_run": [PackageSelector], "noarch_to_run": [PackageSelector] } @@ -444,19 +446,17 @@ We define the following translation between this schema and previous versions of | `exports:` | `run_exports:` (v0) | `run_exports:` (v1) | |---|---|--| -| `build_to_build:` | IGNORED | IGNORED | | `build_to_constraints:` | `strong_constrains:` | `strong_constraints:` | | `build_to_host:` | `strong:` | `strong:` | | `build_to_run:` | `strong:` | `strong:` | | `host_to_constraints:` | `weak_constrains:` | `weak_constraints:` | -| `host_to_host:` | IGNORED | IGNORED | | `host_to_run:` | `weak:` | `weak:` | | `noarch_to_run:` | `noarch:` | `noarch:` | Build tools MUST populate the output-level `run_exports.json` file with the payload of the `exports:` object (for the respective output in the rendered recipe) as follows: they MUST translate keys to the v0 schema per -the table above (while leaving the corresponding values unchanged), and MUST omit keys marked "IGNORED" above -(including the corresponding values). If both `build_to_host:` and `build_to_run:` have non-empty values for +the table above (while leaving the corresponding values unchanged). +If both `build_to_host:` and `build_to_run:` have non-empty values for the output in question, those values MUST be concatenated into `strong:`. Tools MUST apply the same list of normalization steps, as specified for `exports.json` above, before creating `run_exports.json`. @@ -476,9 +476,6 @@ have `exports.json` metadata, the values in `exports:` MUST be populated from th "packages": { "package-version-build.conda": { # or package-version-build.tar.bz "exports": { - "build_to_build": [ - "string", - ], "build_to_constraints": [ "string", ], @@ -491,9 +488,6 @@ have `exports.json` metadata, the values in `exports:` MUST be populated from th "host_to_constraints": [ "string", ], - "host_to_host": [ - "string", - ], "host_to_run": [ "string", ], From 4d03093bd8108727dc2f32c8530a1b366d42d57f Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sun, 21 Dec 2025 16:49:25 +1100 Subject: [PATCH 28/39] update with learnings about chained and ignored exports --- cep-XXXX.md | 251 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 201 insertions(+), 50 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 71dafb16..97edd42d 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -208,10 +208,15 @@ requirements: host_to_constraints: # matches `weak_constrains:` (v0) / `weak_constraints:` (v1) - a_run_constraint build_to_constraints: # matches `strong_constrains:` (v0) / `strong_constraints:` (v1) - - a_transitive_dependency + - a_run_constraint # all the above _do not_ apply when building `noarch: generic` or `noarch: python` packages noarch_to_run: # matches `noarch:` run-export; _does_ apply when building noarch packages - a_dependency_exported_when_consumer_is_noarch + # potential future additions, not proposed by this CEP, but included for completeness + build_to_build: + - a_transitive_dependency + host_to_host: + - a_transitive_dependency ``` As indicated by the comments, `host_to_run:` matches the existing weak run-export. If taken together with @@ -235,74 +240,192 @@ noarch packages at all, it's seems natural to consider noarch as just another co during the build phase of a dependent project, or not. The overall rule still remains easy to summarise as "`exports:` do not apply when building noarch packages, unless the export is of type `noarch_to_run:`." -### Other modifications +Combined with the consistent `_to_` naming scheme, `exports: noarch_to_run: ...` is also more +self-explanatory than the current `run_exports: noarch: ...`. -Additionally, to keep the `_to_` pattern (aside from -the special case for `noarch`), we rename +### Omitted combinations + +One could ask about a possible `host_to_build:` key. While this would arguably be an even better +fit for the C++/Fortran modules ABI issue described above, the reason this proposal refrains from suggesting +such a key is to limit implementation complexity, by having an explicit order of environment resolution from +`build:` to `host:` to `run:`. Allowing both `host_to_build:` as well as `build_to_*:` would complicate this +process unnecessarily, and we believe the relevant use-cases are fully expressible using `build_to_host:` +together with constraints (such as `_fortran_modules_abi`) attached to packages that appear in `host:`. + +### Exporting happens more than once during the build + +While the `build:` to `host:` to `run:` order described above reduces procedural complexity, it still enlarges +the space of possible scenarios. Specifically, there are now at least two distinct points where exports happen, +once for `build_to_*:` and once for `host_to_run:`. This leads to new questions, of how chained exports interact. + +For example, if we have ```yaml +# output: a_complicated_package requirements: - constraints: # changed from run_constraints - - [...] + exports: + build_to_host: + - some_package_with_a_run_export ``` -because constraints only make sense when that package gets installed somewhere in any case, so the -"run_" is superfluous (aside from being inconsistent with the proposed pattern for export variants). -On top of that, the "run_" can also be confusing, because the name might be misinterpreted as being -specific to the `run:` environment, when in actuality, the `run_constraints:` of a package still -take effect also when installed into a `host:` or `build:` environment. +and -Likewise we rename `ignore_run_exports` +```yaml +# output: some_package_with_a_run_export +requirements: + exports: + host_to_run: + - the_export_in_question +``` + +and then consume it ```yaml - requirements: - ignore_exports: # changed from ignore_run_exports - from_package: - - zlib - by_name: - - libzlib +# output: mypkg +requirements: + build: + - a_complicated_package + host: + # from a_complicated_package's build_to_host export + # - some_package_with_a_run_export + run: + # ...should the host_to_run export of some_package_with_a_run_export get triggered here?! + # - the_export_in_question ``` -which should ignore any exports into `run:` or `constraints:` matching the conditions (whether -by originating package or by name of the export), regardless of which export type it comes from. +the question arises how to handle export of a package _that's not explicitly named_ in the recipe. -### Transitive compilation requirements +Obviously this is quite an impactful question, as it is certainly not desirable to trigger the exports of all +packages that happen to transitively make it into a build or host environment. -A case can be made for `host_to_host:` and `build_to_build:`, despite the natural follow-up question -why whatever is being exported in such a manner could not be a direct (run-)dependency of the package. -The answer is that there may be transitive dependencies *at compilation time* that we do not want -consumers to inherit at runtime (similar to the situation with the Fortran modules ABI). +Taken in isolation, it would be understandable to conclude from the above that the rule "only packages named +explicitly in the recipe may have their exports applied" should remain sacrosanct, and that consequently, the +answer to the above question would be that `the_export_in_question` should not get added. -An example of this is if a library `foo` depends on the headers of another library `bar` at compile-time -(possibly with restrictive version constraints), but we do not want packages built atop of `foo` to carry -along those `bar` headers or related constraints (because at that point they're not needed anymore; -the concern is about a compile-time quantity, which only concerns either `build:` and/or `host:`). +There is however an important other use-case on the horizon that needs to be taken into account for the design +here. There are technical constraints (dependencies on other efforts) why this CEP does not propose the resulting +mechanism directly, but that should not hinder us from approaching the design space holistically. -While these exports were proposed in a previous iteration of this CEP, they have now been removed. -The key reason for this is that it complicates the environment resolution process in a way that is -difficult to disentangle: for a given set of dependencies, we would have to check whether they have -any `X_to_X:` exports, but to do so, we already have to resolve the environment, in order to determine -the exact set of artefacts for which we would look up the respective `exports.json`. Once the initial -solve has been made, adding further constraints might change the environment, which may go as far as -removing the artefact that was responsible for the `X_to_X:` export in the first place! +### Future-proofing: self-exports -Such use-cases are very likely better solved by conditional dependencies, which can be resolved in a -single pass, and would avoid substantial implementation complexity stemming from the unavoidable -multi-phase resolution (and attendant issues), if this were done on the same level as cross-environment -exports. +Consider the following use-case: `libB` depends on `libA` as usual, but compiling against `libB` needs +*the specific version* of `libA` used to build `libB`, e.g. due to the way ABI and headers between +`libA` and `libB` interact. This is a scenario that happens with increasing frequency for C++ for example, +where the adoption of templates and `contexpr` functions effectively push dependencies into public headers. -### Other omitted combinations +This can be regarded as a "compile-time-only" dependency of `libB`; `libA` is not necessary at runtime, but +_compiling against_ `libB` requires the correct `libA` to be present. For example, if `libB 1.0.0 *_1` is built +against `libA=1` and `libB 1.0.0 *_2` is built against `libA=2`, then a package `mypkg` that (generically) depends +on `libB` needs to match the `libA` version that was used to build the *specific* artefact of `libB` that is in +`host:` at build time of `mypkg`. -Furthermore, one could ask about a possible `host_to_build:` key. While this would arguably be an even better -fit for the C++/Fortran modules ABI issue described above, the reason this proposal refrains from suggesting -such a key is to limit implementation complexity, by having an implicit order of environment resolution from -`build:` to `host:` to `run:`. Allowing both `host_to_build:` as well as `build_to_*:` would complicate this -process unnecessarily, and we believe the relevant use-cases are fully expressible using `build_to_host:` -together with constraints (such as `_fortran_modules_abi`) attached to packages that appear in `host:`. +To illustrate this better, let us look at some example recipes: -Summing up, `build:` can export to all others (i.e. `host:`, `run:`, `constraints:`), `host:` can export -to `run:` and `constraints:`, while nothing can be exported from either `run:` or `constraints:`. None -of these exports apply when building noarch packages, which only take into account `noarch_to_run:` exports. +```yaml +# output: libA +requirements: + # omitted: regular build/host/run environments + exports: + host_to_run: + - ${{ pin_compatible("libA") }} +``` + +This matches the operations of `run_export:` so far. Before we consider the recipe of `libB`, let us consider +`mypkg` first, because it illustrates that using `libB` _necessarily_ needs to inject a constraint on `libA` +that's not present in the consuming recipe: + +```yaml +# output: mypkg +requirements: + # omitted: regular build environment + host: + - libB + # injected! + # - libA # matches libA-constraint of libB + run: + # regular run-export from libB + # - libB >={{ver_B}},<{{next_ver_B}} + # run-export from injected libA! + # - libA >={{ver_A}},<{{next_ver_A}} + - some_other_regular_dependency +``` + +Clearly, this is an usual situation deserving of very explicit syntax. Fortunately, the `_to_` +pattern provides a very natural way to do this: + +```yaml +# output: libB +requirements: + # omitted: regular build environment + host: + - libA + run: + # regular host_to_run export from libA + # - libA >={{ver_A}},<{{next_ver_A}} + exports: + host_to_run: + - ${{ pin_compatible("libB") }} + # export that injects libA whenever libB is a named dependency in a host environment! + host_to_host: + - ${{ pin_compatible("libA") }} +``` + +The main additional complication this introduces (and why it is not being proposed in this CEP yet), is that +one cannot simply follow the previous process of: +- resolve environment +- collect any exports of concrete packages +- inject those packages into next environment + +anymore, because the resolution process becomes entangled with the exports. Doing this iteratively is not an +option because arbitrarily pathological behaviour is possible (e.g. the addition of a `host_to_host:` export +could add constraints that end up evicting the package that caused the export in the first place!). + +This can be solved in conjunction with conditional dependencies, and will be handled in a separate CEP. + +### Applying exports transitively + +The fact that there are important use cases where it is necessary to apply exports of packages not explicitly +named in the recipe provides us a concrete answer to the question about what should happen with `the_export_in_question` +in the example further up: it should be applied. + +This provides consistent behaviour between `build_to_host:` with how `build_to_build:` resp. `host_to_host:` +necessarily must operate. Instead of only applying exports from packages named in the respective environment, +the consistent rule then becomes: +"exports are applied from packages that are either explicitly named in the environment, or have been injected via exports." + +### Ignoring exports + +For the most common use, it suffices to rename `ignore_run_exports` + +```yaml + requirements: + ignore_exports: # changed from ignore_run_exports + from_package: + - zlib + by_name: + - libzlib +``` + +which should ignore the application of any exports matching the conditions (whether +by originating package or by name of the export), regardless of which export type it comes from. + +For cases where the same package is exported into several environments, or the same package is exporting +_from_ different environments, it may be necessary to ignore exports more granularly. For this purpose, +we allow qualifying the same ignore schema (i.e. `from_package:` / `by_name:`) by the target environment +(which is a more natural fit here than the originating environment of a given export). +```yaml +requirements: + ignore_exports: + to_host: + from_package: + - zlib + by_name: + - libzlib + to_run: # covers exports to both `run:` and `constraints:` + [...] + to_build: # for completeness; only relevant once `build_to_build:` exports exist + [...] +``` ### No convenience shorthand @@ -337,6 +460,34 @@ For one, it complicates the schema definition and handling unnecessarily, and sa the resulting ambiguity. Finally, using `host_to_run:` improves clarity for the recipe reader and will naturally (we believe) lead to understanding the other export types (or even run-exports as a concept in the first place). +### Other modifications + +Additionally, to keep the `_to_` pattern (aside from +the special case for `noarch`), we rename + +```yaml +requirements: + constraints: # changed from run_constraints + - [...] +``` + +because constraints only make sense when that package gets installed somewhere in any case, so the +"run_" is superfluous (aside from being inconsistent with the proposed pattern for export variants). +On top of that, the "run_" can also be confusing, because the name might be misinterpreted as being +specific to the `run:` environment, when in actuality, the `run_constraints:` of a package still +take effect also when installed into a `host:` or `build:` environment. + +### Summary + +To recap, `build:` can export to all others (i.e. `host:`, `run:`, `constraints:`), `host:` can export +to `run:` and `constraints:`, while nothing can be exported from either `run:` or `constraints:`. None +of these exports apply when building noarch packages, which only take into account `noarch_to_run:` exports. + +Packages that have been added through an export to an environment will in turn have their own exports be +applied, even if not explicitly named in the recipe. + +Finally, granular facilities exist for ignoring exports overall or into specific environments. + ## Impacts on package and channel metadata ### Background From 7fca00d54ab282206e8885cae85086f6aa1971a4 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sun, 21 Dec 2025 17:42:09 +1100 Subject: [PATCH 29/39] introduce ignore_exports.to_any; forbid bare ignore_exports --- cep-XXXX.md | 34 ++++++++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 8 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 97edd42d..f3f26963 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -395,24 +395,40 @@ the consistent rule then becomes: ### Ignoring exports -For the most common use, it suffices to rename `ignore_run_exports` +As a natural counterpart to increased variety of exports, we need to consider the facilities for overriding +exports where necessary. For the most common use, it would be tempting to simply rename `ignore_run_exports` ```yaml requirements: ignore_exports: # changed from ignore_run_exports - from_package: + from_package: # NOT PROPOSED, see below - zlib by_name: - libzlib ``` -which should ignore the application of any exports matching the conditions (whether -by originating package or by name of the export), regardless of which export type it comes from. +However, more complicated scenarios need to be expressible now (see below). Together with a desire to avoid +dynamic schemas where not absolutely necessary (c.f. also the next section about convenience shorthands), +this leads us to propose the following syntax for the general case: + +```yaml + requirements: + ignore_exports: # changed from ignore_run_exports + to_any: + from_package: + - zlib + by_name: + - libzlib +``` + +Unsurprisingly, the above instructs the build tool to ignore the application of any exports matching the +conditions (whether by originating package or by name of the export), regardless of where it comes from. + +For more complicated cases, e.g. where the same package is exported into several environments, or where the +same package is exporting _from_ different environments, it may be necessary to ignore exports more granularly. +For this purpose, we allow qualifying the same ignore schema (i.e. `from_package:` / `by_name:`) by the target +environment (which is a more natural fit here than the originating environment of a given export). -For cases where the same package is exported into several environments, or the same package is exporting -_from_ different environments, it may be necessary to ignore exports more granularly. For this purpose, -we allow qualifying the same ignore schema (i.e. `from_package:` / `by_name:`) by the target environment -(which is a more natural fit here than the originating environment of a given export). ```yaml requirements: ignore_exports: @@ -425,6 +441,8 @@ requirements: [...] to_build: # for completeness; only relevant once `build_to_build:` exports exist [...] + to_any: # trivially compatible with "general" ignore_exports + [...] ``` ### No convenience shorthand From 2f3d23f17b05b923e2a5edd278c4a2389e766cde Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sun, 21 Dec 2025 19:22:38 +1100 Subject: [PATCH 30/39] add draft for self-exports CEP --- cep-XXXY.md | 199 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 199 insertions(+) create mode 100644 cep-XXXY.md diff --git a/cep-XXXY.md b/cep-XXXY.md new file mode 100644 index 00000000..5eb41dc4 --- /dev/null +++ b/cep-XXXY.md @@ -0,0 +1,199 @@ +# CEP XXXY - Self-exports + + + + + + + + + +
Title Self-exports
Status Draft
Author(s) Axel Obermeier <h.vetinari@gmx.com>
Created Dec 21, 2025
Discussion TBD
Implementation TBD
Requires https://github.com/conda/ceps/pull/111, https://github.com/conda/ceps/pull/129
+ +## Abstract + +This CEP builds on https://github.com/conda/ceps/pull/129 and is currently just a draft. + +## Motivation + +Consider the following use-case: `libB` depends on `libA` as usual, but compiling against `libB` needs +*the specific version* of `libA` used to build `libB`, e.g. due to the way ABI and headers between +`libA` and `libB` interact. This is a scenario that happens with increasing frequency for C++ for example, +where the adoption of templates and `contexpr` functions effectively push dependencies into public headers. + +This can be regarded as a "compile-time-only" dependency of `libB`; `libA` is not necessary at runtime, but +_compiling against_ `libB` requires the correct `libA` to be present. For example, if `libB 1.0.0 *_1` is built +against `libA=1` and `libB 1.0.0 *_2` is built against `libA=2`, then a package `mypkg` that (generically) depends +on `libB` needs to match the `libA` version that was used to build the *specific* artefact of `libB` that is in +`host:` at build time of `mypkg`. + +Other use-cases that require such machinery involve the compiler stack, where various constraints of the compiler +runtimes or standard libraries should be transitive to consumers of the (non-compiler) package being built. + +To illustrate this better, let us look at some example recipes: + +```yaml +# output: libA +requirements: + # omitted: regular build/host/run environments + exports: + host_to_run: + - ${{ pin_compatible("libA") }} +``` + +This is a standard `host_to_run:` export (previously `run_export:`). Before we consider the recipe of `libB`, +let us consider `mypkg` first, because it illustrates that using `libB` _necessarily_ needs to inject a +constraint on `libA` that's not present in the consuming recipe: + +```yaml +# output: mypkg +requirements: + # omitted: regular build environment + host: + - libB + # injected! + # - libA # matches libA-constraint of libB + run: + # regular run-export from libB + # - libB >={{ver_B}},<{{next_ver_B}} + # run-export from injected libA! + # - libA >={{ver_A}},<{{next_ver_A}} + - some_other_regular_dependency +``` + +Clearly, this is an usual situation deserving of very explicit syntax. Fortunately, the `_to_` +pattern from CEP XXXX provides a very natural way to do this: + +```yaml +# output: libB +requirements: + # omitted: regular build environment + host: + - libA + run: + # regular host_to_run export from libA + # - libA >={{ver_A}},<{{next_ver_A}} + exports: + host_to_run: + - ${{ pin_compatible("libB") }} + # export that injects libA whenever libB is a named dependency in a host environment! + host_to_host: + - ${{ pin_compatible("libA") }} +``` + +The main additional complication this introduces, is that one cannot simply follow the previous process of: +- resolve environment +- collect any exports of concrete packages +- inject those packages into next environment + +anymore, because the resolution process becomes entangled with the exports. Doing this iteratively is not an +option because arbitrarily pathological behaviour is possible (e.g. the addition of a `host_to_host:` export +could add constraints that end up evicting the package that caused the export in the first place!). + +## Design + +The design of CEP XXXX explicitly considered self-exports from the beginning, so the changes are purely additive: + +```yaml +requirements: + exports: + host_to_run: + - a_shared_library + build_to_host: + - a_host_constraint =*=*foo + build_to_run: + - a_compiler_runtime + host_to_constraints: + - a_run_constraint + build_to_constraints: + - a_run_constraint + build_to_build: # NEW + - a_transitive_dependency + host_to_host: # NEW + - a_transitive_dependency + # all the above _do not_ apply when building `noarch: generic` or `noarch: python` packages + noarch_to_run: + - a_dependency_exported_when_consumer_is_noarch +``` + +Likewise, the expanded form of `ignore_exports` gains a key: +```yaml +requirements: + ignore_exports: + to_build: # NEW + from_package: + - zlib + by_name: + - libzlib + to_host: + [...] # same inner schema + to_run: + [...] + to_any: + [...] +``` + +The changes to `exports.json` follow the existing pattern and just need to add the respective keys. + +## Implementation strategy + +The key question with self-exports is how to avoid the need for multiple solves to inject the exports +in the environment being resolved. Any number of solves greater than one can trivially lead to pathological +behaviour and must therefore be avoided. + +As hinted at by "compile-time-only dependency" in the motivation section, CEP XXX1 (conditional dependencies) +provides the right tool for this. The trick to avoid multiple solves is to implement self-exports as conditional +dependencies, where the condition is a question along the lines of "is the target environment of type `host:`?". + +This allows applying (or ignoring) self-exports in the same pass that is necessary anyway to process repodata +with conditional dependencies, before it can be handed over to the SAT solver. + +To clarify this further, we need to look at the process for how this would work concretely. Assuming an +already-parsed recipe for some package `mypkg`, let us follow the resolution order mandated by CEP XXXX and +thus begin begin with a `build:`, which contains some named dependencies `[foo, baz, qux]`. Let's call this +set `B`. Further, let `foo` have `exports: build_to_build: bar`, `bar` have `exports: build_to_build: fizz`, +`fizz` have `exports: build_to_host: bang` and `bang` have `exports: host_to_run: boink`. Finally, let `baz` +have `exports: build_to_run: bla`. + +Then, the way to build `mypkg` works as follows. The tags distinguish between metadata fetching (`[meta]`), +handling self-exports (`[self]`), solver interactions (`[solve]`), as well as regular cross-environment +exports (`[cross]`) and saving the package metadata (`[save]`). + +- [meta] Fetch metadata + - Either for entire channel (before CEP 16) + - Or for sharded repodata (CEP 16), incrementally for all named dependencies, including all builds of their + transitive dependencies, as well as all of their exports and their transitive dependencies. + - Do the same for `host:` and `run:` dependencies, including their exports and all transitive dependencies. +- [self] Filter conditional dependencies of all involved packages (or the entire channel) based on current context + (here: the fact that we're solving for a `build:` environment) before handing to the SAT solver. + - In the example above, this pre-processing turns `bar` and `fizz` into regular transitive dependencies of `foo`. + - Compare with `ignore_exports.{to_build,to_any}` and remove `bar` and/or `fizz` again if they match. + - Let `B2B` be the set of `build_to_build:` exports that were not ignored. + - Translate other conditional dependencies (unrelated to self-exports) into solver constraints, see + [here](https://github.com/prefix-dev/resolvo/blob/main/src/solver/conditions.rs). +- [solve] Solve constraints to receive a concrete set of artefacts per dependency for the final `build:` environment. + - Loosely speaking, we're passing `B+B2B` to the solver. + - Assuming no matching `ignore_exports` being specified in the `mypkg` recipe, this would amount to + `[foo, baz, qux, bar, fizz]` (including their various constraints) for the above example. +- [cross] Determine `exports:` for the concrete artefacts in resolved `build:` environment. + - Collect exports, but only for packages in the set `B+B2B` (this is where we extend the "only named packages + have their exports applied" rule to also include exports of packages injected through exports themselves). + - Filter the resulting `build_to_host:` exports by `ignore_exports.{to_host,to_any}`, call this set `B2H`. + - Do the same (including filtering by `ignore_exports:`) for `build_to_run:` exports, call this set `B2R`. + - Do the same for `build_to_constraints:`, call the resulting set `B2C`. + - In the above example, `B2H=[bang]`, `B2R=[bla]` and `B2C` is empty. +- [self] Filter conditional dependencies of `host:` packages (as for `build:` above) + - Calculate `H2H` as the set of `host_to_host:` exports, not including those that match a corresponding + `ignore_export` rule. + - In the above example, `H2H` is empty. +- [solve] Solve constraints for named `host:` dependencies + `B2H` + `H2H` to get the final `host:` environment +- [cross] Determine `exports: host_to_run` for the concrete artefacts in resolved `host:` environment + - Collect exports, but only for packages in the set `H+B2H+H2H`. + - Filter `host_to_run:` exports by `ignore_exports.{to_run,to_any}`, call the resulting set `H2R`. + - Do the same for `host_to_constraints:`, call the resulting set `H2C` + - In the above example, `H2R=[boink]` and `H2C` is empty. +- [save] Save `run:` + `B2R` + `H2R` as dependencies of `mypkg`, save `B2C` + `H2C` as constraints. + +## Specification + +TODO From 735736b01a8dfcc643a2b83c3d14271c5bfced56 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sun, 21 Dec 2025 22:04:18 +1100 Subject: [PATCH 31/39] resort keys in self-exports CEP --- cep-XXXY.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/cep-XXXY.md b/cep-XXXY.md index 5eb41dc4..d7dd0a6e 100644 --- a/cep-XXXY.md +++ b/cep-XXXY.md @@ -97,20 +97,20 @@ The design of CEP XXXX explicitly considered self-exports from the beginning, so ```yaml requirements: exports: - host_to_run: - - a_shared_library + build_to_build: # NEW + - a_transitive_dependency build_to_host: - a_host_constraint =*=*foo build_to_run: - a_compiler_runtime - host_to_constraints: - - a_run_constraint build_to_constraints: - a_run_constraint - build_to_build: # NEW - - a_transitive_dependency host_to_host: # NEW - a_transitive_dependency + host_to_run: + - a_shared_library + host_to_constraints: + - a_run_constraint # all the above _do not_ apply when building `noarch: generic` or `noarch: python` packages noarch_to_run: - a_dependency_exported_when_consumer_is_noarch From 2825c2004898690ef3fce609e62cf3cdb466c7d4 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Sun, 21 Dec 2025 22:17:28 +1100 Subject: [PATCH 32/39] minor fixes --- cep-XXXY.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/cep-XXXY.md b/cep-XXXY.md index d7dd0a6e..d878a28a 100644 --- a/cep-XXXY.md +++ b/cep-XXXY.md @@ -117,6 +117,7 @@ requirements: ``` Likewise, the expanded form of `ignore_exports` gains a key: + ```yaml requirements: ignore_exports: @@ -150,7 +151,7 @@ with conditional dependencies, before it can be handed over to the SAT solver. To clarify this further, we need to look at the process for how this would work concretely. Assuming an already-parsed recipe for some package `mypkg`, let us follow the resolution order mandated by CEP XXXX and -thus begin begin with a `build:`, which contains some named dependencies `[foo, baz, qux]`. Let's call this +thus begin begin with `build:`, which shall contain some named dependencies `[foo, baz, qux]`. Let's call this set `B`. Further, let `foo` have `exports: build_to_build: bar`, `bar` have `exports: build_to_build: fizz`, `fizz` have `exports: build_to_host: bang` and `bang` have `exports: host_to_run: boink`. Finally, let `baz` have `exports: build_to_run: bla`. @@ -167,8 +168,8 @@ exports (`[cross]`) and saving the package metadata (`[save]`). - [self] Filter conditional dependencies of all involved packages (or the entire channel) based on current context (here: the fact that we're solving for a `build:` environment) before handing to the SAT solver. - In the example above, this pre-processing turns `bar` and `fizz` into regular transitive dependencies of `foo`. - - Compare with `ignore_exports.{to_build,to_any}` and remove `bar` and/or `fizz` again if they match. - - Let `B2B` be the set of `build_to_build:` exports that were not ignored. + - Compare with `ignore_exports.{to_build,to_any}` and remove any exports that match. + - Let `B2B` be the set of `build_to_build:` exports that were not ignored; here `B2B=[bar, fizz]`. - Translate other conditional dependencies (unrelated to self-exports) into solver constraints, see [here](https://github.com/prefix-dev/resolvo/blob/main/src/solver/conditions.rs). - [solve] Solve constraints to receive a concrete set of artefacts per dependency for the final `build:` environment. From cfa17a9e3b299be13b72cd63bafb11609dc6f37c Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Mon, 22 Dec 2025 09:19:27 +1100 Subject: [PATCH 33/39] distinguish "transitive" from self-exports --- cep-XXXX.md | 4 ++-- cep-XXXY.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index f3f26963..a61e3e9e 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -214,9 +214,9 @@ requirements: - a_dependency_exported_when_consumer_is_noarch # potential future additions, not proposed by this CEP, but included for completeness build_to_build: - - a_transitive_dependency + - a_compile_time_only_dependency host_to_host: - - a_transitive_dependency + - a_compile_time_only_dependency ``` As indicated by the comments, `host_to_run:` matches the existing weak run-export. If taken together with diff --git a/cep-XXXY.md b/cep-XXXY.md index d878a28a..01908112 100644 --- a/cep-XXXY.md +++ b/cep-XXXY.md @@ -98,7 +98,7 @@ The design of CEP XXXX explicitly considered self-exports from the beginning, so requirements: exports: build_to_build: # NEW - - a_transitive_dependency + - a_compile_time_only_dependency build_to_host: - a_host_constraint =*=*foo build_to_run: @@ -106,7 +106,7 @@ requirements: build_to_constraints: - a_run_constraint host_to_host: # NEW - - a_transitive_dependency + - a_compile_time_only_dependency host_to_run: - a_shared_library host_to_constraints: From 5c37964fd50f9569062a401fca052750ac4ac753 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Mon, 22 Dec 2025 09:21:53 +1100 Subject: [PATCH 34/39] improve comments in ignore_exports section --- cep-XXXX.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index a61e3e9e..937783ff 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -413,8 +413,8 @@ this leads us to propose the following syntax for the general case: ```yaml requirements: - ignore_exports: # changed from ignore_run_exports - to_any: + ignore_exports: # changed from ignore_run_exports + to_any: # additional key! from_package: - zlib by_name: @@ -438,10 +438,10 @@ requirements: by_name: - libzlib to_run: # covers exports to both `run:` and `constraints:` + [...] # same inner schema + to_build: # potential future addition; only relevant once `build_to_build:` exports exist [...] - to_build: # for completeness; only relevant once `build_to_build:` exports exist - [...] - to_any: # trivially compatible with "general" ignore_exports + to_any: # trivially combineable with "general" ignore_exports [...] ``` From 013701872810e81c2da5c2e08cefc665c5b9ade9 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Mon, 22 Dec 2025 09:29:33 +1100 Subject: [PATCH 35/39] avoid the term "run-export" when talking about future --- cep-XXXX.md | 7 ++++--- cep-XXXY.md | 5 +++-- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 937783ff..38d4dd41 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -343,9 +343,9 @@ requirements: # injected! # - libA # matches libA-constraint of libB run: - # regular run-export from libB + # regular host_to_run export from libB # - libB >={{ver_B}},<{{next_ver_B}} - # run-export from injected libA! + # host_to_run export from injected libA! # - libA >={{ver_A}},<{{next_ver_A}} - some_other_regular_dependency ``` @@ -372,6 +372,7 @@ requirements: The main additional complication this introduces (and why it is not being proposed in this CEP yet), is that one cannot simply follow the previous process of: + - resolve environment - collect any exports of concrete packages - inject those packages into next environment @@ -476,7 +477,7 @@ requirements: For one, it complicates the schema definition and handling unnecessarily, and saving a few characters is not worth the resulting ambiguity. Finally, using `host_to_run:` improves clarity for the recipe reader and will naturally -(we believe) lead to understanding the other export types (or even run-exports as a concept in the first place). +(we believe) lead to understanding the other export types (or even exports as a concept in the first place). ### Other modifications diff --git a/cep-XXXY.md b/cep-XXXY.md index 01908112..e9d17405 100644 --- a/cep-XXXY.md +++ b/cep-XXXY.md @@ -54,9 +54,9 @@ requirements: # injected! # - libA # matches libA-constraint of libB run: - # regular run-export from libB + # regular host_to_run export from libB # - libB >={{ver_B}},<{{next_ver_B}} - # run-export from injected libA! + # host_to_run export from injected libA! # - libA >={{ver_A}},<{{next_ver_A}} - some_other_regular_dependency ``` @@ -82,6 +82,7 @@ requirements: ``` The main additional complication this introduces, is that one cannot simply follow the previous process of: + - resolve environment - collect any exports of concrete packages - inject those packages into next environment From 37ce53c73243edc7ded354e8259b498d0e0f7fe1 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Mon, 22 Dec 2025 10:25:03 +1100 Subject: [PATCH 36/39] minor fixes for self-exports --- cep-XXXY.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/cep-XXXY.md b/cep-XXXY.md index e9d17405..ac40c0cc 100644 --- a/cep-XXXY.md +++ b/cep-XXXY.md @@ -93,7 +93,8 @@ could add constraints that end up evicting the package that caused the export in ## Design -The design of CEP XXXX explicitly considered self-exports from the beginning, so the changes are purely additive: +CEP XXXX explicitly considered self-exports from the beginning for the overall design of the exports overhaul, +so the changes are purely additive: ```yaml requirements: @@ -168,14 +169,15 @@ exports (`[cross]`) and saving the package metadata (`[save]`). - Do the same for `host:` and `run:` dependencies, including their exports and all transitive dependencies. - [self] Filter conditional dependencies of all involved packages (or the entire channel) based on current context (here: the fact that we're solving for a `build:` environment) before handing to the SAT solver. - - In the example above, this pre-processing turns `bar` and `fizz` into regular transitive dependencies of `foo`. + - In the example above, this pre-processing turns `bar` into a regular dependency of `foo`, while `fizz` becomes + a regular dependency of `bar` (and thus a transitive dependency of `foo`). - Compare with `ignore_exports.{to_build,to_any}` and remove any exports that match. - Let `B2B` be the set of `build_to_build:` exports that were not ignored; here `B2B=[bar, fizz]`. - Translate other conditional dependencies (unrelated to self-exports) into solver constraints, see [here](https://github.com/prefix-dev/resolvo/blob/main/src/solver/conditions.rs). - [solve] Solve constraints to receive a concrete set of artefacts per dependency for the final `build:` environment. - Loosely speaking, we're passing `B+B2B` to the solver. - - Assuming no matching `ignore_exports` being specified in the `mypkg` recipe, this would amount to + - Assuming no matching `ignore_exports` in the `mypkg` recipe, this would amount to `[foo, baz, qux, bar, fizz]` (including their various constraints) for the above example. - [cross] Determine `exports:` for the concrete artefacts in resolved `build:` environment. - Collect exports, but only for packages in the set `B+B2B` (this is where we extend the "only named packages @@ -184,16 +186,19 @@ exports (`[cross]`) and saving the package metadata (`[save]`). - Do the same (including filtering by `ignore_exports:`) for `build_to_run:` exports, call this set `B2R`. - Do the same for `build_to_constraints:`, call the resulting set `B2C`. - In the above example, `B2H=[bang]`, `B2R=[bla]` and `B2C` is empty. -- [self] Filter conditional dependencies of `host:` packages (as for `build:` above) +- [self] Filter conditional dependencies of `host:` packages (as for `build:` above). - Calculate `H2H` as the set of `host_to_host:` exports, not including those that match a corresponding `ignore_export` rule. - In the above example, `H2H` is empty. -- [solve] Solve constraints for named `host:` dependencies + `B2H` + `H2H` to get the final `host:` environment +- [solve] Solve constraints for `H` (named `host:` dependencies) + `B2H` + `H2H` to get the final `host:` + environment - [cross] Determine `exports: host_to_run` for the concrete artefacts in resolved `host:` environment - Collect exports, but only for packages in the set `H+B2H+H2H`. - Filter `host_to_run:` exports by `ignore_exports.{to_run,to_any}`, call the resulting set `H2R`. - Do the same for `host_to_constraints:`, call the resulting set `H2C` - - In the above example, `H2R=[boink]` and `H2C` is empty. + - For the above example we did not specify `host:` dependencies or their exports; we can assume them to be + empty. Nevertheless, `H2R=[boink]` would not be empty, due to a chain of exports originating from the + `build:` environment. Since there was no `host_to_constraints:` export in the example, `H2C` remains empty. - [save] Save `run:` + `B2R` + `H2R` as dependencies of `mypkg`, save `B2C` + `H2C` as constraints. ## Specification From c68696993e854172521e68cd5de5fec916ba6241 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Mon, 22 Dec 2025 10:27:15 +1100 Subject: [PATCH 37/39] add a mermaid graph --- cep-XXXY.md | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/cep-XXXY.md b/cep-XXXY.md index ac40c0cc..24f32e68 100644 --- a/cep-XXXY.md +++ b/cep-XXXY.md @@ -158,6 +158,62 @@ set `B`. Further, let `foo` have `exports: build_to_build: bar`, `bar` have `exp `fizz` have `exports: build_to_host: bang` and `bang` have `exports: host_to_run: boink`. Finally, let `baz` have `exports: build_to_run: bla`. +To illustrate this, the following graph indicates named dependencies in white, exported dependencies in black, +and exports as red arrows: + +```mermaid +flowchart TB + classDef yamlSection fill:#b0b0b0,stroke:#e6e6e6,stroke-width:0.5px,font-family:monospace,color:#000; + classDef namedDep fill:#ffffff,stroke:#ffffff,font-family:monospace,color:#000; + classDef implicitDep fill:#000000,stroke:#00000,font-family:monospace,color:#fff; + + subgraph build["build:"] + direction TB + foo["- foo"] + bar["# - bar"] + fizz["# - fizz"] + baz["- baz"] + qux["- qux"] + end + + subgraph host["host:"] + direction TB + bang["# - bang"] + end + + subgraph run["run:"] + direction TB + bla["# - bla"] + boink["# - boink"] + end + + %% main exports + foo-->bar + bar-->fizz + fizz-->bang + bang-->boink + + %% enforce vertical ordering + baz~~~host + qux~~~host + foo~~~bar + baz~~~qux + bar~~~fizz + fizz~~~baz + bang~~~bla + bla~~~boink + + %% main export, but needs sorting at the end to appear on RHS of graph + baz-->bla + + %% apply line style; corresponding to index of edges as they appear in definition + linkStyle 0,1,2,3,12 stroke:#ff0000,stroke-width:3px; + + class build,host,run yamlSection; + class foo,baz,qux namedDep; + class bar,fizz,bang,boink,bla implicitDep; +``` + Then, the way to build `mypkg` works as follows. The tags distinguish between metadata fetching (`[meta]`), handling self-exports (`[self]`), solver interactions (`[solve]`), as well as regular cross-environment exports (`[cross]`) and saving the package metadata (`[save]`). From f0dccba7d7541417b967d3eff67aa632d5cac9ea Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Mon, 22 Dec 2025 10:48:59 +1100 Subject: [PATCH 38/39] add a missing word --- cep-XXXY.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-XXXY.md b/cep-XXXY.md index 24f32e68..c8f929be 100644 --- a/cep-XXXY.md +++ b/cep-XXXY.md @@ -255,7 +255,7 @@ exports (`[cross]`) and saving the package metadata (`[save]`). - For the above example we did not specify `host:` dependencies or their exports; we can assume them to be empty. Nevertheless, `H2R=[boink]` would not be empty, due to a chain of exports originating from the `build:` environment. Since there was no `host_to_constraints:` export in the example, `H2C` remains empty. -- [save] Save `run:` + `B2R` + `H2R` as dependencies of `mypkg`, save `B2C` + `H2C` as constraints. +- [save] Save `run:` + `B2R` + `H2R` as dependencies of `mypkg`, save `constraints:` + `B2C` + `H2C` as constraints. ## Specification From 59a119af57ddf4b89f5f94c0ce0522db137e7357 Mon Sep 17 00:00:00 2001 From: "H. Vetinari" Date: Fri, 20 Feb 2026 16:23:11 +1100 Subject: [PATCH 39/39] fold `ignore_exports:` under `exports:`, reflect inheritance necessary for cache CEP --- cep-XXXX.md | 234 ++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 162 insertions(+), 72 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 38d4dd41..2c1cd028 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -169,6 +169,23 @@ the ground up based on all the lessons learned in the last decade would be much lowering the barrier to entry into the ecosystem, and helping existing contributors navigate packaging challenges more easily. +### The many uses of exports + +What began as a simple idea (e.g. to inject a runtime dependency on a shared library from `host:` +to `run:`) expressible with a single verb, _to export_ (and its nominalization into `run_exports:`) +has turned into a key concept across many use-cases. As a consequence, exports have become ubiquitous. + +Not only that, but we need to describe fundamentally different processes about exports: + +- Attaching an export (e.g. to a package `foo`) +- Applying an export (e.g. when building a package `bar` depending on `foo`) +- Ignoring an export (e.g. if `bar` needs `foo` present, but doesn't want the export itself) +- Inheriting an export (when talking about `package:`-global builds influencing their `outputs:`, + which may be `files:`-based and thus have no `build:`/`host:` environments of their own, yet + still need to reflect the runtime dependencies of their content) + +These actions are currently either implicit or dispersed incongruously across the recipe. + ## Design We begin with the following observations based on the above: @@ -186,6 +203,7 @@ We begin with the following observations based on the above: which have so far been under-served by the existing run-export infrastructure. - Run-exports are not applied when building `noarch` packages, except if the export is of type `noarch:`. - The v1 recipe format unified all requirement-related topics (including run-exports) under `requirements:`. +- Exports and their associated actions should be consolidated as well. Based on this, we propose the following pattern: @@ -199,28 +217,60 @@ requirements: - [...] # relying on the surrounding "requirements" key for context exports: - host_to_run: # matches `weak:` run-export - - a_shared_library - build_to_host: # "host-export" - - a_host_constraint =*=*foo - build_to_run: # produces same effect as `strong:` run-export when used together with build_to_host - - a_compiler_runtime - host_to_constraints: # matches `weak_constrains:` (v0) / `weak_constraints:` (v1) - - a_run_constraint - build_to_constraints: # matches `strong_constrains:` (v0) / `strong_constraints:` (v1) - - a_run_constraint - # all the above _do not_ apply when building `noarch: generic` or `noarch: python` packages - noarch_to_run: # matches `noarch:` run-export; _does_ apply when building noarch packages - - a_dependency_exported_when_consumer_is_noarch - # potential future additions, not proposed by this CEP, but included for completeness - build_to_build: - - a_compile_time_only_dependency - host_to_host: - - a_compile_time_only_dependency + # exports that the output itself will apply when *used* for building other packages + attach: + host_to_run: # matches `weak:` run-export + - a_shared_library + build_to_host: # "host-export" + - a_host_constraint =*=*foo + build_to_run: # produces same effect as `strong:` run-export when used together with build_to_host + - a_compiler_runtime + host_to_constraints: # matches `weak_constrains:` (v0) / `weak_constraints:` (v1) + - a_run_constraint + build_to_constraints: # matches `strong_constrains:` (v0) / `strong_constraints:` (v1) + - a_run_constraint + # all the above _do not_ apply when building `noarch: generic` or `noarch: python` packages + noarch_to_run: # matches `noarch:` run-export; _does_ apply when building noarch packages + - a_dependency_exported_when_consumer_is_noarch + # potential future additions (see CEP-XXXY), but included for completeness + build_to_build: + - a_compile_time_only_dependency + host_to_host: + - a_compile_time_only_dependency + + # exports from dependencies in the current output that should be ignored when *building* + ignore: + to_any: # default; `exports: ignore: foo` maps to `exports: ignore: to_any: foo` + from: + - zlib + by_name: + - libzlib + to_run: + [...] # same inner schema + to_host: + [...] # same inner schema + # potential future addition (see CEP-XXXY) + to_build: + [...] + + # exports to be inherited from staging output while *building* the output; + # NOT PROPOSED here, but relevant for https://github.com/conda/ceps/pull/102 + inherit: + to_any: # default; `exports: inherit: foo` maps to `exports: inherit: to_any: foo` + from: + - name-of-staging-output + to_{run,host,build}: + [...] # same inner schema ``` -As indicated by the comments, `host_to_run:` matches the existing weak run-export. If taken together with -`build_to_run:` this produces the same effect of a strong run-export. Similarly for `host_to_constraints:` +This unifies all export-related things under `requirements.exports`, as well as collecting the +different actions `{attach,ignore}` (plus potentially `inherit` in the future) in one place. We'll look +at the `ignore:` side further down; let us start with the heart of the matter: to `attach:` an export +to the package being built. + +As indicated by the comments, `host_to_run:` matches the existing weak run-export attached to a package. +If taken together with `build_to_run:`, this produces the same effect of a strong run-export. +Similarly for `host_to_constraints:` and `build_to_constraints:`. The other keys introduce new functionality, which is explained below. This design has the advantage that it's @@ -256,7 +306,7 @@ together with constraints (such as `_fortran_modules_abi`) attached to packages While the `build:` to `host:` to `run:` order described above reduces procedural complexity, it still enlarges the space of possible scenarios. Specifically, there are now at least two distinct points where exports happen, -once for `build_to_*:` and once for `host_to_run:`. This leads to new questions, of how chained exports interact. +once for `build_to_*:` and once for `host_to_run:`. This leads to new questions how chained exports interact. For example, if we have @@ -264,8 +314,9 @@ For example, if we have # output: a_complicated_package requirements: exports: - build_to_host: - - some_package_with_a_run_export + attach: + build_to_host: + - some_package_with_a_run_export ``` and @@ -274,8 +325,9 @@ and # output: some_package_with_a_run_export requirements: exports: - host_to_run: - - the_export_in_question + attach: + host_to_run: + - the_export_in_question ``` and then consume it @@ -295,7 +347,7 @@ requirements: the question arises how to handle export of a package _that's not explicitly named_ in the recipe. -Obviously this is quite an impactful question, as it is certainly not desirable to trigger the exports of all +Obviously this is quite an impactful question, as it is certainly not desirable to apply the exports of all packages that happen to transitively make it into a build or host environment. Taken in isolation, it would be understandable to conclude from the above that the rule "only packages named @@ -326,8 +378,9 @@ To illustrate this better, let us look at some example recipes: requirements: # omitted: regular build/host/run environments exports: - host_to_run: - - ${{ pin_compatible("libA") }} + attach: + host_to_run: + - ${{ pin_compatible("libA") }} ``` This matches the operations of `run_export:` so far. Before we consider the recipe of `libB`, let us consider @@ -363,11 +416,12 @@ requirements: # regular host_to_run export from libA # - libA >={{ver_A}},<{{next_ver_A}} exports: - host_to_run: - - ${{ pin_compatible("libB") }} - # export that injects libA whenever libB is a named dependency in a host environment! - host_to_host: - - ${{ pin_compatible("libA") }} + attach: + host_to_run: + - ${{ pin_compatible("libB") }} + # export that injects libA whenever libB is a named dependency in a host environment! + host_to_host: + - ${{ pin_compatible("libA") }} ``` The main additional complication this introduces (and why it is not being proposed in this CEP yet), is that @@ -397,53 +451,74 @@ the consistent rule then becomes: ### Ignoring exports As a natural counterpart to increased variety of exports, we need to consider the facilities for overriding -exports where necessary. For the most common use, it would be tempting to simply rename `ignore_run_exports` +exports where necessary. For the most common use, we simply rename `ignore_run_exports` ```yaml requirements: - ignore_exports: # changed from ignore_run_exports - from_package: # NOT PROPOSED, see below - - zlib - by_name: - - libzlib + exports: + ignore: # changed from ignore_run_exports + from: # changed from "from_package" + - zlib + by_name: + - libzlib +``` + +The reason to also change `from_package:` to `from:` is that this will make it structurally obvious how +complementary ignored exports are with inherited ones (see https://github.com/conda/ceps/pull/102). +More specifically, for a staging output `libxml2-split` that gets split into different components (e.g. +`libxml2`, `libxml2-devel`, etc.), the following would be a no-op: + + +```yaml +requirements: + exports: + inherit: + from: + - libxml2-split + ignore: + from: + - libxml2-split ``` -However, more complicated scenarios need to be expressible now (see below). Together with a desire to avoid -dynamic schemas where not absolutely necessary (c.f. also the next section about convenience shorthands), -this leads us to propose the following syntax for the general case: +This illustrates the duality of the two operations, and should make it easy for recipe authors to +dial in the desired behaviour by inheriting or ignoring exports more granularly. + +To enable the necessary flexibility, we propose the following syntax for the general case: ```yaml requirements: - ignore_exports: # changed from ignore_run_exports - to_any: # additional key! - from_package: - - zlib - by_name: - - libzlib + exports: + ignore: + to_any: # additional key! + from: + - zlib + by_name: + - libzlib ``` Unsurprisingly, the above instructs the build tool to ignore the application of any exports matching the conditions (whether by originating package or by name of the export), regardless of where it comes from. -For more complicated cases, e.g. where the same package is exported into several environments, or where the -same package is exporting _from_ different environments, it may be necessary to ignore exports more granularly. -For this purpose, we allow qualifying the same ignore schema (i.e. `from_package:` / `by_name:`) by the target +In more complicated cases, e.g. where the same dependency is exported into several environments, or where same +dependency is being exported _from_ different environments, it may be necessary to ignore exports more granularly. +For this purpose, we allow qualifying the same ignore schema (i.e. `from:` / `by_name:`) by the target environment (which is a more natural fit here than the originating environment of a given export). ```yaml requirements: - ignore_exports: - to_host: - from_package: - - zlib - by_name: - - libzlib - to_run: # covers exports to both `run:` and `constraints:` - [...] # same inner schema - to_build: # potential future addition; only relevant once `build_to_build:` exports exist - [...] - to_any: # trivially combineable with "general" ignore_exports - [...] + exports: + ignore: + to_host: + from: + - zlib + by_name: + - libzlib + to_run: # covers exports to both `run:` and `constraints:` + [...] # same inner schema as `to_host:` + to_build: # potential future addition (CEP-XXXY) + [...] + to_any: # trivially combineable with universally ignored exports + [...] ``` ### No convenience shorthand @@ -472,7 +547,17 @@ we do not believe it is worth allowing a similar shortcut for `exports.host_to_r ```yaml requirements: exports: - - libfoo # NOT PROPOSED! + - libfoo # NOT PROPOSED! +``` + +The above would be fundamentally at odds with unifying `attach:` & `ignore:` under `exports:`. +Likewise, we don't propose: + +```yaml +requirements: + exports: + attach: + - libfoo # NOT PROPOSED! ``` For one, it complicates the schema definition and handling unnecessarily, and saving a few characters is not worth @@ -531,6 +616,9 @@ do not even appear in the regular metadata, only in `run_exports.json`. CEP 21 (building on top of [CEP 16](cep-0016.md)) added channel-level run-export information, though in contrast to CEP 12, added this to the physically sharded but logically unified repodata. +Finally, only the exports under `exports.attach:` actually have any effect after building the package in question, +so these are the only quantities worth reflecting in the channel metadata. + ### Transition plan It's easy to map the new export structure to the respective metadata; the complexity lies in providing a smooth @@ -543,7 +631,7 @@ simplified. We suggest to: - Output-level: - Add another `exports.json` next to `repodata_record.json`, to be preferred over `run_exports.json` by tools which know how to handle it. - - Populate `run_exports.json` with "compatible" metadata derived from `exports:` (see below). + - Populate `run_exports.json` with "compatible" metadata derived from `exports.attach:` (see below). - Channel-level: - Add a `exports.json` file to the monolithic channel metadata, to be preferred over `run_exports.json` by tools which know how to handle it. @@ -588,10 +676,11 @@ TODO! ### Package and Channel Metadata -On output-level, if there are any non-empty `exports:` specified, build tools MUST produce an `exports.json` -in the root of the artefact (next to `index.json` etc.), and populate the values with the exports as specified -in the rendered recipe for that output. If the output has no (or empty) `exports:`, creation of `exports.json` -MAY be omitted. For the value of each key under `exports:`, before creating `exports.json` and in the following +On output-level, if there are any non-empty `exports.attach:` sections specified, build tools MUST produce an +`exports.json` in the root of the artefact (next to `index.json` etc.), and populate the values with the +attached exports as specified +in the rendered recipe for that output. If the output has no (or empty) `exports.attach:`, creation of `exports.json` +MAY be omitted. For the value of each key under `exports.attach:`, before creating `exports.json` and in the following order, tools: - MUST error on illegal `PackageSelector`s (as defined in CEP 14), @@ -614,7 +703,7 @@ If the file `exports.json` gets created, its content MUST be a valid JSON object We define the following translation between this schema and previous versions of the `run_exports:` schema: -| `exports:` | `run_exports:` (v0) | `run_exports:` (v1) | +| `exports.attach:` | `run_exports:` (v0) | `run_exports:` (v1) | |---|---|--| | `build_to_constraints:` | `strong_constrains:` | `strong_constraints:` | | `build_to_host:` | `strong:` | `strong:` | @@ -623,7 +712,7 @@ We define the following translation between this schema and previous versions of | `host_to_run:` | `weak:` | `weak:` | | `noarch_to_run:` | `noarch:` | `noarch:` | -Build tools MUST populate the output-level `run_exports.json` file with the payload of the `exports:` object +Build tools MUST populate the output-level `run_exports.json` file with the payload of the `exports.attach:` object (for the respective output in the rendered recipe) as follows: they MUST translate keys to the v0 schema per the table above (while leaving the corresponding values unchanged). If both `build_to_host:` and `build_to_run:` have non-empty values for @@ -673,7 +762,8 @@ have `exports.json` metadata, the values in `exports:` MUST be populated from th Indexers MUST (continue to) populate the channel-level `run_exports.json` from the output-level `run_exports.json`. For sharded repodata following CEP 16 & 21, indexers MUST add an `exports:` key and populate it with the respective -output-level metadata. Where outputs do not yet provide `exports.json`, the values of `exports:` MUST be populated +output-level `exports.attach` metadata. Where outputs do not yet provide `exports.json`, the values of +`exports:` in the sharded repodata MUST be populated from the respective keys in `run_exports:` according to the above schema mapping. Furthermore, indexers MUST (continue to) populate the value of `run_exports:` with the aggregation of output-level `run_exports.json`.