diff --git a/cep-xxxx.md b/cep-xxxx.md new file mode 100644 index 00000000..3ca38c31 --- /dev/null +++ b/cep-xxxx.md @@ -0,0 +1,258 @@ +# CEP XXXX - Specification of conda_build_config.yaml variant configuration files + + + + + + + + + + +
Title Specification of conda_build_config.yaml variant configuration files
Status Draft
Author(s) Ryan Keith <rkeith@anaconda.com>
Created Jul 17, 2026
Updated Jul 17, 2026
Discussion NA
Implementation NA
Requires NA
+ +> The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119][RFC2119] when, and only when, they appear in all capitals, as shown here. +> +> [RFC2119]: https://datatracker.ietf.org/doc/html/rfc2119 + +## Abstract + +This CEP standardizes the format of conda build variant configuration YAML files (most commonly named `conda_build_config.yaml` or CBC as a shorthand). The goal being a single, testable contract shared by `conda-build`, `rattler-build`, and channel-wide configuration (for example conda-forge’s and Anaconda’s global CBCs). + +## Motivation + +Historically, important conda ecosystem files standards have not been fully spec'd. In particular, the CBC file is used for `conda-build` and is a valid input for `rattler-build`. It is used in both recipe and channel specs. It is foundational configuration of `main`, `conda-forge` and `bioconda` in order to set channel global settings so the channel ecosystem can solve globally. + +During a recent bug fix for conda-build, it was noticed that the exact same rules were being applied to the CBC file as package recipe files. Though similar, it is important to distinguish their roles and thus provide a rules and a schema for documentation and linters. + +Variant configuration drives the build matrix, compiler pins, `zip_keys` coupling, and platform-specific defaults across the ecosystem. Behavior is described primarily in conda-build documentation and implemented in a couple tools, but no CEP defines the file format today. + +This CEP proposes a standard for CBC files in order to officially declare what the exact standards are and reduce drift for evolving standards. + +## Nomenclature + +- **CBC**: informal shorthand for `conda_build_config.yaml`. +- **cbc.yaml**: also another shortand seen in some documentation and in code repositories. Not used in this document. +- **Variant configuration file** (VCF): a YAML document that supplies the variant dictionary used at render/build time. +- **Canonical filename**: `conda_build_config.yaml` (underscores). This is the name recognized by conda-build discovery, rattler-build legacy loading, and both: + - [AnacondaRecipes/aggregate](https://github.com/AnacondaRecipes/aggregate/blob/master/conda_build_config.yaml) + - [conda-forge-pinning-feedstock](https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml) + +## Relationship to other standards + +| Standard | Relationship | +|----------|----------------| +| [CEP 24](./cep-0024.md) | The `environment.yml` specification standard. Both use comment-based `# []` selectors. | +| [CEP 39](./cep-0039.md) | v1 recipes consume variant keys; CBC is the usual source for global/channel keys like `c_compiler`. | +| [CEP 40](./cep-0040.md) | `variant_config.yaml` in artifacts records keys *used* in one build, not the full CBC. | +| [CEP 34](./cep-0034.md) | `./info/hash_input.json` stores the resolved variant subset that affected the hash. | +| [CEP 14](./cep-0014.md) | Per-recipe `variant:` overrides complement but do not replace CBC. | + +## Rationale + +The documented purpose of `conda_build_config.yaml` (CBC) is to define *variants*: building a binary package multiple times against different pinned dependencies to support different usage environments (for example, building against different NumPy or Python C ABIs). This is a per-recipe, matrix-generating mechanism, and it is the only purpose the upstream conda-build documentation describes. + +A second purpose is built on the same machinery: a centralized CBC serves as a channel-wide ABI/binary-compatibility baseline. By declaring the canonical version of each shared dependency (e.g. `openssl`, `libabseil`, `hdf5`) in a single file, a channel establishes the ABI that all of its packages build against. + +The altnerative to not having a centeralized CBC file would be updating many individual recipes. This would become difficult to maintain over time and over a large number of package recipes. + + Both Anaconda's `main` channel and `conda-forge` build from their own global CBC files (`AnacondaRecipes/aggregate` and `conda-forge-pinning`, respectively) and because those baselines can differ, packages from the two channels are not always ABI-compatible. This is one reason mixing the channels in a single environment can cause problems. + +These two purposes operate on different axes. The first answers *what to build* (the matrix); the second answers *what to build against* (the baseline). CBC is the shared substrate for both, which is why the baseline role has remained implicit in the upstream documentation despite being the dominant real-world use. + +### Document requirements + +- A variant configuration file (VCF) is a single YAML document whose root is a mapping. +- The canonical filename is `conda_build_config.yaml`. CBC is informal shorthand used in prose and documentation, not a filename the format defines. +- Keys at the root are either variant keys (arbitrary names that become variables in recipe templating) or one of the reserved keys defined in Reserved top-level keys. + +### Variant keys + +Variants are defined as name/value pairs. Each such name becomes a variable available to recipe templating, and its value supplies what that variable can be. (Names with special meaning — the [reserved keys](#reserved-top-level-keys) — are the exception.) + +- A variant key's name must be a valid Jinja identifier: letters, digits, and underscores, not starting with a digit. **No hyphens**. Any case is fine (`openssl`, `CONDA_BUILD_SYSROOT`). +- Its value is a scalar, or a list of scalars. A bare scalar means a one-element list. +- A scalar is an opaque token — a version (`"1.14.6"`), a `version build` pair (`"3.9.* *netlib"`), a name (`gcc`), a path, an int, or a boolean. It is substituted verbatim; match-spec operators (`>=`, `<`) are not used. +- Quote version-like values so YAML keeps them as strings (`python: ["3.10", "3.11"]`); unquoted, `3.10` parses as the float `3.1`. +- Values are data, not templates — no Jinja (`{{ }}`). + +### Build matrix + +A variant key with more than one value defines an axis of variation. When several keys each carry multiple values, the builds produced are the Cartesian product of those axes. This is the per-recipe "define variants" purpose: building the same recipe against, for example: + +- Building a package against several versions of Python (probably the most common use case) +- Several Python versions crossed with different modalites of a library (such as CPU and GPU verisons) + +In practice, channel-wide baselines exercise this very little. In both `AnacondaRecipes/aggregate` and `conda-forge-pinning`, nearly every key holds a single value, and the few multi-valued keys (such as `python`/`numpy`) are coupled with [`zip_keys`](#reserved-top-level-keys) rather than multiplied. The matrix collapses to a near-single point, and the file acts as a set of pinned defaults. + +Two mechanisms alter the default product: + +- [`zip_keys`](#reserved-top-level-keys) couples axes so they advance together instead of multiplying. +- [Preprocessing selectors](#preprocessing-selectors) remove values that do not apply to the target platform. + +Which combinations a tool actually materializes — build order, skipping, deduplication — is outside this specification. + +### Reserved top-level keys + +Five root-level keys have defined meaning and are not treated as variant keys. They are used to control config file behavior. This set is fixed (conda-build handles them specially in variant parsing): + +| Key | Value shape | Role | +|-----|-------------|------| +| `zip_keys` | list of key names, or list of lists of names | Couples variant keys so their value lists advance together by index rather than forming a Cartesian product. | +| `pin_run_as_build` | map of package name → `{min_pin, max_pin}` | Sets the default run pin for packages that are both build and run dependencies. | +| `ignore_version` | list of key names | Suppresses automatic version pinning for the listed keys. | +| `ignore_build_only_deps` | list of key names | Excludes the listed keys from affecting the build environment only. Defaults to `python`, `numpy`. | +| `extend_keys` | list of key names | Names keys whose values aggregate across combined sources instead of being replaced. Includes the reserved keys above and is itself extendable. | + +Package names inside `pin_run_as_build` may contain hyphens (e.g. `r-base`); the no-hyphen rule below applies only to top-level key names. + +#### `zip_keys` forms + +A single group is a flat list: + + zip_keys: + - python + - numpy + +Multiple independent groups are a list of lists: + + zip_keys: + - [c_compiler_version, cxx_compiler_version] + - [python, is_python_min] + +Also valid for multiple independent groups (more common as well): + + zip_keys: + - + - c_compiler_version + - cxx_compiler_version + - + - python + - is_python_min + +A CBC MUST satisfy, for `zip_keys`: + +- every key named in a group is defined in the configuration; +- no key appears in more than one group; +- all keys within a group have value lists of equal length. + +### Preprocessing selectors + +A line in a CBC may carry a trailing selector, `# []`. Before the YAML is parsed, each such line is kept if `` is truthy and dropped otherwise. Selection is line-by-line on raw text, prior to loading. + +`` is a restricted expression. Comparisons, `and`/`or`/`not`, `in`, attribute access, and a fixed set of calls are allowed, not arbitrary code. + +#### Available names + +Selectors evaluate against names external to the file: + +- **Platform and architecture predicates** (booleans): `linux`, `osx` `win`, `unix`, `linux64`, `win64`, `arm`, `x86`, per-subdir OS/arch names, and `build_platform`. +- **Environment variables**: every set variable is available both as a bare name and via `environ` / `os.environ.get(...)`. + +An unknown name (one not defined above) evaluates as `False`, dropping its line. This is how an optional flag works: `# [SOME_FLAG]` is false when unset and true when the variable is present. + +Note the truthiness trap with bare flags: a bare `# [FLAG]` is true whenever the variable is a non-empty string — so `FLAG=False` is *truthy* and keeps the line. To gate on an explicit value, compare it: `# [os.environ.get("FLAG", "False") == "True"]`. Both idioms appear in practice — aggregate uses the bare form, conda-forge the explicit `== "True"` form for its CUDA gating. + +#### Selectors must not branch on file-defined values + +A VCF exists to *declare* the build axes and pins. A selector in a VCF MUST NOT branch on a value the same file defines. In particular, the interpreter-derived names, such as, `py`, `py2k`, `py3k`, `py26`–`py315`, `np`, `pl`, `lua`, `luajit`, MUST NOT be used: they are computed from the `python`, `numpy`, `perl`, and `lua` variant keys, so branching on them re-derives, inside the file, a value the file is meant to set. That circularity does not belong in a definition. + +Per-interpreter conditionals belong in recipe metadata, not in a VCF. Within a VCF, a value that must vary across an axis is expressed by defining the axis directly and, where values must correspond, coupling keys with [`zip_keys`](#reserved-top-level-keys). + +> *Why this matters.* Limiting VCF selectors to platform and environment predicates keeps the +> file evaluable from the build target and environment alone — no interpreter/variant namespace, +> no Jinja — which permits a simpler, faster VCF reader than full recipe rendering requires. + +### Disallowed in CBCs + +- **Jinja templating.** A VCF MUST NOT contain Jinja: neither `{{ ... }}` expressions nor `{% ... %}` statements. A VCF is not rendered as a template — it supplies the values that recipe Jinja later consumes. Selectors (`# [...]`) are the only reprocessing a VCF undergoes. +- **Hyphens in key names.** A top-level variant key name MUST NOT contain `-` (the name is used as a template variable). Hyphenated package names MAY appear as keys under pin_run_as_build (e.g. netcdf-cxx4, r-base). The no-hyphen rule applies only to top-level CBC keys. +- **Interpreter-derived selectors.** `py`, `np`, and the other variant-derived selector names MUST NOT appear in a VCF (see [Preprocessing selectors](#preprocessing-selectors)). + +### Merging + +A build may draw variant configuration from more than one CBC. This CEP specifies how an ordered sequence of sources combines into a single variant dictionary. + +Sources are applied in order. A later source's value **replaces** an earlier one for that key. + +Example (later source `b` wins for both keys): + +```python +a = {"python": ["2.7", "3.5"], "numpy": ["1.10", "1.11"]} +b = {"python": ["3.14", "3.15"], "numpy": ["2.3", "2.5"]} +# merged = {"python": ["3.14", "3.15"], "numpy": ["2.3", "2.5"]} +``` + +#### Reserved keys + +Reserved keys are **additive** across sources (they do not last-wins overwrite): + +| Key | Accumulation | +|-----|----------------| +| `pin_run_as_build` | Map merge (later entry for the same package name wins) | +| `ignore_version`, `ignore_build_only_deps`, `extend_keys` | List concat, then de-duplicated | +| `zip_keys` | Whole groups are appended and de-duplicated; existing groups are not grown or edited in place | + +#### `extend_keys` + +Keys listed in `extend_keys` accumulate across sources. Values are concatenated and then de-duplicated (order not preserved). + +Every source that sets an extended key MUST also list that key under `extend_keys`, or the merge MUST fail. + +## Backwards compatibility + +This CEP largely specifies existing conda-build behavior; most of it is descriptive and introduces no change. The normative constraints have limited impact: + +- **Jinja prohibition** and the **no-hyphen rule** for key names match what conda-build already does +- **Both channel baselines already conform.** `AnacondaRecipes/aggregate` and `conda-forge-pinning` use only platform and environment-driven selectors; neither uses interpreter-derived selectors. The selector restriction costs them nothing. +- **The one potential breakage** is a recipe-local CBC that uses an interpreter-derived selector (`# [py