From 8c545ba3469ace150a9166e22b61d3512496bdfb Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Tue, 23 Dec 2025 15:09:34 -0500 Subject: [PATCH 01/38] Add CEP for Repodata Wheel Support Co-authored-by: Travis Hathaway --- cep-XXXX.md | 561 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 561 insertions(+) create mode 100644 cep-XXXX.md diff --git a/cep-XXXX.md b/cep-XXXX.md new file mode 100644 index 00000000..b769d2a1 --- /dev/null +++ b/cep-XXXX.md @@ -0,0 +1,561 @@ +# CEP XXXX - Repodata Wheel Support + + + + + + + + + +N/A +
Title Repodata Wheel Support
Status Draft
Author(s) + Dan Yeaw <dyeaw@anaconda.com>
+ Travis Hathaway <travis.j.hathaway@gmail.com> +
Created Dec 23, 2025
Updated Dec 23, 2025
Discussion https://github.com/conda/ceps/pull/
Implementation TBD
Requires
+ +> The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", +"RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as +described in [RFC2119][RFC2119] when, and only when, they appear in all capitals, as shown here. + +## Abstract + +This CEP outlines how native support for pure Python wheel packages will be achieved by adding support for them to conda's package index (repodata). When implemented, conda clients will be able to seamlessly install conda packages and pure Python wheels from enabled channels. We explicitly limit the scope of this CEP to pure Python wheels to avoid platform-specific binary compatibility issues, but even with this limitation, we dramatically expand the number of available packages for users in the conda ecosystem. + +## Motivation + +While conda remains a language-agnostic packaging distribution, installing packages for use with Python represents the majority of conda usage. Users frequently encounter packages only available as wheels on PyPI, forcing them to use hybrid workflows that mix conda and pip installations. This approach creates several problems: + +- Mixing of conda and pip or uv can result in overwritten files and broken environments +- Users must understand two package managers, their interaction points, and which of their dependencies are available from which ecosystem +- Where there is support for mixing environments, it requires multiple solves which reduces overall package installation performance + +By adding native support for pure Python wheels to repodata, conda clients can: + +- Resolve dependencies across conda and PyPI packages in a single solve +- Provide users with transparent access to the broader Python ecosystem +- Maintain environment consistency and reproducibility +- Eliminate the cognitive burden of managing two package managers +- Fill gaps in conda package availability without requiring new conda builds +- Reduce the maintenance burden by fully or partially eliminating the need to create and maintain conda recipes for pure-Python packages + +## Specification + +### Add a new top-level key `packages.whl` to list wheels in a channel + +According to the current draft schema for [repodata.json][repodata-schema], repodata consists of five top +level keys: + +- repodata_version +- info +- packages +- packages.conda +- removed +- signatures + +This CEP proposes the addition of a new `packages.whl` section to account for the wheel format. This key points to a mapping that MUST contain [repodata record][repodata-record-schema] objects. The key of this mapping MUST be the wheel file name, and the contents of the package record MUST contain metadata related to that wheel. + +### Record structure and naming convention + +To support sparse repodata processing and maintain compatibility with conda's existing infrastructure, wheel records MUST use conda-style naming conventions rather than wheel filenames as keys. This allows solvers to efficiently filter packages by name without parsing the full package record. + +### Key naming requirements + +The key for each entry in `packages.whl` must follow this format: + +``` +{name}-{version} +``` + +Examples: + +- `httpx-0.28.1` +- `typing_extensions-4.15.0` +- `lazy-loader-0.4` + +### Why wheel names may differ from conda names + +Several factors can cause wheel names to differ from conda-style names: + +1. Name normalization: PyPI normalizes dashes to underscores (PEP 503) + - Example: `lazy-loader` (conda-forge) vs `lazy_loader` (PyPI wheel) +2. Python-specific clarification: PyPI packages are implicitly Python libraries + - Example: `authzed-py` (conda-forge) vs `authzed` (PyPI) +3. Variant differences: Conda may offer multiple variants with different dependencies + - Example: `seaborn-base` (conda-forge) vs `seaborn` (PyPI) +4. Cross-channel naming: Different conda channels may use different names + - Example: `pyperformance` (conda-forge) vs `performance` (main) + +### Naming standard + +When there are naming differences between channels, wheel records MUST use the conda-forge package name as the standard. This choice is made because: + +- conda-forge is the largest community channel with the most packages +- conda-forge naming conventions are well-established and documented +- Using a single standard prevents ambiguity and tooling complexity +- Channel operators adding wheels to repodata MUST determine the appropriate conda-style name by: + - Checking if the package already exists in conda-forge and using that exact name + - If not in conda-forge, applying conda naming conventions per [CEP 26 - Identifying Packages and Channels in the conda Ecosystem][cep-26] + - Documenting any naming mappings for their channel + +### Wheel download URLs + +This CEP introduces a new optional `artifact_url` field in package records to specify download locations for individual packages. + +> Note for this draft: The `artifact_url` field could also be added as a separate CEP to allow it for other record types. + +When present, the url field follows these semantics: + +- If `base_url` is defined in the repodata info object (per [CEP 15][cep-15]), `artifact_url` contains the path relative to `base_url` +- If `base_url` is not defined, `artifact_url` contains the complete download URL +- If `artifact_url` is not present (`null`), the download location is constructed from `base_url` and `fn` (existing behavior) + +This approach allows packages to be served from: + +- A shared relative or absolute `base_url` with all wheels in the same directory, by populating the `base_url` field and leaving the `artifact_url` field empty. +- A manual PyPI repository with wheels in directories by the package name by populating the absolute URL in the `artifact_url` field, or the `base_url` and a relative path in the `artifact_url` field. +- External PyPI mirrors or CDNs using absolute URLs by populating the `artifact_url` field, for example to https://files.pythonhosted.org/packages/.../package-1.0.0-py3-none-any.whl +- Mixed sources within the same repodata file + +### Wheel-Specific Record Values + +When populating repodata records for pure Python wheels: + +- `build: MUST be py3_0 +- `build_number`: MUST be 0 for the initial addition of a wheel version. MAY be incremented for subsequent rebuilds of the same - wheel version (e.g., to correct dependencies or metadata) +- `fn`: MUST be the wheel filename (e.g., package-1.0.0-py3-none-any.whl) +- `subdir`: MUST be noarch +- `noarch`: MUST be "python" +- `artifact_url`: MUST be present and follow the semantics described above + +### Pure Python wheel validation + +Before adding a wheel to packages.whl, channel operators MUST verify: + +- The wheel platform tag is any (e.g., py3-none-any, py2.py3-none-any) +- The wheel contains no compiled extensions (.so, .pyd, .dylib files) +- The wheel METADATA file is present and valid + +Wheels that fail these checks MUST NOT be added to packages.whl. + +### Dependency conversion + +Wheel dependencies (from METADATA file's Requires-Dist entries) MUST be converted to conda format following these rules: + +- **Package names:** Names per [CEP 26][cep-26] and match existing conda-forge package names where they exist +- **Version specifiers:** Map PEP 440 [version specifiers][version-specifiers] to conda format: + - `==X.Y.Z` → `X.Y.Z` (exact pin) + - `>=X.Y.Z` → `>=X.Y.Z` + - `<=X.Y.Z` → `<=X.Y.Z` + - `X.Y.Z` → `>X.Y.Z` + - `~=X.Y` → `>=X.Y,=1.0,<2.0) +- **Python version requirements:** Convert Requires-Python to explicit python dependency +- **Environment markers:** Ignore markers other than Python version (pure Python assumption) + +Example conversion: + +``` +# Wheel METADATA +Requires-Python: >=3.8 +Requires-Dist: requests (>=2.20.0,<3.0.0) +Requires-Dist: click (>=7.0) +Requires-Dist: importlib-metadata (>=1.0) ; python_version < '3.8' +``` + +Resulting conda depends: + +``` +depends: + - python >=3.8 + - requests >=2.20.0,<3.0 + - click >=7.0 +``` + +> **Note:** The `importlib-metadata` dependency is omitted because the `Requires-Python: >=3.8 makes its environment marker always false because the package requires Python >=3.8, making the python_version < '3.8' marker always false. + +### Handling conditional and extra dependencies + +Like in the example above of only requiring `importlib-metadata` for certain Python versions, conditional and extra dependencies MUST be supported to enable full interoperability between the ecosystems. This will be supported through a separate CEP: https://github.com/conda/ceps/pull/111. + +### Solver behavior and package preference + +#### Dependency resolution + +Solvers MUST treat pure Python wheels as valid package candidates during dependency resolution with these constraints: + +- **Exclusivity:** Solvers MUST NOT install both a conda package and wheel for the same package name. +- **Dependency satisfaction:** When a wheel is selected, its `depends` list MUST be satisfied like any conda package. +- **Platform matching:** Since all wheels in `packages.whl` are pure Python (noarch), no platform filtering is needed. + +#### User control of precedence + +By default, channels with conda packages MUST prefer conda packages when they are available. Users MAY override default precedence through: + +- Channel priority configuration (prefer channels with wheels). +- Explicit wheel requests through `::` syntax. + +### Limitations + +This CEP has the following known limitations: + +1. No support for version exclusions: Conda does not support `!=` version constraints. Packages with exclusion requirements will have those constraints omitted, which may lead to incompatible versions being selected in rare cases. +2. Pure Python only: This CEP explicitly does not address wheels with binary extensions, which require platform-specific compatibility guarantees beyond the current scope. Conda’s strength is binary compatibility, so using conda packages may be the optimal solution. +3. Environment markers: Only Python version markers are converted to dependencies. Other environment markers (OS, platform, etc.) are ignored based on the pure Python assumption. +4. Conditionals and Extras: Conditional dependencies and extras are specified by a separate CEP. +5. Repodata size: Supporting a significant portion of pure Python packages from PyPI (potentially hundreds of thousands of packages with multiple versions each) will substantially increase repodata size. Channels adopting wheel support at scale will need to implement sharded repodata ([CEP 16][cep-16]) to maintain acceptable performance. Alternatively, channels may choose to curate a subset of popular or requested packages rather than mirroring all of PyPI. + +## Implementation Notes + +### For conda clients + +Clients implementing this CEP should: + +- Parse the new `packages.whl` section alongside existing package sections +- Apply the same filtering and preference logic used for conda packages +- Extract wheel metadata during solving to populate dependency information +- Provide the ability to natively install wheels or on-the-fly convert wheels to conda packages for installation + +### For channel operators + +Channel operators adding wheel support should: + +- Implement validation to ensure only pure Python wheels are included +- Maintain a mapping of PyPI to conda-style names for their channel +- Consider automation to keep wheel metadata synchronized with PyPI +- Document any naming conventions specific to their channel + +## Examples + +### Download wheels from the default location + +Below represents the default behavior and shows when the `artifact_url` field is not set: + +```json +{ + "packages.whl": { + "requests-2.32.5": { + "record_version": 3, + "name": "requests", + "version": "2.32.5", + "build": "py3_0", + "build_number": 0, + "depends": [ + "charset-normalizer <4,>=2", + "idna <4,>=2.5", + "urllib3 <3,>=1.21.1", + "certifi >=2017.4.17", + "python >=3.9" + ], + "fn": "requests-2.32.5-py3-none-any.whl", + "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", + "size": 6899, + "subdir": "noarch", + "timestamp": 1764005009, + "noarch": "python", + "artifact_url": null + } + } +} +``` + +With this configuration, the wheel file will be downloaded from the following location (assuming we are hosting this from `https://repo.example.com/channel`): + +- `https://repo.example.com/channel/noarch/requests-2.32.5-py3-none-any.whl` + + +### Downloading wheels from a relative location with `base_url` + +The `artifact_url` can also be relative as described above. Here's an example of what that looks like combined with setting the `base_url` property at the top level: + +```json +{ + "info": { + "subdir": "noarch", + "base_url": "https://repo.example.com/channel/" + }, + "packages.whl": { + "requests-2.32.5": { + "record_version": 3, + "name": "requests", + "version": "2.32.5", + "build": "py3_0", + "build_number": 0, + "depends": [ + "charset-normalizer <4,>=2", + "idna <4,>=2.5", + "urllib3 <3,>=1.21.1", + "certifi >=2017.4.17", + "python >=3.9" + ], + "fn": "requests-2.32.5-py3-none-any.whl", + "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", + "size": 6899, + "subdir": "noarch", + "timestamp": 1764005009, + "noarch": "python", + "artifact_url": "requests/requests-2.32.5-py3-none-any.whl" + } + } +} +``` + +This would result in the following being fetched by conda clients: + +- `https://repo.example.com/channel/noarch/requests/requests-2.32.5-py3-none-any.whl` + +### Downloading wheels from external location + +The following shows an example of using an external location to download the wheel from PyPI's file hosting: + +```json +{ + "packages.whl": { + "requests-2.32.5": { + "record_version": 3, + "name": "requests", + "version": "2.32.5", + "build": "py3_0", + "build_number": 0, + "depends": [ + "charset-normalizer <4,>=2", + "idna <4,>=2.5", + "urllib3 <3,>=1.21.1", + "certifi >=2017.4.17", + "python >=3.9" + ], + "fn": "requests-2.32.5-py3-none-any.whl", + "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", + "size": 6899, + "subdir": "noarch", + "timestamp": 1764005009, + "noarch": "python", + "artifact_url": "https://files.pythonhosted.org/packages/1e/db/4254e3eabe8020b458f1a747140d32277ec7a271daf1d235b70dc0b4e6e3/requests-2.32.5-py3-none-any.whl" + } + } +} +``` + +### Name mapping + +Here is an example of name mapping and normalization of the record name and dependencies. + +```json +{ + "packages.whl": { + "annotated-types-0.7.0": { + "record_version": 3, + "name": "annotated-types", + "version": "0.7.0", + "build": "py3_0", + "build_number": 0, + "depends": [ + "typing_extensions >=4.0.0", + "python >=3.8" + ], + "fn": "annotated_types-0.7.0-py3-none-any.whl", + "sha256": "1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53", + "size": 13643, + "subdir": "noarch", + "timestamp": 1756405206, + "noarch": "python", + "artifact_url": "https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl" + } + } +} +``` + +This example demonstrates two types of name normalization: + +1. Record key format: The package is indexed using the `{name}-{version}` format: `annotated-types-0.7.0`. The conda name uses hyphens which matches the canonical PyPI package name. The actual wheel filename in the fn field (`annotated_types-0.7.0-py3-none-any.whl`) normalizes the package name to underscores per PEP 427. +2. Dependency name mapping: This package depends on `typing_extensions`, which is listed in the `depends` field. On PyPI, this package is named `typing-extensions` (with a hyphen), but it has been mapped to the name `typing_extensions` (with an underscore) to match the existing conda-forge package name. + +This example also demonstrates conditional dependencies. The original `METADATA` file from the wheel has the following dependency information: + +``` +Requires-Python: >=3.8 +Requires-Dist: typing-extensions>=4.0.0; python_version < '3.9' +``` + +Once conditional dependencies are supported in a separate CEP, the package record should only declare `typing_extensions` as a dependency when the Python version is < 3.9. In this example, the dependency is declared unconditionally for all Python versions. The Python version constraint of >=3.8 is directly mapped. + +## Rejected ideas + +### Only install Python and pip inside conda environments + +In this scenario, users only install Python and pip inside a clean conda environment. Here, we simply use conda as an environment manager and let pip manage the project dependencies. + +This is what that typically looks like: + +``` +conda create -n pip-environment python pip +conda activate pip-environment +pip install +``` +Despite its safety for environment management, relying solely on conda for this purpose prevents leveraging the package distribution capabilities of the conda ecosystem. + +### Editable installs with conda for dependencies only + +Conda provides all the dependencies of a given package. Then that package is installed on top in editable mode, without addressing dependencies to make sure we don’t accidentally overwrite conda files: + +``` +git clone https://github.com/owner/package.git +conda create -n editable-install package --deps-only +conda activate editable-install +pip install -e . --no-deps +``` + +### Add more conda packages + +Create and maintain new conda packages for each PyPI dependency needed. Tools like [Grayskull] exist to make this easier to convert. However, this is a significant workload for the community, with over half of all conda-forge packages being pure Python. Even with more dedicated resources, creating recipes for over 400 thousand pure Python packages is not achievable. + +### Add interoperability to tools through pip dependency scanning + +The original version of the conda-pypi plugin called `pip` with the `--dry-run` option to analyze the solution to install a package. With a list of all the dependencies needed, the plugin installed everything available in conda channels first and what was left over would be installed with `pip install --no-deps`. Disadvantages of this approach include: + +- Users still have to know which packages they want from PyPI and then have to `conda pip install` them. +- There is no guarantee that the conda and pip packages installed have ABI compatibility. +- Calling pip and conda multiple times is slow. + +### Add interoperability to tools through on-the-fly conversion + +This is the approach that the [conda-pupa][conda-pupa] plugin used and was then implemented in [conda-pypi][conda-pypi]. When `conda pypi install ` is called, it fetches its set of required dependencies iteratively from PyPI just like `pip`. Similar to the dependency scanning option above, it then attempts to install as many dependencies as it can from conda. After this is where these two approaches start to differ. While conda-pypi simply used `pip` to install the remain Python dependencies, conda-pupa converts wheel packages to conda and stores it in a local channel, essentially caching these converted wheels on disk. This means that a repodata.json is also generated allowing us to perform a solve entirely in conda. + +Unfortunately, there are also disadvantages with this approach. Like the solution above, users still have to know which packages they want from PyPI and then have to run `conda pypi install` to install them. Additionally, the following problems also arise: + +- Package conversion can take some time, especially for larger packages. +- Users must rely on a local cache for installing wheels, and this cache cannot easily be shared. +- The current version must solve multiple times which is slow, although this could be optimized. + +### Add interoperability to tools through uv integration + +Pixi has integrated uv for installing packages from PyPI. The user adds the dependency through `pixi add --pypi `. Then, when Pixi is solving the environment, it solves the conda packages using Rattler, and then calls uv to solve the PyPI dependencies. Disadvantages of this approach include: + +- Like the solutions above, users still have to know which packages they want from PyPI and then have to `pixi add --pypi` them. +- Although Pixi and uv are both very fast, it is still slower than performing a single solve of the environment. + +### Direct PyPI communication without repodata + +Another alternative would be for conda clients to query PyPI's API directly during solving, fetching wheel metadata on-demand rather than including it in repodata. This idea was rejected due to: + +- Conda's SAT-based solvers (libsolv, resolvo) require complete package metadata upfront to solve in a single pass, while PyPI's API is designed for lazy evaluation with incremental fetching and backtracking (as pip and uv do). Adapting conda's solvers would require fundamental architectural changes. + +- On-demand fetching would also require hundreds of sequential HTTP requests during solving, making solves slow and network-dependent while breaking offline capability and reproducibility guarantees. + +### Automatic wheel to conda package conversion and hosting + +Another alternative would be establishing a build farm to automatically convert wheels to conda packages and host them on a channel, with conversion triggered by popularity, community requests, or on-demand. + +#### Advantages of automatic conversion +- Unified format: All packages are conda packages, simplifying client implementation +- Leverages mature conda infrastructure and tooling without client changes +- Enables metadata enrichment, quality control, and validation before publishing + +#### Disadvantages and why native wheel support was chosen + +Despite these advantages, this approach was rejected because: + +- Infrastructure burden: Requires significant storage and bandwidth to host and serve converted packages that duplicate PyPI's CDN infrastructure, plus ongoing maintenance of the conversion pipeline and synchronization automation +- Resource inefficiency: Wheels are already an excellent format for pure Python packages; conversion adds no technical value and wastes resources +- Timeliness: Creates lag between PyPI publication and availability, with curation decisions needed about which packages to convert +- Maintenance complexity: Every wheel format change requires tooling updates; bugs affect all converted packages +- Ecosystem misalignment: Converting away from wheels moves conda further from Python packaging standards + +Native wheel support provides the same user experience (transparent PyPI access) while avoiding infrastructure burden and maintaining alignment with Python packaging standards. Channel operators who prefer converted packages can continue building conda packages from PyPI sources. + +## History + +The desire for better interoperability is not new and there has been a discussion over the last 12 years about balancing conda's core strengths (reproducibility, binary packages, cross-language support) with the Python community's expectations for seamless PyPI access. + +### Early Vision (2012-2014) + +The conda project's earliest discussions reveal a consistent vision for broad package manager interoperability that extends beyond just PyPI integration: + +- Issue [#307](https://github.com/conda/conda/issue/307) (2013) proposed conda should work with pip, npm, gem, rpm, and brew, envisioning conda as a universal package manager interface +- Issue [#292](https://github.com/conda/conda/issue/292) (2013) explicitly titled "Making conda the ultimate package manager" outlined ambitious cross-ecosystem integration +- Issue [#224](https://github.com/conda/conda/issue/224) (2012) discussed deprecating pip commands in favor of native conda functionality + +### The PyPI Integration Debate (2013-2016) + +Two competing philosophies emerged on how or if we should provide better integration with PyPI. + +#### Pro-integration + +Issue [#262](https://github.com/conda/conda/issue/262) (2013): conda should install directly from PyPI to reduce duplication and improve package availability. + +Arguments for this approach: + +- Users frequently need packages not available on conda channels +- Building conda packages for every PyPI package is unsustainable + +#### Status quo + +Arguments to for maintaining an independent packaging ecosystem prevailed at the time for the following reasons: + +- Conda's value proposition is reproducible, binary-focused environments with precise dependency resolution +- PyPI's source distributions and pip's resolver could compromise conda's guarantees +- The conda-forge community successfully scaled recipe creation + +### Wheel Support Proposal (2017) + +Issue [#5202](https://github.com/conda/conda/issue/5202) proposed direct wheel installation support, recognizing: + +- Wheels provide binary distributions similar to conda packages +- Growing wheel availability on PyPI reduced build complexity +- Could bridge the gap between conda's reliability and PyPI's breadth + +### Broader Context (2018) + +The Python Discourse thread on [packaging scope boundaries](https://discuss.python.org/t/drawing-a-line-to-the-scope-of-python-packaging/883) reflects the larger ecosystem's struggle with: + +- Multiple competing package management tools (pip, conda, poetry, pipenv) +- Unclear responsibilities and interoperability expectations +- Need for clearer standards and communication between tools + +### conda-pypi development starts (2022) + +Jaime Rodríguez-Guerra starts development on the conda-pypi plugin aimed to improve conda and PyPI interoperability. + +### Rip development starts (2023) + +Prefix.dev starts a barebones pip implemented in Rust to resolve and install PyPI dependencies with Pixi. + +### Pixi Integrates with uv (Jan 2024) + +Pixi changes course to use uv directly instead of rip, which unlocks features like editable installations, and git and path dependencies. + +### conda-pupa creates on-the-fly conversion plugin (July 2024) + +Daniel Holth creates a conda plugin that supports on-the-fly conversion of conda packages to wheels in a local channel. + +### conda-pypi merges in conda-pupa functionality (Oct 2025) + +conda-pupa is merged into conda-pypi which adds a `conda pypi install ` command and support for editable installations. + + +## References + +- [Adopting uv in pixi][uv-in-pixi] +- [rip][rip] +- [conda-pypi project][conda-pypi] +- [conda-pupa][conda-pupa] + +## Copyright + +All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/). + + +[RFC2119]: https://datatracker.ietf.org/doc/html/rfc2119 +[repodata-schema]: https://schemas.conda.org/repodata-1.schema.json +[repodata-record-schema]: https://schemas.conda.org/repodata-record-1.schema.json +[cep-15]: https://conda.org/learn/ceps/cep-0015 +[cep-16]: https://conda.org/learn/ceps/cep-0016 +[cep-26]: https://conda.org/learn/ceps/cep-0026 +[version-specifiers]: https://packaging.python.org/en/latest/specifications/version-specifiers/#id5 +[conda-pypi]: https://github.com/conda-incubator/conda-pypi +[conda-pupa]: https://github.com/dholth/conda-pupa +[uv-in-pixi]: https://prefix.dev/blog/uv_in_pixi +[rip]: https://github.com/prefix-dev/rip +[grayskull]: https://conda.github.io/grayskull/ From f1a6a9c621e4412b258ce7f8139007271d87966b Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Tue, 23 Dec 2025 22:04:54 -0500 Subject: [PATCH 02/38] Add conda-whl-channel to the history timeline --- cep-XXXX.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/cep-XXXX.md b/cep-XXXX.md index b769d2a1..4edaa9e2 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -530,10 +530,17 @@ Pixi changes course to use uv directly instead of rip, which unlocks features li Daniel Holth creates a conda plugin that supports on-the-fly conversion of conda packages to wheels in a local channel. +### conda-whl-channel creates proof-of-concept wheel channel from repodata (Nov 2024) + +Jonathan Helmus and Anil Kulkarni begin development on [conda-whl-channel](https://github.com/Anaconda/conda-whl-channel, a proof-of-concept that adds wheels to the `packages` section of repodata and patches conda to recognize them. The implementation supports conditional dependencies through additional meta packages. In February 2025, the conda-specific functionality is extracted into a separate plugin called [conda-whl-support](https://github.com/Anaconda/conda-whl-support. + ### conda-pypi merges in conda-pupa functionality (Oct 2025) conda-pupa is merged into conda-pypi which adds a `conda pypi install ` command and support for editable installations. +### conda-pypi integrates parts of conda-whl-support (Nov 2025) + +Conda wheel detection logic is added to conda-pypi which is the functionality needed besides the solver changes required for the `packages.whl` section proposed in this CEP. ## References From 9c74c8f227a58df145579a23ea99fe7215e1854d Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Tue, 23 Dec 2025 22:08:41 -0500 Subject: [PATCH 03/38] Grammar fixup --- cep-XXXX.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 4edaa9e2..aaf593c9 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -540,7 +540,7 @@ conda-pupa is merged into conda-pypi which adds a `conda pypi install ` ### conda-pypi integrates parts of conda-whl-support (Nov 2025) -Conda wheel detection logic is added to conda-pypi which is the functionality needed besides the solver changes required for the `packages.whl` section proposed in this CEP. +Conda-pypi incorporates the wheel detection logic from conda-whl-support, providing the core functionality needed beyond the solver and index changes required to support the `packages.whl` section proposed in this CEP. ## References From 42e75da481ccb808540e41fe9d3518757f1bca37 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Wed, 24 Dec 2025 08:12:04 -0500 Subject: [PATCH 04/38] Apply consistency and style updates Co-authored-by: Travis Hathaway --- cep-XXXX.md | 52 ++++++++++++++++++++++++++-------------------------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index aaf593c9..49a390bd 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -53,7 +53,7 @@ level keys: - removed - signatures -This CEP proposes the addition of a new `packages.whl` section to account for the wheel format. This key points to a mapping that MUST contain [repodata record][repodata-record-schema] objects. The key of this mapping MUST be the wheel file name, and the contents of the package record MUST contain metadata related to that wheel. +This CEP proposes the addition of a new `packages.whl` section to account for the wheel format. This key points to a mapping that MUST contain [repodata record][repodata-record-schema] objects. The key of this mapping MUST follow the specification described below in [Key naming requirements](#key-naming-requirements). ### Record structure and naming convention @@ -61,7 +61,7 @@ To support sparse repodata processing and maintain compatibility with conda's ex ### Key naming requirements -The key for each entry in `packages.whl` must follow this format: +The key for each entry in `packages.whl` MUST follow this format: ``` {name}-{version} @@ -77,13 +77,13 @@ Examples: Several factors can cause wheel names to differ from conda-style names: -1. Name normalization: PyPI normalizes dashes to underscores (PEP 503) +1. **Name normalization:** PyPI normalizes dashes to underscores (PEP 503) - Example: `lazy-loader` (conda-forge) vs `lazy_loader` (PyPI wheel) -2. Python-specific clarification: PyPI packages are implicitly Python libraries +2. **Python-specific clarification:** PyPI packages are implicitly Python libraries - Example: `authzed-py` (conda-forge) vs `authzed` (PyPI) -3. Variant differences: Conda may offer multiple variants with different dependencies +3. **Variant differences:** Conda may offer multiple variants with different dependencies - Example: `seaborn-base` (conda-forge) vs `seaborn` (PyPI) -4. Cross-channel naming: Different conda channels may use different names +4. **Cross-channel naming:** Different conda channels may use different names - Example: `pyperformance` (conda-forge) vs `performance` (main) ### Naming standard @@ -91,7 +91,6 @@ Several factors can cause wheel names to differ from conda-style names: When there are naming differences between channels, wheel records MUST use the conda-forge package name as the standard. This choice is made because: - conda-forge is the largest community channel with the most packages -- conda-forge naming conventions are well-established and documented - Using a single standard prevents ambiguity and tooling complexity - Channel operators adding wheels to repodata MUST determine the appropriate conda-style name by: - Checking if the package already exists in conda-forge and using that exact name @@ -104,7 +103,7 @@ This CEP introduces a new optional `artifact_url` field in package records to sp > Note for this draft: The `artifact_url` field could also be added as a separate CEP to allow it for other record types. -When present, the url field follows these semantics: +When present, the `artifact_url` field follows these semantics: - If `base_url` is defined in the repodata info object (per [CEP 15][cep-15]), `artifact_url` contains the path relative to `base_url` - If `base_url` is not defined, `artifact_url` contains the complete download URL @@ -121,10 +120,10 @@ This approach allows packages to be served from: When populating repodata records for pure Python wheels: -- `build: MUST be py3_0 +- `build`: MUST be py3_0 - `build_number`: MUST be 0 for the initial addition of a wheel version. MAY be incremented for subsequent rebuilds of the same - wheel version (e.g., to correct dependencies or metadata) - `fn`: MUST be the wheel filename (e.g., package-1.0.0-py3-none-any.whl) -- `subdir`: MUST be noarch +- `subdir`: MUST be "noarch" - `noarch`: MUST be "python" - `artifact_url`: MUST be present and follow the semantics described above @@ -196,22 +195,23 @@ By default, channels with conda packages MUST prefer conda packages when they ar - Channel priority configuration (prefer channels with wheels). - Explicit wheel requests through `::` syntax. +- Explicit configuration in the client itself (e.g. `prefer_conda` or `prefer_wheel`) ### Limitations This CEP has the following known limitations: -1. No support for version exclusions: Conda does not support `!=` version constraints. Packages with exclusion requirements will have those constraints omitted, which may lead to incompatible versions being selected in rare cases. -2. Pure Python only: This CEP explicitly does not address wheels with binary extensions, which require platform-specific compatibility guarantees beyond the current scope. Conda’s strength is binary compatibility, so using conda packages may be the optimal solution. -3. Environment markers: Only Python version markers are converted to dependencies. Other environment markers (OS, platform, etc.) are ignored based on the pure Python assumption. -4. Conditionals and Extras: Conditional dependencies and extras are specified by a separate CEP. -5. Repodata size: Supporting a significant portion of pure Python packages from PyPI (potentially hundreds of thousands of packages with multiple versions each) will substantially increase repodata size. Channels adopting wheel support at scale will need to implement sharded repodata ([CEP 16][cep-16]) to maintain acceptable performance. Alternatively, channels may choose to curate a subset of popular or requested packages rather than mirroring all of PyPI. +1. **No support for version exclusions:** Conda does not support `!=` version constraints. Packages with exclusion requirements will have those constraints omitted, which may lead to incompatible versions being selected in rare cases. +2. **Pure Python only:** This CEP explicitly does not address wheels with binary extensions, which require platform-specific compatibility guarantees beyond the current scope. Conda’s strength is binary compatibility, so using conda packages may be the optimal solution. +3. **Environment markers:** Only Python version markers are converted to dependencies. Other environment markers (OS, platform, etc.) are ignored based on the pure Python assumption. +4. **Conditionals and Extras:** Conditional dependencies and extras are specified by a separate CEP. +5. **Repodata size:** Supporting a significant portion of pure Python packages from PyPI (potentially hundreds of thousands of packages with multiple versions each) will substantially increase repodata size. Channels adopting wheel support at scale will need to implement sharded repodata ([CEP 16][cep-16]) to maintain acceptable performance. Alternatively, channels may choose to curate a subset of popular or requested packages rather than mirroring all of PyPI. ## Implementation Notes ### For conda clients -Clients implementing this CEP should: +Clients implementing this CEP SHOULD: - Parse the new `packages.whl` section alongside existing package sections - Apply the same filtering and preference logic used for conda packages @@ -398,7 +398,7 @@ Despite its safety for environment management, relying solely on conda for this ### Editable installs with conda for dependencies only -Conda provides all the dependencies of a given package. Then that package is installed on top in editable mode, without addressing dependencies to make sure we don’t accidentally overwrite conda files: +Conda provides all the dependencies of a given package. Then that package is installed on top in editable mode, without addressing dependencies to make sure conda files aren't accidentally overwritten: ``` git clone https://github.com/owner/package.git @@ -415,18 +415,18 @@ Create and maintain new conda packages for each PyPI dependency needed. Tools li The original version of the conda-pypi plugin called `pip` with the `--dry-run` option to analyze the solution to install a package. With a list of all the dependencies needed, the plugin installed everything available in conda channels first and what was left over would be installed with `pip install --no-deps`. Disadvantages of this approach include: -- Users still have to know which packages they want from PyPI and then have to `conda pip install` them. +- Users still have to know which packages they want from PyPI and then have to run `conda pip install` to install them. - There is no guarantee that the conda and pip packages installed have ABI compatibility. - Calling pip and conda multiple times is slow. ### Add interoperability to tools through on-the-fly conversion -This is the approach that the [conda-pupa][conda-pupa] plugin used and was then implemented in [conda-pypi][conda-pypi]. When `conda pypi install ` is called, it fetches its set of required dependencies iteratively from PyPI just like `pip`. Similar to the dependency scanning option above, it then attempts to install as many dependencies as it can from conda. After this is where these two approaches start to differ. While conda-pypi simply used `pip` to install the remain Python dependencies, conda-pupa converts wheel packages to conda and stores it in a local channel, essentially caching these converted wheels on disk. This means that a repodata.json is also generated allowing us to perform a solve entirely in conda. +This is the approach that the [conda-pupa][conda-pupa] plugin used and was then implemented in [conda-pypi][conda-pypi]. When `conda pypi install ` is called, it fetches its set of required dependencies iteratively from PyPI just like `pip`. Similar to the dependency scanning option above, it then attempts to install as many dependencies as it can from conda. Subsequently, this is where these two approaches start to differ. While conda-pypi simply used `pip` to install the remaining Python dependencies, conda-pupa converts wheel packages to conda and stores them in a local channel, essentially caching these converted wheels on disk. This means that a repodata.json is also generated allowing us to perform a solve entirely in conda. Unfortunately, there are also disadvantages with this approach. Like the solution above, users still have to know which packages they want from PyPI and then have to run `conda pypi install` to install them. Additionally, the following problems also arise: - Package conversion can take some time, especially for larger packages. -- Users must rely on a local cache for installing wheels, and this cache cannot easily be shared. +- Users must rely on a local cache for installing wheels, and this cache cannot easily be shared across computers. - The current version must solve multiple times which is slow, although this could be optimized. ### Add interoperability to tools through uv integration @@ -449,7 +449,7 @@ Another alternative would be for conda clients to query PyPI's API directly duri Another alternative would be establishing a build farm to automatically convert wheels to conda packages and host them on a channel, with conversion triggered by popularity, community requests, or on-demand. #### Advantages of automatic conversion -- Unified format: All packages are conda packages, simplifying client implementation +- All packages are conda packages, simplifying client implementation - Leverages mature conda infrastructure and tooling without client changes - Enables metadata enrichment, quality control, and validation before publishing @@ -457,11 +457,11 @@ Another alternative would be establishing a build farm to automatically convert Despite these advantages, this approach was rejected because: -- Infrastructure burden: Requires significant storage and bandwidth to host and serve converted packages that duplicate PyPI's CDN infrastructure, plus ongoing maintenance of the conversion pipeline and synchronization automation -- Resource inefficiency: Wheels are already an excellent format for pure Python packages; conversion adds no technical value and wastes resources -- Timeliness: Creates lag between PyPI publication and availability, with curation decisions needed about which packages to convert -- Maintenance complexity: Every wheel format change requires tooling updates; bugs affect all converted packages -- Ecosystem misalignment: Converting away from wheels moves conda further from Python packaging standards +- **Infrastructure burden:** Requires significant storage and bandwidth to host and serve converted packages that duplicate PyPI's CDN infrastructure, plus ongoing maintenance of the conversion pipeline and synchronization automation +- **Resource inefficiency:** Wheels are already an excellent format for pure Python packages; conversion adds no technical value and wastes resources +- **Timeliness:** Creates lag between PyPI publication and availability, with curation decisions needed about which packages to convert +- **Maintenance complexity:** Every wheel format change requires tooling updates; bugs affect all converted packages +- **Ecosystem misalignment:** Converting away from wheels moves conda further from Python packaging standards Native wheel support provides the same user experience (transparent PyPI access) while avoiding infrastructure burden and maintaining alignment with Python packaging standards. Channel operators who prefer converted packages can continue building conda packages from PyPI sources. From bdd643e5cc567254ed213b867f212ffdf0c43bad Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Wed, 24 Dec 2025 08:28:11 -0500 Subject: [PATCH 05/38] Fix lint errors --- cep-XXXX.md | 42 ++++++++++++++++++++++-------------------- 1 file changed, 22 insertions(+), 20 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 49a390bd..8b738148 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -20,7 +20,8 @@ described in [RFC2119][RFC2119] when, and only when, they appear in all capitals ## Abstract -This CEP outlines how native support for pure Python wheel packages will be achieved by adding support for them to conda's package index (repodata). When implemented, conda clients will be able to seamlessly install conda packages and pure Python wheels from enabled channels. We explicitly limit the scope of this CEP to pure Python wheels to avoid platform-specific binary compatibility issues, but even with this limitation, we dramatically expand the number of available packages for users in the conda ecosystem. +This CEP outlines how native support for pure Python wheel packages will be achieved by adding support for them to conda's package index (repodata). When implemented, conda clients will be able to seamlessly install conda packages and pure Python wheels from enabled channels. +We explicitly limit the scope of this CEP to pure Python wheels to avoid platform-specific binary compatibility issues, but even with this limitation, we dramatically expand the number of available packages for users in the conda ecosystem. ## Motivation @@ -61,11 +62,7 @@ To support sparse repodata processing and maintain compatibility with conda's ex ### Key naming requirements -The key for each entry in `packages.whl` MUST follow this format: - -``` -{name}-{version} -``` +The key for each entry in `packages.whl` MUST follow the format `{name}-{version}`. Examples: @@ -78,13 +75,16 @@ Examples: Several factors can cause wheel names to differ from conda-style names: 1. **Name normalization:** PyPI normalizes dashes to underscores (PEP 503) - - Example: `lazy-loader` (conda-forge) vs `lazy_loader` (PyPI wheel) + - Example: `lazy-loader` (conda-forge) vs `lazy_loader` (PyPI wheel) + 2. **Python-specific clarification:** PyPI packages are implicitly Python libraries - - Example: `authzed-py` (conda-forge) vs `authzed` (PyPI) + - Example: `authzed-py` (conda-forge) vs `authzed` (PyPI) + 3. **Variant differences:** Conda may offer multiple variants with different dependencies - - Example: `seaborn-base` (conda-forge) vs `seaborn` (PyPI) + - Example: `seaborn-base` (conda-forge) vs `seaborn` (PyPI) + 4. **Cross-channel naming:** Different conda channels may use different names - - Example: `pyperformance` (conda-forge) vs `performance` (main) + - Example: `pyperformance` (conda-forge) vs `performance` (main) ### Naming standard @@ -113,7 +113,7 @@ This approach allows packages to be served from: - A shared relative or absolute `base_url` with all wheels in the same directory, by populating the `base_url` field and leaving the `artifact_url` field empty. - A manual PyPI repository with wheels in directories by the package name by populating the absolute URL in the `artifact_url` field, or the `base_url` and a relative path in the `artifact_url` field. -- External PyPI mirrors or CDNs using absolute URLs by populating the `artifact_url` field, for example to https://files.pythonhosted.org/packages/.../package-1.0.0-py3-none-any.whl +- External PyPI mirrors or CDNs using absolute URLs by populating the `artifact_url` field, for example to - Mixed sources within the same repodata file ### Wheel-Specific Record Values @@ -156,7 +156,7 @@ Wheel dependencies (from METADATA file's Requires-Dist entries) MUST be converte Example conversion: -``` +```text # Wheel METADATA Requires-Python: >=3.8 Requires-Dist: requests (>=2.20.0,<3.0.0) @@ -166,7 +166,7 @@ Requires-Dist: importlib-metadata (>=1.0) ; python_version < '3.8' Resulting conda depends: -``` +```text depends: - python >=3.8 - requests >=2.20.0,<3.0 @@ -177,7 +177,7 @@ depends: ### Handling conditional and extra dependencies -Like in the example above of only requiring `importlib-metadata` for certain Python versions, conditional and extra dependencies MUST be supported to enable full interoperability between the ecosystems. This will be supported through a separate CEP: https://github.com/conda/ceps/pull/111. +Like in the example above of only requiring `importlib-metadata` for certain Python versions, conditional and extra dependencies MUST be supported to enable full interoperability between the ecosystems. This will be supported through a separate CEP: . ### Solver behavior and package preference @@ -265,7 +265,6 @@ With this configuration, the wheel file will be downloaded from the following lo - `https://repo.example.com/channel/noarch/requests-2.32.5-py3-none-any.whl` - ### Downloading wheels from a relative location with `base_url` The `artifact_url` can also be relative as described above. Here's an example of what that looks like combined with setting the `base_url` property at the top level: @@ -374,7 +373,7 @@ This example demonstrates two types of name normalization: This example also demonstrates conditional dependencies. The original `METADATA` file from the wheel has the following dependency information: -``` +```text Requires-Python: >=3.8 Requires-Dist: typing-extensions>=4.0.0; python_version < '3.9' ``` @@ -389,18 +388,19 @@ In this scenario, users only install Python and pip inside a clean conda environ This is what that typically looks like: -``` +```bash conda create -n pip-environment python pip conda activate pip-environment pip install ``` + Despite its safety for environment management, relying solely on conda for this purpose prevents leveraging the package distribution capabilities of the conda ecosystem. ### Editable installs with conda for dependencies only Conda provides all the dependencies of a given package. Then that package is installed on top in editable mode, without addressing dependencies to make sure conda files aren't accidentally overwritten: -``` +```bash git clone https://github.com/owner/package.git conda create -n editable-install package --deps-only conda activate editable-install @@ -421,7 +421,8 @@ The original version of the conda-pypi plugin called `pip` with the `--dry-run` ### Add interoperability to tools through on-the-fly conversion -This is the approach that the [conda-pupa][conda-pupa] plugin used and was then implemented in [conda-pypi][conda-pypi]. When `conda pypi install ` is called, it fetches its set of required dependencies iteratively from PyPI just like `pip`. Similar to the dependency scanning option above, it then attempts to install as many dependencies as it can from conda. Subsequently, this is where these two approaches start to differ. While conda-pypi simply used `pip` to install the remaining Python dependencies, conda-pupa converts wheel packages to conda and stores them in a local channel, essentially caching these converted wheels on disk. This means that a repodata.json is also generated allowing us to perform a solve entirely in conda. +This is the approach that the [conda-pupa][conda-pupa] plugin used and was then implemented in [conda-pypi][conda-pypi]. When `conda pypi install ` is called, it fetches its set of required dependencies iteratively from PyPI just like `pip`. Similar to the dependency scanning option above, it then attempts to install as many dependencies as it can from conda. +However, this is where these two approaches start to differ. While conda-pypi simply used `pip` to install the remaining Python dependencies, conda-pupa converts wheel packages to conda and stores them in a local channel, essentially caching these converted wheels on disk. This means that a repodata.json is also generated allowing us to perform a solve entirely in conda. Unfortunately, there are also disadvantages with this approach. Like the solution above, users still have to know which packages they want from PyPI and then have to run `conda pypi install` to install them. Additionally, the following problems also arise: @@ -449,6 +450,7 @@ Another alternative would be for conda clients to query PyPI's API directly duri Another alternative would be establishing a build farm to automatically convert wheels to conda packages and host them on a channel, with conversion triggered by popularity, community requests, or on-demand. #### Advantages of automatic conversion + - All packages are conda packages, simplifying client implementation - Leverages mature conda infrastructure and tooling without client changes - Enables metadata enrichment, quality control, and validation before publishing @@ -532,7 +534,7 @@ Daniel Holth creates a conda plugin that supports on-the-fly conversion of conda ### conda-whl-channel creates proof-of-concept wheel channel from repodata (Nov 2024) -Jonathan Helmus and Anil Kulkarni begin development on [conda-whl-channel](https://github.com/Anaconda/conda-whl-channel, a proof-of-concept that adds wheels to the `packages` section of repodata and patches conda to recognize them. The implementation supports conditional dependencies through additional meta packages. In February 2025, the conda-specific functionality is extracted into a separate plugin called [conda-whl-support](https://github.com/Anaconda/conda-whl-support. +Jonathan Helmus and Anil Kulkarni begin development on [conda-whl-channel](, a proof-of-concept that adds wheels to the `packages` section of repodata and patches conda to recognize them. The implementation supports conditional dependencies through additional meta packages. In February 2025, the conda-specific functionality is extracted into a separate plugin called [conda-whl-support](. ### conda-pypi merges in conda-pupa functionality (Oct 2025) From 93e78bf926882417677416aa2546753c317d3048 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Tue, 30 Dec 2025 14:39:29 -0500 Subject: [PATCH 06/38] Updates to mapping, version exclusions, and PyPI deletion Updates include: - Clarifications on naming standards - Channel mapping - Patching capabilities for dependency management # Conflicts: # cep-XXXX.md --- cep-XXXX.md | 105 ++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 81 insertions(+), 24 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 8b738148..87727737 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -37,8 +37,10 @@ By adding native support for pure Python wheels to repodata, conda clients can: - Provide users with transparent access to the broader Python ecosystem - Maintain environment consistency and reproducibility - Eliminate the cognitive burden of managing two package managers -- Fill gaps in conda package availability without requiring new conda builds -- Reduce the maintenance burden by fully or partially eliminating the need to create and maintain conda recipes for pure-Python packages +- Fill gaps in conda package availability for simpler pure-Python packages +- Reduce the maintenance burden for straightforward pure-Python packages that don't require metadata modifications + +Note that this CEP does not eliminate the need for conda recipes entirely. Many packages will still require proper conda feedstocks due to metadata differences, dependency corrections, or other ecosystem-specific needs. This CEP provides an additional tool for handling simpler pure-Python cases more efficiently. ## Specification @@ -86,16 +88,28 @@ Several factors can cause wheel names to differ from conda-style names: 4. **Cross-channel naming:** Different conda channels may use different names - Example: `pyperformance` (conda-forge) vs `performance` (main) -### Naming standard +### Naming standard and channel mapping -When there are naming differences between channels, wheel records MUST use the conda-forge package name as the standard. This choice is made because: +When there are naming differences between PyPI wheels and conda packages, channel operators MUST determine the appropriate conda-style name by applying conda naming conventions per [CEP 26 - Identifying Packages and Channels in the conda Ecosystem][cep-26]. -- conda-forge is the largest community channel with the most packages -- Using a single standard prevents ambiguity and tooling complexity -- Channel operators adding wheels to repodata MUST determine the appropriate conda-style name by: - - Checking if the package already exists in conda-forge and using that exact name - - If not in conda-forge, applying conda naming conventions per [CEP 26 - Identifying Packages and Channels in the conda Ecosystem][cep-26] - - Documenting any naming mappings for their channel +To help users understand which channel's naming conventions are being used, channels MAY declare an optional `name_mapping_channel` field in the `info` section of repodata. This field indicates which channel's package names are used as the standard for wheel-to-conda name mapping in this repodata. + +```json +{ + "info": { + "subdir": "noarch", + "base_url": "https://repo.example.com/channel/", + "name_mapping_channel": "conda-forge" + } +} +``` + +When `name_mapping_channel` is present, channel operators SHOULD: +- Use the declared channel's package names as the standard for name mapping +- Document any naming mappings specific to their channel +- Ensure consistency with the declared channel's naming conventions + +Channel operators MUST document any naming conventions and mappings specific to their channel, regardless of whether `name_mapping_channel` is declared. ### Wheel download URLs @@ -137,6 +151,24 @@ Before adding a wheel to packages.whl, channel operators MUST verify: Wheels that fail these checks MUST NOT be added to packages.whl. +### Repodata patching support + +Channel operators MAY patch `packages.whl` entries in repodata, just like conda packages, to: + +- Correct dependency metadata errors or conflicts +- Apply immediate fixes without waiting for upstream releases +- Adjust version constraints for conda ecosystem compatibility +- Correct dependency name mappings that were incorrectly converted +- Add or modify metadata fields as needed + +This patching capability is essential for: +- **Quality control**: Test and validate packages before exposing to users +- **Ecosystem needs**: Resolve naming conflicts and dependency incompatibilities +- **Reproducibility**: Optionally mirror and store wheel artifacts locally to prevent external deletions +- **Downstream fixes**: Address metadata issues that would break environments + +Channels SHOULD document their patching policies and maintain transparency about which packages have been modified from their upstream PyPI metadata. + ### Dependency conversion Wheel dependencies (from METADATA file's Requires-Dist entries) MUST be converted to conda format following these rules: @@ -149,7 +181,7 @@ Wheel dependencies (from METADATA file's Requires-Dist entries) MUST be converte - `X.Y.Z` → `>X.Y.Z` - `~=X.Y` → `>=X.Y,=1.0,<2.0) - **Python version requirements:** Convert Requires-Python to explicit python dependency - **Environment markers:** Ignore markers other than Python version (pure Python assumption) @@ -161,17 +193,24 @@ Example conversion: Requires-Python: >=3.8 Requires-Dist: requests (>=2.20.0,<3.0.0) Requires-Dist: click (>=7.0) +Requires-Dist: numpy (>=1.20.0,!=1.24.0) Requires-Dist: importlib-metadata (>=1.0) ; python_version < '3.8' ``` -Resulting conda depends: +Resulting conda record: -```text -depends: - - python >=3.8 - - requests >=2.20.0,<3.0 - - click >=7.0 -``` +```json +{ + "depends": [ + "python >=3.8", + "requests >=2.20.0,<3.0", + "click >=7.0", + "numpy >=1.20.0" + ], + "constrains": [ + "numpy !=1.24.0" + ] +} > **Note:** The `importlib-metadata` dependency is omitted because the `Requires-Python: >=3.8 makes its environment marker always false because the package requires Python >=3.8, making the python_version < '3.8' marker always false. @@ -201,11 +240,11 @@ By default, channels with conda packages MUST prefer conda packages when they ar This CEP has the following known limitations: -1. **No support for version exclusions:** Conda does not support `!=` version constraints. Packages with exclusion requirements will have those constraints omitted, which may lead to incompatible versions being selected in rare cases. -2. **Pure Python only:** This CEP explicitly does not address wheels with binary extensions, which require platform-specific compatibility guarantees beyond the current scope. Conda’s strength is binary compatibility, so using conda packages may be the optimal solution. -3. **Environment markers:** Only Python version markers are converted to dependencies. Other environment markers (OS, platform, etc.) are ignored based on the pure Python assumption. -4. **Conditionals and Extras:** Conditional dependencies and extras are specified by a separate CEP. -5. **Repodata size:** Supporting a significant portion of pure Python packages from PyPI (potentially hundreds of thousands of packages with multiple versions each) will substantially increase repodata size. Channels adopting wheel support at scale will need to implement sharded repodata ([CEP 16][cep-16]) to maintain acceptable performance. Alternatively, channels may choose to curate a subset of popular or requested packages rather than mirroring all of PyPI. +1. **Pure Python only:** This CEP explicitly does not address wheels with binary extensions, which require platform-specific compatibility guarantees beyond the current scope. Conda’s strength is binary compatibility, so using conda packages may be the optimal solution. +2. **Environment markers**:** Only Python version markers are converted to dependencies. Other environment markers (OS, platform, etc.) are ignored based on the pure Python assumption. +3. **Conditionals and Extras:** Conditional dependencies and extras are specified by a separate CEP. +4. **Repodata size:** Supporting a significant portion of pure Python packages from PyPI (potentially hundreds of thousands of packages with multiple versions each) will substantially increase repodata size. Channels adopting wheel support at scale will need to implement sharded repodata ([CEP 16][cep-16]) to maintain acceptable performance. Alternatively, channels may choose to curate a subset of popular or requested packages rather than mirroring all of PyPI. +5. **PyPI package deletion:** PyPI allows package maintainers to delete packages (as opposed to just yanking them), which can break locked environments that reference those packages. Channels using external PyPI URLs directly are subject to this risk. For production use and reproducible environments, channels SHOULD mirror and store wheel artifacts locally rather than relying solely on external PyPI URLs. This ensures that packages referenced in lock files remain available even if deleted from PyPI. ## Implementation Notes @@ -226,6 +265,19 @@ Channel operators adding wheel support should: - Maintain a mapping of PyPI to conda-style names for their channel - Consider automation to keep wheel metadata synchronized with PyPI - Document any naming conventions specific to their channel +- For production channels, consider mirroring wheel artifacts locally to ensure reproducibility and protect against PyPI deletions +- Establish patching workflows to correct metadata issues and resolve dependency conflicts +- Document patching policies and maintain transparency about modified packages + +### Implementation approaches + +Channels may implement wheel support through various approaches: + +- **Manual curation**: Channels can manually add specific pure-Python wheels via repodata patching, providing full control over which packages are included and allowing for quality control and metadata validation before exposure to users. +- **Semi-automated**: Channels can implement automated processes to discover and add wheels, with manual review and patching workflows for quality assurance. +- **Full automation**: Channels can automatically mirror and index wheels from PyPI, though this requires robust validation, patching infrastructure, and consideration of repodata size impacts. + +A phased approach starting with manual curation and moving toward increased automation as processes mature is RECOMMENDED. This balances the lower barrier to entry (no recipes needed for many packages) with quality control. Complex packages with metadata conflicts or special requirements should still use traditional conda feedstocks. ## Examples @@ -249,6 +301,7 @@ Below represents the default behavior and shows when the `artifact_url` field is "certifi >=2017.4.17", "python >=3.9" ], + "constrains": [], "fn": "requests-2.32.5-py3-none-any.whl", "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", "size": 6899, @@ -273,7 +326,8 @@ The `artifact_url` can also be relative as described above. Here's an example of { "info": { "subdir": "noarch", - "base_url": "https://repo.example.com/channel/" + "base_url": "https://repo.example.com/channel/", + "name_mapping_channel": "conda-forge" }, "packages.whl": { "requests-2.32.5": { @@ -289,6 +343,7 @@ The `artifact_url` can also be relative as described above. Here's an example of "certifi >=2017.4.17", "python >=3.9" ], + "constrains": [], "fn": "requests-2.32.5-py3-none-any.whl", "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", "size": 6899, @@ -325,6 +380,7 @@ The following shows an example of using an external location to download the whe "certifi >=2017.4.17", "python >=3.9" ], + "constrains": [], "fn": "requests-2.32.5-py3-none-any.whl", "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", "size": 6899, @@ -354,6 +410,7 @@ Here is an example of name mapping and normalization of the record name and depe "typing_extensions >=4.0.0", "python >=3.8" ], + "constrains": [], "fn": "annotated_types-0.7.0-py3-none-any.whl", "sha256": "1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53", "size": 13643, From 337d31ef9ae1963fe70f06006d5dfbc2f3318234 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Tue, 30 Dec 2025 14:43:18 -0500 Subject: [PATCH 07/38] Add section for centralized name mapping in PEP804 --- cep-XXXX.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/cep-XXXX.md b/cep-XXXX.md index 87727737..6708086d 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -111,6 +111,8 @@ When `name_mapping_channel` is present, channel operators SHOULD: Channel operators MUST document any naming conventions and mappings specific to their channel, regardless of whether `name_mapping_channel` is declared. +If a centralized name mapping standard becomes available (such as the external dependency registry proposed in [PEP 804][pep-804]), channels MAY use that standard for name mapping instead of or in addition to channel-specific conventions. This would provide a unified approach to name mapping across ecosystems and reduce discrepancies. + ### Wheel download URLs This CEP introduces a new optional `artifact_url` field in package records to specify download locations for individual packages. @@ -625,3 +627,4 @@ All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdo [uv-in-pixi]: https://prefix.dev/blog/uv_in_pixi [rip]: https://github.com/prefix-dev/rip [grayskull]: https://conda.github.io/grayskull/ +[pep-804]: https://peps.python.org/pep-0804/ From 9ace46e4636dc2e581a54b138a59a2dad263690a Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Tue, 30 Dec 2025 14:53:37 -0500 Subject: [PATCH 08/38] Small formatting fix --- cep-XXXX.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 6708086d..654dbe53 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -105,6 +105,7 @@ To help users understand which channel's naming conventions are being used, chan ``` When `name_mapping_channel` is present, channel operators SHOULD: + - Use the declared channel's package names as the standard for name mapping - Document any naming mappings specific to their channel - Ensure consistency with the declared channel's naming conventions @@ -164,10 +165,11 @@ Channel operators MAY patch `packages.whl` entries in repodata, just like conda - Add or modify metadata fields as needed This patching capability is essential for: -- **Quality control**: Test and validate packages before exposing to users -- **Ecosystem needs**: Resolve naming conflicts and dependency incompatibilities -- **Reproducibility**: Optionally mirror and store wheel artifacts locally to prevent external deletions -- **Downstream fixes**: Address metadata issues that would break environments + +- **Quality control:** Test and validate packages before exposing to users +- **Ecosystem needs:** Resolve naming conflicts and dependency incompatibilities +- **Reproducibility:** Optionally mirror and store wheel artifacts locally to prevent external deletions +- **Downstream fixes:** Address metadata issues that would break environments Channels SHOULD document their patching policies and maintain transparency about which packages have been modified from their upstream PyPI metadata. @@ -619,7 +621,6 @@ All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdo [repodata-schema]: https://schemas.conda.org/repodata-1.schema.json [repodata-record-schema]: https://schemas.conda.org/repodata-record-1.schema.json [cep-15]: https://conda.org/learn/ceps/cep-0015 -[cep-16]: https://conda.org/learn/ceps/cep-0016 [cep-26]: https://conda.org/learn/ceps/cep-0026 [version-specifiers]: https://packaging.python.org/en/latest/specifications/version-specifiers/#id5 [conda-pypi]: https://github.com/conda-incubator/conda-pypi From 80359376db75380b95c06753635f0e67d0c83854 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Tue, 30 Dec 2025 15:09:34 -0500 Subject: [PATCH 09/38] Fix code block not closed and missing URL --- cep-XXXX.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/cep-XXXX.md b/cep-XXXX.md index 654dbe53..91362dd4 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -215,6 +215,7 @@ Resulting conda record: "numpy !=1.24.0" ] } +``` > **Note:** The `importlib-metadata` dependency is omitted because the `Requires-Python: >=3.8 makes its environment marker always false because the package requires Python >=3.8, making the python_version < '3.8' marker always false. @@ -621,6 +622,7 @@ All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdo [repodata-schema]: https://schemas.conda.org/repodata-1.schema.json [repodata-record-schema]: https://schemas.conda.org/repodata-record-1.schema.json [cep-15]: https://conda.org/learn/ceps/cep-0015 +[cep-16]: https://conda.org/learn/ceps/cep-0016 [cep-26]: https://conda.org/learn/ceps/cep-0026 [version-specifiers]: https://packaging.python.org/en/latest/specifications/version-specifiers/#id5 [conda-pypi]: https://github.com/conda-incubator/conda-pypi From 9da5728bf775d973177434cd02a20f6221cc067e Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Mon, 5 Jan 2026 19:26:47 -0500 Subject: [PATCH 10/38] Clarifications based on @JeanChristopheMorinPerso feedback --- cep-XXXX.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 91362dd4..2ede5b19 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -120,11 +120,12 @@ This CEP introduces a new optional `artifact_url` field in package records to sp > Note for this draft: The `artifact_url` field could also be added as a separate CEP to allow it for other record types. -When present, the `artifact_url` field follows these semantics: +When present, the `artifact_url` field SHALL follow these semantics: - If `base_url` is defined in the repodata info object (per [CEP 15][cep-15]), `artifact_url` contains the path relative to `base_url` - If `base_url` is not defined, `artifact_url` contains the complete download URL -- If `artifact_url` is not present (`null`), the download location is constructed from `base_url` and `fn` (existing behavior) + +When not present (`null`), the download location is constructed from `base_url` and `fn` (existing behavior). This approach allows packages to be served from: @@ -137,7 +138,7 @@ This approach allows packages to be served from: When populating repodata records for pure Python wheels: -- `build`: MUST be py3_0 +- `build`: MUST be py`PY_MAJOR_VERSION`_`build_number` (e.g. py3_0) - `build_number`: MUST be 0 for the initial addition of a wheel version. MAY be incremented for subsequent rebuilds of the same - wheel version (e.g., to correct dependencies or metadata) - `fn`: MUST be the wheel filename (e.g., package-1.0.0-py3-none-any.whl) - `subdir`: MUST be "noarch" @@ -249,7 +250,7 @@ This CEP has the following known limitations: 2. **Environment markers**:** Only Python version markers are converted to dependencies. Other environment markers (OS, platform, etc.) are ignored based on the pure Python assumption. 3. **Conditionals and Extras:** Conditional dependencies and extras are specified by a separate CEP. 4. **Repodata size:** Supporting a significant portion of pure Python packages from PyPI (potentially hundreds of thousands of packages with multiple versions each) will substantially increase repodata size. Channels adopting wheel support at scale will need to implement sharded repodata ([CEP 16][cep-16]) to maintain acceptable performance. Alternatively, channels may choose to curate a subset of popular or requested packages rather than mirroring all of PyPI. -5. **PyPI package deletion:** PyPI allows package maintainers to delete packages (as opposed to just yanking them), which can break locked environments that reference those packages. Channels using external PyPI URLs directly are subject to this risk. For production use and reproducible environments, channels SHOULD mirror and store wheel artifacts locally rather than relying solely on external PyPI URLs. This ensures that packages referenced in lock files remain available even if deleted from PyPI. +5. **PyPI package deletion:** PyPI allows package maintainers to delete packages (as opposed to just yanking them), which can break locked environments that reference those packages. Channels using external PyPI URLs directly are subject to this risk. For production use and reproducible environments, channels MAY mirror and store wheel artifacts locally rather than relying solely on external PyPI URLs. ## Implementation Notes @@ -264,11 +265,11 @@ Clients implementing this CEP SHOULD: ### For channel operators -Channel operators adding wheel support should: +Channel operators adding wheel support SHOULD: - Implement validation to ensure only pure Python wheels are included - Maintain a mapping of PyPI to conda-style names for their channel -- Consider automation to keep wheel metadata synchronized with PyPI +- Consider automation to keep the repodata up to date with newer releases on PyPI - Document any naming conventions specific to their channel - For production channels, consider mirroring wheel artifacts locally to ensure reproducibility and protect against PyPI deletions - Establish patching workflows to correct metadata issues and resolve dependency conflicts @@ -276,7 +277,7 @@ Channel operators adding wheel support should: ### Implementation approaches -Channels may implement wheel support through various approaches: +Channels MAY implement wheel support through various approaches: - **Manual curation**: Channels can manually add specific pure-Python wheels via repodata patching, providing full control over which packages are included and allowing for quality control and metadata validation before exposure to users. - **Semi-automated**: Channels can implement automated processes to discover and add wheels, with manual review and patching workflows for quality assurance. From 5c28f04f81a454beabc6d2b9728fa6de846004c3 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Mon, 5 Jan 2026 19:49:33 -0500 Subject: [PATCH 11/38] Add clarification about using wheel metadata names --- cep-XXXX.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 2ede5b19..62388ddb 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -76,8 +76,8 @@ Examples: Several factors can cause wheel names to differ from conda-style names: -1. **Name normalization:** PyPI normalizes dashes to underscores (PEP 503) - - Example: `lazy-loader` (conda-forge) vs `lazy_loader` (PyPI wheel) +1. **Wheel filename normalization:** Wheel filenames normalize hyphens to underscores per PEP 427, but the METADATA `Name` field contains the canonical name (which may have hyphens). + - Example: `lazy-loader` (METADATA name) vs `lazy_loader` (wheel filename) 2. **Python-specific clarification:** PyPI packages are implicitly Python libraries - Example: `authzed-py` (conda-forge) vs `authzed` (PyPI) @@ -90,6 +90,8 @@ Several factors can cause wheel names to differ from conda-style names: ### Naming standard and channel mapping +The key name for wheel records MUST be derived from the wheel's METADATA file (the `Name` field), normalized according to conda naming conventions per [CEP 26 - Identifying Packages and Channels in the conda Ecosystem][cep-26]. The wheel filename (stored in the `fn` field) MUST NOT be used as the key, as wheel filenames may differ from the canonical package name due to PEP 427 normalization. + When there are naming differences between PyPI wheels and conda packages, channel operators MUST determine the appropriate conda-style name by applying conda naming conventions per [CEP 26 - Identifying Packages and Channels in the conda Ecosystem][cep-26]. To help users understand which channel's naming conventions are being used, channels MAY declare an optional `name_mapping_channel` field in the `info` section of repodata. This field indicates which channel's package names are used as the standard for wheel-to-conda name mapping in this repodata. From bad61c5037c499a7877e8a8df276c865b7f32d7a Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Mon, 5 Jan 2026 19:37:05 -0500 Subject: [PATCH 12/38] Change from MUST to SHOULD --- cep-XXXX.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 62388ddb..371d49ea 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -112,7 +112,7 @@ When `name_mapping_channel` is present, channel operators SHOULD: - Document any naming mappings specific to their channel - Ensure consistency with the declared channel's naming conventions -Channel operators MUST document any naming conventions and mappings specific to their channel, regardless of whether `name_mapping_channel` is declared. +Channel operators SHOULD document any naming conventions and mappings specific to their channel, regardless of whether `name_mapping_channel` is declared. If a centralized name mapping standard becomes available (such as the external dependency registry proposed in [PEP 804][pep-804]), channels MAY use that standard for name mapping instead of or in addition to channel-specific conventions. This would provide a unified approach to name mapping across ecosystems and reduce discrepancies. From d7f018d78e08c9cf03dfddb588b44769533b9dd3 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Tue, 6 Jan 2026 21:01:11 -0500 Subject: [PATCH 13/38] Add clarifications from @baszalmstra and @wolfv --- cep-XXXX.md | 107 +++++++++++++++++++++++++++++++--------------------- 1 file changed, 65 insertions(+), 42 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 371d49ea..9ef4ace8 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -64,13 +64,18 @@ To support sparse repodata processing and maintain compatibility with conda's ex ### Key naming requirements -The key for each entry in `packages.whl` MUST follow the format `{name}-{version}`. +The key for each entry in `packages.whl` SHALL follow the format `{name}-{version}-{build}-{abi_tag}-{platform_tag}`, where: + +- `{name}` is derived from the wheel's METADATA file (the `Name` field), normalized according to conda naming conventions per [CEP 26 - Identifying Packages and Channels in the conda Ecosystem][cep-26] and any name mappings specified in `name_mapping_channel` +- `{version}` is the package version from METADATA +- `{build}` is the build string (e.g., `py3_0`, `py3_1`) from the `build` field +- `{abi_tag}-{platform_tag}` are extracted from the wheel filename (stored in the `fn` field) Examples: -- `httpx-0.28.1` -- `typing_extensions-4.15.0` -- `lazy-loader-0.4` +- `httpx-0.28.1-py3_0-none-any` +- `typing_extensions-4.15.0-py3_0-none-any` (METADATA name `typing-extensions` mapped to conda name `typing_extensions`) +- `requests-2.32.5-py3_1-none-any` (rebuild with build_number 1) ### Why wheel names may differ from conda names @@ -90,32 +95,55 @@ Several factors can cause wheel names to differ from conda-style names: ### Naming standard and channel mapping -The key name for wheel records MUST be derived from the wheel's METADATA file (the `Name` field), normalized according to conda naming conventions per [CEP 26 - Identifying Packages and Channels in the conda Ecosystem][cep-26]. The wheel filename (stored in the `fn` field) MUST NOT be used as the key, as wheel filenames may differ from the canonical package name due to PEP 427 normalization. +The key name for wheel records SHALL be constructed by combining the conda-style name (derived from METADATA and normalized per conda conventions) with the version, build string, and wheel tags (abi_tag-platform_tag) extracted from the filename. The python_tag is omitted as it is redundant with the build string. This ensures the name portion follows conda naming conventions, supports multiple rebuilds, and the tags portion aligns with the wheel filename structure. When there are naming differences between PyPI wheels and conda packages, channel operators MUST determine the appropriate conda-style name by applying conda naming conventions per [CEP 26 - Identifying Packages and Channels in the conda Ecosystem][cep-26]. -To help users understand which channel's naming conventions are being used, channels MAY declare an optional `name_mapping_channel` field in the `info` section of repodata. This field indicates which channel's package names are used as the standard for wheel-to-conda name mapping in this repodata. +To help users understand which naming conventions are being used, channels MAY declare an optional `name_mapping_channel` field in the `info` section of repodata. This field SHALL be a URL to a JSON file containing package name mappings used for wheel-to-conda name translation in this repodata. + +The URL MAY be absolute or relative to `base_url` (following the same semantics as `artifact_url`). The JSON file SHALL contain a JSON object mapping PyPI package names to conda package names. + +Example with a relative URL: ```json { "info": { "subdir": "noarch", "base_url": "https://repo.example.com/channel/", - "name_mapping_channel": "conda-forge" + "name_mapping_channel": "name-mappings.json" } } ``` +Example with an absolute URL: + +```json +{ + "info": { + "subdir": "noarch", + "base_url": "https://repo.example.com/channel/", + "name_mapping_channel": "https://example.com/mappings/conda-forge-mappings.json" + } +} +``` + +Example JSON file structure (`name-mappings.json`): + +```json +{ + "typing-extensions": "typing_extensions", + "importlib-metadata": "importlib_metadata", + "setuptools-scm": "setuptools_scm" +} +``` + When `name_mapping_channel` is present, channel operators SHOULD: -- Use the declared channel's package names as the standard for name mapping -- Document any naming mappings specific to their channel -- Ensure consistency with the declared channel's naming conventions +- Use the mappings from the JSON file as the standard for name mapping +- Ensure the JSON file is accessible and kept up to date Channel operators SHOULD document any naming conventions and mappings specific to their channel, regardless of whether `name_mapping_channel` is declared. -If a centralized name mapping standard becomes available (such as the external dependency registry proposed in [PEP 804][pep-804]), channels MAY use that standard for name mapping instead of or in addition to channel-specific conventions. This would provide a unified approach to name mapping across ecosystems and reduce discrepancies. - ### Wheel download URLs This CEP introduces a new optional `artifact_url` field in package records to specify download locations for individual packages. @@ -124,8 +152,8 @@ This CEP introduces a new optional `artifact_url` field in package records to sp When present, the `artifact_url` field SHALL follow these semantics: -- If `base_url` is defined in the repodata info object (per [CEP 15][cep-15]), `artifact_url` contains the path relative to `base_url` -- If `base_url` is not defined, `artifact_url` contains the complete download URL +- If `artifact_url` is an absolute URL, use it as is. +- If `artifact_url` is a relative URL, append it to the `base_url`. When not present (`null`), the download location is constructed from `base_url` and `fn` (existing behavior). @@ -145,17 +173,18 @@ When populating repodata records for pure Python wheels: - `fn`: MUST be the wheel filename (e.g., package-1.0.0-py3-none-any.whl) - `subdir`: MUST be "noarch" - `noarch`: MUST be "python" -- `artifact_url`: MUST be present and follow the semantics described above +- `artifact_url`: MAY be present and follow the semantics described above ### Pure Python wheel validation -Before adding a wheel to packages.whl, channel operators MUST verify: +Before adding a wheel to `packages.whl`, channel operators MUST verify: -- The wheel platform tag is any (e.g., py3-none-any, py2.py3-none-any) -- The wheel contains no compiled extensions (.so, .pyd, .dylib files) -- The wheel METADATA file is present and valid +- The wheel's platform tag is `any` (e.g., `py3-none-any`, `py2.py3-none-any`) +- The wheel's ABI tag is `none` +- The wheel contains no compiled extensions (`.so`, `.pyd`, `.dylib` files) +- The wheel's `METADATA` file is present and valid -Wheels that fail these checks MUST NOT be added to packages.whl. +Wheels that fail any of these checks MUST NOT be added to `packages.whl`. ### Repodata patching support @@ -187,8 +216,8 @@ Wheel dependencies (from METADATA file's Requires-Dist entries) MUST be converte - `<=X.Y.Z` → `<=X.Y.Z` - `X.Y.Z` → `>X.Y.Z` - - `~=X.Y` → `>=X.Y,=1.0,<2.0) - **Python version requirements:** Convert Requires-Python to explicit python dependency - **Environment markers:** Ignore markers other than Python version (pure Python assumption) @@ -212,16 +241,11 @@ Resulting conda record: "python >=3.8", "requests >=2.20.0,<3.0", "click >=7.0", - "numpy >=1.20.0" - ], - "constrains": [ - "numpy !=1.24.0" + "numpy >=1.20.0,!=1.24.0" ] } ``` -> **Note:** The `importlib-metadata` dependency is omitted because the `Requires-Python: >=3.8 makes its environment marker always false because the package requires Python >=3.8, making the python_version < '3.8' marker always false. - ### Handling conditional and extra dependencies Like in the example above of only requiring `importlib-metadata` for certain Python versions, conditional and extra dependencies MUST be supported to enable full interoperability between the ecosystems. This will be supported through a separate CEP: . @@ -233,7 +257,7 @@ Like in the example above of only requiring `importlib-metadata` for certain Pyt Solvers MUST treat pure Python wheels as valid package candidates during dependency resolution with these constraints: - **Exclusivity:** Solvers MUST NOT install both a conda package and wheel for the same package name. -- **Dependency satisfaction:** When a wheel is selected, its `depends` list MUST be satisfied like any conda package. +- **Dependency satisfaction:** When a wheel is selected, its `depends` list MUST be satisfied like any conda package. Dependencies in the `depends` list MAY be satisfied by either wheels or conda packages. - **Platform matching:** Since all wheels in `packages.whl` are pure Python (noarch), no platform filtering is needed. #### User control of precedence @@ -296,7 +320,7 @@ Below represents the default behavior and shows when the `artifact_url` field is ```json { "packages.whl": { - "requests-2.32.5": { + "requests-2.32.5-py3_0-none-any": { "record_version": 3, "name": "requests", "version": "2.32.5", @@ -335,10 +359,10 @@ The `artifact_url` can also be relative as described above. Here's an example of "info": { "subdir": "noarch", "base_url": "https://repo.example.com/channel/", - "name_mapping_channel": "conda-forge" + "name_mapping_channel": "name-mappings.json" }, "packages.whl": { - "requests-2.32.5": { + "requests-2.32.5-py3_0-none-any": { "record_version": 3, "name": "requests", "version": "2.32.5", @@ -375,7 +399,7 @@ The following shows an example of using an external location to download the whe ```json { "packages.whl": { - "requests-2.32.5": { + "requests-2.32.5-py3_0-none-any": { "record_version": 3, "name": "requests", "version": "2.32.5", @@ -408,14 +432,14 @@ Here is an example of name mapping and normalization of the record name and depe ```json { "packages.whl": { - "annotated-types-0.7.0": { + "annotated_types-0.7.0-py3_0-none-any": { "record_version": 3, "name": "annotated-types", "version": "0.7.0", "build": "py3_0", "build_number": 0, "depends": [ - "typing_extensions >=4.0.0", + "typing_extensions >=4.0.0; if python < 3.9", "python >=3.8" ], "constrains": [], @@ -433,7 +457,8 @@ Here is an example of name mapping and normalization of the record name and depe This example demonstrates two types of name normalization: -1. Record key format: The package is indexed using the `{name}-{version}` format: `annotated-types-0.7.0`. The conda name uses hyphens which matches the canonical PyPI package name. The actual wheel filename in the fn field (`annotated_types-0.7.0-py3-none-any.whl`) normalizes the package name to underscores per PEP 427. +1. Record key format: The package is indexed using the format `{conda_name}-{version}-{build}-{abi_tag}-{platform_tag}`: `annotated_types-0.7.0-py3_0-none-any`. The name portion (`annotated_types`) comes from the METADATA `Name` field (`annotated-types`), normalized to conda conventions (mapped to `annotated_types` to match conda-forge naming). +The build portion (`py3_0`) comes from the `build` field. The tags portion (`none-any`) is extracted from the wheel filename (`annotated_types-0.7.0-py3-none-any.whl`), which normalizes the package name to underscores per PEP 427. 2. Dependency name mapping: This package depends on `typing_extensions`, which is listed in the `depends` field. On PyPI, this package is named `typing-extensions` (with a hyphen), but it has been mapped to the name `typing_extensions` (with an underscore) to match the existing conda-forge package name. This example also demonstrates conditional dependencies. The original `METADATA` file from the wheel has the following dependency information: @@ -443,7 +468,7 @@ Requires-Python: >=3.8 Requires-Dist: typing-extensions>=4.0.0; python_version < '3.9' ``` -Once conditional dependencies are supported in a separate CEP, the package record should only declare `typing_extensions` as a dependency when the Python version is < 3.9. In this example, the dependency is declared unconditionally for all Python versions. The Python version constraint of >=3.8 is directly mapped. +The package record uses conditional dependency syntax (`; if python < 3.9`) to declare `typing_extensions` only when the Python version is < 3.9, matching the original wheel METADATA. The Python version constraint of >=3.8 is directly mapped. ## Rejected ideas @@ -506,9 +531,9 @@ Pixi has integrated uv for installing packages from PyPI. The user adds the depe Another alternative would be for conda clients to query PyPI's API directly during solving, fetching wheel metadata on-demand rather than including it in repodata. This idea was rejected due to: -- Conda's SAT-based solvers (libsolv, resolvo) require complete package metadata upfront to solve in a single pass, while PyPI's API is designed for lazy evaluation with incremental fetching and backtracking (as pip and uv do). Adapting conda's solvers would require fundamental architectural changes. +- While resolvo (used by Rattler) supports dynamic metadata fetching during solving (as showcased in [rip](https://github.com/prefix-dev/rip)), libsolv requires complete package metadata upfront. This inconsistency across solvers would complicate implementation and limit compatibility. -- On-demand fetching would also require hundreds of sequential HTTP requests during solving, making solves slow and network-dependent while breaking offline capability and reproducibility guarantees. +- While on-demand fetching works well for pip and uv, using repodata provides consistency with conda's existing infrastructure, enables better caching strategies, and allows channels to curate and validate packages before they're available to users. ### Automatic wheel to conda package conversion and hosting @@ -567,7 +592,7 @@ Arguments to for maintaining an independent packaging ecosystem prevailed at the ### Wheel Support Proposal (2017) -Issue [#5202](https://github.com/conda/conda/issue/5202) proposed direct wheel installation support, recognizing: +Issue [#5202](https://github.com/conda/conda/issues/5202) proposed direct wheel installation support, recognizing: - Wheels provide binary distributions similar to conda packages - Growing wheel availability on PyPI reduced build complexity @@ -624,7 +649,6 @@ All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdo [RFC2119]: https://datatracker.ietf.org/doc/html/rfc2119 [repodata-schema]: https://schemas.conda.org/repodata-1.schema.json [repodata-record-schema]: https://schemas.conda.org/repodata-record-1.schema.json -[cep-15]: https://conda.org/learn/ceps/cep-0015 [cep-16]: https://conda.org/learn/ceps/cep-0016 [cep-26]: https://conda.org/learn/ceps/cep-0026 [version-specifiers]: https://packaging.python.org/en/latest/specifications/version-specifiers/#id5 @@ -633,4 +657,3 @@ All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdo [uv-in-pixi]: https://prefix.dev/blog/uv_in_pixi [rip]: https://github.com/prefix-dev/rip [grayskull]: https://conda.github.io/grayskull/ -[pep-804]: https://peps.python.org/pep-0804/ From 026a22ea771108938d617903bf104314bbeb17ca Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Tue, 6 Jan 2026 21:22:56 -0500 Subject: [PATCH 14/38] Reduce cons of wheel to conda package conversion --- cep-XXXX.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 9ef4ace8..fe9530b5 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -549,13 +549,10 @@ Another alternative would be establishing a build farm to automatically convert Despite these advantages, this approach was rejected because: -- **Infrastructure burden:** Requires significant storage and bandwidth to host and serve converted packages that duplicate PyPI's CDN infrastructure, plus ongoing maintenance of the conversion pipeline and synchronization automation +- **Infrastructure burden:** Requires significant storage and bandwidth to host and serve converted packages that duplicate PyPI's CDN infrastructure - **Resource inefficiency:** Wheels are already an excellent format for pure Python packages; conversion adds no technical value and wastes resources -- **Timeliness:** Creates lag between PyPI publication and availability, with curation decisions needed about which packages to convert -- **Maintenance complexity:** Every wheel format change requires tooling updates; bugs affect all converted packages -- **Ecosystem misalignment:** Converting away from wheels moves conda further from Python packaging standards -Native wheel support provides the same user experience (transparent PyPI access) while avoiding infrastructure burden and maintaining alignment with Python packaging standards. Channel operators who prefer converted packages can continue building conda packages from PyPI sources. +Native wheel support provides the same user experience (transparent PyPI access) while avoiding the infrastructure burden and resource inefficiency of conversion. Channel operators who prefer converted packages can continue building conda packages from PyPI sources. ## History From 9c51042f006a97b392b0111ba0412383494d5c0c Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Tue, 6 Jan 2026 22:08:44 -0500 Subject: [PATCH 15/38] Rename mapping field to name_mappings --- cep-XXXX.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index fe9530b5..8dd1c27e 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -66,7 +66,7 @@ To support sparse repodata processing and maintain compatibility with conda's ex The key for each entry in `packages.whl` SHALL follow the format `{name}-{version}-{build}-{abi_tag}-{platform_tag}`, where: -- `{name}` is derived from the wheel's METADATA file (the `Name` field), normalized according to conda naming conventions per [CEP 26 - Identifying Packages and Channels in the conda Ecosystem][cep-26] and any name mappings specified in `name_mapping_channel` +- `{name}` is derived from the wheel's METADATA file (the `Name` field), normalized according to conda naming conventions per [CEP 26 - Identifying Packages and Channels in the conda Ecosystem][cep-26] and any name mappings specified in `name_mappings` - `{version}` is the package version from METADATA - `{build}` is the build string (e.g., `py3_0`, `py3_1`) from the `build` field - `{abi_tag}-{platform_tag}` are extracted from the wheel filename (stored in the `fn` field) @@ -99,7 +99,7 @@ The key name for wheel records SHALL be constructed by combining the conda-style When there are naming differences between PyPI wheels and conda packages, channel operators MUST determine the appropriate conda-style name by applying conda naming conventions per [CEP 26 - Identifying Packages and Channels in the conda Ecosystem][cep-26]. -To help users understand which naming conventions are being used, channels MAY declare an optional `name_mapping_channel` field in the `info` section of repodata. This field SHALL be a URL to a JSON file containing package name mappings used for wheel-to-conda name translation in this repodata. +To help users understand which naming conventions are being used, channels MAY declare an optional `name_mappings` field in the `info` section of repodata. This field SHALL be a URL to a JSON file containing package name mappings used for wheel-to-conda name translation in this repodata. The URL MAY be absolute or relative to `base_url` (following the same semantics as `artifact_url`). The JSON file SHALL contain a JSON object mapping PyPI package names to conda package names. @@ -110,7 +110,7 @@ Example with a relative URL: "info": { "subdir": "noarch", "base_url": "https://repo.example.com/channel/", - "name_mapping_channel": "name-mappings.json" + "name_mappings": "name-mappings.json" } } ``` @@ -122,7 +122,7 @@ Example with an absolute URL: "info": { "subdir": "noarch", "base_url": "https://repo.example.com/channel/", - "name_mapping_channel": "https://example.com/mappings/conda-forge-mappings.json" + "name_mappings": "https://example.com/mappings/conda-forge-mappings.json" } } ``` @@ -137,12 +137,12 @@ Example JSON file structure (`name-mappings.json`): } ``` -When `name_mapping_channel` is present, channel operators SHOULD: +When `name_mappings` is present, channel operators SHOULD: - Use the mappings from the JSON file as the standard for name mapping - Ensure the JSON file is accessible and kept up to date -Channel operators SHOULD document any naming conventions and mappings specific to their channel, regardless of whether `name_mapping_channel` is declared. +Channel operators SHOULD document any naming conventions and mappings specific to their channel, regardless of whether `name_mappings` is declared. ### Wheel download URLs @@ -359,7 +359,7 @@ The `artifact_url` can also be relative as described above. Here's an example of "info": { "subdir": "noarch", "base_url": "https://repo.example.com/channel/", - "name_mapping_channel": "name-mappings.json" + "name_mappings": "name-mappings.json" }, "packages.whl": { "requests-2.32.5-py3_0-none-any": { From ee3b1a30eb60ba7a85003b291e5f613b17c227db Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Wed, 7 Jan 2026 14:22:48 -0500 Subject: [PATCH 16/38] Replace JSON name mapping with a parent channel --- cep-XXXX.md | 56 ++++++++--------------------------------------------- 1 file changed, 8 insertions(+), 48 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 8dd1c27e..d6397b84 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -66,7 +66,7 @@ To support sparse repodata processing and maintain compatibility with conda's ex The key for each entry in `packages.whl` SHALL follow the format `{name}-{version}-{build}-{abi_tag}-{platform_tag}`, where: -- `{name}` is derived from the wheel's METADATA file (the `Name` field), normalized according to conda naming conventions per [CEP 26 - Identifying Packages and Channels in the conda Ecosystem][cep-26] and any name mappings specified in `name_mappings` +- `{name}` is derived from the wheel's METADATA file (the `Name` field), normalized according to conda naming conventions per [CEP 26 - Identifying Packages and Channels in the conda Ecosystem][cep-26] and any name mappings inherited from a parent channel (see [Naming standard and channel mapping](#naming-standard-and-channel-mapping)) - `{version}` is the package version from METADATA - `{build}` is the build string (e.g., `py3_0`, `py3_1`) from the `build` field - `{abi_tag}-{platform_tag}` are extracted from the wheel filename (stored in the `fn` field) @@ -99,50 +99,10 @@ The key name for wheel records SHALL be constructed by combining the conda-style When there are naming differences between PyPI wheels and conda packages, channel operators MUST determine the appropriate conda-style name by applying conda naming conventions per [CEP 26 - Identifying Packages and Channels in the conda Ecosystem][cep-26]. -To help users understand which naming conventions are being used, channels MAY declare an optional `name_mappings` field in the `info` section of repodata. This field SHALL be a URL to a JSON file containing package name mappings used for wheel-to-conda name translation in this repodata. +To help users understand which naming conventions are being used, channels MAY reference a parent channel that defines the naming conventions and mappings for wheel-to-conda name translation. Parent/child channel relationships will be defined in a separate CEP, which will specify how channels can declare a `parent_channel` field in the `info` section of repodata to inherit naming conventions and mappings from another channel. +For example, a wheel channel could reference conda-forge as its parent channel, inheriting conda-forge's established naming conventions and package name mappings. This allows channels to avoid duplicating name mapping definitions and ensures consistency across related channels. -The URL MAY be absolute or relative to `base_url` (following the same semantics as `artifact_url`). The JSON file SHALL contain a JSON object mapping PyPI package names to conda package names. - -Example with a relative URL: - -```json -{ - "info": { - "subdir": "noarch", - "base_url": "https://repo.example.com/channel/", - "name_mappings": "name-mappings.json" - } -} -``` - -Example with an absolute URL: - -```json -{ - "info": { - "subdir": "noarch", - "base_url": "https://repo.example.com/channel/", - "name_mappings": "https://example.com/mappings/conda-forge-mappings.json" - } -} -``` - -Example JSON file structure (`name-mappings.json`): - -```json -{ - "typing-extensions": "typing_extensions", - "importlib-metadata": "importlib_metadata", - "setuptools-scm": "setuptools_scm" -} -``` - -When `name_mappings` is present, channel operators SHOULD: - -- Use the mappings from the JSON file as the standard for name mapping -- Ensure the JSON file is accessible and kept up to date - -Channel operators SHOULD document any naming conventions and mappings specific to their channel, regardless of whether `name_mappings` is declared. +Channel operators SHOULD document any naming conventions and mappings specific to their channel, including whether they reference a parent channel for naming conventions. ### Wheel download URLs @@ -294,7 +254,7 @@ Clients implementing this CEP SHOULD: Channel operators adding wheel support SHOULD: - Implement validation to ensure only pure Python wheels are included -- Maintain a mapping of PyPI to conda-style names for their channel +- Maintain a mapping of PyPI to conda-style names for their channel, or reference a parent channel that provides these mappings (see [Naming standard and channel mapping](#naming-standard-and-channel-mapping)) - Consider automation to keep the repodata up to date with newer releases on PyPI - Document any naming conventions specific to their channel - For production channels, consider mirroring wheel artifacts locally to ensure reproducibility and protect against PyPI deletions @@ -352,14 +312,14 @@ With this configuration, the wheel file will be downloaded from the following lo ### Downloading wheels from a relative location with `base_url` -The `artifact_url` can also be relative as described above. Here's an example of what that looks like combined with setting the `base_url` property at the top level: +The `artifact_url` can also be relative as described above. Here's an example of what that looks like combined with setting the `base_url` property at the top level, and also showing how a channel can reference a parent channel: ```json { "info": { "subdir": "noarch", "base_url": "https://repo.example.com/channel/", - "name_mappings": "name-mappings.json" + "parent_channel": "https://conda.anaconda.org/conda-forge" }, "packages.whl": { "requests-2.32.5-py3_0-none-any": { @@ -459,7 +419,7 @@ This example demonstrates two types of name normalization: 1. Record key format: The package is indexed using the format `{conda_name}-{version}-{build}-{abi_tag}-{platform_tag}`: `annotated_types-0.7.0-py3_0-none-any`. The name portion (`annotated_types`) comes from the METADATA `Name` field (`annotated-types`), normalized to conda conventions (mapped to `annotated_types` to match conda-forge naming). The build portion (`py3_0`) comes from the `build` field. The tags portion (`none-any`) is extracted from the wheel filename (`annotated_types-0.7.0-py3-none-any.whl`), which normalizes the package name to underscores per PEP 427. -2. Dependency name mapping: This package depends on `typing_extensions`, which is listed in the `depends` field. On PyPI, this package is named `typing-extensions` (with a hyphen), but it has been mapped to the name `typing_extensions` (with an underscore) to match the existing conda-forge package name. +2. Dependency name mapping: This package depends on `typing_extensions`, which is listed in the `depends` field. On PyPI, this package is named `typing-extensions` (with a hyphen), but it has been mapped to the name `typing_extensions` (with an underscore) to match the existing conda-forge package name. Such mappings may be inherited from a parent channel (see [Naming standard and channel mapping](#naming-standard-and-channel-mapping)). This example also demonstrates conditional dependencies. The original `METADATA` file from the wheel has the following dependency information: From 4fd08bfd909a114438b428b80a6d430052f7efe1 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Wed, 7 Jan 2026 14:30:04 -0500 Subject: [PATCH 17/38] Simplify version specifier language --- cep-XXXX.md | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index d6397b84..18b821a1 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -171,13 +171,10 @@ Wheel dependencies (from METADATA file's Requires-Dist entries) MUST be converte - **Package names:** Names per [CEP 26][cep-26] and match existing conda-forge package names where they exist - **Version specifiers:** Map PEP 440 [version specifiers][version-specifiers] to conda format: - - `==X.Y.Z` → `X.Y.Z` (exact pin) - - `>=X.Y.Z` → `>=X.Y.Z` - - `<=X.Y.Z` → `<=X.Y.Z` - - `X.Y.Z` → `>X.Y.Z` - - `~=X.Y` → `~=X.Y` (compatible release) - - `!=X.Y.Z` → `!=X.Y.Z` + - The `==` operator is converted to an exact pin (removing the `==`) + - All other PEP 440 operators (`>=`, `<=`, `<`, `>`, `~=`, `!=`) are used as-is + - Multiple version specifiers are combined with commas (e.g., `>=1.0,<2.0`) + - **Multiple specifiers:** Combine with commas (e.g., >=1.0,<2.0) - **Python version requirements:** Convert Requires-Python to explicit python dependency - **Environment markers:** Ignore markers other than Python version (pure Python assumption) From 0747ed0471007202486d1b10bc9efb1477d662ed Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Wed, 7 Jan 2026 21:50:53 -0500 Subject: [PATCH 18/38] Updates from feedback from @jaimegrp --- cep-XXXX.md | 46 +++++++++++++++++++--------------------------- 1 file changed, 19 insertions(+), 27 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 18b821a1..97c8367e 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -1,7 +1,7 @@ # CEP XXXX - Repodata Wheel Support - + - + -N/A +
Title Repodata Wheel Support
Title Repodata Wheel Support
Status Draft
Author(s) Dan Yeaw <dyeaw@anaconda.com>
@@ -9,9 +9,9 @@
Created Dec 23, 2025
Updated Dec 23, 2025
Discussion https://github.com/conda/ceps/pull/
Discussion https://github.com/conda/ceps/pull/145
Implementation TBD
Requires
Requires N/A
> The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", @@ -25,7 +25,7 @@ We explicitly limit the scope of this CEP to pure Python wheels to avoid platfor ## Motivation -While conda remains a language-agnostic packaging distribution, installing packages for use with Python represents the majority of conda usage. Users frequently encounter packages only available as wheels on PyPI, forcing them to use hybrid workflows that mix conda and pip installations. This approach creates several problems: +While conda remains a language-agnostic packaging distribution, installing packages for use with Python represents the majority of conda usage. Users frequently encounter packages only available as wheels on PyPI, forcing them to use hybrid workflows that mix conda and PyPI installations. This approach creates several problems: - Mixing of conda and pip or uv can result in overwritten files and broken environments - Users must understand two package managers, their interaction points, and which of their dependencies are available from which ecosystem @@ -33,10 +33,10 @@ While conda remains a language-agnostic packaging distribution, installing packa By adding native support for pure Python wheels to repodata, conda clients can: -- Resolve dependencies across conda and PyPI packages in a single solve +- Eliminate the cognitive burden of managing two package managers - Provide users with transparent access to the broader Python ecosystem - Maintain environment consistency and reproducibility -- Eliminate the cognitive burden of managing two package managers +- Resolve dependencies across conda and PyPI packages in a single solve - Fill gaps in conda package availability for simpler pure-Python packages - Reduce the maintenance burden for straightforward pure-Python packages that don't require metadata modifications @@ -64,9 +64,9 @@ To support sparse repodata processing and maintain compatibility with conda's ex ### Key naming requirements -The key for each entry in `packages.whl` SHALL follow the format `{name}-{version}-{build}-{abi_tag}-{platform_tag}`, where: +The key for each entry in `packages.whl` MUST follow the format `{name}-{version}-{build}__{abi_tag}__{platform_tag}`, where: -- `{name}` is derived from the wheel's METADATA file (the `Name` field), normalized according to conda naming conventions per [CEP 26 - Identifying Packages and Channels in the conda Ecosystem][cep-26] and any name mappings inherited from a parent channel (see [Naming standard and channel mapping](#naming-standard-and-channel-mapping)) +- `{name}` is derived from the wheel's METADATA file (the `Name` field), normalized according to conda naming conventions per [CEP 26][cep-26] and any name mappings inherited from a parent channel (see [Naming standard and channel mapping](#naming-standard-and-channel-mapping)) - `{version}` is the package version from METADATA - `{build}` is the build string (e.g., `py3_0`, `py3_1`) from the `build` field - `{abi_tag}-{platform_tag}` are extracted from the wheel filename (stored in the `fn` field) @@ -146,25 +146,6 @@ Before adding a wheel to `packages.whl`, channel operators MUST verify: Wheels that fail any of these checks MUST NOT be added to `packages.whl`. -### Repodata patching support - -Channel operators MAY patch `packages.whl` entries in repodata, just like conda packages, to: - -- Correct dependency metadata errors or conflicts -- Apply immediate fixes without waiting for upstream releases -- Adjust version constraints for conda ecosystem compatibility -- Correct dependency name mappings that were incorrectly converted -- Add or modify metadata fields as needed - -This patching capability is essential for: - -- **Quality control:** Test and validate packages before exposing to users -- **Ecosystem needs:** Resolve naming conflicts and dependency incompatibilities -- **Reproducibility:** Optionally mirror and store wheel artifacts locally to prevent external deletions -- **Downstream fixes:** Address metadata issues that would break environments - -Channels SHOULD document their patching policies and maintain transparency about which packages have been modified from their upstream PyPI metadata. - ### Dependency conversion Wheel dependencies (from METADATA file's Requires-Dist entries) MUST be converted to conda format following these rules: @@ -492,6 +473,17 @@ Another alternative would be for conda clients to query PyPI's API directly duri - While on-demand fetching works well for pip and uv, using repodata provides consistency with conda's existing infrastructure, enables better caching strategies, and allows channels to curate and validate packages before they're available to users. +### Magic local channel approach + +The "magic local channel" approach () was considered but rejected. This approach involves: + +- A local FastAPI app intercepts the CLI and detects PyPI specs +- Fetches metadata on the fly and converts it to repodata (following some of the ideas discussed above) +- Downloads the wheels and converts them to .conda via whl2conda +- Caches and installs the .conda artifacts + +While this approach provides on-demand conversion and caching, it requires a separate service to be running and adds complexity to the user workflow. The chosen approach of native repodata wheel support provides a more seamless experience where wheels are pre-indexed in channels and work with standard conda workflows without requiring additional services. + ### Automatic wheel to conda package conversion and hosting Another alternative would be establishing a build farm to automatically convert wheels to conda packages and host them on a channel, with conversion triggered by popularity, community requests, or on-demand. From d20b1c00deff29610a10602723bd27b16609dd31 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Wed, 7 Jan 2026 22:25:04 -0500 Subject: [PATCH 19/38] Make dictionary structure more explicit --- cep-XXXX.md | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 97c8367e..12663577 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -54,13 +54,28 @@ level keys: - packages - packages.conda - removed -- signatures -This CEP proposes the addition of a new `packages.whl` section to account for the wheel format. This key points to a mapping that MUST contain [repodata record][repodata-record-schema] objects. The key of this mapping MUST follow the specification described below in [Key naming requirements](#key-naming-requirements). - -### Record structure and naming convention - -To support sparse repodata processing and maintain compatibility with conda's existing infrastructure, wheel records MUST use conda-style naming conventions rather than wheel filenames as keys. This allows solvers to efficiently filter packages by name without parsing the full package record. +This CEP proposes the addition of a new `packages.whl` section to account for the wheel format. This key points to a mapping that MUST contain [repodata record][repodata-record-schema] objects. + +### `packages.whl` dictionary structure + +The `packages.whl` dictionary maps conda-like filenames to repodata records. The key MUST follow the format specified in [Key naming requirements](#key-naming-requirements). The value MUST be a repodata record object that conforms to the [repodata record schema][repodata-record-schema] with the following field specifications: + +- **`name`**: Taken from the wheel's METADATA `Name` field, normalized per [CEP 26][cep-26] and any parent channel name mappings. +- **`version`**: Taken from the wheel's METADATA `Version` field, normalized per PEP 440. +- **`build`**: Format `py{PY_MAJOR_VERSION}_{build_number}` (e.g., `py3_0`), conforming to CEP 26 build string conventions. +- **`build_number`**: As in regular conda packages. MUST be 0 initially; MAY be incremented for rebuilds. +- **`depends`**: Array including: + - `python` dependency from `Requires-Python` (if present), converted to conda format + - All `Requires-Dist` entries from METADATA, converted from PEP 440 to conda format per [Dependency conversion](#dependency-conversion) + - Package names normalized to conda-style names per [CEP 26][cep-26] +- **`constrains`**: Contains `!=` version specifiers from PEP 440 (not included in `depends`). +- **`fn`**: The wheel filename (e.g., `package-1.0.0-py3-none-any.whl`). +- **`subdir`**: MUST be `"noarch"`. +- **`noarch`**: MUST be `"python"`. +- **`artifact_url`**: MAY be present. See [Wheel download URLs](#wheel-download-urls) for semantics. +- **`sha256`**, **`size`**, **`timestamp`**: Standard repodata fields for the wheel file. +- **`record_version`**: MUST be present (currently 3). ### Key naming requirements From 44831ce6a52b9e585d0b1840652eb4280fb320d0 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Wed, 7 Jan 2026 22:31:00 -0500 Subject: [PATCH 20/38] Move content to a rationale section --- cep-XXXX.md | 35 +++++++++++++++++++---------------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 12663577..a95979c0 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -92,22 +92,6 @@ Examples: - `typing_extensions-4.15.0-py3_0-none-any` (METADATA name `typing-extensions` mapped to conda name `typing_extensions`) - `requests-2.32.5-py3_1-none-any` (rebuild with build_number 1) -### Why wheel names may differ from conda names - -Several factors can cause wheel names to differ from conda-style names: - -1. **Wheel filename normalization:** Wheel filenames normalize hyphens to underscores per PEP 427, but the METADATA `Name` field contains the canonical name (which may have hyphens). - - Example: `lazy-loader` (METADATA name) vs `lazy_loader` (wheel filename) - -2. **Python-specific clarification:** PyPI packages are implicitly Python libraries - - Example: `authzed-py` (conda-forge) vs `authzed` (PyPI) - -3. **Variant differences:** Conda may offer multiple variants with different dependencies - - Example: `seaborn-base` (conda-forge) vs `seaborn` (PyPI) - -4. **Cross-channel naming:** Different conda channels may use different names - - Example: `pyperformance` (conda-forge) vs `performance` (main) - ### Naming standard and channel mapping The key name for wheel records SHALL be constructed by combining the conda-style name (derived from METADATA and normalized per conda conventions) with the version, build string, and wheel tags (abi_tag-platform_tag) extracted from the filename. The python_tag is omitted as it is redundant with the build string. This ensures the name portion follows conda naming conventions, supports multiple rebuilds, and the tags portion aligns with the wheel filename structure. @@ -231,6 +215,24 @@ This CEP has the following known limitations: 4. **Repodata size:** Supporting a significant portion of pure Python packages from PyPI (potentially hundreds of thousands of packages with multiple versions each) will substantially increase repodata size. Channels adopting wheel support at scale will need to implement sharded repodata ([CEP 16][cep-16]) to maintain acceptable performance. Alternatively, channels may choose to curate a subset of popular or requested packages rather than mirroring all of PyPI. 5. **PyPI package deletion:** PyPI allows package maintainers to delete packages (as opposed to just yanking them), which can break locked environments that reference those packages. Channels using external PyPI URLs directly are subject to this risk. For production use and reproducible environments, channels MAY mirror and store wheel artifacts locally rather than relying solely on external PyPI URLs. +## Rationale + +### Why wheel names may differ from conda names + +Several factors can cause wheel names to differ from conda-style names: + +1. **Wheel filename normalization:** Wheel filenames normalize hyphens to underscores per PEP 427, but the METADATA `Name` field contains the canonical name (which may have hyphens). + - Example: `lazy-loader` (METADATA name) vs `lazy_loader` (wheel filename) + +2. **Python-specific clarification:** PyPI packages are implicitly Python libraries + - Example: `authzed-py` (conda-forge) vs `authzed` (PyPI) + +3. **Variant differences:** Conda may offer multiple variants with different dependencies + - Example: `seaborn-base` (conda-forge) vs `seaborn` (PyPI) + +4. **Cross-channel naming:** Different conda channels may use different names + - Example: `pyperformance` (conda-forge) vs `performance` (main) + ## Implementation Notes ### For conda clients @@ -247,6 +249,7 @@ Clients implementing this CEP SHOULD: Channel operators adding wheel support SHOULD: - Implement validation to ensure only pure Python wheels are included +- Ensure that dependencies are solvable, including that compiled dependencies exist on the parent conda channel - Maintain a mapping of PyPI to conda-style names for their channel, or reference a parent channel that provides these mappings (see [Naming standard and channel mapping](#naming-standard-and-channel-mapping)) - Consider automation to keep the repodata up to date with newer releases on PyPI - Document any naming conventions specific to their channel From ec31b64f82dcf5a7f5c4cb4e5be4b02f354d1ac3 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Fri, 9 Jan 2026 13:10:37 -0500 Subject: [PATCH 21/38] Update build string format --- cep-XXXX.md | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index a95979c0..ee46acc3 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -63,7 +63,7 @@ The `packages.whl` dictionary maps conda-like filenames to repodata records. The - **`name`**: Taken from the wheel's METADATA `Name` field, normalized per [CEP 26][cep-26] and any parent channel name mappings. - **`version`**: Taken from the wheel's METADATA `Version` field, normalized per PEP 440. -- **`build`**: Format `py{PY_MAJOR_VERSION}_{build_number}` (e.g., `py3_0`), conforming to CEP 26 build string conventions. +- **`build`**: Format `py{PY_MAJOR_VERSION}_{abi_tag}_{platform_tag}_{build_number}` (e.g., `py3_none_any_0`), conforming to CEP 26 build string conventions and the repodata record schema pattern `^([a-z0-9_.]+_)?[0-9]+$`. The build number MUST be at the end of the build string. The `{abi_tag}` and `{platform_tag}` are extracted from the wheel filename. - **`build_number`**: As in regular conda packages. MUST be 0 initially; MAY be incremented for rebuilds. - **`depends`**: Array including: - `python` dependency from `Requires-Python` (if present), converted to conda format @@ -79,22 +79,22 @@ The `packages.whl` dictionary maps conda-like filenames to repodata records. The ### Key naming requirements -The key for each entry in `packages.whl` MUST follow the format `{name}-{version}-{build}__{abi_tag}__{platform_tag}`, where: +The key for each entry in `packages.whl` MUST follow the standard conda distribution string format per [CEP 26][cep-26]: `{name}-{version}-{build string}`, where: - `{name}` is derived from the wheel's METADATA file (the `Name` field), normalized according to conda naming conventions per [CEP 26][cep-26] and any name mappings inherited from a parent channel (see [Naming standard and channel mapping](#naming-standard-and-channel-mapping)) - `{version}` is the package version from METADATA -- `{build}` is the build string (e.g., `py3_0`, `py3_1`) from the `build` field -- `{abi_tag}-{platform_tag}` are extracted from the wheel filename (stored in the `fn` field) +- `{build}` is the build string (e.g., `py3_none_any_0`, `py3_none_any_1`) from the `build` field, which includes the Python version, ABI tag, platform tag, and build number (with build number at the end) Examples: -- `httpx-0.28.1-py3_0-none-any` -- `typing_extensions-4.15.0-py3_0-none-any` (METADATA name `typing-extensions` mapped to conda name `typing_extensions`) -- `requests-2.32.5-py3_1-none-any` (rebuild with build_number 1) +- `httpx-0.28.1-py3_none_any_0` +- `typing_extensions-4.15.0-py3_none_any_0` (METADATA name `typing-extensions` mapped to conda name `typing_extensions`) +- `requests-2.32.5-py3_none_any_1` (rebuild with build_number 1) ### Naming standard and channel mapping -The key name for wheel records SHALL be constructed by combining the conda-style name (derived from METADATA and normalized per conda conventions) with the version, build string, and wheel tags (abi_tag-platform_tag) extracted from the filename. The python_tag is omitted as it is redundant with the build string. This ensures the name portion follows conda naming conventions, supports multiple rebuilds, and the tags portion aligns with the wheel filename structure. +The key name for wheel records SHALL follow the standard conda distribution string format `{name}-{version}-{build}` per [CEP 26][cep-26]. The build string includes the Python version, ABI tag, platform tag, and build number (e.g., `py3_none_any_0`), where the build number is at the end per the repodata record schema pattern. +The ABI and platform tags are extracted from the wheel filename. The python_tag from the wheel filename is omitted as it is redundant with the Python version in the build string. This ensures compatibility with standard conda parsing while preserving wheel tag information within the build string. When there are naming differences between PyPI wheels and conda packages, channel operators MUST determine the appropriate conda-style name by applying conda naming conventions per [CEP 26 - Identifying Packages and Channels in the conda Ecosystem][cep-26]. @@ -127,8 +127,8 @@ This approach allows packages to be served from: When populating repodata records for pure Python wheels: -- `build`: MUST be py`PY_MAJOR_VERSION`_`build_number` (e.g. py3_0) -- `build_number`: MUST be 0 for the initial addition of a wheel version. MAY be incremented for subsequent rebuilds of the same - wheel version (e.g., to correct dependencies or metadata) +- `build`: MUST be py`PY_MAJOR_VERSION`_`abi_tag`_`platform_tag`_`build_number` (e.g. `py3_none_any_0`), where `{abi_tag}` and `{platform_tag}` are extracted from the wheel filename, and the build number MUST be at the end of the build string per the repodata record schema pattern +- `build_number`: MUST be 0 for the initial addition of a wheel version. MAY be incremented for subsequent rebuilds of the same wheel version (e.g., to correct dependencies or metadata) - `fn`: MUST be the wheel filename (e.g., package-1.0.0-py3-none-any.whl) - `subdir`: MUST be "noarch" - `noarch`: MUST be "python" @@ -276,11 +276,11 @@ Below represents the default behavior and shows when the `artifact_url` field is ```json { "packages.whl": { - "requests-2.32.5-py3_0-none-any": { + "requests-2.32.5-py3_none_any_0": { "record_version": 3, "name": "requests", "version": "2.32.5", - "build": "py3_0", + "build": "py3_none_any_0", "build_number": 0, "depends": [ "charset-normalizer <4,>=2", @@ -318,11 +318,11 @@ The `artifact_url` can also be relative as described above. Here's an example of "parent_channel": "https://conda.anaconda.org/conda-forge" }, "packages.whl": { - "requests-2.32.5-py3_0-none-any": { + "requests-2.32.5-py3_none_any_0": { "record_version": 3, "name": "requests", "version": "2.32.5", - "build": "py3_0", + "build": "py3_none_any_0", "build_number": 0, "depends": [ "charset-normalizer <4,>=2", @@ -355,11 +355,11 @@ The following shows an example of using an external location to download the whe ```json { "packages.whl": { - "requests-2.32.5-py3_0-none-any": { + "requests-2.32.5-py3_none_any_0": { "record_version": 3, "name": "requests", "version": "2.32.5", - "build": "py3_0", + "build": "py3_none_any_0", "build_number": 0, "depends": [ "charset-normalizer <4,>=2", @@ -388,11 +388,11 @@ Here is an example of name mapping and normalization of the record name and depe ```json { "packages.whl": { - "annotated_types-0.7.0-py3_0-none-any": { + "annotated_types-0.7.0-py3_none_any_0": { "record_version": 3, "name": "annotated-types", "version": "0.7.0", - "build": "py3_0", + "build": "py3_none_any_0", "build_number": 0, "depends": [ "typing_extensions >=4.0.0; if python < 3.9", @@ -413,8 +413,8 @@ Here is an example of name mapping and normalization of the record name and depe This example demonstrates two types of name normalization: -1. Record key format: The package is indexed using the format `{conda_name}-{version}-{build}-{abi_tag}-{platform_tag}`: `annotated_types-0.7.0-py3_0-none-any`. The name portion (`annotated_types`) comes from the METADATA `Name` field (`annotated-types`), normalized to conda conventions (mapped to `annotated_types` to match conda-forge naming). -The build portion (`py3_0`) comes from the `build` field. The tags portion (`none-any`) is extracted from the wheel filename (`annotated_types-0.7.0-py3-none-any.whl`), which normalizes the package name to underscores per PEP 427. +1. Record key format: The package is indexed using the standard conda distribution string format `{conda_name}-{version}-{build}` per [CEP 26][cep-26]: `annotated_types-0.7.0-py3_none_any_0`. The name portion (`annotated_types`) comes from the METADATA `Name` field (`annotated-types`), normalized to conda conventions (mapped to `annotated_types` to match conda-forge naming). +The build string (`py3_none_any_0`) includes the Python version, ABI tag (`none`), platform tag (`any`), and build number (`0` at the end), all extracted from the wheel filename (`annotated_types-0.7.0-py3-none-any.whl`), which normalizes the package name to underscores per PEP 427. 2. Dependency name mapping: This package depends on `typing_extensions`, which is listed in the `depends` field. On PyPI, this package is named `typing-extensions` (with a hyphen), but it has been mapped to the name `typing_extensions` (with an underscore) to match the existing conda-forge package name. Such mappings may be inherited from a parent channel (see [Naming standard and channel mapping](#naming-standard-and-channel-mapping)). This example also demonstrates conditional dependencies. The original `METADATA` file from the wheel has the following dependency information: From a330d9d687f76c2b1e0f6969f892003cc552f81a Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Fri, 9 Jan 2026 13:13:26 -0500 Subject: [PATCH 22/38] Rename artifact_url to url --- cep-XXXX.md | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index ee46acc3..09944166 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -73,7 +73,7 @@ The `packages.whl` dictionary maps conda-like filenames to repodata records. The - **`fn`**: The wheel filename (e.g., `package-1.0.0-py3-none-any.whl`). - **`subdir`**: MUST be `"noarch"`. - **`noarch`**: MUST be `"python"`. -- **`artifact_url`**: MAY be present. See [Wheel download URLs](#wheel-download-urls) for semantics. +- **`url`**: MAY be present. See [Wheel download URLs](#wheel-download-urls) for semantics. - **`sha256`**, **`size`**, **`timestamp`**: Standard repodata fields for the wheel file. - **`record_version`**: MUST be present (currently 3). @@ -105,22 +105,22 @@ Channel operators SHOULD document any naming conventions and mappings specific t ### Wheel download URLs -This CEP introduces a new optional `artifact_url` field in package records to specify download locations for individual packages. +This CEP introduces a new optional `url` field in package records to specify download locations for individual packages. -> Note for this draft: The `artifact_url` field could also be added as a separate CEP to allow it for other record types. +> Note for this draft: The `url` field could also be added as a separate CEP to allow it for other record types. -When present, the `artifact_url` field SHALL follow these semantics: +When present, the `url` field SHALL follow these semantics: -- If `artifact_url` is an absolute URL, use it as is. -- If `artifact_url` is a relative URL, append it to the `base_url`. +- If `url` is an absolute URL, use it as is. +- If `url` is a relative URL, append it to the `base_url`. When not present (`null`), the download location is constructed from `base_url` and `fn` (existing behavior). This approach allows packages to be served from: -- A shared relative or absolute `base_url` with all wheels in the same directory, by populating the `base_url` field and leaving the `artifact_url` field empty. -- A manual PyPI repository with wheels in directories by the package name by populating the absolute URL in the `artifact_url` field, or the `base_url` and a relative path in the `artifact_url` field. -- External PyPI mirrors or CDNs using absolute URLs by populating the `artifact_url` field, for example to +- A shared relative or absolute `base_url` with all wheels in the same directory, by populating the `base_url` field and leaving the `url` field empty. +- A manual PyPI repository with wheels in directories by the package name by populating the absolute URL in the `url` field, or the `base_url` and a relative path in the `url` field. +- External PyPI mirrors or CDNs using absolute URLs by populating the `url` field, for example to - Mixed sources within the same repodata file ### Wheel-Specific Record Values @@ -132,7 +132,7 @@ When populating repodata records for pure Python wheels: - `fn`: MUST be the wheel filename (e.g., package-1.0.0-py3-none-any.whl) - `subdir`: MUST be "noarch" - `noarch`: MUST be "python" -- `artifact_url`: MAY be present and follow the semantics described above +- `url`: MAY be present and follow the semantics described above ### Pure Python wheel validation @@ -271,7 +271,7 @@ A phased approach starting with manual curation and moving toward increased auto ### Download wheels from the default location -Below represents the default behavior and shows when the `artifact_url` field is not set: +Below represents the default behavior and shows when the `url` field is not set: ```json { @@ -296,7 +296,7 @@ Below represents the default behavior and shows when the `artifact_url` field is "subdir": "noarch", "timestamp": 1764005009, "noarch": "python", - "artifact_url": null + "url": null } } } @@ -308,7 +308,7 @@ With this configuration, the wheel file will be downloaded from the following lo ### Downloading wheels from a relative location with `base_url` -The `artifact_url` can also be relative as described above. Here's an example of what that looks like combined with setting the `base_url` property at the top level, and also showing how a channel can reference a parent channel: +The `url` can also be relative as described above. Here's an example of what that looks like combined with setting the `base_url` property at the top level, and also showing how a channel can reference a parent channel: ```json { @@ -338,7 +338,7 @@ The `artifact_url` can also be relative as described above. Here's an example of "subdir": "noarch", "timestamp": 1764005009, "noarch": "python", - "artifact_url": "requests/requests-2.32.5-py3-none-any.whl" + "url": "requests/requests-2.32.5-py3-none-any.whl" } } } @@ -375,7 +375,7 @@ The following shows an example of using an external location to download the whe "subdir": "noarch", "timestamp": 1764005009, "noarch": "python", - "artifact_url": "https://files.pythonhosted.org/packages/1e/db/4254e3eabe8020b458f1a747140d32277ec7a271daf1d235b70dc0b4e6e3/requests-2.32.5-py3-none-any.whl" + "url": "https://files.pythonhosted.org/packages/1e/db/4254e3eabe8020b458f1a747140d32277ec7a271daf1d235b70dc0b4e6e3/requests-2.32.5-py3-none-any.whl" } } } @@ -405,7 +405,7 @@ Here is an example of name mapping and normalization of the record name and depe "subdir": "noarch", "timestamp": 1756405206, "noarch": "python", - "artifact_url": "https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl" + "url": "https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl" } } } From 6bff9df4dffb2e435d1cee27afdc1ee42a50b7c9 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Mon, 2 Feb 2026 15:43:40 -0500 Subject: [PATCH 23/38] Remove unnecessary fn field --- cep-XXXX.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 09944166..ce3ee8fb 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -8,7 +8,7 @@ Travis Hathaway <travis.j.hathaway@gmail.com> Created Dec 23, 2025 - Updated Dec 23, 2025 + Updated Feb 2, 2026 Discussion https://github.com/conda/ceps/pull/145 Implementation TBD Requires N/A @@ -70,7 +70,6 @@ The `packages.whl` dictionary maps conda-like filenames to repodata records. The - All `Requires-Dist` entries from METADATA, converted from PEP 440 to conda format per [Dependency conversion](#dependency-conversion) - Package names normalized to conda-style names per [CEP 26][cep-26] - **`constrains`**: Contains `!=` version specifiers from PEP 440 (not included in `depends`). -- **`fn`**: The wheel filename (e.g., `package-1.0.0-py3-none-any.whl`). - **`subdir`**: MUST be `"noarch"`. - **`noarch`**: MUST be `"python"`. - **`url`**: MAY be present. See [Wheel download URLs](#wheel-download-urls) for semantics. @@ -129,7 +128,6 @@ When populating repodata records for pure Python wheels: - `build`: MUST be py`PY_MAJOR_VERSION`_`abi_tag`_`platform_tag`_`build_number` (e.g. `py3_none_any_0`), where `{abi_tag}` and `{platform_tag}` are extracted from the wheel filename, and the build number MUST be at the end of the build string per the repodata record schema pattern - `build_number`: MUST be 0 for the initial addition of a wheel version. MAY be incremented for subsequent rebuilds of the same wheel version (e.g., to correct dependencies or metadata) -- `fn`: MUST be the wheel filename (e.g., package-1.0.0-py3-none-any.whl) - `subdir`: MUST be "noarch" - `noarch`: MUST be "python" - `url`: MAY be present and follow the semantics described above @@ -290,7 +288,6 @@ Below represents the default behavior and shows when the `url` field is not set: "python >=3.9" ], "constrains": [], - "fn": "requests-2.32.5-py3-none-any.whl", "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", "size": 6899, "subdir": "noarch", @@ -332,7 +329,6 @@ The `url` can also be relative as described above. Here's an example of what tha "python >=3.9" ], "constrains": [], - "fn": "requests-2.32.5-py3-none-any.whl", "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", "size": 6899, "subdir": "noarch", @@ -369,7 +365,6 @@ The following shows an example of using an external location to download the whe "python >=3.9" ], "constrains": [], - "fn": "requests-2.32.5-py3-none-any.whl", "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", "size": 6899, "subdir": "noarch", @@ -399,7 +394,6 @@ Here is an example of name mapping and normalization of the record name and depe "python >=3.8" ], "constrains": [], - "fn": "annotated_types-0.7.0-py3-none-any.whl", "sha256": "1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53", "size": 13643, "subdir": "noarch", From fb64b68724b71434d7ff3a830e0c0ecaae81bc54 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Mon, 2 Feb 2026 15:44:25 -0500 Subject: [PATCH 24/38] Reference separate URL field CEP draft --- cep-XXXX.md | 20 +++----------------- 1 file changed, 3 insertions(+), 17 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index ce3ee8fb..ba754ea1 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -11,7 +11,7 @@ Updated Feb 2, 2026 Discussion https://github.com/conda/ceps/pull/145 Implementation TBD - Requires N/A + Requires https://github.com/conda/ceps/pull/151 https://github.com/conda/ceps/pull/146 > The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", @@ -104,23 +104,9 @@ Channel operators SHOULD document any naming conventions and mappings specific t ### Wheel download URLs -This CEP introduces a new optional `url` field in package records to specify download locations for individual packages. +This CEP depends on the `url` field for package records ([CEP XXXX](https://github.com/conda/ceps/pull/151)), which enables custom download locations. This is essential for wheel support since PyPI wheels are commonly hosted in per-package subdirectories or served from CDNs. -> Note for this draft: The `url` field could also be added as a separate CEP to allow it for other record types. - -When present, the `url` field SHALL follow these semantics: - -- If `url` is an absolute URL, use it as is. -- If `url` is a relative URL, append it to the `base_url`. - -When not present (`null`), the download location is constructed from `base_url` and `fn` (existing behavior). - -This approach allows packages to be served from: - -- A shared relative or absolute `base_url` with all wheels in the same directory, by populating the `base_url` field and leaving the `url` field empty. -- A manual PyPI repository with wheels in directories by the package name by populating the absolute URL in the `url` field, or the `base_url` and a relative path in the `url` field. -- External PyPI mirrors or CDNs using absolute URLs by populating the `url` field, for example to -- Mixed sources within the same repodata file +Both the `packages.whl` field and the `url` field will be introduced following the backwards-compatible update strategy ([CEP XXXX](https://github.com/conda/ceps/pull/146)). ### Wheel-Specific Record Values From bcece12a8ab3439bbc0fb0276dc73c59533fb15d Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Thu, 16 Apr 2026 09:27:46 -0400 Subject: [PATCH 25/38] Use backwards compatible repodata update strategy --- cep-XXXX.md | 205 +++++++++++++++++++++++++++------------------------- 1 file changed, 108 insertions(+), 97 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index ba754ea1..a0874f0d 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -8,7 +8,7 @@ Travis Hathaway <travis.j.hathaway@gmail.com> Created Dec 23, 2025 - Updated Feb 2, 2026 + Updated Apr 16, 2026 Discussion https://github.com/conda/ceps/pull/145 Implementation TBD Requires https://github.com/conda/ceps/pull/151 https://github.com/conda/ceps/pull/146 @@ -44,10 +44,9 @@ Note that this CEP does not eliminate the need for conda recipes entirely. Many ## Specification -### Add a new top-level key `packages.whl` to list wheels in a channel +### Expose wheels via `whl` inside a repodata revision (`v{revision}`) -According to the current draft schema for [repodata.json][repodata-schema], repodata consists of five top -level keys: +According to the current draft schema for [repodata.json][repodata-schema], the repodata object is traditionally organized around these top-level keys (the surface unchanged clients rely on): - repodata_version - info @@ -55,11 +54,11 @@ level keys: - packages.conda - removed -This CEP proposes the addition of a new `packages.whl` section to account for the wheel format. This key points to a mapping that MUST contain [repodata record][repodata-record-schema] objects. +Wheel support MUST follow [A backwards-compatible repodata update strategy](https://github.com/conda/ceps/pull/146): publishers SHALL register the revision in `info.repodata_revisions` and SHALL place the wheel index under the matching top-level `v{revision}` dictionary (where `revision` is the integer listed for that entry). This CEP specifies a `whl` member inside that `v{revision}` object. The value MUST be a mapping whose entries are [repodata record][repodata-record-schema] objects. -### `packages.whl` dictionary structure +### `whl` dictionary structure -The `packages.whl` dictionary maps conda-like filenames to repodata records. The key MUST follow the format specified in [Key naming requirements](#key-naming-requirements). The value MUST be a repodata record object that conforms to the [repodata record schema][repodata-record-schema] with the following field specifications: +The `whl` dictionary maps conda-like filenames to repodata records. The key MUST follow the format specified in [Key naming requirements](#key-naming-requirements). The value MUST be a repodata record object that conforms to the [repodata record schema][repodata-record-schema] with the following field specifications: - **`name`**: Taken from the wheel's METADATA `Name` field, normalized per [CEP 26][cep-26] and any parent channel name mappings. - **`version`**: Taken from the wheel's METADATA `Version` field, normalized per PEP 440. @@ -74,11 +73,10 @@ The `packages.whl` dictionary maps conda-like filenames to repodata records. The - **`noarch`**: MUST be `"python"`. - **`url`**: MAY be present. See [Wheel download URLs](#wheel-download-urls) for semantics. - **`sha256`**, **`size`**, **`timestamp`**: Standard repodata fields for the wheel file. -- **`record_version`**: MUST be present (currently 3). ### Key naming requirements -The key for each entry in `packages.whl` MUST follow the standard conda distribution string format per [CEP 26][cep-26]: `{name}-{version}-{build string}`, where: +The key for each entry in `whl` MUST follow the standard conda distribution string format per [CEP 26][cep-26]: `{name}-{version}-{build string}`, where: - `{name}` is derived from the wheel's METADATA file (the `Name` field), normalized according to conda naming conventions per [CEP 26][cep-26] and any name mappings inherited from a parent channel (see [Naming standard and channel mapping](#naming-standard-and-channel-mapping)) - `{version}` is the package version from METADATA @@ -106,7 +104,8 @@ Channel operators SHOULD document any naming conventions and mappings specific t This CEP depends on the `url` field for package records ([CEP XXXX](https://github.com/conda/ceps/pull/151)), which enables custom download locations. This is essential for wheel support since PyPI wheels are commonly hosted in per-package subdirectories or served from CDNs. -Both the `packages.whl` field and the `url` field will be introduced following the backwards-compatible update strategy ([CEP XXXX](https://github.com/conda/ceps/pull/146)). +The `whl` mapping (inside the `v{revision}` payload) and the per-record `url` field SHALL follow +the backwards-compatible update strategy ([CEP XXXX](https://github.com/conda/ceps/pull/146)). ### Wheel-Specific Record Values @@ -120,14 +119,14 @@ When populating repodata records for pure Python wheels: ### Pure Python wheel validation -Before adding a wheel to `packages.whl`, channel operators MUST verify: +Before adding a wheel to `whl`, channel operators MUST verify: - The wheel's platform tag is `any` (e.g., `py3-none-any`, `py2.py3-none-any`) - The wheel's ABI tag is `none` - The wheel contains no compiled extensions (`.so`, `.pyd`, `.dylib` files) - The wheel's `METADATA` file is present and valid -Wheels that fail any of these checks MUST NOT be added to `packages.whl`. +Wheels that fail any of these checks MUST NOT be added to `whl`. ### Dependency conversion @@ -179,7 +178,7 @@ Solvers MUST treat pure Python wheels as valid package candidates during depende - **Exclusivity:** Solvers MUST NOT install both a conda package and wheel for the same package name. - **Dependency satisfaction:** When a wheel is selected, its `depends` list MUST be satisfied like any conda package. Dependencies in the `depends` list MAY be satisfied by either wheels or conda packages. -- **Platform matching:** Since all wheels in `packages.whl` are pure Python (noarch), no platform filtering is needed. +- **Platform matching:** Since all wheels in `whl` are pure Python (noarch), no platform filtering is needed. #### User control of precedence @@ -223,7 +222,8 @@ Several factors can cause wheel names to differ from conda-style names: Clients implementing this CEP SHOULD: -- Parse the new `packages.whl` section alongside existing package sections +- Parse the `whl` mapping inside each supported `v{revision}` object alongside existing `packages` / + `packages.conda` sections - Apply the same filtering and preference logic used for conda packages - Extract wheel metadata during solving to populate dependency information - Provide the ability to natively install wheels or on-the-fly convert wheels to conda packages for installation @@ -253,35 +253,41 @@ A phased approach starting with manual curation and moving toward increased auto ## Examples +The JSON fragments below use revision `3` as an example (`v3`); the integer MUST match an entry in +`info.repodata_revisions` per [the backwards-compatible repodata update strategy](https://github.com/conda/ceps/pull/146). +A complete channel index also includes the traditional top-level keys (`repodata_version`, `packages`, +`packages.conda`, `removed`, and so on). A full generated example is checked in with [conda-pypi][conda-pypi-example-repodata]. + ### Download wheels from the default location Below represents the default behavior and shows when the `url` field is not set: ```json { - "packages.whl": { - "requests-2.32.5-py3_none_any_0": { - "record_version": 3, - "name": "requests", - "version": "2.32.5", - "build": "py3_none_any_0", - "build_number": 0, - "depends": [ - "charset-normalizer <4,>=2", - "idna <4,>=2.5", - "urllib3 <3,>=1.21.1", - "certifi >=2017.4.17", - "python >=3.9" - ], - "constrains": [], - "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", - "size": 6899, - "subdir": "noarch", - "timestamp": 1764005009, - "noarch": "python", - "url": null + "v3": { + "whl": { + "requests-2.32.5-py3_none_any_0": { + "name": "requests", + "version": "2.32.5", + "build": "py3_none_any_0", + "build_number": 0, + "depends": [ + "charset-normalizer <4,>=2", + "idna <4,>=2.5", + "urllib3 <3,>=1.21.1", + "certifi >=2017.4.17", + "python >=3.9" + ], + "constrains": [], + "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", + "size": 6899, + "subdir": "noarch", + "timestamp": 1764005009, + "noarch": "python", + "url": null + } } - } + } } ``` @@ -300,27 +306,28 @@ The `url` can also be relative as described above. Here's an example of what tha "base_url": "https://repo.example.com/channel/", "parent_channel": "https://conda.anaconda.org/conda-forge" }, - "packages.whl": { - "requests-2.32.5-py3_none_any_0": { - "record_version": 3, - "name": "requests", - "version": "2.32.5", - "build": "py3_none_any_0", - "build_number": 0, - "depends": [ - "charset-normalizer <4,>=2", - "idna <4,>=2.5", - "urllib3 <3,>=1.21.1", - "certifi >=2017.4.17", - "python >=3.9" - ], - "constrains": [], - "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", - "size": 6899, - "subdir": "noarch", - "timestamp": 1764005009, - "noarch": "python", - "url": "requests/requests-2.32.5-py3-none-any.whl" + "v3": { + "whl": { + "requests-2.32.5-py3_none_any_0": { + "name": "requests", + "version": "2.32.5", + "build": "py3_none_any_0", + "build_number": 0, + "depends": [ + "charset-normalizer <4,>=2", + "idna <4,>=2.5", + "urllib3 <3,>=1.21.1", + "certifi >=2017.4.17", + "python >=3.9" + ], + "constrains": [], + "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", + "size": 6899, + "subdir": "noarch", + "timestamp": 1764005009, + "noarch": "python", + "url": "requests/requests-2.32.5-py3-none-any.whl" + } } } } @@ -336,27 +343,28 @@ The following shows an example of using an external location to download the whe ```json { - "packages.whl": { - "requests-2.32.5-py3_none_any_0": { - "record_version": 3, - "name": "requests", - "version": "2.32.5", - "build": "py3_none_any_0", - "build_number": 0, - "depends": [ - "charset-normalizer <4,>=2", - "idna <4,>=2.5", - "urllib3 <3,>=1.21.1", - "certifi >=2017.4.17", - "python >=3.9" - ], - "constrains": [], - "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", - "size": 6899, - "subdir": "noarch", - "timestamp": 1764005009, - "noarch": "python", - "url": "https://files.pythonhosted.org/packages/1e/db/4254e3eabe8020b458f1a747140d32277ec7a271daf1d235b70dc0b4e6e3/requests-2.32.5-py3-none-any.whl" + "v3": { + "whl": { + "requests-2.32.5-py3_none_any_0": { + "name": "requests", + "version": "2.32.5", + "build": "py3_none_any_0", + "build_number": 0, + "depends": [ + "charset-normalizer <4,>=2", + "idna <4,>=2.5", + "urllib3 <3,>=1.21.1", + "certifi >=2017.4.17", + "python >=3.9" + ], + "constrains": [], + "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", + "size": 6899, + "subdir": "noarch", + "timestamp": 1764005009, + "noarch": "python", + "url": "https://files.pythonhosted.org/packages/1e/db/4254e3eabe8020b458f1a747140d32277ec7a271daf1d235b70dc0b4e6e3/requests-2.32.5-py3-none-any.whl" + } } } } @@ -368,24 +376,25 @@ Here is an example of name mapping and normalization of the record name and depe ```json { - "packages.whl": { - "annotated_types-0.7.0-py3_none_any_0": { - "record_version": 3, - "name": "annotated-types", - "version": "0.7.0", - "build": "py3_none_any_0", - "build_number": 0, - "depends": [ - "typing_extensions >=4.0.0; if python < 3.9", - "python >=3.8" - ], - "constrains": [], - "sha256": "1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53", - "size": 13643, - "subdir": "noarch", - "timestamp": 1756405206, - "noarch": "python", - "url": "https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl" + "v3": { + "whl": { + "annotated_types-0.7.0-py3_none_any_0": { + "name": "annotated-types", + "version": "0.7.0", + "build": "py3_none_any_0", + "build_number": 0, + "depends": [ + "typing_extensions >=4.0.0; if python < 3.9", + "python >=3.8" + ], + "constrains": [], + "sha256": "1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53", + "size": 13643, + "subdir": "noarch", + "timestamp": 1756405206, + "noarch": "python", + "url": "https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl" + } } } } @@ -576,13 +585,14 @@ conda-pupa is merged into conda-pypi which adds a `conda pypi install ` ### conda-pypi integrates parts of conda-whl-support (Nov 2025) -Conda-pypi incorporates the wheel detection logic from conda-whl-support, providing the core functionality needed beyond the solver and index changes required to support the `packages.whl` section proposed in this CEP. +Conda-pypi incorporates the wheel detection logic from conda-whl-support, providing the core functionality needed beyond the solver and index changes required to support the `whl` section proposed in this CEP. ## References - [Adopting uv in pixi][uv-in-pixi] - [rip][rip] - [conda-pypi project][conda-pypi] +- [Example `repodata.json` (conda-pypi test channel)][conda-pypi-example-repodata] - [conda-pupa][conda-pupa] ## Copyright @@ -597,6 +607,7 @@ All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdo [cep-26]: https://conda.org/learn/ceps/cep-0026 [version-specifiers]: https://packaging.python.org/en/latest/specifications/version-specifiers/#id5 [conda-pypi]: https://github.com/conda-incubator/conda-pypi +[conda-pypi-example-repodata]: https://github.com/conda-incubator/conda-pypi/blob/main/tests/conda_local_channel/noarch/repodata.json [conda-pupa]: https://github.com/dholth/conda-pupa [uv-in-pixi]: https://prefix.dev/blog/uv_in_pixi [rip]: https://github.com/prefix-dev/rip From f6fb31650dbf4e73500fcd13579518b5b5a6d8eb Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Thu, 16 Apr 2026 09:59:10 -0400 Subject: [PATCH 26/38] Reference the channel relations in repodata CEP draft --- cep-XXXX.md | 41 +++++++++++++++++++++++------------------ 1 file changed, 23 insertions(+), 18 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index a0874f0d..dae4602d 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -11,7 +11,7 @@ Updated Apr 16, 2026 Discussion https://github.com/conda/ceps/pull/145 Implementation TBD - Requires https://github.com/conda/ceps/pull/151 https://github.com/conda/ceps/pull/146 + Requires https://github.com/conda/ceps/pull/151 https://github.com/conda/ceps/pull/146 https://github.com/conda/ceps/pull/155 > The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", @@ -60,10 +60,10 @@ Wheel support MUST follow [A backwards-compatible repodata update strategy](http The `whl` dictionary maps conda-like filenames to repodata records. The key MUST follow the format specified in [Key naming requirements](#key-naming-requirements). The value MUST be a repodata record object that conforms to the [repodata record schema][repodata-record-schema] with the following field specifications: -- **`name`**: Taken from the wheel's METADATA `Name` field, normalized per [CEP 26][cep-26] and any parent channel name mappings. +- **`name`**: Taken from the wheel's METADATA `Name` field, normalized per [CEP 26][cep-26] and any name mappings aligned with the channel declared in `info.channel_relations.base` when present (see [Naming standard and channel mapping](#naming-standard-and-channel-mapping)). - **`version`**: Taken from the wheel's METADATA `Version` field, normalized per PEP 440. - **`build`**: Format `py{PY_MAJOR_VERSION}_{abi_tag}_{platform_tag}_{build_number}` (e.g., `py3_none_any_0`), conforming to CEP 26 build string conventions and the repodata record schema pattern `^([a-z0-9_.]+_)?[0-9]+$`. The build number MUST be at the end of the build string. The `{abi_tag}` and `{platform_tag}` are extracted from the wheel filename. -- **`build_number`**: As in regular conda packages. MUST be 0 initially; MAY be incremented for rebuilds. +- **`build_number`**: As in regular conda packages. MUST be 0 initially. MAY be incremented for rebuilds. - **`depends`**: Array including: - `python` dependency from `Requires-Python` (if present), converted to conda format - All `Requires-Dist` entries from METADATA, converted from PEP 440 to conda format per [Dependency conversion](#dependency-conversion) @@ -78,7 +78,7 @@ The `whl` dictionary maps conda-like filenames to repodata records. The key MUST The key for each entry in `whl` MUST follow the standard conda distribution string format per [CEP 26][cep-26]: `{name}-{version}-{build string}`, where: -- `{name}` is derived from the wheel's METADATA file (the `Name` field), normalized according to conda naming conventions per [CEP 26][cep-26] and any name mappings inherited from a parent channel (see [Naming standard and channel mapping](#naming-standard-and-channel-mapping)) +- `{name}` is derived from the wheel's METADATA file (the `Name` field), normalized according to conda naming conventions per [CEP 26][cep-26] and any name mappings inherited from the base channel when `info.channel_relations.base` is declared (see [Naming standard and channel mapping](#naming-standard-and-channel-mapping)) - `{version}` is the package version from METADATA - `{build}` is the build string (e.g., `py3_none_any_0`, `py3_none_any_1`) from the `build` field, which includes the Python version, ABI tag, platform tag, and build number (with build number at the end) @@ -95,10 +95,11 @@ The ABI and platform tags are extracted from the wheel filename. The python_tag When there are naming differences between PyPI wheels and conda packages, channel operators MUST determine the appropriate conda-style name by applying conda naming conventions per [CEP 26 - Identifying Packages and Channels in the conda Ecosystem][cep-26]. -To help users understand which naming conventions are being used, channels MAY reference a parent channel that defines the naming conventions and mappings for wheel-to-conda name translation. Parent/child channel relationships will be defined in a separate CEP, which will specify how channels can declare a `parent_channel` field in the `info` section of repodata to inherit naming conventions and mappings from another channel. -For example, a wheel channel could reference conda-forge as its parent channel, inheriting conda-forge's established naming conventions and package name mappings. This allows channels to avoid duplicating name mapping definitions and ensures consistency across related channels. +To help users understand which naming conventions are being used, a wheel channel MAY declare a **base** channel using `info.channel_relations.base` as specified in [Channel relations in repodata][cep-channel-relations]. +The base channel is the one clients load with higher priority. It is also the natural source of conda-style naming conventions and PyPI-to-conda name mappings for wheels published alongside that stack (for example, `conda-forge` as the base for a specialized wheel index on the same channel host). +Channel references MUST use relative paths as required by that CEP (for example `"base": "../conda-forge"`). Absolute URLs in `info` are not valid channel references for relations. -Channel operators SHOULD document any naming conventions and mappings specific to their channel, including whether they reference a parent channel for naming conventions. +Channel operators SHOULD document any naming conventions and mappings specific to their channel, including how they relate to their declared `channel_relations`. ### Wheel download URLs @@ -233,8 +234,8 @@ Clients implementing this CEP SHOULD: Channel operators adding wheel support SHOULD: - Implement validation to ensure only pure Python wheels are included -- Ensure that dependencies are solvable, including that compiled dependencies exist on the parent conda channel -- Maintain a mapping of PyPI to conda-style names for their channel, or reference a parent channel that provides these mappings (see [Naming standard and channel mapping](#naming-standard-and-channel-mapping)) +- Ensure that dependencies are solvable, including that compiled dependencies exist on the conda channel declared as `info.channel_relations.base` when wheels depend on conda packages from that stack +- Maintain a mapping of PyPI to conda-style names for their channel, or declare `channel_relations.base` so clients load a base channel that defines those mappings (see [Naming standard and channel mapping](#naming-standard-and-channel-mapping) and [Channel relations in repodata][cep-channel-relations]) - Consider automation to keep the repodata up to date with newer releases on PyPI - Document any naming conventions specific to their channel - For production channels, consider mirroring wheel artifacts locally to ensure reproducibility and protect against PyPI deletions @@ -253,7 +254,7 @@ A phased approach starting with manual curation and moving toward increased auto ## Examples -The JSON fragments below use revision `3` as an example (`v3`); the integer MUST match an entry in +The JSON fragments below use revision `3` as an example (`v3`). The integer MUST match an entry in `info.repodata_revisions` per [the backwards-compatible repodata update strategy](https://github.com/conda/ceps/pull/146). A complete channel index also includes the traditional top-level keys (`repodata_version`, `packages`, `packages.conda`, `removed`, and so on). A full generated example is checked in with [conda-pypi][conda-pypi-example-repodata]. @@ -297,14 +298,16 @@ With this configuration, the wheel file will be downloaded from the following lo ### Downloading wheels from a relative location with `base_url` -The `url` can also be relative as described above. Here's an example of what that looks like combined with setting the `base_url` property at the top level, and also showing how a channel can reference a parent channel: +The `url` can also be relative as described above. Here's an example of what that looks like combined with setting the `base_url` property at the top level, and also showing how a channel can declare a base channel for naming and dependencies using `channel_relations` ([Channel relations in repodata][cep-channel-relations]): ```json { "info": { "subdir": "noarch", - "base_url": "https://repo.example.com/channel/", - "parent_channel": "https://conda.anaconda.org/conda-forge" + "base_url": "https://packages.example.org/wheel-extra/", + "channel_relations": { + "base": "../core" + } }, "v3": { "whl": { @@ -335,7 +338,7 @@ The `url` can also be relative as described above. Here's an example of what tha This would result in the following being fetched by conda clients: -- `https://repo.example.com/channel/noarch/requests/requests-2.32.5-py3-none-any.whl` +- `https://packages.example.org/wheel-extra/noarch/requests/requests-2.32.5-py3-none-any.whl` ### Downloading wheels from external location @@ -404,7 +407,7 @@ This example demonstrates two types of name normalization: 1. Record key format: The package is indexed using the standard conda distribution string format `{conda_name}-{version}-{build}` per [CEP 26][cep-26]: `annotated_types-0.7.0-py3_none_any_0`. The name portion (`annotated_types`) comes from the METADATA `Name` field (`annotated-types`), normalized to conda conventions (mapped to `annotated_types` to match conda-forge naming). The build string (`py3_none_any_0`) includes the Python version, ABI tag (`none`), platform tag (`any`), and build number (`0` at the end), all extracted from the wheel filename (`annotated_types-0.7.0-py3-none-any.whl`), which normalizes the package name to underscores per PEP 427. -2. Dependency name mapping: This package depends on `typing_extensions`, which is listed in the `depends` field. On PyPI, this package is named `typing-extensions` (with a hyphen), but it has been mapped to the name `typing_extensions` (with an underscore) to match the existing conda-forge package name. Such mappings may be inherited from a parent channel (see [Naming standard and channel mapping](#naming-standard-and-channel-mapping)). +2. Dependency name mapping: This package depends on `typing_extensions`, which is listed in the `depends` field. On PyPI, this package is named `typing-extensions` (with a hyphen), but it has been mapped to the name `typing_extensions` (with an underscore) to match the existing conda-forge package name. Such mappings may be aligned with the channel declared in `info.channel_relations.base` (see [Naming standard and channel mapping](#naming-standard-and-channel-mapping)). This example also demonstrates conditional dependencies. The original `METADATA` file from the wheel has the following dependency information: @@ -413,7 +416,7 @@ Requires-Python: >=3.8 Requires-Dist: typing-extensions>=4.0.0; python_version < '3.9' ``` -The package record uses conditional dependency syntax (`; if python < 3.9`) to declare `typing_extensions` only when the Python version is < 3.9, matching the original wheel METADATA. The Python version constraint of >=3.8 is directly mapped. +The package record uses conda-style conditional markers on the dependency so `typing_extensions` applies only when the Python version is less than 3.9, matching the original wheel METADATA. The Python version constraint of >=3.8 is directly mapped. ## Rejected ideas @@ -506,7 +509,7 @@ Another alternative would be establishing a build farm to automatically convert Despite these advantages, this approach was rejected because: - **Infrastructure burden:** Requires significant storage and bandwidth to host and serve converted packages that duplicate PyPI's CDN infrastructure -- **Resource inefficiency:** Wheels are already an excellent format for pure Python packages; conversion adds no technical value and wastes resources +- **Resource inefficiency:** Wheels are already an excellent format for pure Python packages. Conversion adds no technical value and wastes resources Native wheel support provides the same user experience (transparent PyPI access) while avoiding the infrastructure burden and resource inefficiency of conversion. Channel operators who prefer converted packages can continue building conda packages from PyPI sources. @@ -585,7 +588,7 @@ conda-pupa is merged into conda-pypi which adds a `conda pypi install ` ### conda-pypi integrates parts of conda-whl-support (Nov 2025) -Conda-pypi incorporates the wheel detection logic from conda-whl-support, providing the core functionality needed beyond the solver and index changes required to support the `whl` section proposed in this CEP. +Conda-pypi incorporates the wheel detection logic from conda-whl-support, providing the core functionality needed beyond the solver and index changes required to support the `whl` section proposed in this CEP, together with channel relation metadata when channels adopt [Channel relations in repodata][cep-channel-relations]. ## References @@ -593,6 +596,7 @@ Conda-pypi incorporates the wheel detection logic from conda-whl-support, provid - [rip][rip] - [conda-pypi project][conda-pypi] - [Example `repodata.json` (conda-pypi test channel)][conda-pypi-example-repodata] +- [Channel relations in repodata (PR 155)][cep-channel-relations] - [conda-pupa][conda-pupa] ## Copyright @@ -608,6 +612,7 @@ All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdo [version-specifiers]: https://packaging.python.org/en/latest/specifications/version-specifiers/#id5 [conda-pypi]: https://github.com/conda-incubator/conda-pypi [conda-pypi-example-repodata]: https://github.com/conda-incubator/conda-pypi/blob/main/tests/conda_local_channel/noarch/repodata.json +[cep-channel-relations]: https://github.com/conda/ceps/pull/155 [conda-pupa]: https://github.com/dholth/conda-pupa [uv-in-pixi]: https://prefix.dev/blog/uv_in_pixi [rip]: https://github.com/prefix-dev/rip From 46c8faab7fb124f09c4660f311b352f7f1ac6303 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Thu, 16 Apr 2026 11:04:24 -0400 Subject: [PATCH 27/38] Reference conditional and extras CEP --- cep-XXXX.md | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index dae4602d..bb3f402a 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -11,7 +11,7 @@ Updated Apr 16, 2026 Discussion https://github.com/conda/ceps/pull/145 Implementation TBD - Requires https://github.com/conda/ceps/pull/151 https://github.com/conda/ceps/pull/146 https://github.com/conda/ceps/pull/155 + Requires https://github.com/conda/ceps/pull/151 https://github.com/conda/ceps/pull/146 https://github.com/conda/ceps/pull/155 https://github.com/conda/ceps/pull/111 > The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", @@ -46,7 +46,7 @@ Note that this CEP does not eliminate the need for conda recipes entirely. Many ### Expose wheels via `whl` inside a repodata revision (`v{revision}`) -According to the current draft schema for [repodata.json][repodata-schema], the repodata object is traditionally organized around these top-level keys (the surface unchanged clients rely on): +According to the current schema for [repodata.json][repodata-schema], the repodata object is traditionally organized around these top-level keys (the surface unchanged clients rely on): - repodata_version - info @@ -141,7 +141,9 @@ Wheel dependencies (from METADATA file's Requires-Dist entries) MUST be converte - **Multiple specifiers:** Combine with commas (e.g., >=1.0,<2.0) - **Python version requirements:** Convert Requires-Python to explicit python dependency -- **Environment markers:** Ignore markers other than Python version (pure Python assumption) +- **Environment markers:** Map Python-version markers on `Requires-Dist` to conditional `MatchSpec` `when=` as specified in [PR 111][pr-111]. Other markers are out of scope for the default conversion rules in this CEP (see [Limitations](#limitations)). + +For a non-normative, per-variable description of how **conda-pypi** maps PEP 508 markers to `when=` fragments (and related repodata fields), see [PEP 508 marker conversion][conda-pypi-marker-conversion] in the conda-pypi documentation. Example conversion: @@ -162,14 +164,17 @@ Resulting conda record: "python >=3.8", "requests >=2.20.0,<3.0", "click >=7.0", - "numpy >=1.20.0,!=1.24.0" + "numpy >=1.20.0,!=1.24.0", + "importlib-metadata>=1.0[when=\"python<3.8\"]" ] } ``` -### Handling conditional and extra dependencies +### Handling conditional dependencies and extras + +Wheel `depends` entries that encode PEP 508 environment markers MUST use conditional `MatchSpec` syntax with `when=` as specified in [PR 111][pr-111]. -Like in the example above of only requiring `importlib-metadata` for certain Python versions, conditional and extra dependencies MUST be supported to enable full interoperability between the ecosystems. This will be supported through a separate CEP: . +PyPI **extras** (optional dependency groups) MUST follow the `extras` repodata record model in [PR 111][pr-111]. A default install does not activate extras unless the user requests them (for example with `extras=` in `MatchSpec` as described there). ### Solver behavior and package preference @@ -194,8 +199,8 @@ By default, channels with conda packages MUST prefer conda packages when they ar This CEP has the following known limitations: 1. **Pure Python only:** This CEP explicitly does not address wheels with binary extensions, which require platform-specific compatibility guarantees beyond the current scope. Conda’s strength is binary compatibility, so using conda packages may be the optimal solution. -2. **Environment markers**:** Only Python version markers are converted to dependencies. Other environment markers (OS, platform, etc.) are ignored based on the pure Python assumption. -3. **Conditionals and Extras:** Conditional dependencies and extras are specified by a separate CEP. +2. **Environment markers:** This CEP’s default conversion rules focus on Python-version markers mapped to `when=` per [PR 111][pr-111]. Other markers (for example `sys_platform`) are out of scope for those default rules and are not converted here, though some MAY be expressible with `when=` once clients implement [PR 111][pr-111]. +3. **Conditionals and extras:** Normative syntax and record fields for `when=` and optional dependency groups are specified in [PR 111][pr-111], which this wheel CEP depends on for PyPI-aligned conditionals and extras. 4. **Repodata size:** Supporting a significant portion of pure Python packages from PyPI (potentially hundreds of thousands of packages with multiple versions each) will substantially increase repodata size. Channels adopting wheel support at scale will need to implement sharded repodata ([CEP 16][cep-16]) to maintain acceptable performance. Alternatively, channels may choose to curate a subset of popular or requested packages rather than mirroring all of PyPI. 5. **PyPI package deletion:** PyPI allows package maintainers to delete packages (as opposed to just yanking them), which can break locked environments that reference those packages. Channels using external PyPI URLs directly are subject to this risk. For production use and reproducible environments, channels MAY mirror and store wheel artifacts locally rather than relying solely on external PyPI URLs. @@ -227,6 +232,9 @@ Clients implementing this CEP SHOULD: `packages.conda` sections - Apply the same filtering and preference logic used for conda packages - Extract wheel metadata during solving to populate dependency information +- Evaluate `when=` on wheel `depends` entries per [PR 111][pr-111] +- Honor optional dependency groups when the user selects `extras` per [PR 111][pr-111] +- Treat wheel records that use `when=` or `extras` metadata as part of the same `v{revision}` repodata payload as other new-syntax records, per the [backwards-compatible repodata update strategy](https://github.com/conda/ceps/pull/146) - Provide the ability to natively install wheels or on-the-fly convert wheels to conda packages for installation ### For channel operators @@ -387,7 +395,7 @@ Here is an example of name mapping and normalization of the record name and depe "build": "py3_none_any_0", "build_number": 0, "depends": [ - "typing_extensions >=4.0.0; if python < 3.9", + "typing_extensions>=4.0.0[when=\"python<3.9\"]", "python >=3.8" ], "constrains": [], @@ -416,7 +424,7 @@ Requires-Python: >=3.8 Requires-Dist: typing-extensions>=4.0.0; python_version < '3.9' ``` -The package record uses conda-style conditional markers on the dependency so `typing_extensions` applies only when the Python version is less than 3.9, matching the original wheel METADATA. The Python version constraint of >=3.8 is directly mapped. +The package record expresses the conditional with `when=` so `typing_extensions` applies only when the Python version is less than 3.9, matching the original wheel METADATA and [PR 111][pr-111]. The Python version constraint of >=3.8 is directly mapped. ## Rejected ideas @@ -595,8 +603,10 @@ Conda-pypi incorporates the wheel detection logic from conda-whl-support, provid - [Adopting uv in pixi][uv-in-pixi] - [rip][rip] - [conda-pypi project][conda-pypi] +- [PEP 508 marker conversion (conda-pypi developer docs)][conda-pypi-marker-conversion] - [Example `repodata.json` (conda-pypi test channel)][conda-pypi-example-repodata] - [Channel relations in repodata (PR 155)][cep-channel-relations] +- [PR 111 – Conditional dependencies and optional groups][pr-111] - [conda-pupa][conda-pupa] ## Copyright @@ -611,8 +621,10 @@ All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdo [cep-26]: https://conda.org/learn/ceps/cep-0026 [version-specifiers]: https://packaging.python.org/en/latest/specifications/version-specifiers/#id5 [conda-pypi]: https://github.com/conda-incubator/conda-pypi +[conda-pypi-marker-conversion]: https://conda.github.io/conda-pypi/developer/marker-conversion/#pep-508-variables [conda-pypi-example-repodata]: https://github.com/conda-incubator/conda-pypi/blob/main/tests/conda_local_channel/noarch/repodata.json [cep-channel-relations]: https://github.com/conda/ceps/pull/155 +[pr-111]: https://github.com/conda/ceps/pull/111 [conda-pupa]: https://github.com/dholth/conda-pupa [uv-in-pixi]: https://prefix.dev/blog/uv_in_pixi [rip]: https://github.com/prefix-dev/rip From 8dd4cc7105b7056f20074cd310443e28692ca740 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Thu, 16 Apr 2026 11:51:33 -0400 Subject: [PATCH 28/38] Add extra_depends nomenclature --- cep-XXXX.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index bb3f402a..87a88ae2 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -69,6 +69,7 @@ The `whl` dictionary maps conda-like filenames to repodata records. The key MUST - All `Requires-Dist` entries from METADATA, converted from PEP 440 to conda format per [Dependency conversion](#dependency-conversion) - Package names normalized to conda-style names per [CEP 26][cep-26] - **`constrains`**: Contains `!=` version specifiers from PEP 440 (not included in `depends`). +- **`extra_depends`**: MAY be present. When present, MUST be an object mapping extra names to lists of dependency strings for optional groups, per [PR 111][pr-111]. When absent or empty, the record declares no optional groups beyond `depends`. - **`subdir`**: MUST be `"noarch"`. - **`noarch`**: MUST be `"python"`. - **`url`**: MAY be present. See [Wheel download URLs](#wheel-download-urls) for semantics. @@ -143,7 +144,7 @@ Wheel dependencies (from METADATA file's Requires-Dist entries) MUST be converte - **Python version requirements:** Convert Requires-Python to explicit python dependency - **Environment markers:** Map Python-version markers on `Requires-Dist` to conditional `MatchSpec` `when=` as specified in [PR 111][pr-111]. Other markers are out of scope for the default conversion rules in this CEP (see [Limitations](#limitations)). -For a non-normative, per-variable description of how **conda-pypi** maps PEP 508 markers to `when=` fragments (and related repodata fields), see [PEP 508 marker conversion][conda-pypi-marker-conversion] in the conda-pypi documentation. +For a non-normative, per-variable description of how **conda-pypi** maps PEP 508 markers to `when=` fragments and **`extra_depends`** for optional groups, see [PEP 508 marker conversion][conda-pypi-marker-conversion] in the conda-pypi documentation. Example conversion: @@ -174,7 +175,7 @@ Resulting conda record: Wheel `depends` entries that encode PEP 508 environment markers MUST use conditional `MatchSpec` syntax with `when=` as specified in [PR 111][pr-111]. -PyPI **extras** (optional dependency groups) MUST follow the `extras` repodata record model in [PR 111][pr-111]. A default install does not activate extras unless the user requests them (for example with `extras=` in `MatchSpec` as described there). +PyPI **extras** (optional dependency groups) MUST be represented on the wheel repodata record using an **`extra_depends`** object (a mapping from extra name to lists of dependency strings) as specified in [PR 111][pr-111]. A default install uses only `depends` and MUST NOT union in `extra_depends` entries unless the user selects optional groups (for example with `extras=` in `MatchSpec` as described in [PR 111][pr-111]). ### Solver behavior and package preference @@ -200,7 +201,7 @@ This CEP has the following known limitations: 1. **Pure Python only:** This CEP explicitly does not address wheels with binary extensions, which require platform-specific compatibility guarantees beyond the current scope. Conda’s strength is binary compatibility, so using conda packages may be the optimal solution. 2. **Environment markers:** This CEP’s default conversion rules focus on Python-version markers mapped to `when=` per [PR 111][pr-111]. Other markers (for example `sys_platform`) are out of scope for those default rules and are not converted here, though some MAY be expressible with `when=` once clients implement [PR 111][pr-111]. -3. **Conditionals and extras:** Normative syntax and record fields for `when=` and optional dependency groups are specified in [PR 111][pr-111], which this wheel CEP depends on for PyPI-aligned conditionals and extras. +3. **Conditionals and extras:** Normative syntax and record fields for `when=` on `depends` and for optional groups in **`extra_depends`** are specified in [PR 111][pr-111], which this wheel CEP depends on for PyPI-aligned conditionals and extras. 4. **Repodata size:** Supporting a significant portion of pure Python packages from PyPI (potentially hundreds of thousands of packages with multiple versions each) will substantially increase repodata size. Channels adopting wheel support at scale will need to implement sharded repodata ([CEP 16][cep-16]) to maintain acceptable performance. Alternatively, channels may choose to curate a subset of popular or requested packages rather than mirroring all of PyPI. 5. **PyPI package deletion:** PyPI allows package maintainers to delete packages (as opposed to just yanking them), which can break locked environments that reference those packages. Channels using external PyPI URLs directly are subject to this risk. For production use and reproducible environments, channels MAY mirror and store wheel artifacts locally rather than relying solely on external PyPI URLs. @@ -233,8 +234,8 @@ Clients implementing this CEP SHOULD: - Apply the same filtering and preference logic used for conda packages - Extract wheel metadata during solving to populate dependency information - Evaluate `when=` on wheel `depends` entries per [PR 111][pr-111] -- Honor optional dependency groups when the user selects `extras` per [PR 111][pr-111] -- Treat wheel records that use `when=` or `extras` metadata as part of the same `v{revision}` repodata payload as other new-syntax records, per the [backwards-compatible repodata update strategy](https://github.com/conda/ceps/pull/146) +- Honor **`extra_depends`** when the user selects optional groups (for example with `extras=` in `MatchSpec`) per [PR 111][pr-111] +- Treat wheel records that use `when=` on `depends` or **`extra_depends`** metadata as part of the same `v{revision}` repodata payload as other new-syntax records, per the [backwards-compatible repodata update strategy](https://github.com/conda/ceps/pull/146) - Provide the ability to natively install wheels or on-the-fly convert wheels to conda packages for installation ### For channel operators From 549b023241d751cfec578712080734f1ec6291d3 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Thu, 16 Apr 2026 12:08:06 -0400 Subject: [PATCH 29/38] Align fields: add fn, remove constrains, timestamp --- cep-XXXX.md | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 87a88ae2..3048673b 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -68,12 +68,12 @@ The `whl` dictionary maps conda-like filenames to repodata records. The key MUST - `python` dependency from `Requires-Python` (if present), converted to conda format - All `Requires-Dist` entries from METADATA, converted from PEP 440 to conda format per [Dependency conversion](#dependency-conversion) - Package names normalized to conda-style names per [CEP 26][cep-26] -- **`constrains`**: Contains `!=` version specifiers from PEP 440 (not included in `depends`). - **`extra_depends`**: MAY be present. When present, MUST be an object mapping extra names to lists of dependency strings for optional groups, per [PR 111][pr-111]. When absent or empty, the record declares no optional groups beyond `depends`. - **`subdir`**: MUST be `"noarch"`. - **`noarch`**: MUST be `"python"`. - **`url`**: MAY be present. See [Wheel download URLs](#wheel-download-urls) for semantics. -- **`sha256`**, **`size`**, **`timestamp`**: Standard repodata fields for the wheel file. +- **`fn`**: MUST be the wheel filename (`.whl`), as in standard conda repodata records (for example `requests-2.32.5-py3-none-any.whl`). +- **`sha256`**, **`size`**: Standard repodata fields for the wheel file. ### Key naming requirements @@ -117,6 +117,7 @@ When populating repodata records for pure Python wheels: - `build_number`: MUST be 0 for the initial addition of a wheel version. MAY be incremented for subsequent rebuilds of the same wheel version (e.g., to correct dependencies or metadata) - `subdir`: MUST be "noarch" - `noarch`: MUST be "python" +- `fn`: MUST be the wheel filename (`.whl`) - `url`: MAY be present and follow the semantics described above ### Pure Python wheel validation @@ -288,11 +289,10 @@ Below represents the default behavior and shows when the `url` field is not set: "certifi >=2017.4.17", "python >=3.9" ], - "constrains": [], + "fn": "requests-2.32.5-py3-none-any.whl", "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", "size": 6899, "subdir": "noarch", - "timestamp": 1764005009, "noarch": "python", "url": null } @@ -332,11 +332,10 @@ The `url` can also be relative as described above. Here's an example of what tha "certifi >=2017.4.17", "python >=3.9" ], - "constrains": [], + "fn": "requests-2.32.5-py3-none-any.whl", "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", "size": 6899, "subdir": "noarch", - "timestamp": 1764005009, "noarch": "python", "url": "requests/requests-2.32.5-py3-none-any.whl" } @@ -369,11 +368,10 @@ The following shows an example of using an external location to download the whe "certifi >=2017.4.17", "python >=3.9" ], - "constrains": [], + "fn": "requests-2.32.5-py3-none-any.whl", "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", "size": 6899, "subdir": "noarch", - "timestamp": 1764005009, "noarch": "python", "url": "https://files.pythonhosted.org/packages/1e/db/4254e3eabe8020b458f1a747140d32277ec7a271daf1d235b70dc0b4e6e3/requests-2.32.5-py3-none-any.whl" } @@ -399,11 +397,10 @@ Here is an example of name mapping and normalization of the record name and depe "typing_extensions>=4.0.0[when=\"python<3.9\"]", "python >=3.8" ], - "constrains": [], + "fn": "annotated_types-0.7.0-py3-none-any.whl", "sha256": "1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53", "size": 13643, "subdir": "noarch", - "timestamp": 1756405206, "noarch": "python", "url": "https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl" } From cd43c24b6e5aabf825578d1d7344e32edf28d313 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Thu, 16 Apr 2026 12:19:41 -0400 Subject: [PATCH 30/38] Reference download URL draft CEP --- cep-XXXX.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 3048673b..8f547988 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -71,7 +71,7 @@ The `whl` dictionary maps conda-like filenames to repodata records. The key MUST - **`extra_depends`**: MAY be present. When present, MUST be an object mapping extra names to lists of dependency strings for optional groups, per [PR 111][pr-111]. When absent or empty, the record declares no optional groups beyond `depends`. - **`subdir`**: MUST be `"noarch"`. - **`noarch`**: MUST be `"python"`. -- **`url`**: MAY be present. See [Wheel download URLs](#wheel-download-urls) for semantics. +- **`url`**: MAY be present. Semantics are defined in [PR 151][pr-151] (see [Wheel download URLs](#wheel-download-urls)). - **`fn`**: MUST be the wheel filename (`.whl`), as in standard conda repodata records (for example `requests-2.32.5-py3-none-any.whl`). - **`sha256`**, **`size`**: Standard repodata fields for the wheel file. @@ -104,10 +104,10 @@ Channel operators SHOULD document any naming conventions and mappings specific t ### Wheel download URLs -This CEP depends on the `url` field for package records ([CEP XXXX](https://github.com/conda/ceps/pull/151)), which enables custom download locations. This is essential for wheel support since PyPI wheels are commonly hosted in per-package subdirectories or served from CDNs. +This CEP depends on the optional per-record **`url`** field as specified in [PR 151][pr-151], which enables custom download locations. This is essential for wheel support since PyPI wheels are commonly hosted in per-package subdirectories or served from CDNs. The `whl` mapping (inside the `v{revision}` payload) and the per-record `url` field SHALL follow -the backwards-compatible update strategy ([CEP XXXX](https://github.com/conda/ceps/pull/146)). +the backwards-compatible update strategy ([PR 146](https://github.com/conda/ceps/pull/146)). ### Wheel-Specific Record Values @@ -118,7 +118,7 @@ When populating repodata records for pure Python wheels: - `subdir`: MUST be "noarch" - `noarch`: MUST be "python" - `fn`: MUST be the wheel filename (`.whl`) -- `url`: MAY be present and follow the semantics described above +- `url`: MAY be present and follow the semantics in [PR 151][pr-151] and [Wheel download URLs](#wheel-download-urls) ### Pure Python wheel validation @@ -604,6 +604,7 @@ Conda-pypi incorporates the wheel detection logic from conda-whl-support, provid - [PEP 508 marker conversion (conda-pypi developer docs)][conda-pypi-marker-conversion] - [Example `repodata.json` (conda-pypi test channel)][conda-pypi-example-repodata] - [Channel relations in repodata (PR 155)][cep-channel-relations] +- [PR 151 – URL field for package records][pr-151] - [PR 111 – Conditional dependencies and optional groups][pr-111] - [conda-pupa][conda-pupa] @@ -622,6 +623,7 @@ All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdo [conda-pypi-marker-conversion]: https://conda.github.io/conda-pypi/developer/marker-conversion/#pep-508-variables [conda-pypi-example-repodata]: https://github.com/conda-incubator/conda-pypi/blob/main/tests/conda_local_channel/noarch/repodata.json [cep-channel-relations]: https://github.com/conda/ceps/pull/155 +[pr-151]: https://github.com/conda/ceps/pull/151 [pr-111]: https://github.com/conda/ceps/pull/111 [conda-pupa]: https://github.com/dholth/conda-pupa [uv-in-pixi]: https://prefix.dev/blog/uv_in_pixi From bdb1d40c0947de7660b2a5c9acc17179257ba4e5 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Thu, 16 Apr 2026 14:33:54 -0400 Subject: [PATCH 31/38] Add details about the first pip subcommand --- cep-XXXX.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/cep-XXXX.md b/cep-XXXX.md index 8f547988..ebb475ff 100644 --- a/cep-XXXX.md +++ b/cep-XXXX.md @@ -521,15 +521,16 @@ Native wheel support provides the same user experience (transparent PyPI access) ## History -The desire for better interoperability is not new and there has been a discussion over the last 12 years about balancing conda's core strengths (reproducibility, binary packages, cross-language support) with the Python community's expectations for seamless PyPI access. +PyPI integration has been debated for years in tension with conda's emphasis on reproducible, binary-first environments. ### Early Vision (2012-2014) -The conda project's earliest discussions reveal a consistent vision for broad package manager interoperability that extends beyond just PyPI integration: +Early conda assumed environments could mix conda-installed packages with software brought in by other tools, and tried to fold those installs into environments managed by conda. The first concrete step was PyPI: within weeks of conda's first release, it shipped a `pip` subcommand. +This feature took a snapshot of untracked files, ran that environment's `pip install`, created a diff, packed the new files into one `.tar.bz2`, and then reinstalled them with conda. However, folding arbitrary pip installs into a single conda record was a poor fit for dependency identity, upgrades, and reuse. The subcommand was removed in 1.8 for that model and to reduce confusion between `conda pip` and ordinary `pip` in an activated environment. -- Issue [#307](https://github.com/conda/conda/issue/307) (2013) proposed conda should work with pip, npm, gem, rpm, and brew, envisioning conda as a universal package manager interface -- Issue [#292](https://github.com/conda/conda/issue/292) (2013) explicitly titled "Making conda the ultimate package manager" outlined ambitious cross-ecosystem integration -- Issue [#224](https://github.com/conda/conda/issue/224) (2012) discussed deprecating pip commands in favor of native conda functionality +Issue [#327](https://github.com/conda/conda/issues/327) added `--use-pypi`. conda 4.6.0 introduced experimental `prefix_data_interoperability` to reconcile pip-installed metadata with conda's when enabled; it remained off by default because of the performance cost. + +Representative issues from the same period: [#307](https://github.com/conda/conda/issues/307) (2013, pip, npm, gems, rpm, brew), [#292](https://github.com/conda/conda/issues/292) (2013, "ultimate package manager"), [#224](https://github.com/conda/conda/issues/224) (2012, native conda vs pip commands). ### The PyPI Integration Debate (2013-2016) @@ -537,7 +538,7 @@ Two competing philosophies emerged on how or if we should provide better integra #### Pro-integration -Issue [#262](https://github.com/conda/conda/issue/262) (2013): conda should install directly from PyPI to reduce duplication and improve package availability. +Issue [#262](https://github.com/conda/conda/issues/262) (2013): conda should install directly from PyPI to reduce duplication and improve package availability. Arguments for this approach: @@ -546,7 +547,7 @@ Arguments for this approach: #### Status quo -Arguments to for maintaining an independent packaging ecosystem prevailed at the time for the following reasons: +Arguments for maintaining an independent packaging ecosystem prevailed at the time for the following reasons: - Conda's value proposition is reproducible, binary-focused environments with precise dependency resolution - PyPI's source distributions and pip's resolver could compromise conda's guarantees From fd69873b2001535192a8ec4b789d8eb9a541f316 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Fri, 24 Apr 2026 15:42:44 -0400 Subject: [PATCH 32/38] Add informative overview CEP --- cep-XXX0.md | 245 +++++++++++++++++++++++++++++++++ cep-XXXX.md => cep-XXX1.md | 269 ++++++------------------------------- cep-XXX2.md | 174 ++++++++++++++++++++++++ 3 files changed, 461 insertions(+), 227 deletions(-) create mode 100644 cep-XXX0.md rename cep-XXXX.md => cep-XXX1.md (57%) create mode 100644 cep-XXX2.md diff --git a/cep-XXX0.md b/cep-XXX0.md new file mode 100644 index 00000000..11d2e5db --- /dev/null +++ b/cep-XXX0.md @@ -0,0 +1,245 @@ +# CEP XXXX - Wheel support in conda (overview) + + + + + + + + + + +
Title Wheel support in conda (overview)
Status Draft
Author(s) + Dan Yeaw <dyeaw@anaconda.com> +
Created Apr 24, 2026
Updated Apr 24, 2026
Discussion https://github.com/conda/ceps/pull/145
Implementation TBD
See also [CEP XXX1 – Repodata wheel support](cep-XXX1.md), [CEP XXX2 – Wheel conda client support](cep-XXX2.md)
+ +> This document is **informative context** for the normative split between [CEP XXX1 – Repodata wheel support][cep-xxx1] (channel index / `whl` records) and [CEP XXX2 – Wheel conda client support][cep-xxx2] (solver, download, install). It collects motivation, history, and rejected alternatives. Unless stated otherwise, it does not add new SHALL requirements beyond pointing readers to those CEPs. + +## Abstract + +Pure Python wheels can be indexed in conda channel repodata so users solve environments in one pass with conda packages. This overview explains **why** that direction was chosen for conda, traces **historical** discussion and tooling, and documents **rejected ideas** and ecosystem alternatives. The repodata format and publisher rules live in [CEP XXX1][cep-xxx1]; conda client behavior lives in [CEP XXX2][cep-xxx2]. + +## Companion CEPs + +| CEP | Role | +|---------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [CEP XXX1 – Repodata wheel support][cep-xxx1] | Normative specification of the `whl` mapping under `v{revision}`, record fields, `METADATA` conversion, validation, and channel-operator guidance. | +| [CEP XXX2 – Wheel conda client support][cep-xxx2] | Normative specification of merging `whl` into the solver index, exclusivity and preference, conditional dependencies and extras, download, and prefix integration as `noarch: python` conda packages. | + +Together, the three documents describe **native** wheel support: wheels are first-class index entries, not a parallel pip-only install path. Scope is limited to **pure Python** wheels (`py*-none-any` class) in [CEP XXX1][cep-xxx1] to avoid platform-specific binary compatibility questions in this round of work. + +## Motivation + +While conda remains a language-agnostic packaging distribution, installing packages for use with Python represents the majority of the current conda usage. Users frequently encounter packages only available as wheels on PyPI, forcing them to use hybrid workflows that mix conda and PyPI installations. This approach creates several problems: + +- Mixing of conda and pip or uv can result in overwritten files and broken environments +- Users must understand two package managers, their interaction points, and which of their dependencies are available from which ecosystem +- Where there is support for mixing environments, it requires multiple solves which reduces overall package installation performance + +By adding native support for pure Python wheels—**indexed in repodata** and **consumed by conda clients** per [CEP XXX1][cep-xxx1] and [CEP XXX2][cep-xxx2]—conda can: + +- Reduce the cognitive burden of managing two package managers +- Provide users with transparent access to a broader slice of the Python ecosystem through channels that choose to publish wheels +- Maintain environment consistency and reproducibility under conda's package model +- Resolve dependencies across conda and wheel records in a single solve where channels publish both +- Fill gaps in conda package availability for simpler pure-Python packages +- Reduce the maintenance burden for straightforward pure-Python packages that do not require recipe-level metadata fixes + +[CEP XXX1][cep-xxx1] does not eliminate the need for conda recipes. Many packages still require feedstocks due to metadata differences, dependency corrections, compiled dependencies, or ecosystem-specific needs. Indexed wheels are an additional tool for simpler pure-Python cases. + +## History + +PyPI integration has been debated for years in tension with conda's emphasis on reproducible, binary-first environments. + +### Early Vision (2012-2014) + +Early conda assumed environments could mix conda-installed packages with software brought in by other tools, and tried to fold those installs into environments managed by conda. The first concrete step was PyPI: within weeks of conda's first release, it shipped a `pip` subcommand. +This feature took a snapshot of untracked files, ran that environment's `pip install`, created a diff, packed the new files into one `.tar.bz2`, and then reinstalled them with conda. However, folding arbitrary pip installs into a single conda record was a poor fit for dependency identity, upgrades, and reuse. The subcommand was removed in 1.8 for that model and to reduce confusion between `conda pip` and ordinary `pip` in an activated environment. + +Issue [#327](https://github.com/conda/conda/issues/327) added `--use-pypi`. conda 4.6.0 introduced experimental `prefix_data_interoperability` to reconcile pip-installed metadata with conda's when enabled; it remained off by default because of the performance cost. + +Representative issues from the same period: [#307](https://github.com/conda/conda/issues/307) (2013, pip, npm, gems, rpm, brew), [#292](https://github.com/conda/conda/issues/292) (2013, "ultimate package manager"), [#224](https://github.com/conda/conda/issues/224) (2012, native conda vs pip commands). + +### The PyPI Integration Debate (2013-2016) + +Two competing philosophies emerged on how or if we should provide better integration with PyPI. + +#### Pro-integration + +Issue [#262](https://github.com/conda/conda/issues/262) (2013): conda should install directly from PyPI to reduce duplication and improve package availability. + +Arguments for this approach: + +- Users frequently need packages not available on conda channels +- Building conda packages for every PyPI package is unsustainable + +#### Status quo + +Arguments for maintaining an independent packaging ecosystem prevailed at the time for the following reasons: + +- Conda's value proposition is reproducible, binary-focused environments with precise dependency resolution +- PyPI's source distributions and pip's resolver could compromise conda's guarantees +- The conda-forge community successfully scaled recipe creation + +### Wheel Support Proposal (2017) + +Issue [#5202](https://github.com/conda/conda/issues/5202) proposed direct wheel installation support, recognizing: + +- Wheels provide binary distributions similar to conda packages +- Growing wheel availability on PyPI reduced build complexity +- Could bridge the gap between conda's reliability and PyPI's breadth + +### Broader Context (2018) + +The Python Discourse thread on [packaging scope boundaries](https://discuss.python.org/t/drawing-a-line-to-the-scope-of-python-packaging/883) reflects the larger ecosystem's struggle with: + +- Multiple competing package management tools (pip, conda, poetry, pipenv) +- Unclear responsibilities and interoperability expectations +- Need for clearer standards and communication between tools + +### conda-pypi development starts (2022) + +Jaime Rodríguez-Guerra starts development on the conda-pypi plugin aimed to improve conda and PyPI interoperability. + +### Rip development starts (2023) + +Prefix.dev starts a barebones pip implemented in Rust to resolve and install PyPI dependencies with Pixi. + +### Pixi Integrates with uv (Jan 2024) + +Pixi changes course to use uv directly instead of rip, which unlocks features like editable installations, and git and path dependencies. + +### conda-pupa creates on-the-fly conversion plugin (July 2024) + +Daniel Holth creates a conda plugin that supports on-the-fly conversion of conda packages to wheels in a local channel. + +### conda-whl-channel creates proof-of-concept wheel channel from repodata (Nov 2024) + +Jonathan Helmus and Anil Kulkarni begin development on [conda-whl-channel](https://github.com/Anaconda/conda-whl-channel), a proof-of-concept that adds wheels to the `packages` section of repodata and patches conda to recognize them. The implementation supports conditional dependencies through additional meta packages. In February 2025, the conda-specific functionality is extracted into a separate plugin called [conda-whl-support](https://github.com/Anaconda/conda-whl-support). + +### conda-pypi merges in conda-pupa functionality (Oct 2025) + +conda-pupa is merged into conda-pypi which adds a `conda pypi install ` command and support for editable installations. + +### conda-pypi integrates parts of conda-whl-support (Nov 2025) + +Conda-pypi incorporates the wheel detection logic from conda-whl-support, providing core functionality beyond the solver and index changes required to support the `whl` section proposed in [CEP XXX1][cep-xxx1], together with channel relation metadata when channels adopt [Channel relations in repodata][cep-channel-relations]. + +## Rejected ideas + +### Only install Python and pip inside conda environments + +In this scenario, users only install Python and pip inside a clean conda environment. Here, we simply use conda as an environment manager and let pip manage the project dependencies. + +This is what that typically looks like: + +```bash +conda create -n pip-environment python pip +conda activate pip-environment +pip install +``` + +Despite its safety for environment management, relying solely on conda for this purpose prevents leveraging the package distribution capabilities of the conda ecosystem. + +### Editable installs with conda for dependencies only + +Conda provides all the dependencies of a given package. Then that package is installed on top in editable mode, without addressing dependencies to make sure conda files aren't accidentally overwritten: + +```bash +git clone https://github.com/owner/package.git +conda create -n editable-install package --deps-only +conda activate editable-install +pip install -e . --no-deps +``` + +### Add more conda packages + +Create and maintain new conda packages for each PyPI dependency needed. Tools like [Grayskull][grayskull] exist to make this easier to convert. However, this is a significant workload for the community, with over half of all conda-forge packages being pure Python. Even with more dedicated resources, creating recipes for over 400 thousand pure Python packages is not achievable. + +### Add interoperability to tools through pip dependency scanning + +The original version of the conda-pypi plugin called `pip` with the `--dry-run` option to analyze the solution to install a package. With a list of all the dependencies needed, the plugin installed everything available in conda channels first and what was left over would be installed with `pip install --no-deps`. Disadvantages of this approach include: + +- Users still have to know which packages they want from PyPI and then have to run `conda pip install` to install them. +- There is no guarantee that the conda and pip packages installed have ABI compatibility. +- Calling pip and conda multiple times is slow. + +### Add interoperability to tools through on-the-fly conversion + +This is the approach that the [conda-pupa][conda-pupa] plugin used and was then implemented in [conda-pypi][conda-pypi]. When `conda pypi install ` is called, it fetches its set of required dependencies iteratively from PyPI just like `pip`. Similar to the dependency scanning option above, it then attempts to install as many dependencies as it can from conda. +However, this is where these two approaches start to differ. While conda-pypi simply used `pip` to install the remaining Python dependencies, conda-pupa converts wheel packages to conda and stores them in a local channel, essentially caching these converted wheels on disk. This means that a repodata.json is also generated allowing us to perform a solve entirely in conda. + +Unfortunately, there are also disadvantages with this approach. Like the solution above, users still have to know which packages they want from PyPI and then have to run `conda pypi install` to install them. Additionally, the following problems also arise: + +- Package conversion can take some time, especially for larger packages. +- Users must rely on a local cache for installing wheels, and this cache cannot easily be shared across computers. +- The current version must solve multiple times which is slow, although this could be optimized. + +### Add interoperability to tools through uv integration + +Pixi has integrated uv for installing packages from PyPI. The user adds the dependency through `pixi add --pypi `. Then, when Pixi is solving the environment, it solves the conda packages using Rattler, and then calls uv to solve the PyPI dependencies. Disadvantages of this approach include: + +- Like the solutions above, users still have to know which packages they want from PyPI and then have to `pixi add --pypi` them. +- Although Pixi and uv are both very fast, it is still slower than performing a single solve of the environment. + +### Direct PyPI communication without repodata + +Another alternative would be for conda clients to query PyPI's API directly during solving, fetching wheel metadata on-demand rather than including it in repodata. This idea was rejected due to: + +- While resolvo (used by Rattler) supports dynamic metadata fetching during solving (as showcased in [rip](https://github.com/prefix-dev/rip)), libsolv requires complete package metadata upfront. This inconsistency across solvers would complicate implementation and limit compatibility. + +- While on-demand fetching works well for pip and uv, using repodata provides consistency with conda's existing infrastructure, enables better caching strategies, and allows channels to curate and validate packages before they're available to users. + +### Magic local channel approach + +The "magic local channel" approach () was considered but rejected. This approach involves: + +- A local FastAPI app intercepts the CLI and detects PyPI specs +- Fetches metadata on the fly and converts it to repodata (following some of the ideas discussed above) +- Downloads the wheels and converts them to .conda via whl2conda +- Caches and installs the .conda artifacts + +While this approach provides on-demand conversion and caching, it requires a separate service to be running and adds complexity to the user workflow. The chosen approach of native repodata wheel support provides a more seamless experience where wheels are pre-indexed in channels and work with standard conda workflows without requiring additional services. + +### Automatic wheel to conda package conversion and hosting + +Another alternative would be establishing a build farm to automatically convert wheels to conda packages and host them on a channel, with conversion triggered by popularity, community requests, or on-demand. + +#### Advantages of automatic conversion + +- All packages are conda packages, simplifying client implementation +- Leverages mature conda infrastructure and tooling without client changes +- Enables metadata enrichment, quality control, and validation before publishing + +#### Disadvantages and why native wheel support was chosen + +Despite these advantages, this approach was rejected because: + +- **Infrastructure burden:** Requires significant storage and bandwidth to host and serve converted packages that duplicate PyPI's CDN infrastructure +- **Resource inefficiency:** Wheels are already an excellent format for pure Python packages. Conversion adds no technical value and wastes resources + +Native wheel support provides the same user experience (transparent PyPI access through channels that publish wheels) while avoiding the infrastructure burden and resource inefficiency of conversion. Channel operators who prefer converted packages can continue building conda packages from PyPI sources. + +## References + +- [CEP XXX1 – Repodata wheel support][cep-xxx1] +- [CEP XXX2 – Wheel conda client support][cep-xxx2] +- [Adopting uv in pixi][uv-in-pixi] +- [rip][rip] +- [conda-pypi project][conda-pypi] +- [Channel relations in repodata (PR 155)][cep-channel-relations] +- [conda-pupa][conda-pupa] + +## Copyright + +All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/). + + +[cep-xxx1]: cep-XXX1.md +[cep-xxx2]: cep-XXX2.md +[conda-pypi]: https://github.com/conda-incubator/conda-pypi +[cep-channel-relations]: https://github.com/conda/ceps/pull/155 +[conda-pupa]: https://github.com/dholth/conda-pupa +[uv-in-pixi]: https://prefix.dev/blog/uv_in_pixi +[rip]: https://github.com/prefix-dev/rip +[grayskull]: https://conda.github.io/grayskull/ diff --git a/cep-XXXX.md b/cep-XXX1.md similarity index 57% rename from cep-XXXX.md rename to cep-XXX1.md index ebb475ff..33fa1822 100644 --- a/cep-XXXX.md +++ b/cep-XXX1.md @@ -8,10 +8,11 @@ Travis Hathaway <travis.j.hathaway@gmail.com> Created Dec 23, 2025 - Updated Apr 16, 2026 + Updated Apr 24, 2026 Discussion https://github.com/conda/ceps/pull/145 Implementation TBD - Requires https://github.com/conda/ceps/pull/151 https://github.com/conda/ceps/pull/146 https://github.com/conda/ceps/pull/155 https://github.com/conda/ceps/pull/111 + Requires https://github.com/conda/ceps/pull/151 https://github.com/conda/ceps/pull/146 https://github.com/conda/ceps/pull/155 https://github.com/conda/ceps/pull/111 [CEP XXX2 (Wheel conda client support)](cep-XXX2.md) + See also [CEP XXX0 – Wheel support in conda (overview)](cep-XXX0.md) > The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", @@ -20,27 +21,15 @@ described in [RFC2119][RFC2119] when, and only when, they appear in all capitals ## Abstract -This CEP outlines how native support for pure Python wheel packages will be achieved by adding support for them to conda's package index (repodata). When implemented, conda clients will be able to seamlessly install conda packages and pure Python wheels from enabled channels. -We explicitly limit the scope of this CEP to pure Python wheels to avoid platform-specific binary compatibility issues, but even with this limitation, we dramatically expand the number of available packages for users in the conda ecosystem. +This CEP specifies how channel operators publish pure Python wheel packages in conda's package index (repodata): the `whl` member under `v{revision}`, record fields, publisher-side dependency conversion from wheel `METADATA`, and validation rules. +It does not define conda client solver, download, or install-time behavior; those are specified in [CEP XXX2 – Wheel conda client support][cep-xxx2]. +Motivation, ecosystem history, and rejected alternatives are in [CEP XXX0 – Wheel support in conda (overview)][cep-xxx0]. +We limit the scope to pure Python wheels to avoid platform-specific binary compatibility issues; even so, publishing wheels in repodata greatly increases the packages channels can expose without building conda recipes for every case. -## Motivation +## Relationship to companion CEPs -While conda remains a language-agnostic packaging distribution, installing packages for use with Python represents the majority of conda usage. Users frequently encounter packages only available as wheels on PyPI, forcing them to use hybrid workflows that mix conda and PyPI installations. This approach creates several problems: - -- Mixing of conda and pip or uv can result in overwritten files and broken environments -- Users must understand two package managers, their interaction points, and which of their dependencies are available from which ecosystem -- Where there is support for mixing environments, it requires multiple solves which reduces overall package installation performance - -By adding native support for pure Python wheels to repodata, conda clients can: - -- Eliminate the cognitive burden of managing two package managers -- Provide users with transparent access to the broader Python ecosystem -- Maintain environment consistency and reproducibility -- Resolve dependencies across conda and PyPI packages in a single solve -- Fill gaps in conda package availability for simpler pure-Python packages -- Reduce the maintenance burden for straightforward pure-Python packages that don't require metadata modifications - -Note that this CEP does not eliminate the need for conda recipes entirely. Many packages will still require proper conda feedstocks due to metadata differences, dependency corrections, or other ecosystem-specific needs. This CEP provides an additional tool for handling simpler pure-Python cases more efficiently. +- **[CEP XXX0 – Wheel support in conda (overview)][cep-xxx0]** — Why native wheel support, historical context, and discussion of rejected ideas (informative). +- **[CEP XXX2 – Wheel conda client support][cep-xxx2]** — Normative client behavior: loading `whl` into the solve, exclusivity and preference, download, and installation into the prefix. ## Specification @@ -145,8 +134,6 @@ Wheel dependencies (from METADATA file's Requires-Dist entries) MUST be converte - **Python version requirements:** Convert Requires-Python to explicit python dependency - **Environment markers:** Map Python-version markers on `Requires-Dist` to conditional `MatchSpec` `when=` as specified in [PR 111][pr-111]. Other markers are out of scope for the default conversion rules in this CEP (see [Limitations](#limitations)). -For a non-normative, per-variable description of how **conda-pypi** maps PEP 508 markers to `when=` fragments and **`extra_depends`** for optional groups, see [PEP 508 marker conversion][conda-pypi-marker-conversion] in the conda-pypi documentation. - Example conversion: ```text @@ -172,37 +159,48 @@ Resulting conda record: } ``` -### Handling conditional dependencies and extras +### PEP 508 marker translation guidance -Wheel `depends` entries that encode PEP 508 environment markers MUST use conditional `MatchSpec` syntax with `when=` as specified in [PR 111][pr-111]. +This subsection layers **interoperable expectations** for the wire format, **non-normative** reference material for richer mappings, and **lossy translation** policy. It does not duplicate a full PEP 508 variable matrix in this CEP; maintaining that table is left to living documentation and tooling (see below). -PyPI **extras** (optional dependency groups) MUST be represented on the wheel repodata record using an **`extra_depends`** object (a mapping from extra name to lists of dependency strings) as specified in [PR 111][pr-111]. A default install uses only `depends` and MUST NOT union in `extra_depends` entries unless the user selects optional groups (for example with `extras=` in `MatchSpec` as described in [PR 111][pr-111]). +#### Interoperable minimum -### Solver behavior and package preference +For published `whl` records, the following conversions are the **normative floor** aligned with [Dependency conversion](#dependency-conversion) and [PR 111][pr-111]: + +- **`Requires-Python`:** MUST appear as an explicit `python` dependency in `depends`. +- **Python-version conditions on `Requires-Dist`:** MUST be represented with conditional `MatchSpec` `when=` on the relevant dependency strings per [PR 111][pr-111]. +- **`extra` (optional groups):** MUST be represented with **`extra_depends`** and optional-group selection per [PR 111][pr-111], not only as opaque marker text on `depends`. + +Channels and tools MAY apply additional, stricter mappings where they can express them in repodata. + +#### Non-normative extended mappings -#### Dependency resolution +Per-variable translation of other PEP 508 markers (for example `sys_platform`, `platform_system`) to `when=` fragments, virtual packages, or omissions is **not** fully specified in this CEP. For a non-normative, per-variable description—including how **conda-pypi** maps markers to `when=` and **`extra_depends`**—see [PEP 508 marker conversion][conda-pypi-marker-conversion] in the conda-pypi documentation. -Solvers MUST treat pure Python wheels as valid package candidates during dependency resolution with these constraints: +#### Lossy translation -- **Exclusivity:** Solvers MUST NOT install both a conda package and wheel for the same package name. -- **Dependency satisfaction:** When a wheel is selected, its `depends` list MUST be satisfied like any conda package. Dependencies in the `depends` list MAY be satisfied by either wheels or conda packages. -- **Platform matching:** Since all wheels in `whl` are pure Python (noarch), no platform filtering is needed. +When a channel cannot encode a PEP 508 dimension in repodata (for example no virtual package for an architecture string, or unsupported boolean branches), the operator MAY **omit** the condition, **simplify** the requirement, or **widen** the dependency (for example listing a package in `depends` without `when=` so it applies in more environments than PEP 508 would). +Channel operators SHOULD document their policy for such cases (including any conservative “include unconditionally” behavior). They MUST NOT claim that published repodata reproduces **full** PEP 508 semantics when a lossy transform was applied. -#### User control of precedence +### Handling conditional dependencies and extras + +Wheel `depends` entries that encode PEP 508 environment markers MUST use conditional `MatchSpec` syntax with `when=` as specified in [PR 111][pr-111]. -By default, channels with conda packages MUST prefer conda packages when they are available. Users MAY override default precedence through: +PyPI **extras** (optional dependency groups) MUST be represented on the wheel repodata record using an **`extra_depends`** object (a mapping from extra name to lists of dependency strings) as specified in [PR 111][pr-111]. A default install uses only `depends` and MUST NOT union in `extra_depends` entries unless the user selects optional groups (for example with `extras=` in `MatchSpec` as described in [PR 111][pr-111]). + +### Solver behavior and package preference -- Channel priority configuration (prefer channels with wheels). -- Explicit wheel requests through `::` syntax. -- Explicit configuration in the client itself (e.g. `prefer_conda` or `prefer_wheel`) +Normative rules for how conda clients treat `whl` records during dependency resolution (candidate pool, exclusivity, default preference for conda vs wheel, and user overrides) are defined in [CEP XXX2 – Wheel conda client support][cep-xxx2]. +This CEP only requires that published `whl` records are valid [repodata records][repodata-record-schema] so that clients can consume them alongside `packages` and `packages.conda`. ### Limitations This CEP has the following known limitations: 1. **Pure Python only:** This CEP explicitly does not address wheels with binary extensions, which require platform-specific compatibility guarantees beyond the current scope. Conda’s strength is binary compatibility, so using conda packages may be the optimal solution. -2. **Environment markers:** This CEP’s default conversion rules focus on Python-version markers mapped to `when=` per [PR 111][pr-111]. Other markers (for example `sys_platform`) are out of scope for those default rules and are not converted here, though some MAY be expressible with `when=` once clients implement [PR 111][pr-111]. -3. **Conditionals and extras:** Normative syntax and record fields for `when=` on `depends` and for optional groups in **`extra_depends`** are specified in [PR 111][pr-111], which this wheel CEP depends on for PyPI-aligned conditionals and extras. +2. **Environment markers (publisher defaults):** Default conversion rules in this CEP focus on Python-version markers on `Requires-Dist`, mapped to `when=` per [PR 111][pr-111]. Other markers (for example `sys_platform`) are out of scope for those default publisher rules and are not required to be converted into repodata here; see [PEP 508 marker translation guidance](#pep-508-marker-translation-guidance). +How conda clients evaluate `when=` and optional groups at solve time (including environment context) is specified in [CEP XXX2][cep-xxx2] together with [PR 111][pr-111]. +3. **Conditionals and extras:** Normative syntax and record fields for `when=` on `depends` and for optional groups in **`extra_depends`** are specified in [PR 111][pr-111], on which this repodata CEP and [CEP XXX2][cep-xxx2] rely for PyPI-aligned conditionals and extras in published records and at client solve time, respectively. 4. **Repodata size:** Supporting a significant portion of pure Python packages from PyPI (potentially hundreds of thousands of packages with multiple versions each) will substantially increase repodata size. Channels adopting wheel support at scale will need to implement sharded repodata ([CEP 16][cep-16]) to maintain acceptable performance. Alternatively, channels may choose to curate a subset of popular or requested packages rather than mirroring all of PyPI. 5. **PyPI package deletion:** PyPI allows package maintainers to delete packages (as opposed to just yanking them), which can break locked environments that reference those packages. Channels using external PyPI URLs directly are subject to this risk. For production use and reproducible environments, channels MAY mirror and store wheel artifacts locally rather than relying solely on external PyPI URLs. @@ -228,16 +226,7 @@ Several factors can cause wheel names to differ from conda-style names: ### For conda clients -Clients implementing this CEP SHOULD: - -- Parse the `whl` mapping inside each supported `v{revision}` object alongside existing `packages` / - `packages.conda` sections -- Apply the same filtering and preference logic used for conda packages -- Extract wheel metadata during solving to populate dependency information -- Evaluate `when=` on wheel `depends` entries per [PR 111][pr-111] -- Honor **`extra_depends`** when the user selects optional groups (for example with `extras=` in `MatchSpec`) per [PR 111][pr-111] -- Treat wheel records that use `when=` on `depends` or **`extra_depends`** metadata as part of the same `v{revision}` repodata payload as other new-syntax records, per the [backwards-compatible repodata update strategy](https://github.com/conda/ceps/pull/146) -- Provide the ability to natively install wheels or on-the-fly convert wheels to conda packages for installation +Client behavior (loading `whl` into the solve, preference and exclusivity, download, and installation into the environment prefix) is specified in [CEP XXX2 – Wheel conda client support][cep-xxx2]. Channel operators need not implement that CEP; they only publish repodata that conforms to this document. ### For channel operators @@ -424,183 +413,10 @@ Requires-Dist: typing-extensions>=4.0.0; python_version < '3.9' The package record expresses the conditional with `when=` so `typing_extensions` applies only when the Python version is less than 3.9, matching the original wheel METADATA and [PR 111][pr-111]. The Python version constraint of >=3.8 is directly mapped. -## Rejected ideas - -### Only install Python and pip inside conda environments - -In this scenario, users only install Python and pip inside a clean conda environment. Here, we simply use conda as an environment manager and let pip manage the project dependencies. - -This is what that typically looks like: - -```bash -conda create -n pip-environment python pip -conda activate pip-environment -pip install -``` - -Despite its safety for environment management, relying solely on conda for this purpose prevents leveraging the package distribution capabilities of the conda ecosystem. - -### Editable installs with conda for dependencies only - -Conda provides all the dependencies of a given package. Then that package is installed on top in editable mode, without addressing dependencies to make sure conda files aren't accidentally overwritten: - -```bash -git clone https://github.com/owner/package.git -conda create -n editable-install package --deps-only -conda activate editable-install -pip install -e . --no-deps -``` - -### Add more conda packages - -Create and maintain new conda packages for each PyPI dependency needed. Tools like [Grayskull] exist to make this easier to convert. However, this is a significant workload for the community, with over half of all conda-forge packages being pure Python. Even with more dedicated resources, creating recipes for over 400 thousand pure Python packages is not achievable. - -### Add interoperability to tools through pip dependency scanning - -The original version of the conda-pypi plugin called `pip` with the `--dry-run` option to analyze the solution to install a package. With a list of all the dependencies needed, the plugin installed everything available in conda channels first and what was left over would be installed with `pip install --no-deps`. Disadvantages of this approach include: - -- Users still have to know which packages they want from PyPI and then have to run `conda pip install` to install them. -- There is no guarantee that the conda and pip packages installed have ABI compatibility. -- Calling pip and conda multiple times is slow. - -### Add interoperability to tools through on-the-fly conversion - -This is the approach that the [conda-pupa][conda-pupa] plugin used and was then implemented in [conda-pypi][conda-pypi]. When `conda pypi install ` is called, it fetches its set of required dependencies iteratively from PyPI just like `pip`. Similar to the dependency scanning option above, it then attempts to install as many dependencies as it can from conda. -However, this is where these two approaches start to differ. While conda-pypi simply used `pip` to install the remaining Python dependencies, conda-pupa converts wheel packages to conda and stores them in a local channel, essentially caching these converted wheels on disk. This means that a repodata.json is also generated allowing us to perform a solve entirely in conda. - -Unfortunately, there are also disadvantages with this approach. Like the solution above, users still have to know which packages they want from PyPI and then have to run `conda pypi install` to install them. Additionally, the following problems also arise: - -- Package conversion can take some time, especially for larger packages. -- Users must rely on a local cache for installing wheels, and this cache cannot easily be shared across computers. -- The current version must solve multiple times which is slow, although this could be optimized. - -### Add interoperability to tools through uv integration - -Pixi has integrated uv for installing packages from PyPI. The user adds the dependency through `pixi add --pypi `. Then, when Pixi is solving the environment, it solves the conda packages using Rattler, and then calls uv to solve the PyPI dependencies. Disadvantages of this approach include: - -- Like the solutions above, users still have to know which packages they want from PyPI and then have to `pixi add --pypi` them. -- Although Pixi and uv are both very fast, it is still slower than performing a single solve of the environment. - -### Direct PyPI communication without repodata - -Another alternative would be for conda clients to query PyPI's API directly during solving, fetching wheel metadata on-demand rather than including it in repodata. This idea was rejected due to: - -- While resolvo (used by Rattler) supports dynamic metadata fetching during solving (as showcased in [rip](https://github.com/prefix-dev/rip)), libsolv requires complete package metadata upfront. This inconsistency across solvers would complicate implementation and limit compatibility. - -- While on-demand fetching works well for pip and uv, using repodata provides consistency with conda's existing infrastructure, enables better caching strategies, and allows channels to curate and validate packages before they're available to users. - -### Magic local channel approach - -The "magic local channel" approach () was considered but rejected. This approach involves: - -- A local FastAPI app intercepts the CLI and detects PyPI specs -- Fetches metadata on the fly and converts it to repodata (following some of the ideas discussed above) -- Downloads the wheels and converts them to .conda via whl2conda -- Caches and installs the .conda artifacts - -While this approach provides on-demand conversion and caching, it requires a separate service to be running and adds complexity to the user workflow. The chosen approach of native repodata wheel support provides a more seamless experience where wheels are pre-indexed in channels and work with standard conda workflows without requiring additional services. - -### Automatic wheel to conda package conversion and hosting - -Another alternative would be establishing a build farm to automatically convert wheels to conda packages and host them on a channel, with conversion triggered by popularity, community requests, or on-demand. - -#### Advantages of automatic conversion - -- All packages are conda packages, simplifying client implementation -- Leverages mature conda infrastructure and tooling without client changes -- Enables metadata enrichment, quality control, and validation before publishing - -#### Disadvantages and why native wheel support was chosen - -Despite these advantages, this approach was rejected because: - -- **Infrastructure burden:** Requires significant storage and bandwidth to host and serve converted packages that duplicate PyPI's CDN infrastructure -- **Resource inefficiency:** Wheels are already an excellent format for pure Python packages. Conversion adds no technical value and wastes resources - -Native wheel support provides the same user experience (transparent PyPI access) while avoiding the infrastructure burden and resource inefficiency of conversion. Channel operators who prefer converted packages can continue building conda packages from PyPI sources. - -## History - -PyPI integration has been debated for years in tension with conda's emphasis on reproducible, binary-first environments. - -### Early Vision (2012-2014) - -Early conda assumed environments could mix conda-installed packages with software brought in by other tools, and tried to fold those installs into environments managed by conda. The first concrete step was PyPI: within weeks of conda's first release, it shipped a `pip` subcommand. -This feature took a snapshot of untracked files, ran that environment's `pip install`, created a diff, packed the new files into one `.tar.bz2`, and then reinstalled them with conda. However, folding arbitrary pip installs into a single conda record was a poor fit for dependency identity, upgrades, and reuse. The subcommand was removed in 1.8 for that model and to reduce confusion between `conda pip` and ordinary `pip` in an activated environment. - -Issue [#327](https://github.com/conda/conda/issues/327) added `--use-pypi`. conda 4.6.0 introduced experimental `prefix_data_interoperability` to reconcile pip-installed metadata with conda's when enabled; it remained off by default because of the performance cost. - -Representative issues from the same period: [#307](https://github.com/conda/conda/issues/307) (2013, pip, npm, gems, rpm, brew), [#292](https://github.com/conda/conda/issues/292) (2013, "ultimate package manager"), [#224](https://github.com/conda/conda/issues/224) (2012, native conda vs pip commands). - -### The PyPI Integration Debate (2013-2016) - -Two competing philosophies emerged on how or if we should provide better integration with PyPI. - -#### Pro-integration - -Issue [#262](https://github.com/conda/conda/issues/262) (2013): conda should install directly from PyPI to reduce duplication and improve package availability. - -Arguments for this approach: - -- Users frequently need packages not available on conda channels -- Building conda packages for every PyPI package is unsustainable - -#### Status quo - -Arguments for maintaining an independent packaging ecosystem prevailed at the time for the following reasons: - -- Conda's value proposition is reproducible, binary-focused environments with precise dependency resolution -- PyPI's source distributions and pip's resolver could compromise conda's guarantees -- The conda-forge community successfully scaled recipe creation - -### Wheel Support Proposal (2017) - -Issue [#5202](https://github.com/conda/conda/issues/5202) proposed direct wheel installation support, recognizing: - -- Wheels provide binary distributions similar to conda packages -- Growing wheel availability on PyPI reduced build complexity -- Could bridge the gap between conda's reliability and PyPI's breadth - -### Broader Context (2018) - -The Python Discourse thread on [packaging scope boundaries](https://discuss.python.org/t/drawing-a-line-to-the-scope-of-python-packaging/883) reflects the larger ecosystem's struggle with: - -- Multiple competing package management tools (pip, conda, poetry, pipenv) -- Unclear responsibilities and interoperability expectations -- Need for clearer standards and communication between tools - -### conda-pypi development starts (2022) - -Jaime Rodríguez-Guerra starts development on the conda-pypi plugin aimed to improve conda and PyPI interoperability. - -### Rip development starts (2023) - -Prefix.dev starts a barebones pip implemented in Rust to resolve and install PyPI dependencies with Pixi. - -### Pixi Integrates with uv (Jan 2024) - -Pixi changes course to use uv directly instead of rip, which unlocks features like editable installations, and git and path dependencies. - -### conda-pupa creates on-the-fly conversion plugin (July 2024) - -Daniel Holth creates a conda plugin that supports on-the-fly conversion of conda packages to wheels in a local channel. - -### conda-whl-channel creates proof-of-concept wheel channel from repodata (Nov 2024) - -Jonathan Helmus and Anil Kulkarni begin development on [conda-whl-channel](, a proof-of-concept that adds wheels to the `packages` section of repodata and patches conda to recognize them. The implementation supports conditional dependencies through additional meta packages. In February 2025, the conda-specific functionality is extracted into a separate plugin called [conda-whl-support](. - -### conda-pypi merges in conda-pupa functionality (Oct 2025) - -conda-pupa is merged into conda-pypi which adds a `conda pypi install ` command and support for editable installations. - -### conda-pypi integrates parts of conda-whl-support (Nov 2025) - -Conda-pypi incorporates the wheel detection logic from conda-whl-support, providing the core functionality needed beyond the solver and index changes required to support the `whl` section proposed in this CEP, together with channel relation metadata when channels adopt [Channel relations in repodata][cep-channel-relations]. - ## References -- [Adopting uv in pixi][uv-in-pixi] -- [rip][rip] +- [CEP XXX0 – Wheel support in conda (overview)][cep-xxx0] +- [CEP XXX2 – Wheel conda client support][cep-xxx2] - [conda-pypi project][conda-pypi] - [PEP 508 marker conversion (conda-pypi developer docs)][conda-pypi-marker-conversion] - [Example `repodata.json` (conda-pypi test channel)][conda-pypi-example-repodata] @@ -627,6 +443,5 @@ All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdo [pr-151]: https://github.com/conda/ceps/pull/151 [pr-111]: https://github.com/conda/ceps/pull/111 [conda-pupa]: https://github.com/dholth/conda-pupa -[uv-in-pixi]: https://prefix.dev/blog/uv_in_pixi -[rip]: https://github.com/prefix-dev/rip -[grayskull]: https://conda.github.io/grayskull/ +[cep-xxx0]: cep-XXX0.md +[cep-xxx2]: cep-XXX2.md diff --git a/cep-XXX2.md b/cep-XXX2.md new file mode 100644 index 00000000..9d6a4c7c --- /dev/null +++ b/cep-XXX2.md @@ -0,0 +1,174 @@ +# CEP XXXX - Wheel conda client support + + + + + + + + + + + +
Title Wheel conda client support
Status Draft
Author(s) + Dan Yeaw <dyeaw@anaconda.com> +
Created Apr 24, 2026
Updated Apr 24, 2026
Discussion https://github.com/conda/ceps/pull/145
Implementation TBD
Requires [CEP XXX1 – Repodata wheel support](cep-XXX1.md) https://github.com/conda/ceps/pull/151 https://github.com/conda/ceps/pull/146 https://github.com/conda/ceps/pull/155 https://github.com/conda/ceps/pull/111
See also [CEP XXX0 – Wheel support in conda (overview)](cep-XXX0.md)
+ +> The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", +"RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as +described in [RFC2119][RFC2119] when, and only when, they appear in all capitals, as shown here. + +## Abstract + +This CEP specifies how conda clients consume wheel entries published in repodata: loading `v{revision}.whl` into the package index used by the solver, applying exclusivity and preference rules, evaluating conditional dependencies and extras per [PR 111][pr-111], downloading wheel artifacts, and integrating them into the environment **as equivalent `noarch: python` conda packages** (layout, package database, and lifecycle), not as a separate pip-style install path. +The shape of the `whl` index and publisher-side `METADATA` conversion are defined in [CEP XXX1 – Repodata wheel support][cep-xxx1]. This CEP applies to pure Python wheels in the sense of that document (e.g. `py3-none-any`); it does not define binary wheel support. +Background on **why** channels and clients pursue native wheel support, plus historical context and rejected ecosystem alternatives, is in [CEP XXX0 – Wheel support in conda (overview)][cep-xxx0]. + +## Motivation + +[CEP XXX1][cep-xxx1] makes wheel metadata available in channel repodata so environments can be solved in one pass together with conda packages. +That is not sufficient by itself: users need a single, well-defined **client** story for (1) which packages the solver may choose, (2) how optional groups and `when=` interact with the live environment, (3) where artifacts are downloaded from, and (4) how wheel contents map onto the prefix (site-packages, scripts, and conda metadata) without corrupting the environment or duplicating work already covered by conda package conventions. + +## Specification + +### Relationship to [CEP XXX1][cep-xxx1] + +This CEP assumes repodata that conforms to [CEP XXX1 – Repodata wheel support][cep-xxx1], including the `whl` mapping under a registered `v{revision}` and valid [repodata record][repodata-record-schema] fields for each entry. + +### Repodata consumption + +Conda clients that implement wheel support MUST merge entries from the `whl` mapping in each supported [repodata revision](https://github.com/conda/ceps/pull/146) with candidates derived from `packages` and `packages.conda` for the same revision, so that a single index feeds dependency resolution. Clients SHOULD apply the same subdir and channel filtering hooks used for other repodata records unless a separate policy is explicitly documented. + +Wheel records for pure Python packages use `subdir: "noarch"` per [CEP XXX1][cep-xxx1]; no additional platform filter is required for those wheels beyond the normal noarch and virtual-package logic for the target prefix. + +### Solver: dependency resolution and package preference + +Solvers MUST treat pure Python wheel records in `whl` as valid package candidates during dependency resolution with these constraints: + +- **Exclusivity:** Solvers MUST NOT install both a conda package and a wheel for the same [package name][cep-26] (as used in `MatchSpec` / the solver index). +- **Dependency satisfaction:** When a wheel is selected, its `depends` list MUST be satisfied the same as for any other package record. Dependencies in `depends` MAY be satisfied by either wheel or conda package records. +- **Platform matching:** For wheel records that conform to the pure-Python rules in [CEP XXX1][cep-xxx1] (`noarch: python` and the stated tag checks), no extra platform tag filtering of the wheel artifact is required, because channels only add that class of wheel to `whl` under those validation rules. + +#### User control of precedence + +By default, when both a conda package and a wheel are available for the same package name, the solver SHOULD prefer the conda package unless channel priority or an explicit spec selects the wheel. Users MAY override default precedence through: + +- Channel priority configuration (for example, preferring channels that publish wheels in `whl`); +- Explicit requests using `::` (or other channel-pinned syntax supported by the client); +- Client configuration (for example, flags or settings such as `prefer_conda` or `prefer_wheel`) where the implementation documents behavior clearly. + +The exact default when priorities tie is a client quality-of-implementation choice but MUST remain consistent with **Exclusivity** above (never two artifacts for the same name). + +### Conditional dependencies and extras + +For wheel records, [PR 111][pr-111] normatively defines `when=` on `depends` and the **`extra_depends`** object for PyPI optional groups. Clients that implement this CEP MUST: + +- Evaluate `when=` for wheel `depends` entries according to the same rules as for other new-syntax records, using an environment model that provides at least Python version and, where applicable, virtual packages (see [Marker handling](#marker-handling)). +- Honor **`extra_depends`** when the user selects optional groups (for example with `extras=` in `MatchSpec`) per [PR 111][pr-111], and MUST NOT pull in `extra_depends` groups on a default install. +- Load wheel records that use `when=` or `extra_depends` in the same `v{revision}` payload and revision-handling as other records introduced under the [backwards-compatible repodata update strategy](https://github.com/conda/ceps/pull/146). + +The strings on a wheel’s `depends` and `extra_depends` are produced by channel operators from wheel `METADATA` per [CEP XXX1][cep-xxx1]; the client’s job is to interpret them, not to re-parse the wheel for dependency resolution (though reading `METADATA` for validation or error messages is allowed). + +### Marker handling + +**Publisher output (in repodata):** [CEP XXX1][cep-xxx1] requires converting certain PEP 508 markers on `Requires-Dist` into `when=` and optional-group fields. Other markers are out of scope for the default *publisher* conversion rules; repodata may omit or simplify them. + +**Client (solve time):** The solver MUST use the MatchSpec and PR 111 features carried on the repodata record: `when=` subexpressions, `python` constraints, and virtual packages, so that the effective dependency graph matches the published strings for the current environment. Clients MAY map additional environment dimensions to virtual packages where documented (non-normative per-variable notes appear in the [conda-pypi marker conversion][conda-pypi-marker-conversion] documentation). + +**Out-of-scope dimensions:** Some PEP 508 variables (for example, `implementation_name`, or markers tied to a specific wheel build matrix beyond the pure-Python subset) do not have a defined mapping in the normative [CEP XXX1][cep-xxx1] record; behavior for such wheels remains undefined unless a future CEP or channel policy encodes them in repodata. Clients MUST NOT claim full PEP 508 equivalence beyond what the record expresses. + +### Download + +After the solver selects a wheel record, the client MUST download the `.whl` file using the same rules as for other `noarch` artifacts with a `fn` and optional `url`: + +- Resolve the URL per [PR 151][pr-151] and the download examples in [CEP XXX1][cep-xxx1] (channel `base_url`, default `noarch` layout, or absolute per-record `url`). +- Verify integrity using `sha256` and `size` from the repodata record when those fields are present, consistent with other conda downloads. + +### Installation layout and prefix placement + +Wheels selected from repodata MUST be integrated into the target environment **as the equivalent of a `noarch: python` conda package** ([CEP 20][cep-20], [CEP 34][cep-34]): the same style of layout under the prefix, registration in conda's package database, and `conda list` / `conda remove` / upgrade behavior that matches other packages from the index—not a standalone "pip install this wheel" interaction. + +Importable code MUST end up under the environment's [site-packages][cep-20] (or the site-packages / path layout for `noarch: python` described in that package’s `info/index.json` and related files per [CEP 34][cep-34]). Headers, data, and other install-tree paths from the wheel MUST be placed under prefix locations that are tracked and removed the same way as in a built conda package. +The repodata record points at a wheel file as the artifact to obtain; the **normative** requirement is conda-equivalent integration as in the preceding paragraph, not a wheel-only layout profile considered in isolation from conda's package model. + +- **Metadata:** The install MUST create a valid `.dist-info` (or legacy `.egg-info` if present in the wheel) so `importlib.metadata` can load the distribution, consistent with other Python packages in the environment. +- **RECORD:** The client SHOULD verify the wheel's `RECORD` where present to detect tampering or corruption; it MUST reject path entries that escape the intended prefix or package staging area. +- **Console and GUI entry points:** Application entry points SHOULD be materialized like other conda `noarch: python` packages: through `info/link.json` and conda's link machinery per [CEP 20][cep-20] and [CEP 34][cep-34], so shebangs and `bin` scripts remain under conda's control. A client implementation MAY avoid emitting duplicate script files from the wheel when `link.json` (or an equivalent) defines the public commands. + +## Rationale + +### One solve, predictable layout + +Native wheel support in the client avoids separate PyPI or pip phases for the packages indexed in [CEP XXX1][cep-xxx1], matches user expectations for where importable code lives, and reuses conda's environment model. Keeping download rules aligned with [PR 151][pr-151] matches existing channel URL behavior. + +### Native unpack vs always converting to conda packages + +On-the-fly conversion to a `.conda` artifact can still be a useful optimization or fallback (for example, sharing on a local channel). Either way, the user-visible result remains a normal conda-managed install. +This CEP requires that materializing the environment from a **wheel file** (without requiring a pre-built conda package on the channel) be possible so channels do not need a second hosting pipeline for the same file PyPI already ships as a wheel, and to avoid inflating storage with redundant formats for pure-Python content. + +## Implementation Notes + +### For conda client implementers + +- **Index:** Parse `whl` inside each supported `v{revision}` object alongside `packages` / `packages.conda` and feed merged candidates to the solver. +- **PR 111:** Implement evaluation of `when=` and merging of `extra_depends` when the user requests extras, consistent with the published record strings. +- **Revision gating:** Treat new-syntax wheel records with the same `v{revision}` handling as the rest of the PR 146 strategy (do not expect older repodata only clients to read `whl`). +- **Download:** Reuse the channel and fetch stack used for other artifacts, including `url` and checksum verification. +- **Install:** Integrate wheels using the same prefix layout and metadata model as `noarch: python` conda packages ([CEP 20][cep-20], [CEP 34][cep-34]), including `info/link.json` for entry points when using conda's link pipeline. The [conda-pypi][conda-pypi] project demonstrates populating a conda package shape from a wheel (for example, suppressing duplicate `bin` script generation when `link.json` is used) as **non-normative** prior art. +- **Testing:** Add integration tests for exclusivity, preference, conditional edges, a wheel-only dependency, and a conda–wheel mixed graph. + +## Examples + +### Solver constraints (illustrative) + +For a request that could be satisfied by either a conda `requests` or a wheel in `whl` with the same name, a configuration that prefers conda SHOULD select the conda build unless the user pins the wheel channel or package record. + +### Install outcome (illustrative) + +After installing `requests-2.32.5-py3-none-any.whl` from a wheel repodata record, the environment’s site-packages should contain the `requests` import package and a `requests-2.32.5.dist-info` (or equivalent) directory such that `import requests` and `importlib.metadata.version("requests")` succeed for that prefix’s Python. + +## Backwards compatibility + +- **Repodata shape:** The `whl` key appears only under a registered `v{revision}` (see [PR 146](https://github.com/conda/ceps/pull/146)). Conda clients that do not support that revision or the `whl` index MUST ignore unknown `v{revision}` members or fall back to earlier revisions per channel policy, and MUST NOT break on the presence of `whl` in repodata for revisions they do not load. +- **Record syntax:** Records that use `when=` and `extra_depends` are part of the new-syntax repodata world [PR 111][pr-111] already depends on; wheel rows are not a separate syntax fork. + +## Rejected ideas + +Ecosystem-level alternatives (relying on pip only, on-the-fly PyPI without repodata, build farms that convert every wheel) are discussed under **Rejected ideas** in [CEP XXX0 – Wheel support in conda (overview)][cep-xxx0]. This CEP is the client-side complement to the chosen approach: indexed wheels plus native client install. + +## References + +- [CEP XXX0 – Wheel support in conda (overview)][cep-xxx0] +- [CEP XXX1 – Repodata wheel support][cep-xxx1] +- [CEP 20 – `noarch` (python) packages and site-packages][cep-20] +- [CEP 34 – Package metadata and `info/`](./cep-0034.md) (`index.json`, `link.json`, etc.) +- [CEP 26 – Identifying packages and channels][cep-26] +- [PR 111 – Conditional dependencies and optional groups][pr-111] +- [PR 151 – URL field for package records][pr-151] +- [PEP 508 marker conversion (conda-pypi developer docs)][conda-pypi-marker-conversion] +- [Example `repodata.json` (conda-pypi test channel)][conda-pypi-example-repodata] +- [conda-pypi project][conda-pypi] +- [Adopting uv in pixi][uv-in-pixi] +- [rip][rip] +- [conda-pupa][conda-pupa] + +## Copyright + +All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/). + + +[RFC2119]: https://datatracker.ietf.org/doc/html/rfc2119 +[repodata-record-schema]: https://schemas.conda.org/repodata-record-1.schema.json +[cep-26]: https://conda.org/learn/ceps/cep-0026 +[cep-20]: ./cep-0020.md +[cep-34]: ./cep-0034.md +[cep-xxx0]: cep-XXX0.md +[cep-xxx1]: cep-XXX1.md +[conda-pypi]: https://github.com/conda-incubator/conda-pypi +[conda-pypi-marker-conversion]: https://conda.github.io/conda-pypi/developer/marker-conversion/#pep-508-variables +[conda-pypi-example-repodata]: https://github.com/conda-incubator/conda-pypi/blob/main/tests/conda_local_channel/noarch/repodata.json +[pr-151]: https://github.com/conda/ceps/pull/151 +[pr-111]: https://github.com/conda/ceps/pull/111 +[conda-pupa]: https://github.com/dholth/conda-pupa +[uv-in-pixi]: https://prefix.dev/blog/uv_in_pixi +[rip]: https://github.com/prefix-dev/rip From ef7f5a462263bbaf46c60b1202d93320fce957cd Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Fri, 24 Apr 2026 19:10:30 -0400 Subject: [PATCH 33/38] Cleanup wording --- cep-XXX0.md | 29 ++++++++++++++++++++--------- cep-XXX1.md | 26 ++++++++++++++++---------- cep-XXX2.md | 50 +++++++++++++++----------------------------------- 3 files changed, 51 insertions(+), 54 deletions(-) diff --git a/cep-XXX0.md b/cep-XXX0.md index 11d2e5db..4bdc99b7 100644 --- a/cep-XXX0.md +++ b/cep-XXX0.md @@ -24,7 +24,7 @@ Pure Python wheels can be indexed in conda channel repodata so users solve envir | CEP | Role | |---------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [CEP XXX1 – Repodata wheel support][cep-xxx1] | Normative specification of the `whl` mapping under `v{revision}`, record fields, `METADATA` conversion, validation, and channel-operator guidance. | -| [CEP XXX2 – Wheel conda client support][cep-xxx2] | Normative specification of merging `whl` into the solver index, exclusivity and preference, conditional dependencies and extras, download, and prefix integration as `noarch: python` conda packages. | +| [CEP XXX2 – Wheel conda client support][cep-xxx2] | Normative specification of loading `whl` into the solver index with other repodata records, conditional dependencies and extras, download, and prefix integration as `noarch: python` conda packages. | Together, the three documents describe **native** wheel support: wheels are first-class index entries, not a parallel pip-only install path. Scope is limited to **pure Python** wheels (`py*-none-any` class) in [CEP XXX1][cep-xxx1] to avoid platform-specific binary compatibility questions in this round of work. @@ -36,7 +36,7 @@ While conda remains a language-agnostic packaging distribution, installing packa - Users must understand two package managers, their interaction points, and which of their dependencies are available from which ecosystem - Where there is support for mixing environments, it requires multiple solves which reduces overall package installation performance -By adding native support for pure Python wheels—**indexed in repodata** and **consumed by conda clients** per [CEP XXX1][cep-xxx1] and [CEP XXX2][cep-xxx2]—conda can: +Adding native support for pure Python wheels—**indexed in repodata** and **consumed by conda clients** per [CEP XXX1][cep-xxx1] and [CEP XXX2][cep-xxx2], has the following advantages: - Reduce the cognitive burden of managing two package managers - Provide users with transparent access to a broader slice of the Python ecosystem through channels that choose to publish wheels @@ -53,10 +53,10 @@ PyPI integration has been debated for years in tension with conda's emphasis on ### Early Vision (2012-2014) -Early conda assumed environments could mix conda-installed packages with software brought in by other tools, and tried to fold those installs into environments managed by conda. The first concrete step was PyPI: within weeks of conda's first release, it shipped a `pip` subcommand. +Early `conda` assumed environments could mix conda-installed packages with software brought in by other tools, and tried to fold those installs into environments managed by conda. The first concrete step was PyPI: within weeks of conda's first release, it shipped a `pip` subcommand. This feature took a snapshot of untracked files, ran that environment's `pip install`, created a diff, packed the new files into one `.tar.bz2`, and then reinstalled them with conda. However, folding arbitrary pip installs into a single conda record was a poor fit for dependency identity, upgrades, and reuse. The subcommand was removed in 1.8 for that model and to reduce confusion between `conda pip` and ordinary `pip` in an activated environment. -Issue [#327](https://github.com/conda/conda/issues/327) added `--use-pypi`. conda 4.6.0 introduced experimental `prefix_data_interoperability` to reconcile pip-installed metadata with conda's when enabled; it remained off by default because of the performance cost. +Issue [#327](https://github.com/conda/conda/issues/327) added `--use-pypi`. conda 4.6.0 introduced experimental `prefix_data_interoperability` to reconcile pip-installed metadata with conda's when enabled. However, it remained off by default because of the performance cost. Representative issues from the same period: [#307](https://github.com/conda/conda/issues/307) (2013, pip, npm, gems, rpm, brew), [#292](https://github.com/conda/conda/issues/292) (2013, "ultimate package manager"), [#224](https://github.com/conda/conda/issues/224) (2012, native conda vs pip commands). @@ -105,6 +105,10 @@ Jaime Rodríguez-Guerra starts development on the conda-pypi plugin aimed to imp Prefix.dev starts a barebones pip implemented in Rust to resolve and install PyPI dependencies with Pixi. +### Whl2conda development starts (Aug 2023) + +Christopher Barber develops [whl2conda](https://github.com/zuzukin/whl2conda), a command-line tool that generates conda packages from pure Python wheels (and related packaging workflows) without going through a full `conda-build` recipe for every package. + ### Pixi Integrates with uv (Jan 2024) Pixi changes course to use uv directly instead of rip, which unlocks features like editable installations, and git and path dependencies. @@ -125,6 +129,10 @@ conda-pupa is merged into conda-pypi which adds a `conda pypi install ` Conda-pypi incorporates the wheel detection logic from conda-whl-support, providing core functionality beyond the solver and index changes required to support the `whl` section proposed in [CEP XXX1][cep-xxx1], together with channel relation metadata when channels adopt [Channel relations in repodata][cep-channel-relations]. +### Native wheel unpack in conda-pypi; Rattler reads wheels in repodata (Dec 2025) + +conda-pypi adds native support for unpacking `.whl` artifacts into the target environment (rather than delegating that step to pip). Rattler adds support for reading wheel records from repodata, for example, the `whl` mapping under a registered `v{revision}`, so clients built on Rattler can load the same index surface described in [CEP XXX1][cep-xxx1]. + ## Rejected ideas ### Only install Python and pip inside conda environments @@ -139,9 +147,9 @@ conda activate pip-environment pip install ``` -Despite its safety for environment management, relying solely on conda for this purpose prevents leveraging the package distribution capabilities of the conda ecosystem. +Despite its safety for environment management, relying solely on `conda` for this purpose prevents leveraging the package distribution capabilities of the conda ecosystem. -### Editable installs with conda for dependencies only +### Editable installs with `conda` for dependencies only Conda provides all the dependencies of a given package. Then that package is installed on top in editable mode, without addressing dependencies to make sure conda files aren't accidentally overwritten: @@ -152,9 +160,12 @@ conda activate editable-install pip install -e . --no-deps ``` +This pattern was rejected because it still splits responsibility between `conda` and `pip`: the editable project is not a first-class conda package, so `conda list`, upgrades, removals, and reproducible environment exports do not fully describe what is on disk. +Relying on `--no-deps` avoids one class of conflicts but leaves no single solver pass for the project under development and its dependencies, and it does not remove the risk of accidental `pip install` (with dependencies) overwriting files conda manages. + ### Add more conda packages -Create and maintain new conda packages for each PyPI dependency needed. Tools like [Grayskull][grayskull] exist to make this easier to convert. However, this is a significant workload for the community, with over half of all conda-forge packages being pure Python. Even with more dedicated resources, creating recipes for over 400 thousand pure Python packages is not achievable. +Create and maintain new conda packages for each PyPI dependency needed. Tools like [Grayskull][grayskull] exist to make this easier to convert. However, this is a significant workload for the community, with over half of all conda-forge packages being pure Python. Even with more dedicated resources, creating recipes for over 500 thousand pure Python packages is not achievable. ### Add interoperability to tools through pip dependency scanning @@ -166,7 +177,7 @@ The original version of the conda-pypi plugin called `pip` with the `--dry-run` ### Add interoperability to tools through on-the-fly conversion -This is the approach that the [conda-pupa][conda-pupa] plugin used and was then implemented in [conda-pypi][conda-pypi]. When `conda pypi install ` is called, it fetches its set of required dependencies iteratively from PyPI just like `pip`. Similar to the dependency scanning option above, it then attempts to install as many dependencies as it can from conda. +This is the approach that the [conda-pupa][conda-pupa] plugin used and was then implemented in [conda-pypi][conda-pypi]. When `conda pypi install ` is called, it fetches its set of required dependencies iteratively from PyPI just like `pip`. Similar to the dependency scanning option above, it then attempts to install as many dependencies as it can from `conda`. However, this is where these two approaches start to differ. While conda-pypi simply used `pip` to install the remaining Python dependencies, conda-pupa converts wheel packages to conda and stores them in a local channel, essentially caching these converted wheels on disk. This means that a repodata.json is also generated allowing us to perform a solve entirely in conda. Unfortunately, there are also disadvantages with this approach. Like the solution above, users still have to know which packages they want from PyPI and then have to run `conda pypi install` to install them. Additionally, the following problems also arise: @@ -179,7 +190,7 @@ Unfortunately, there are also disadvantages with this approach. Like the solutio Pixi has integrated uv for installing packages from PyPI. The user adds the dependency through `pixi add --pypi `. Then, when Pixi is solving the environment, it solves the conda packages using Rattler, and then calls uv to solve the PyPI dependencies. Disadvantages of this approach include: -- Like the solutions above, users still have to know which packages they want from PyPI and then have to `pixi add --pypi` them. +- Like the solutions above, users still have to know which packages they want from PyPI and then have to run `pixi add --pypi` on them. - Although Pixi and uv are both very fast, it is still slower than performing a single solve of the environment. ### Direct PyPI communication without repodata diff --git a/cep-XXX1.md b/cep-XXX1.md index 33fa1822..240aeee0 100644 --- a/cep-XXX1.md +++ b/cep-XXX1.md @@ -24,12 +24,12 @@ described in [RFC2119][RFC2119] when, and only when, they appear in all capitals This CEP specifies how channel operators publish pure Python wheel packages in conda's package index (repodata): the `whl` member under `v{revision}`, record fields, publisher-side dependency conversion from wheel `METADATA`, and validation rules. It does not define conda client solver, download, or install-time behavior; those are specified in [CEP XXX2 – Wheel conda client support][cep-xxx2]. Motivation, ecosystem history, and rejected alternatives are in [CEP XXX0 – Wheel support in conda (overview)][cep-xxx0]. -We limit the scope to pure Python wheels to avoid platform-specific binary compatibility issues; even so, publishing wheels in repodata greatly increases the packages channels can expose without building conda recipes for every case. +We limit the scope to pure Python wheels to avoid platform-specific binary compatibility issues. Even so, publishing wheels in repodata greatly increases the packages channels can expose without building conda recipes for every case. ## Relationship to companion CEPs - **[CEP XXX0 – Wheel support in conda (overview)][cep-xxx0]** — Why native wheel support, historical context, and discussion of rejected ideas (informative). -- **[CEP XXX2 – Wheel conda client support][cep-xxx2]** — Normative client behavior: loading `whl` into the solve, exclusivity and preference, download, and installation into the prefix. +- **[CEP XXX2 – Wheel conda client support][cep-xxx2]** — Normative client behavior: loading `whl` into the solve with other repodata records, download, and installation. ## Specification @@ -122,7 +122,9 @@ Wheels that fail any of these checks MUST NOT be added to `whl`. ### Dependency conversion -Wheel dependencies (from METADATA file's Requires-Dist entries) MUST be converted to conda format following these rules: +Channel operators MUST convert Python dependency declarations—`Requires-Dist`, `Requires-Python`, and extras—into conda `depends`, **`extra_depends`** when applicable, and related fields, following the rules below. + +Those declarations MAY be read from the wheel's embedded **`METADATA`** and/or from the [**PyPI JSON API**][pypi-json-api] release payload for the same **project** and **version** as the wheel being indexed. Using the API can simplify batch indexing and mirrors. Regardless of source, the same conversion rules and interoperable minimum in [PEP 508 marker translation guidance](#pep-508-marker-translation-guidance) apply. - **Package names:** Names per [CEP 26][cep-26] and match existing conda-forge package names where they exist - **Version specifiers:** Map PEP 440 [version specifiers][version-specifiers] to conda format: @@ -161,7 +163,7 @@ Resulting conda record: ### PEP 508 marker translation guidance -This subsection layers **interoperable expectations** for the wire format, **non-normative** reference material for richer mappings, and **lossy translation** policy. It does not duplicate a full PEP 508 variable matrix in this CEP; maintaining that table is left to living documentation and tooling (see below). +This subsection layers interoperable expectations for the repodata format, non-normative reference material for richer mappings, and a general translation policy. #### Interoperable minimum @@ -180,7 +182,7 @@ Per-variable translation of other PEP 508 markers (for example `sys_platform`, ` #### Lossy translation When a channel cannot encode a PEP 508 dimension in repodata (for example no virtual package for an architecture string, or unsupported boolean branches), the operator MAY **omit** the condition, **simplify** the requirement, or **widen** the dependency (for example listing a package in `depends` without `when=` so it applies in more environments than PEP 508 would). -Channel operators SHOULD document their policy for such cases (including any conservative “include unconditionally” behavior). They MUST NOT claim that published repodata reproduces **full** PEP 508 semantics when a lossy transform was applied. +Channel operators SHOULD document their policy for such cases (including any conservative “include unconditionally” behavior). ### Handling conditional dependencies and extras @@ -188,9 +190,9 @@ Wheel `depends` entries that encode PEP 508 environment markers MUST use conditi PyPI **extras** (optional dependency groups) MUST be represented on the wheel repodata record using an **`extra_depends`** object (a mapping from extra name to lists of dependency strings) as specified in [PR 111][pr-111]. A default install uses only `depends` and MUST NOT union in `extra_depends` entries unless the user selects optional groups (for example with `extras=` in `MatchSpec` as described in [PR 111][pr-111]). -### Solver behavior and package preference +### Solver behavior -Normative rules for how conda clients treat `whl` records during dependency resolution (candidate pool, exclusivity, default preference for conda vs wheel, and user overrides) are defined in [CEP XXX2 – Wheel conda client support][cep-xxx2]. +Normative rules for how conda clients treat `whl` records during dependency resolution (same candidate pool as other repodata records; no CEP-mandated default between conda builds and wheels) are defined in [CEP XXX2 – Wheel conda client support][cep-xxx2]. This CEP only requires that published `whl` records are valid [repodata records][repodata-record-schema] so that clients can consume them alongside `packages` and `packages.conda`. ### Limitations @@ -201,8 +203,9 @@ This CEP has the following known limitations: 2. **Environment markers (publisher defaults):** Default conversion rules in this CEP focus on Python-version markers on `Requires-Dist`, mapped to `when=` per [PR 111][pr-111]. Other markers (for example `sys_platform`) are out of scope for those default publisher rules and are not required to be converted into repodata here; see [PEP 508 marker translation guidance](#pep-508-marker-translation-guidance). How conda clients evaluate `when=` and optional groups at solve time (including environment context) is specified in [CEP XXX2][cep-xxx2] together with [PR 111][pr-111]. 3. **Conditionals and extras:** Normative syntax and record fields for `when=` on `depends` and for optional groups in **`extra_depends`** are specified in [PR 111][pr-111], on which this repodata CEP and [CEP XXX2][cep-xxx2] rely for PyPI-aligned conditionals and extras in published records and at client solve time, respectively. -4. **Repodata size:** Supporting a significant portion of pure Python packages from PyPI (potentially hundreds of thousands of packages with multiple versions each) will substantially increase repodata size. Channels adopting wheel support at scale will need to implement sharded repodata ([CEP 16][cep-16]) to maintain acceptable performance. Alternatively, channels may choose to curate a subset of popular or requested packages rather than mirroring all of PyPI. -5. **PyPI package deletion:** PyPI allows package maintainers to delete packages (as opposed to just yanking them), which can break locked environments that reference those packages. Channels using external PyPI URLs directly are subject to this risk. For production use and reproducible environments, channels MAY mirror and store wheel artifacts locally rather than relying solely on external PyPI URLs. +4. **Repodata size:** Supporting a significant portion of pure Python packages from PyPI (potentially hundreds of thousands of packages with multiple versions each) will substantially increase repodata size. Channels adopting wheel support at scale SHOULD implement sharded repodata ([CEP 16][cep-16]) to maintain acceptable performance. +5. **PyPI package deletion:** PyPI allows package maintainers to delete releases (as opposed to only yanking them). Releases may also be removed when classified as **malicious** or otherwise pulled by PyPI administrators. Any removal may break locked environments that reference those artifacts. +Channels using external PyPI URLs directly are subject to this risk. For production use and reproducible environments, channels MAY mirror and store wheel artifacts locally rather than relying solely on external PyPI URLs. ## Rationale @@ -226,13 +229,14 @@ Several factors can cause wheel names to differ from conda-style names: ### For conda clients -Client behavior (loading `whl` into the solve, preference and exclusivity, download, and installation into the environment prefix) is specified in [CEP XXX2 – Wheel conda client support][cep-xxx2]. Channel operators need not implement that CEP; they only publish repodata that conforms to this document. +Client behavior (loading `whl` into the solve with other repodata records, download, and installation into the environment prefix) is specified in [CEP XXX2 – Wheel conda client support][cep-xxx2]. Channel operators need not implement that CEP; they only publish repodata that conforms to this document. ### For channel operators Channel operators adding wheel support SHOULD: - Implement validation to ensure only pure Python wheels are included +- When using the [PyPI JSON API][pypi-json-api] for dependency metadata, consider comparing against the wheel's `METADATA` and fall back (or merge) when the API response is incomplete relative to the artifact - Ensure that dependencies are solvable, including that compiled dependencies exist on the conda channel declared as `info.channel_relations.base` when wheels depend on conda packages from that stack - Maintain a mapping of PyPI to conda-style names for their channel, or declare `channel_relations.base` so clients load a base channel that defines those mappings (see [Naming standard and channel mapping](#naming-standard-and-channel-mapping) and [Channel relations in repodata][cep-channel-relations]) - Consider automation to keep the repodata up to date with newer releases on PyPI @@ -424,6 +428,7 @@ The package record expresses the conditional with `when=` so `typing_extensions` - [PR 151 – URL field for package records][pr-151] - [PR 111 – Conditional dependencies and optional groups][pr-111] - [conda-pupa][conda-pupa] +- [PyPI JSON API documentation][pypi-json-api] ## Copyright @@ -443,5 +448,6 @@ All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdo [pr-151]: https://github.com/conda/ceps/pull/151 [pr-111]: https://github.com/conda/ceps/pull/111 [conda-pupa]: https://github.com/dholth/conda-pupa +[pypi-json-api]: https://docs.pypi.org/api/json/ [cep-xxx0]: cep-XXX0.md [cep-xxx2]: cep-XXX2.md diff --git a/cep-XXX2.md b/cep-XXX2.md index 9d6a4c7c..440c0866 100644 --- a/cep-XXX2.md +++ b/cep-XXX2.md @@ -20,8 +20,8 @@ described in [RFC2119][RFC2119] when, and only when, they appear in all capitals ## Abstract -This CEP specifies how conda clients consume wheel entries published in repodata: loading `v{revision}.whl` into the package index used by the solver, applying exclusivity and preference rules, evaluating conditional dependencies and extras per [PR 111][pr-111], downloading wheel artifacts, and integrating them into the environment **as equivalent `noarch: python` conda packages** (layout, package database, and lifecycle), not as a separate pip-style install path. -The shape of the `whl` index and publisher-side `METADATA` conversion are defined in [CEP XXX1 – Repodata wheel support][cep-xxx1]. This CEP applies to pure Python wheels in the sense of that document (e.g. `py3-none-any`); it does not define binary wheel support. +This CEP specifies how conda clients consume wheel entries published in repodata: loading `v{revision}.whl` into the same index as other repodata records, evaluating conditional dependencies and extras per [PR 111][pr-111], downloading wheel artifacts, and integrating them into the environment **as equivalent `noarch: python` conda packages** (layout, package database, and lifecycle). +The shape of the `whl` index and publisher-side `METADATA` conversion are defined in [CEP XXX1 – Repodata wheel support][cep-xxx1]. Background on **why** channels and clients pursue native wheel support, plus historical context and rejected ecosystem alternatives, is in [CEP XXX0 – Wheel support in conda (overview)][cep-xxx0]. ## Motivation @@ -37,27 +37,13 @@ This CEP assumes repodata that conforms to [CEP XXX1 – Repodata wheel support] ### Repodata consumption -Conda clients that implement wheel support MUST merge entries from the `whl` mapping in each supported [repodata revision](https://github.com/conda/ceps/pull/146) with candidates derived from `packages` and `packages.conda` for the same revision, so that a single index feeds dependency resolution. Clients SHOULD apply the same subdir and channel filtering hooks used for other repodata records unless a separate policy is explicitly documented. +Conda clients that implement wheel support MUST load records from the `whl` mapping in each supported [repodata revision](https://github.com/conda/ceps/pull/146) into the same index used for `packages` and `packages.conda` for that revision. Clients SHOULD apply the same subdir and channel filtering hooks used for other repodata records unless a separate policy is explicitly documented. -Wheel records for pure Python packages use `subdir: "noarch"` per [CEP XXX1][cep-xxx1]; no additional platform filter is required for those wheels beyond the normal noarch and virtual-package logic for the target prefix. +Wheel records for pure Python packages use `subdir: "noarch"` per [CEP XXX1][cep-xxx1] and no additional platform filter is required for those wheels beyond the normal noarch and virtual-package logic for the target prefix. -### Solver: dependency resolution and package preference +### Solver: dependency resolution -Solvers MUST treat pure Python wheel records in `whl` as valid package candidates during dependency resolution with these constraints: - -- **Exclusivity:** Solvers MUST NOT install both a conda package and a wheel for the same [package name][cep-26] (as used in `MatchSpec` / the solver index). -- **Dependency satisfaction:** When a wheel is selected, its `depends` list MUST be satisfied the same as for any other package record. Dependencies in `depends` MAY be satisfied by either wheel or conda package records. -- **Platform matching:** For wheel records that conform to the pure-Python rules in [CEP XXX1][cep-xxx1] (`noarch: python` and the stated tag checks), no extra platform tag filtering of the wheel artifact is required, because channels only add that class of wheel to `whl` under those validation rules. - -#### User control of precedence - -By default, when both a conda package and a wheel are available for the same package name, the solver SHOULD prefer the conda package unless channel priority or an explicit spec selects the wheel. Users MAY override default precedence through: - -- Channel priority configuration (for example, preferring channels that publish wheels in `whl`); -- Explicit requests using `::` (or other channel-pinned syntax supported by the client); -- Client configuration (for example, flags or settings such as `prefer_conda` or `prefer_wheel`) where the implementation documents behavior clearly. - -The exact default when priorities tie is a client quality-of-implementation choice but MUST remain consistent with **Exclusivity** above (never two artifacts for the same name). +Wheel records are normal [repodata records][repodata-record-schema] and solvers treat them like other index entries for dependency resolution, `depends`, and (where applicable) subdir / virtual-package filtering—the same model as for `packages` and `packages.conda`. ### Conditional dependencies and extras @@ -91,9 +77,9 @@ Wheels selected from repodata MUST be integrated into the target environment **a Importable code MUST end up under the environment's [site-packages][cep-20] (or the site-packages / path layout for `noarch: python` described in that package’s `info/index.json` and related files per [CEP 34][cep-34]). Headers, data, and other install-tree paths from the wheel MUST be placed under prefix locations that are tracked and removed the same way as in a built conda package. The repodata record points at a wheel file as the artifact to obtain; the **normative** requirement is conda-equivalent integration as in the preceding paragraph, not a wheel-only layout profile considered in isolation from conda's package model. -- **Metadata:** The install MUST create a valid `.dist-info` (or legacy `.egg-info` if present in the wheel) so `importlib.metadata` can load the distribution, consistent with other Python packages in the environment. -- **RECORD:** The client SHOULD verify the wheel's `RECORD` where present to detect tampering or corruption; it MUST reject path entries that escape the intended prefix or package staging area. -- **Console and GUI entry points:** Application entry points SHOULD be materialized like other conda `noarch: python` packages: through `info/link.json` and conda's link machinery per [CEP 20][cep-20] and [CEP 34][cep-34], so shebangs and `bin` scripts remain under conda's control. A client implementation MAY avoid emitting duplicate script files from the wheel when `link.json` (or an equivalent) defines the public commands. +- **Metadata:** The install MUST create a valid `.dist-info` so `importlib.metadata` can load the distribution, consistent with other Python packages in the environment. +- **RECORD:** The client SHOULD verify the wheel's `RECORD` where present to detect tampering or corruption and it MUST reject path entries that escape the intended prefix or package staging area. +- **Console and GUI entry points:** Application entry points SHOULD be materialized like other conda `noarch: python` packages: through `info/link.json` and conda's link machinery per [CEP 20][cep-20] and [CEP 34][cep-34], so shebangs and `bin` scripts remain under the conda client's control. A client implementation MAY avoid emitting duplicate script files from the wheel when `link.json` (or an equivalent) defines the public commands. ## Rationale @@ -101,35 +87,29 @@ The repodata record points at a wheel file as the artifact to obtain; the **norm Native wheel support in the client avoids separate PyPI or pip phases for the packages indexed in [CEP XXX1][cep-xxx1], matches user expectations for where importable code lives, and reuses conda's environment model. Keeping download rules aligned with [PR 151][pr-151] matches existing channel URL behavior. -### Native unpack vs always converting to conda packages +### Native unpack -On-the-fly conversion to a `.conda` artifact can still be a useful optimization or fallback (for example, sharing on a local channel). Either way, the user-visible result remains a normal conda-managed install. -This CEP requires that materializing the environment from a **wheel file** (without requiring a pre-built conda package on the channel) be possible so channels do not need a second hosting pipeline for the same file PyPI already ships as a wheel, and to avoid inflating storage with redundant formats for pure-Python content. +Conda clients install from the `.whl` artifact referenced in repodata and lays out the environment as a normal conda-managed `noarch: python` install (see [Installation layout and prefix placement](#installation-layout-and-prefix-placement)). ## Implementation Notes ### For conda client implementers -- **Index:** Parse `whl` inside each supported `v{revision}` object alongside `packages` / `packages.conda` and feed merged candidates to the solver. -- **PR 111:** Implement evaluation of `when=` and merging of `extra_depends` when the user requests extras, consistent with the published record strings. +- **Index:** Parse `whl` inside each supported `v{revision}` object alongside `packages` / `packages.conda` into the same solver-facing index. +- **Conditional dependencies and extras ([PR 111][pr-111]):** Evaluate `when=` on `depends` and honor **`extra_depends`** when the user requests optional groups, consistent with the published record strings (wheel rows use the same syntax as other PR 111 records). - **Revision gating:** Treat new-syntax wheel records with the same `v{revision}` handling as the rest of the PR 146 strategy (do not expect older repodata only clients to read `whl`). - **Download:** Reuse the channel and fetch stack used for other artifacts, including `url` and checksum verification. -- **Install:** Integrate wheels using the same prefix layout and metadata model as `noarch: python` conda packages ([CEP 20][cep-20], [CEP 34][cep-34]), including `info/link.json` for entry points when using conda's link pipeline. The [conda-pypi][conda-pypi] project demonstrates populating a conda package shape from a wheel (for example, suppressing duplicate `bin` script generation when `link.json` is used) as **non-normative** prior art. -- **Testing:** Add integration tests for exclusivity, preference, conditional edges, a wheel-only dependency, and a conda–wheel mixed graph. +- **Install:** Integrate wheels using the same prefix layout and metadata model as `noarch: python` conda packages ([CEP 20][cep-20], [CEP 34][cep-34]), including `info/link.json` for entry points when using conda's link pipeline. The [conda-pypi][conda-pypi] project demonstrates populating a conda package shape from a wheel (for example, suppressing duplicate `bin` script generation when `link.json` is used) as a non-normative example. ## Examples -### Solver constraints (illustrative) - -For a request that could be satisfied by either a conda `requests` or a wheel in `whl` with the same name, a configuration that prefers conda SHOULD select the conda build unless the user pins the wheel channel or package record. - ### Install outcome (illustrative) After installing `requests-2.32.5-py3-none-any.whl` from a wheel repodata record, the environment’s site-packages should contain the `requests` import package and a `requests-2.32.5.dist-info` (or equivalent) directory such that `import requests` and `importlib.metadata.version("requests")` succeed for that prefix’s Python. ## Backwards compatibility -- **Repodata shape:** The `whl` key appears only under a registered `v{revision}` (see [PR 146](https://github.com/conda/ceps/pull/146)). Conda clients that do not support that revision or the `whl` index MUST ignore unknown `v{revision}` members or fall back to earlier revisions per channel policy, and MUST NOT break on the presence of `whl` in repodata for revisions they do not load. +- **Repodata shape:** The `whl` key appears only under a registered `v{revision}` (see [PR 146](https://github.com/conda/ceps/pull/146)). Conda clients that do not support that revision or the `whl` index MUST ignore unknown `v{revision}` members or fall back to earlier revisions per channel policy. - **Record syntax:** Records that use `when=` and `extra_depends` are part of the new-syntax repodata world [PR 111][pr-111] already depends on; wheel rows are not a separate syntax fork. ## Rejected ideas From e91cca1d5d9850ef96b5832dde759df723ecda62 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Fri, 24 Apr 2026 19:22:32 -0400 Subject: [PATCH 34/38] More corrections for accuracy --- cep-XXX0.md | 6 +++--- cep-XXX1.md | 9 ++++----- cep-XXX2.md | 2 +- 3 files changed, 8 insertions(+), 9 deletions(-) diff --git a/cep-XXX0.md b/cep-XXX0.md index 4bdc99b7..cd4d89a6 100644 --- a/cep-XXX0.md +++ b/cep-XXX0.md @@ -30,7 +30,7 @@ Together, the three documents describe **native** wheel support: wheels are firs ## Motivation -While conda remains a language-agnostic packaging distribution, installing packages for use with Python represents the majority of the current conda usage. Users frequently encounter packages only available as wheels on PyPI, forcing them to use hybrid workflows that mix conda and PyPI installations. This approach creates several problems: +While conda remains a language-agnostic package manager, installing packages for use with Python represents the majority of the current conda usage. Users frequently encounter packages only available as wheels on PyPI, forcing them to use hybrid workflows that mix conda and PyPI installations. This approach creates several problems: - Mixing of conda and pip or uv can result in overwritten files and broken environments - Users must understand two package managers, their interaction points, and which of their dependencies are available from which ecosystem @@ -115,7 +115,7 @@ Pixi changes course to use uv directly instead of rip, which unlocks features li ### conda-pupa creates on-the-fly conversion plugin (July 2024) -Daniel Holth creates a conda plugin that supports on-the-fly conversion of conda packages to wheels in a local channel. +Daniel Holth creates a conda plugin that supports on-the-fly conversion of PyPI wheels to conda packages in a local channel. ### conda-whl-channel creates proof-of-concept wheel channel from repodata (Nov 2024) @@ -127,7 +127,7 @@ conda-pupa is merged into conda-pypi which adds a `conda pypi install ` ### conda-pypi integrates parts of conda-whl-support (Nov 2025) -Conda-pypi incorporates the wheel detection logic from conda-whl-support, providing core functionality beyond the solver and index changes required to support the `whl` section proposed in [CEP XXX1][cep-xxx1], together with channel relation metadata when channels adopt [Channel relations in repodata][cep-channel-relations]. +conda-pypi incorporates the wheel detection logic from conda-whl-support, providing core functionality beyond the solver and index changes required to support the `whl` section proposed in [CEP XXX1][cep-xxx1], together with channel relation metadata when channels adopt [Channel relations in repodata][cep-channel-relations]. ### Native wheel unpack in conda-pypi; Rattler reads wheels in repodata (Dec 2025) diff --git a/cep-XXX1.md b/cep-XXX1.md index 240aeee0..1def0eac 100644 --- a/cep-XXX1.md +++ b/cep-XXX1.md @@ -113,7 +113,7 @@ When populating repodata records for pure Python wheels: Before adding a wheel to `whl`, channel operators MUST verify: -- The wheel's platform tag is `any` (e.g., `py3-none-any`, `py2.py3-none-any`) +- The wheel's platform tag is `any` (i.e., wheel filenames ending in `-none-any.whl`, such as `requests-2.32.5-py3-none-any.whl`) - The wheel's ABI tag is `none` - The wheel contains no compiled extensions (`.so`, `.pyd`, `.dylib` files) - The wheel's `METADATA` file is present and valid @@ -132,7 +132,6 @@ Those declarations MAY be read from the wheel's embedded **`METADATA`** and/or f - All other PEP 440 operators (`>=`, `<=`, `<`, `>`, `~=`, `!=`) are used as-is - Multiple version specifiers are combined with commas (e.g., `>=1.0,<2.0`) -- **Multiple specifiers:** Combine with commas (e.g., >=1.0,<2.0) - **Python version requirements:** Convert Requires-Python to explicit python dependency - **Environment markers:** Map Python-version markers on `Requires-Dist` to conditional `MatchSpec` `when=` as specified in [PR 111][pr-111]. Other markers are out of scope for the default conversion rules in this CEP (see [Limitations](#limitations)). @@ -144,7 +143,7 @@ Requires-Python: >=3.8 Requires-Dist: requests (>=2.20.0,<3.0.0) Requires-Dist: click (>=7.0) Requires-Dist: numpy (>=1.20.0,!=1.24.0) -Requires-Dist: importlib-metadata (>=1.0) ; python_version < '3.8' +Requires-Dist: importlib-metadata (>=1.0) ; python_version < '3.10' ``` Resulting conda record: @@ -156,7 +155,7 @@ Resulting conda record: "requests >=2.20.0,<3.0", "click >=7.0", "numpy >=1.20.0,!=1.24.0", - "importlib-metadata>=1.0[when=\"python<3.8\"]" + "importlib-metadata>=1.0[when=\"python<3.10\"]" ] } ``` @@ -264,7 +263,7 @@ A complete channel index also includes the traditional top-level keys (`repodata ### Download wheels from the default location -Below represents the default behavior and shows when the `url` field is not set: +Below represents the default behavior when the `url` field is omitted: ```json { diff --git a/cep-XXX2.md b/cep-XXX2.md index 440c0866..dba4d7ea 100644 --- a/cep-XXX2.md +++ b/cep-XXX2.md @@ -89,7 +89,7 @@ Native wheel support in the client avoids separate PyPI or pip phases for the pa ### Native unpack -Conda clients install from the `.whl` artifact referenced in repodata and lays out the environment as a normal conda-managed `noarch: python` install (see [Installation layout and prefix placement](#installation-layout-and-prefix-placement)). +Conda clients install from the `.whl` artifact referenced in repodata and lay out the environment as a normal conda-managed `noarch: python` install (see [Installation layout and prefix placement](#installation-layout-and-prefix-placement)). ## Implementation Notes From e0f955eb5d09e7ffaa08cc2186a5e72db72453ae Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Mon, 27 Apr 2026 09:56:28 -0400 Subject: [PATCH 35/38] Update conditional and extras CEP PR links --- cep-XXX1.md | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/cep-XXX1.md b/cep-XXX1.md index 1def0eac..902704f7 100644 --- a/cep-XXX1.md +++ b/cep-XXX1.md @@ -11,7 +11,7 @@ Updated Apr 24, 2026 Discussion https://github.com/conda/ceps/pull/145 Implementation TBD - Requires https://github.com/conda/ceps/pull/151 https://github.com/conda/ceps/pull/146 https://github.com/conda/ceps/pull/155 https://github.com/conda/ceps/pull/111 [CEP XXX2 (Wheel conda client support)](cep-XXX2.md) + Requires https://github.com/conda/ceps/pull/151 https://github.com/conda/ceps/pull/146 https://github.com/conda/ceps/pull/155 https://github.com/conda/ceps/pull/164 https://github.com/conda/ceps/pull/165 [CEP XXX2 (Wheel conda client support)](cep-XXX2.md) See also [CEP XXX0 – Wheel support in conda (overview)](cep-XXX0.md) @@ -57,7 +57,7 @@ The `whl` dictionary maps conda-like filenames to repodata records. The key MUST - `python` dependency from `Requires-Python` (if present), converted to conda format - All `Requires-Dist` entries from METADATA, converted from PEP 440 to conda format per [Dependency conversion](#dependency-conversion) - Package names normalized to conda-style names per [CEP 26][cep-26] -- **`extra_depends`**: MAY be present. When present, MUST be an object mapping extra names to lists of dependency strings for optional groups, per [PR 111][pr-111]. When absent or empty, the record declares no optional groups beyond `depends`. +- **`extra_depends`**: MAY be present. When present, MUST be an object mapping extra names to lists of dependency strings for optional groups, per [PR 165][pr-165]. When absent or empty, the record declares no optional groups beyond `depends`. - **`subdir`**: MUST be `"noarch"`. - **`noarch`**: MUST be `"python"`. - **`url`**: MAY be present. Semantics are defined in [PR 151][pr-151] (see [Wheel download URLs](#wheel-download-urls)). @@ -133,7 +133,7 @@ Those declarations MAY be read from the wheel's embedded **`METADATA`** and/or f - Multiple version specifiers are combined with commas (e.g., `>=1.0,<2.0`) - **Python version requirements:** Convert Requires-Python to explicit python dependency -- **Environment markers:** Map Python-version markers on `Requires-Dist` to conditional `MatchSpec` `when=` as specified in [PR 111][pr-111]. Other markers are out of scope for the default conversion rules in this CEP (see [Limitations](#limitations)). +- **Environment markers:** Map Python-version markers on `Requires-Dist` to conditional `MatchSpec` `when=` as specified in [PR 164][pr-164]. Other markers are out of scope for the default conversion rules in this CEP (see [Limitations](#limitations)). Example conversion: @@ -166,11 +166,11 @@ This subsection layers interoperable expectations for the repodata format, non-n #### Interoperable minimum -For published `whl` records, the following conversions are the **normative floor** aligned with [Dependency conversion](#dependency-conversion) and [PR 111][pr-111]: +For published `whl` records, the following conversions are the **normative floor** aligned with [Dependency conversion](#dependency-conversion) and [PR 164][pr-164] and [PR 165][pr-165]: - **`Requires-Python`:** MUST appear as an explicit `python` dependency in `depends`. -- **Python-version conditions on `Requires-Dist`:** MUST be represented with conditional `MatchSpec` `when=` on the relevant dependency strings per [PR 111][pr-111]. -- **`extra` (optional groups):** MUST be represented with **`extra_depends`** and optional-group selection per [PR 111][pr-111], not only as opaque marker text on `depends`. +- **Python-version conditions on `Requires-Dist`:** MUST be represented with conditional `MatchSpec` `when=` on the relevant dependency strings per [PR 164][pr-164]. +- **`extra` (optional groups):** MUST be represented with **`extra_depends`** and optional-group selection per [PR 165][pr-165], not only as opaque marker text on `depends`. Channels and tools MAY apply additional, stricter mappings where they can express them in repodata. @@ -185,9 +185,9 @@ Channel operators SHOULD document their policy for such cases (including any con ### Handling conditional dependencies and extras -Wheel `depends` entries that encode PEP 508 environment markers MUST use conditional `MatchSpec` syntax with `when=` as specified in [PR 111][pr-111]. +Wheel `depends` entries that encode PEP 508 environment markers MUST use conditional `MatchSpec` syntax with `when=` as specified in [PR 164][pr-164]. -PyPI **extras** (optional dependency groups) MUST be represented on the wheel repodata record using an **`extra_depends`** object (a mapping from extra name to lists of dependency strings) as specified in [PR 111][pr-111]. A default install uses only `depends` and MUST NOT union in `extra_depends` entries unless the user selects optional groups (for example with `extras=` in `MatchSpec` as described in [PR 111][pr-111]). +PyPI **extras** (optional dependency groups) MUST be represented on the wheel repodata record using an **`extra_depends`** object (a mapping from extra name to lists of dependency strings) as specified in [PR 165][pr-165]. A default install uses only `depends` and MUST NOT union in `extra_depends` entries unless the user selects optional groups (for example with `extras=` in `MatchSpec` as described in [PR 165][pr-165]). ### Solver behavior @@ -199,9 +199,9 @@ This CEP only requires that published `whl` records are valid [repodata records] This CEP has the following known limitations: 1. **Pure Python only:** This CEP explicitly does not address wheels with binary extensions, which require platform-specific compatibility guarantees beyond the current scope. Conda’s strength is binary compatibility, so using conda packages may be the optimal solution. -2. **Environment markers (publisher defaults):** Default conversion rules in this CEP focus on Python-version markers on `Requires-Dist`, mapped to `when=` per [PR 111][pr-111]. Other markers (for example `sys_platform`) are out of scope for those default publisher rules and are not required to be converted into repodata here; see [PEP 508 marker translation guidance](#pep-508-marker-translation-guidance). -How conda clients evaluate `when=` and optional groups at solve time (including environment context) is specified in [CEP XXX2][cep-xxx2] together with [PR 111][pr-111]. -3. **Conditionals and extras:** Normative syntax and record fields for `when=` on `depends` and for optional groups in **`extra_depends`** are specified in [PR 111][pr-111], on which this repodata CEP and [CEP XXX2][cep-xxx2] rely for PyPI-aligned conditionals and extras in published records and at client solve time, respectively. +2. **Environment markers (publisher defaults):** Default conversion rules in this CEP focus on Python-version markers on `Requires-Dist`, mapped to `when=` per [PR 164][pr-164]. Other markers (for example `sys_platform`) are out of scope for those default publisher rules and are not required to be converted into repodata here; see [PEP 508 marker translation guidance](#pep-508-marker-translation-guidance). +How conda clients evaluate `when=` and optional groups at solve time (including environment context) is specified in [CEP XXX2][cep-xxx2] together with [PR 164][pr-164] and [PR 165][pr-165]. +3. **Conditionals and extras:** Normative syntax and record fields for `when=` on `depends` and for optional groups in **`extra_depends`** are specified in [PR 164][pr-164] and [PR 165][pr-165], on which this repodata CEP and [CEP XXX2][cep-xxx2] rely for PyPI-aligned conditionals and extras in published records and at client solve time, respectively. 4. **Repodata size:** Supporting a significant portion of pure Python packages from PyPI (potentially hundreds of thousands of packages with multiple versions each) will substantially increase repodata size. Channels adopting wheel support at scale SHOULD implement sharded repodata ([CEP 16][cep-16]) to maintain acceptable performance. 5. **PyPI package deletion:** PyPI allows package maintainers to delete releases (as opposed to only yanking them). Releases may also be removed when classified as **malicious** or otherwise pulled by PyPI administrators. Any removal may break locked environments that reference those artifacts. Channels using external PyPI URLs directly are subject to this risk. For production use and reproducible environments, channels MAY mirror and store wheel artifacts locally rather than relying solely on external PyPI URLs. @@ -414,7 +414,7 @@ Requires-Python: >=3.8 Requires-Dist: typing-extensions>=4.0.0; python_version < '3.9' ``` -The package record expresses the conditional with `when=` so `typing_extensions` applies only when the Python version is less than 3.9, matching the original wheel METADATA and [PR 111][pr-111]. The Python version constraint of >=3.8 is directly mapped. +The package record expresses the conditional with `when=` so `typing_extensions` applies only when the Python version is less than 3.9, matching the original wheel METADATA and [PR 164][pr-164]. The Python version constraint of >=3.8 is directly mapped. ## References @@ -425,7 +425,8 @@ The package record expresses the conditional with `when=` so `typing_extensions` - [Example `repodata.json` (conda-pypi test channel)][conda-pypi-example-repodata] - [Channel relations in repodata (PR 155)][cep-channel-relations] - [PR 151 – URL field for package records][pr-151] -- [PR 111 – Conditional dependencies and optional groups][pr-111] +- [PR 164 – Conditional dependencies][pr-164] +- [PR 165 – Optional dependency groups][pr-165] - [conda-pupa][conda-pupa] - [PyPI JSON API documentation][pypi-json-api] @@ -445,7 +446,8 @@ All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdo [conda-pypi-example-repodata]: https://github.com/conda-incubator/conda-pypi/blob/main/tests/conda_local_channel/noarch/repodata.json [cep-channel-relations]: https://github.com/conda/ceps/pull/155 [pr-151]: https://github.com/conda/ceps/pull/151 -[pr-111]: https://github.com/conda/ceps/pull/111 +[pr-164]: https://github.com/conda/ceps/pull/164 +[pr-165]: https://github.com/conda/ceps/pull/165 [conda-pupa]: https://github.com/dholth/conda-pupa [pypi-json-api]: https://docs.pypi.org/api/json/ [cep-xxx0]: cep-XXX0.md From 4c208ee4dbabaa7a1917570d1dde20f7994e6181 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Mon, 27 Apr 2026 10:02:14 -0400 Subject: [PATCH 36/38] Add reference to conda-forge issue #26 --- cep-XXX0.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/cep-XXX0.md b/cep-XXX0.md index cd4d89a6..04b8a294 100644 --- a/cep-XXX0.md +++ b/cep-XXX0.md @@ -68,6 +68,8 @@ Two competing philosophies emerged on how or if we should provide better integra Issue [#262](https://github.com/conda/conda/issues/262) (2013): conda should install directly from PyPI to reduce duplication and improve package availability. +conda-forge issue [#28](https://github.com/conda-forge/conda-forge.github.io/issues/28) (Feb 2016): Chris Barker raises the pure-Python package gap from the conda-forge community's perspective, proposing either a real-time PyPI bridge (on-the-fly conda package generation from PyPI) or an automated conda skeleton farm to mirror popular PyPI packages with version tracking. + Arguments for this approach: - Users frequently need packages not available on conda channels From 7c63b45f23ef6605125c168c62de91fab2bf7287 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Fri, 31 Jul 2026 09:34:57 -0400 Subject: [PATCH 37/38] Make remote URLs out of scope of wheels CEP --- cep-XXX0.md | 2 +- cep-XXX1.md | 101 +++++++--------------------------------------------- cep-XXX2.md | 13 ++++--- 3 files changed, 19 insertions(+), 97 deletions(-) diff --git a/cep-XXX0.md b/cep-XXX0.md index 04b8a294..7cc0c985 100644 --- a/cep-XXX0.md +++ b/cep-XXX0.md @@ -231,7 +231,7 @@ Despite these advantages, this approach was rejected because: - **Infrastructure burden:** Requires significant storage and bandwidth to host and serve converted packages that duplicate PyPI's CDN infrastructure - **Resource inefficiency:** Wheels are already an excellent format for pure Python packages. Conversion adds no technical value and wastes resources -Native wheel support provides the same user experience (transparent PyPI access through channels that publish wheels) while avoiding the infrastructure burden and resource inefficiency of conversion. Channel operators who prefer converted packages can continue building conda packages from PyPI sources. +Native wheel support provides the same user experience (transparent access to pure-Python packages through channels that index and host wheels like other packages) while avoiding the infrastructure burden and resource inefficiency of conversion. Channel operators who prefer converted packages can continue building conda packages from PyPI sources. ## References diff --git a/cep-XXX1.md b/cep-XXX1.md index 902704f7..dc67caca 100644 --- a/cep-XXX1.md +++ b/cep-XXX1.md @@ -11,7 +11,7 @@ Updated Apr 24, 2026 Discussion https://github.com/conda/ceps/pull/145 Implementation TBD - Requires https://github.com/conda/ceps/pull/151 https://github.com/conda/ceps/pull/146 https://github.com/conda/ceps/pull/155 https://github.com/conda/ceps/pull/164 https://github.com/conda/ceps/pull/165 [CEP XXX2 (Wheel conda client support)](cep-XXX2.md) + Requires https://github.com/conda/ceps/pull/146 https://github.com/conda/ceps/pull/155 https://github.com/conda/ceps/pull/164 https://github.com/conda/ceps/pull/165 [CEP XXX2 (Wheel conda client support)](cep-XXX2.md) See also [CEP XXX0 – Wheel support in conda (overview)](cep-XXX0.md) @@ -60,7 +60,6 @@ The `whl` dictionary maps conda-like filenames to repodata records. The key MUST - **`extra_depends`**: MAY be present. When present, MUST be an object mapping extra names to lists of dependency strings for optional groups, per [PR 165][pr-165]. When absent or empty, the record declares no optional groups beyond `depends`. - **`subdir`**: MUST be `"noarch"`. - **`noarch`**: MUST be `"python"`. -- **`url`**: MAY be present. Semantics are defined in [PR 151][pr-151] (see [Wheel download URLs](#wheel-download-urls)). - **`fn`**: MUST be the wheel filename (`.whl`), as in standard conda repodata records (for example `requests-2.32.5-py3-none-any.whl`). - **`sha256`**, **`size`**: Standard repodata fields for the wheel file. @@ -91,12 +90,11 @@ Channel references MUST use relative paths as required by that CEP (for example Channel operators SHOULD document any naming conventions and mappings specific to their channel, including how they relate to their declared `channel_relations`. -### Wheel download URLs +### Wheel download locations -This CEP depends on the optional per-record **`url`** field as specified in [PR 151][pr-151], which enables custom download locations. This is essential for wheel support since PyPI wheels are commonly hosted in per-package subdirectories or served from CDNs. +Wheel artifacts SHALL be fetched like other conda packages: combine the channel location (or the repodata `info.base_url` when present, per [CEP 15][cep-15] / [CEP 16][cep-16]) with `subdir` and `fn`. Channel operators MUST make the `.whl` file available at that URL (typically `{channel}/noarch/{fn}`). -The `whl` mapping (inside the `v{revision}` payload) and the per-record `url` field SHALL follow -the backwards-compatible update strategy ([PR 146](https://github.com/conda/ceps/pull/146)). +The `whl` mapping (inside the `v{revision}` payload) SHALL follow the backwards-compatible update strategy ([PR 146](https://github.com/conda/ceps/pull/146)). ### Wheel-Specific Record Values @@ -107,7 +105,6 @@ When populating repodata records for pure Python wheels: - `subdir`: MUST be "noarch" - `noarch`: MUST be "python" - `fn`: MUST be the wheel filename (`.whl`) -- `url`: MAY be present and follow the semantics in [PR 151][pr-151] and [Wheel download URLs](#wheel-download-urls) ### Pure Python wheel validation @@ -203,8 +200,6 @@ This CEP has the following known limitations: How conda clients evaluate `when=` and optional groups at solve time (including environment context) is specified in [CEP XXX2][cep-xxx2] together with [PR 164][pr-164] and [PR 165][pr-165]. 3. **Conditionals and extras:** Normative syntax and record fields for `when=` on `depends` and for optional groups in **`extra_depends`** are specified in [PR 164][pr-164] and [PR 165][pr-165], on which this repodata CEP and [CEP XXX2][cep-xxx2] rely for PyPI-aligned conditionals and extras in published records and at client solve time, respectively. 4. **Repodata size:** Supporting a significant portion of pure Python packages from PyPI (potentially hundreds of thousands of packages with multiple versions each) will substantially increase repodata size. Channels adopting wheel support at scale SHOULD implement sharded repodata ([CEP 16][cep-16]) to maintain acceptable performance. -5. **PyPI package deletion:** PyPI allows package maintainers to delete releases (as opposed to only yanking them). Releases may also be removed when classified as **malicious** or otherwise pulled by PyPI administrators. Any removal may break locked environments that reference those artifacts. -Channels using external PyPI URLs directly are subject to this risk. For production use and reproducible environments, channels MAY mirror and store wheel artifacts locally rather than relying solely on external PyPI URLs. ## Rationale @@ -240,7 +235,7 @@ Channel operators adding wheel support SHOULD: - Maintain a mapping of PyPI to conda-style names for their channel, or declare `channel_relations.base` so clients load a base channel that defines those mappings (see [Naming standard and channel mapping](#naming-standard-and-channel-mapping) and [Channel relations in repodata][cep-channel-relations]) - Consider automation to keep the repodata up to date with newer releases on PyPI - Document any naming conventions specific to their channel -- For production channels, consider mirroring wheel artifacts locally to ensure reproducibility and protect against PyPI deletions +- Host wheel artifacts on the channel (under `noarch/` by default) so clients can fetch them with standard conda URL construction - Establish patching workflows to correct metadata issues and resolve dependency conflicts - Document patching policies and maintain transparency about modified packages @@ -261,51 +256,14 @@ The JSON fragments below use revision `3` as an example (`v3`). The integer MUST A complete channel index also includes the traditional top-level keys (`repodata_version`, `packages`, `packages.conda`, `removed`, and so on). A full generated example is checked in with [conda-pypi][conda-pypi-example-repodata]. -### Download wheels from the default location +### Channel-hosted wheel (default location) -Below represents the default behavior when the `url` field is omitted: - -```json -{ - "v3": { - "whl": { - "requests-2.32.5-py3_none_any_0": { - "name": "requests", - "version": "2.32.5", - "build": "py3_none_any_0", - "build_number": 0, - "depends": [ - "charset-normalizer <4,>=2", - "idna <4,>=2.5", - "urllib3 <3,>=1.21.1", - "certifi >=2017.4.17", - "python >=3.9" - ], - "fn": "requests-2.32.5-py3-none-any.whl", - "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", - "size": 6899, - "subdir": "noarch", - "noarch": "python", - "url": null - } - } - } -} -``` - -With this configuration, the wheel file will be downloaded from the following location (assuming we are hosting this from `https://repo.example.com/channel`): - -- `https://repo.example.com/channel/noarch/requests-2.32.5-py3-none-any.whl` - -### Downloading wheels from a relative location with `base_url` - -The `url` can also be relative as described above. Here's an example of what that looks like combined with setting the `base_url` property at the top level, and also showing how a channel can declare a base channel for naming and dependencies using `channel_relations` ([Channel relations in repodata][cep-channel-relations]): +A channel-hosted wheel uses standard conda URL construction. The `.whl` file is served under the channel’s `noarch` directory (or under `info.base_url` when set), the same way as other `noarch` artifacts: ```json { "info": { "subdir": "noarch", - "base_url": "https://packages.example.org/wheel-extra/", "channel_relations": { "base": "../core" } @@ -328,49 +286,16 @@ The `url` can also be relative as described above. Here's an example of what tha "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", "size": 6899, "subdir": "noarch", - "noarch": "python", - "url": "requests/requests-2.32.5-py3-none-any.whl" + "noarch": "python" } } } } ``` -This would result in the following being fetched by conda clients: - -- `https://packages.example.org/wheel-extra/noarch/requests/requests-2.32.5-py3-none-any.whl` - -### Downloading wheels from external location - -The following shows an example of using an external location to download the wheel from PyPI's file hosting: +With this configuration, the wheel file is downloaded from (assuming the channel is at `https://repo.example.com/channel`): -```json -{ - "v3": { - "whl": { - "requests-2.32.5-py3_none_any_0": { - "name": "requests", - "version": "2.32.5", - "build": "py3_none_any_0", - "build_number": 0, - "depends": [ - "charset-normalizer <4,>=2", - "idna <4,>=2.5", - "urllib3 <3,>=1.21.1", - "certifi >=2017.4.17", - "python >=3.9" - ], - "fn": "requests-2.32.5-py3-none-any.whl", - "sha256": "78820a3e5d9d3b25ce8e1c99c1c89cd19caa904a92973a3e50f8426009e8a4b3", - "size": 6899, - "subdir": "noarch", - "noarch": "python", - "url": "https://files.pythonhosted.org/packages/1e/db/4254e3eabe8020b458f1a747140d32277ec7a271daf1d235b70dc0b4e6e3/requests-2.32.5-py3-none-any.whl" - } - } - } -} -``` +- `https://repo.example.com/channel/noarch/requests-2.32.5-py3-none-any.whl` ### Name mapping @@ -393,8 +318,7 @@ Here is an example of name mapping and normalization of the record name and depe "sha256": "1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53", "size": 13643, "subdir": "noarch", - "noarch": "python", - "url": "https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl" + "noarch": "python" } } } @@ -424,7 +348,6 @@ The package record expresses the conditional with `when=` so `typing_extensions` - [PEP 508 marker conversion (conda-pypi developer docs)][conda-pypi-marker-conversion] - [Example `repodata.json` (conda-pypi test channel)][conda-pypi-example-repodata] - [Channel relations in repodata (PR 155)][cep-channel-relations] -- [PR 151 – URL field for package records][pr-151] - [PR 164 – Conditional dependencies][pr-164] - [PR 165 – Optional dependency groups][pr-165] - [conda-pupa][conda-pupa] @@ -438,6 +361,7 @@ All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdo [RFC2119]: https://datatracker.ietf.org/doc/html/rfc2119 [repodata-schema]: https://schemas.conda.org/repodata-1.schema.json [repodata-record-schema]: https://schemas.conda.org/repodata-record-1.schema.json +[cep-15]: https://conda.org/learn/ceps/cep-0015 [cep-16]: https://conda.org/learn/ceps/cep-0016 [cep-26]: https://conda.org/learn/ceps/cep-0026 [version-specifiers]: https://packaging.python.org/en/latest/specifications/version-specifiers/#id5 @@ -445,7 +369,6 @@ All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdo [conda-pypi-marker-conversion]: https://conda.github.io/conda-pypi/developer/marker-conversion/#pep-508-variables [conda-pypi-example-repodata]: https://github.com/conda-incubator/conda-pypi/blob/main/tests/conda_local_channel/noarch/repodata.json [cep-channel-relations]: https://github.com/conda/ceps/pull/155 -[pr-151]: https://github.com/conda/ceps/pull/151 [pr-164]: https://github.com/conda/ceps/pull/164 [pr-165]: https://github.com/conda/ceps/pull/165 [conda-pupa]: https://github.com/dholth/conda-pupa diff --git a/cep-XXX2.md b/cep-XXX2.md index dba4d7ea..6aea69b2 100644 --- a/cep-XXX2.md +++ b/cep-XXX2.md @@ -10,7 +10,7 @@ Updated Apr 24, 2026 Discussion https://github.com/conda/ceps/pull/145 Implementation TBD - Requires [CEP XXX1 – Repodata wheel support](cep-XXX1.md) https://github.com/conda/ceps/pull/151 https://github.com/conda/ceps/pull/146 https://github.com/conda/ceps/pull/155 https://github.com/conda/ceps/pull/111 + Requires [CEP XXX1 – Repodata wheel support](cep-XXX1.md) https://github.com/conda/ceps/pull/146 https://github.com/conda/ceps/pull/155 https://github.com/conda/ceps/pull/111 See also [CEP XXX0 – Wheel support in conda (overview)](cep-XXX0.md) @@ -65,9 +65,9 @@ The strings on a wheel’s `depends` and `extra_depends` are produced by channel ### Download -After the solver selects a wheel record, the client MUST download the `.whl` file using the same rules as for other `noarch` artifacts with a `fn` and optional `url`: +After the solver selects a wheel record, the client MUST download the `.whl` file using the same rules as for other `noarch` artifacts with a `fn`: -- Resolve the URL per [PR 151][pr-151] and the download examples in [CEP XXX1][cep-xxx1] (channel `base_url`, default `noarch` layout, or absolute per-record `url`). +- Construct the download URL from the channel location (or repodata `info.base_url` when present) together with `subdir` and `fn`, as for other conda packages. See [Wheel download locations][cep-xxx1-download] in [CEP XXX1][cep-xxx1]. - Verify integrity using `sha256` and `size` from the repodata record when those fields are present, consistent with other conda downloads. ### Installation layout and prefix placement @@ -85,7 +85,7 @@ The repodata record points at a wheel file as the artifact to obtain; the **norm ### One solve, predictable layout -Native wheel support in the client avoids separate PyPI or pip phases for the packages indexed in [CEP XXX1][cep-xxx1], matches user expectations for where importable code lives, and reuses conda's environment model. Keeping download rules aligned with [PR 151][pr-151] matches existing channel URL behavior. +Native wheel support in the client avoids separate PyPI or pip phases for the packages indexed in [CEP XXX1][cep-xxx1], matches user expectations for where importable code lives, and reuses conda's environment model and fetch stack. ### Native unpack @@ -98,7 +98,7 @@ Conda clients install from the `.whl` artifact referenced in repodata and lay ou - **Index:** Parse `whl` inside each supported `v{revision}` object alongside `packages` / `packages.conda` into the same solver-facing index. - **Conditional dependencies and extras ([PR 111][pr-111]):** Evaluate `when=` on `depends` and honor **`extra_depends`** when the user requests optional groups, consistent with the published record strings (wheel rows use the same syntax as other PR 111 records). - **Revision gating:** Treat new-syntax wheel records with the same `v{revision}` handling as the rest of the PR 146 strategy (do not expect older repodata only clients to read `whl`). -- **Download:** Reuse the channel and fetch stack used for other artifacts, including `url` and checksum verification. +- **Download:** Reuse the channel and fetch stack used for other artifacts (`fn`, channel/`base_url`, and checksum verification). - **Install:** Integrate wheels using the same prefix layout and metadata model as `noarch: python` conda packages ([CEP 20][cep-20], [CEP 34][cep-34]), including `info/link.json` for entry points when using conda's link pipeline. The [conda-pypi][conda-pypi] project demonstrates populating a conda package shape from a wheel (for example, suppressing duplicate `bin` script generation when `link.json` is used) as a non-normative example. ## Examples @@ -124,7 +124,6 @@ Ecosystem-level alternatives (relying on pip only, on-the-fly PyPI without repod - [CEP 34 – Package metadata and `info/`](./cep-0034.md) (`index.json`, `link.json`, etc.) - [CEP 26 – Identifying packages and channels][cep-26] - [PR 111 – Conditional dependencies and optional groups][pr-111] -- [PR 151 – URL field for package records][pr-151] - [PEP 508 marker conversion (conda-pypi developer docs)][conda-pypi-marker-conversion] - [Example `repodata.json` (conda-pypi test channel)][conda-pypi-example-repodata] - [conda-pypi project][conda-pypi] @@ -144,10 +143,10 @@ All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdo [cep-34]: ./cep-0034.md [cep-xxx0]: cep-XXX0.md [cep-xxx1]: cep-XXX1.md +[cep-xxx1-download]: cep-XXX1.md#wheel-download-locations [conda-pypi]: https://github.com/conda-incubator/conda-pypi [conda-pypi-marker-conversion]: https://conda.github.io/conda-pypi/developer/marker-conversion/#pep-508-variables [conda-pypi-example-repodata]: https://github.com/conda-incubator/conda-pypi/blob/main/tests/conda_local_channel/noarch/repodata.json -[pr-151]: https://github.com/conda/ceps/pull/151 [pr-111]: https://github.com/conda/ceps/pull/111 [conda-pupa]: https://github.com/dholth/conda-pupa [uv-in-pixi]: https://prefix.dev/blog/uv_in_pixi From 566f09ebd389a5e9c74ec3fa4b1f29f5ffee93e7 Mon Sep 17 00:00:00 2001 From: Dan Yeaw Date: Fri, 31 Jul 2026 09:35:58 -0400 Subject: [PATCH 38/38] Update conditional and extras references to approved CEPs --- cep-XXX1.md | 32 ++++++++++++++++---------------- cep-XXX2.md | 20 +++++++++++--------- 2 files changed, 27 insertions(+), 25 deletions(-) diff --git a/cep-XXX1.md b/cep-XXX1.md index dc67caca..70a7cca0 100644 --- a/cep-XXX1.md +++ b/cep-XXX1.md @@ -11,7 +11,7 @@ Updated Apr 24, 2026 Discussion https://github.com/conda/ceps/pull/145 Implementation TBD - Requires https://github.com/conda/ceps/pull/146 https://github.com/conda/ceps/pull/155 https://github.com/conda/ceps/pull/164 https://github.com/conda/ceps/pull/165 [CEP XXX2 (Wheel conda client support)](cep-XXX2.md) + Requires https://github.com/conda/ceps/pull/146 https://github.com/conda/ceps/pull/155 [CEP 43](https://conda.org/learn/ceps/cep-0043) [CEP 44](https://conda.org/learn/ceps/cep-0044) [CEP XXX2 (Wheel conda client support)](cep-XXX2.md) See also [CEP XXX0 – Wheel support in conda (overview)](cep-XXX0.md) @@ -57,7 +57,7 @@ The `whl` dictionary maps conda-like filenames to repodata records. The key MUST - `python` dependency from `Requires-Python` (if present), converted to conda format - All `Requires-Dist` entries from METADATA, converted from PEP 440 to conda format per [Dependency conversion](#dependency-conversion) - Package names normalized to conda-style names per [CEP 26][cep-26] -- **`extra_depends`**: MAY be present. When present, MUST be an object mapping extra names to lists of dependency strings for optional groups, per [PR 165][pr-165]. When absent or empty, the record declares no optional groups beyond `depends`. +- **`extra_depends`**: MAY be present. When present, MUST be an object mapping extra names to lists of dependency strings for optional groups, per [CEP 44][cep-44]. When absent or empty, the record declares no optional groups beyond `depends`. - **`subdir`**: MUST be `"noarch"`. - **`noarch`**: MUST be `"python"`. - **`fn`**: MUST be the wheel filename (`.whl`), as in standard conda repodata records (for example `requests-2.32.5-py3-none-any.whl`). @@ -130,7 +130,7 @@ Those declarations MAY be read from the wheel's embedded **`METADATA`** and/or f - Multiple version specifiers are combined with commas (e.g., `>=1.0,<2.0`) - **Python version requirements:** Convert Requires-Python to explicit python dependency -- **Environment markers:** Map Python-version markers on `Requires-Dist` to conditional `MatchSpec` `when=` as specified in [PR 164][pr-164]. Other markers are out of scope for the default conversion rules in this CEP (see [Limitations](#limitations)). +- **Environment markers:** Map Python-version markers on `Requires-Dist` to conditional `MatchSpec` `when=` as specified in [CEP 43][cep-43]. Other markers are out of scope for the default conversion rules in this CEP (see [Limitations](#limitations)). Example conversion: @@ -163,11 +163,11 @@ This subsection layers interoperable expectations for the repodata format, non-n #### Interoperable minimum -For published `whl` records, the following conversions are the **normative floor** aligned with [Dependency conversion](#dependency-conversion) and [PR 164][pr-164] and [PR 165][pr-165]: +For published `whl` records, the following conversions are the **normative floor** aligned with [Dependency conversion](#dependency-conversion) and [CEP 43][cep-43] and [CEP 44][cep-44]: - **`Requires-Python`:** MUST appear as an explicit `python` dependency in `depends`. -- **Python-version conditions on `Requires-Dist`:** MUST be represented with conditional `MatchSpec` `when=` on the relevant dependency strings per [PR 164][pr-164]. -- **`extra` (optional groups):** MUST be represented with **`extra_depends`** and optional-group selection per [PR 165][pr-165], not only as opaque marker text on `depends`. +- **Python-version conditions on `Requires-Dist`:** MUST be represented with conditional `MatchSpec` `when=` on the relevant dependency strings per [CEP 43][cep-43]. +- **`extra` (optional groups):** MUST be represented with **`extra_depends`** and optional-group selection per [CEP 44][cep-44], not only as opaque marker text on `depends`. Channels and tools MAY apply additional, stricter mappings where they can express them in repodata. @@ -182,9 +182,9 @@ Channel operators SHOULD document their policy for such cases (including any con ### Handling conditional dependencies and extras -Wheel `depends` entries that encode PEP 508 environment markers MUST use conditional `MatchSpec` syntax with `when=` as specified in [PR 164][pr-164]. +Wheel `depends` entries that encode PEP 508 environment markers MUST use conditional `MatchSpec` syntax with `when=` as specified in [CEP 43][cep-43]. -PyPI **extras** (optional dependency groups) MUST be represented on the wheel repodata record using an **`extra_depends`** object (a mapping from extra name to lists of dependency strings) as specified in [PR 165][pr-165]. A default install uses only `depends` and MUST NOT union in `extra_depends` entries unless the user selects optional groups (for example with `extras=` in `MatchSpec` as described in [PR 165][pr-165]). +PyPI **extras** (optional dependency groups) MUST be represented on the wheel repodata record using an **`extra_depends`** object (a mapping from extra name to lists of dependency strings) as specified in [CEP 44][cep-44]. A default install uses only `depends` and MUST NOT union in `extra_depends` entries unless the user selects optional groups (for example with `extras=` in `MatchSpec` as described in [CEP 44][cep-44]). ### Solver behavior @@ -196,9 +196,9 @@ This CEP only requires that published `whl` records are valid [repodata records] This CEP has the following known limitations: 1. **Pure Python only:** This CEP explicitly does not address wheels with binary extensions, which require platform-specific compatibility guarantees beyond the current scope. Conda’s strength is binary compatibility, so using conda packages may be the optimal solution. -2. **Environment markers (publisher defaults):** Default conversion rules in this CEP focus on Python-version markers on `Requires-Dist`, mapped to `when=` per [PR 164][pr-164]. Other markers (for example `sys_platform`) are out of scope for those default publisher rules and are not required to be converted into repodata here; see [PEP 508 marker translation guidance](#pep-508-marker-translation-guidance). -How conda clients evaluate `when=` and optional groups at solve time (including environment context) is specified in [CEP XXX2][cep-xxx2] together with [PR 164][pr-164] and [PR 165][pr-165]. -3. **Conditionals and extras:** Normative syntax and record fields for `when=` on `depends` and for optional groups in **`extra_depends`** are specified in [PR 164][pr-164] and [PR 165][pr-165], on which this repodata CEP and [CEP XXX2][cep-xxx2] rely for PyPI-aligned conditionals and extras in published records and at client solve time, respectively. +2. **Environment markers (publisher defaults):** Default conversion rules in this CEP focus on Python-version markers on `Requires-Dist`, mapped to `when=` per [CEP 43][cep-43]. Other markers (for example `sys_platform`) are out of scope for those default publisher rules and are not required to be converted into repodata here; see [PEP 508 marker translation guidance](#pep-508-marker-translation-guidance). +How conda clients evaluate `when=` and optional groups at solve time (including environment context) is specified in [CEP XXX2][cep-xxx2] together with [CEP 43][cep-43] and [CEP 44][cep-44]. +3. **Conditionals and extras:** Normative syntax and record fields for `when=` on `depends` and for optional groups in **`extra_depends`** are specified in [CEP 43][cep-43] and [CEP 44][cep-44], on which this repodata CEP and [CEP XXX2][cep-xxx2] rely for PyPI-aligned conditionals and extras in published records and at client solve time, respectively. 4. **Repodata size:** Supporting a significant portion of pure Python packages from PyPI (potentially hundreds of thousands of packages with multiple versions each) will substantially increase repodata size. Channels adopting wheel support at scale SHOULD implement sharded repodata ([CEP 16][cep-16]) to maintain acceptable performance. ## Rationale @@ -338,7 +338,7 @@ Requires-Python: >=3.8 Requires-Dist: typing-extensions>=4.0.0; python_version < '3.9' ``` -The package record expresses the conditional with `when=` so `typing_extensions` applies only when the Python version is less than 3.9, matching the original wheel METADATA and [PR 164][pr-164]. The Python version constraint of >=3.8 is directly mapped. +The package record expresses the conditional with `when=` so `typing_extensions` applies only when the Python version is less than 3.9, matching the original wheel METADATA and [CEP 43][cep-43]. The Python version constraint of >=3.8 is directly mapped. ## References @@ -348,8 +348,8 @@ The package record expresses the conditional with `when=` so `typing_extensions` - [PEP 508 marker conversion (conda-pypi developer docs)][conda-pypi-marker-conversion] - [Example `repodata.json` (conda-pypi test channel)][conda-pypi-example-repodata] - [Channel relations in repodata (PR 155)][cep-channel-relations] -- [PR 164 – Conditional dependencies][pr-164] -- [PR 165 – Optional dependency groups][pr-165] +- [CEP 43 – Conditional dependencies][cep-43] +- [CEP 44 – Optional dependency groups][cep-44] - [conda-pupa][conda-pupa] - [PyPI JSON API documentation][pypi-json-api] @@ -369,8 +369,8 @@ All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdo [conda-pypi-marker-conversion]: https://conda.github.io/conda-pypi/developer/marker-conversion/#pep-508-variables [conda-pypi-example-repodata]: https://github.com/conda-incubator/conda-pypi/blob/main/tests/conda_local_channel/noarch/repodata.json [cep-channel-relations]: https://github.com/conda/ceps/pull/155 -[pr-164]: https://github.com/conda/ceps/pull/164 -[pr-165]: https://github.com/conda/ceps/pull/165 +[cep-43]: https://conda.org/learn/ceps/cep-0043 +[cep-44]: https://conda.org/learn/ceps/cep-0044 [conda-pupa]: https://github.com/dholth/conda-pupa [pypi-json-api]: https://docs.pypi.org/api/json/ [cep-xxx0]: cep-XXX0.md diff --git a/cep-XXX2.md b/cep-XXX2.md index 6aea69b2..da1160a1 100644 --- a/cep-XXX2.md +++ b/cep-XXX2.md @@ -10,7 +10,7 @@ Updated Apr 24, 2026 Discussion https://github.com/conda/ceps/pull/145 Implementation TBD - Requires [CEP XXX1 – Repodata wheel support](cep-XXX1.md) https://github.com/conda/ceps/pull/146 https://github.com/conda/ceps/pull/155 https://github.com/conda/ceps/pull/111 + Requires [CEP XXX1 – Repodata wheel support](cep-XXX1.md) https://github.com/conda/ceps/pull/146 https://github.com/conda/ceps/pull/155 [CEP 43](https://conda.org/learn/ceps/cep-0043) [CEP 44](https://conda.org/learn/ceps/cep-0044) See also [CEP XXX0 – Wheel support in conda (overview)](cep-XXX0.md) @@ -20,7 +20,7 @@ described in [RFC2119][RFC2119] when, and only when, they appear in all capitals ## Abstract -This CEP specifies how conda clients consume wheel entries published in repodata: loading `v{revision}.whl` into the same index as other repodata records, evaluating conditional dependencies and extras per [PR 111][pr-111], downloading wheel artifacts, and integrating them into the environment **as equivalent `noarch: python` conda packages** (layout, package database, and lifecycle). +This CEP specifies how conda clients consume wheel entries published in repodata: loading `v{revision}.whl` into the same index as other repodata records, evaluating conditional dependencies and extras per [CEP 43][cep-43] and [CEP 44][cep-44], downloading wheel artifacts, and integrating them into the environment **as equivalent `noarch: python` conda packages** (layout, package database, and lifecycle). The shape of the `whl` index and publisher-side `METADATA` conversion are defined in [CEP XXX1 – Repodata wheel support][cep-xxx1]. Background on **why** channels and clients pursue native wheel support, plus historical context and rejected ecosystem alternatives, is in [CEP XXX0 – Wheel support in conda (overview)][cep-xxx0]. @@ -47,10 +47,10 @@ Wheel records are normal [repodata records][repodata-record-schema] and solvers ### Conditional dependencies and extras -For wheel records, [PR 111][pr-111] normatively defines `when=` on `depends` and the **`extra_depends`** object for PyPI optional groups. Clients that implement this CEP MUST: +For wheel records, [CEP 43][cep-43] defines `when=` on `depends` and [CEP 44][cep-44] defines the **`extra_depends`** object for optional groups. Clients that implement this CEP MUST: - Evaluate `when=` for wheel `depends` entries according to the same rules as for other new-syntax records, using an environment model that provides at least Python version and, where applicable, virtual packages (see [Marker handling](#marker-handling)). -- Honor **`extra_depends`** when the user selects optional groups (for example with `extras=` in `MatchSpec`) per [PR 111][pr-111], and MUST NOT pull in `extra_depends` groups on a default install. +- Honor **`extra_depends`** when the user selects optional groups (for example with `extras=` in `MatchSpec`) per [CEP 44][cep-44], and MUST NOT pull in `extra_depends` groups on a default install. - Load wheel records that use `when=` or `extra_depends` in the same `v{revision}` payload and revision-handling as other records introduced under the [backwards-compatible repodata update strategy](https://github.com/conda/ceps/pull/146). The strings on a wheel’s `depends` and `extra_depends` are produced by channel operators from wheel `METADATA` per [CEP XXX1][cep-xxx1]; the client’s job is to interpret them, not to re-parse the wheel for dependency resolution (though reading `METADATA` for validation or error messages is allowed). @@ -59,7 +59,7 @@ The strings on a wheel’s `depends` and `extra_depends` are produced by channel **Publisher output (in repodata):** [CEP XXX1][cep-xxx1] requires converting certain PEP 508 markers on `Requires-Dist` into `when=` and optional-group fields. Other markers are out of scope for the default *publisher* conversion rules; repodata may omit or simplify them. -**Client (solve time):** The solver MUST use the MatchSpec and PR 111 features carried on the repodata record: `when=` subexpressions, `python` constraints, and virtual packages, so that the effective dependency graph matches the published strings for the current environment. Clients MAY map additional environment dimensions to virtual packages where documented (non-normative per-variable notes appear in the [conda-pypi marker conversion][conda-pypi-marker-conversion] documentation). +**Client (solve time):** The solver MUST use the MatchSpec features from [CEP 43][cep-43] and [CEP 44][cep-44] carried on the repodata record: `when=` subexpressions, `python` constraints, and virtual packages, so that the effective dependency graph matches the published strings for the current environment. Clients MAY map additional environment dimensions to virtual packages where documented (non-normative per-variable notes appear in the [conda-pypi marker conversion][conda-pypi-marker-conversion] documentation). **Out-of-scope dimensions:** Some PEP 508 variables (for example, `implementation_name`, or markers tied to a specific wheel build matrix beyond the pure-Python subset) do not have a defined mapping in the normative [CEP XXX1][cep-xxx1] record; behavior for such wheels remains undefined unless a future CEP or channel policy encodes them in repodata. Clients MUST NOT claim full PEP 508 equivalence beyond what the record expresses. @@ -96,7 +96,7 @@ Conda clients install from the `.whl` artifact referenced in repodata and lay ou ### For conda client implementers - **Index:** Parse `whl` inside each supported `v{revision}` object alongside `packages` / `packages.conda` into the same solver-facing index. -- **Conditional dependencies and extras ([PR 111][pr-111]):** Evaluate `when=` on `depends` and honor **`extra_depends`** when the user requests optional groups, consistent with the published record strings (wheel rows use the same syntax as other PR 111 records). +- **Conditional dependencies and extras ([CEP 43][cep-43], [CEP 44][cep-44]):** Evaluate `when=` on `depends` and honor **`extra_depends`** when the user requests optional groups, consistent with the published record strings (wheel rows use the same syntax as other CEP 43 / CEP 44 records). - **Revision gating:** Treat new-syntax wheel records with the same `v{revision}` handling as the rest of the PR 146 strategy (do not expect older repodata only clients to read `whl`). - **Download:** Reuse the channel and fetch stack used for other artifacts (`fn`, channel/`base_url`, and checksum verification). - **Install:** Integrate wheels using the same prefix layout and metadata model as `noarch: python` conda packages ([CEP 20][cep-20], [CEP 34][cep-34]), including `info/link.json` for entry points when using conda's link pipeline. The [conda-pypi][conda-pypi] project demonstrates populating a conda package shape from a wheel (for example, suppressing duplicate `bin` script generation when `link.json` is used) as a non-normative example. @@ -110,7 +110,7 @@ After installing `requests-2.32.5-py3-none-any.whl` from a wheel repodata record ## Backwards compatibility - **Repodata shape:** The `whl` key appears only under a registered `v{revision}` (see [PR 146](https://github.com/conda/ceps/pull/146)). Conda clients that do not support that revision or the `whl` index MUST ignore unknown `v{revision}` members or fall back to earlier revisions per channel policy. -- **Record syntax:** Records that use `when=` and `extra_depends` are part of the new-syntax repodata world [PR 111][pr-111] already depends on; wheel rows are not a separate syntax fork. +- **Record syntax:** Records that use `when=` and `extra_depends` are part of the new-syntax repodata world defined by [CEP 43][cep-43] and [CEP 44][cep-44]; wheel rows are not a separate syntax fork. ## Rejected ideas @@ -123,7 +123,8 @@ Ecosystem-level alternatives (relying on pip only, on-the-fly PyPI without repod - [CEP 20 – `noarch` (python) packages and site-packages][cep-20] - [CEP 34 – Package metadata and `info/`](./cep-0034.md) (`index.json`, `link.json`, etc.) - [CEP 26 – Identifying packages and channels][cep-26] -- [PR 111 – Conditional dependencies and optional groups][pr-111] +- [CEP 43 – Conditional dependencies][cep-43] +- [CEP 44 – Optional dependency groups][cep-44] - [PEP 508 marker conversion (conda-pypi developer docs)][conda-pypi-marker-conversion] - [Example `repodata.json` (conda-pypi test channel)][conda-pypi-example-repodata] - [conda-pypi project][conda-pypi] @@ -147,7 +148,8 @@ All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdo [conda-pypi]: https://github.com/conda-incubator/conda-pypi [conda-pypi-marker-conversion]: https://conda.github.io/conda-pypi/developer/marker-conversion/#pep-508-variables [conda-pypi-example-repodata]: https://github.com/conda-incubator/conda-pypi/blob/main/tests/conda_local_channel/noarch/repodata.json -[pr-111]: https://github.com/conda/ceps/pull/111 +[cep-43]: https://conda.org/learn/ceps/cep-0043 +[cep-44]: https://conda.org/learn/ceps/cep-0044 [conda-pupa]: https://github.com/dholth/conda-pupa [uv-in-pixi]: https://prefix.dev/blog/uv_in_pixi [rip]: https://github.com/prefix-dev/rip