Add CEP for a URL field for package records

[danyeaw]

Checklist for submitter

• I am submitting a new CEP: URL field for package records.
  • I am using the CEP template by creating a copy cep-0000.md named cep-XXXX.md in the root level.
• I am submitting modifications to CEP XX.
• Something else: (add your description here).

Checklist for CEP approvals

• The vote period has ended and the vote has passed the necessary quorum and approval thresholds.
• A new CEP number has been minted. Usually, this is ${greatest-number-in-main} + 1.
• The cep-XXXX.md file has been renamed accordingly.
• The # CEP XXXX - header has been edited accordingly.
• The CEP status in the table has been changed to approved.
• The last modification date in the table has been updated accordingly.
• The table in the README has been updated with the new CEP entry.
• The pre-commit checks are passing.

[danyeaw]

Hi @jaimergp, I would to please move this to the RFC stage, thanks!

[jaimergp]

Dear @conda/steering-council, this CEP is officially entering its Request For Comments (RFC) period. This will last two weeks, until May 12th EOD AoE.

[jaimergp]

Can relative paths traverse parents with ../ or is that forbidden? If not, should we forbid that for security purposes?

[danyeaw]

Hi @jaimergp, thanks for the feedback! I updated the CEP to exclude path traversal to parents. I think it isn't great for security and other methods of declaring base_urls are already flexible enough without it.

[jaimergp]

Won't this mean that these old clients will get 404s for records that define url?

[danyeaw]

We will use the backwards compatible repodata update strategy, I made this more clear in the CEP.

[jaimergp]

Are there any rules where the filename in the url field must match the filename in the repodata key? Should they? Which filename is it used for the local cache if they don't match?

[danyeaw]

Thanks, I added language about the cache key needing to be the basename of the file. This will support both current files and future ones like wheels.

[baszalmstra]

One question I have is what we do with lock-files. Let me explain.

Currently, lockfiles store the packages by their url. But now we have two urls. We have the "channel url" (<channel>/<subdir>/<filename>) and the "download url" as proposed by this CEP. I was first thinking we could just use the "download url" as the url for the package in the lockfile, however, thinking about it more, this poses a small problem. Two metadata entries in repodata can have different metadata but the same download url.

I think we should still use the "channel url" in lock files to uniquely identify the package in the channel/repodata, and also encode the "download url" separately.

I think it would be good to include this information in the CEP as a recommendation. WDYT?

[danyeaw]

Hi @baszalmstra, this is a great point, thanks for bringing it up! I agree on using the channel URL for packages on the channel. But the recommendation needs to handle the case where a record's url is an absolute external URL, for example how we are using it for wheels linked to a CDN. In that case the artifact doesn't live on the channel at all, so the download URL is the primary artifact identifier. What if I update the CEP to say:

• Absolute url: use the download URL as the primary lockfile identifier
• Relative url or no url: use the channel URL as the primary lockfile identifier

Would that be clear and unambiguous?

[baszalmstra]

I think it doesn't pose a problem as long as we keep both urls in the lockfile. We will never download from the "channel url" if a "download url" is present. So in the current pixi.lock it could look like (simplified):

- conda: https://conda.anaconda.org/pypi-test/noarch/foobar-0.1.0-a.whl
  url: files.pypihosted.com/myexamplehash/cp310-cp310-macosx_10_9_x86_64.whl
- conda: https://conda.anaconda.org/pypi-test/noarch/foobar-0.1.0-b.whl
  url: files.pypihosted.com/myexamplehash/cp310-cp310-macosx_10_9_x86_64.whl

Note that the download url is the same but the unique identifier from the channel isnt.

[danyeaw]

Hey @baszalmstra, I could definitely be wrong and might be missing something, but to your example, https://conda.anaconda.org/pypi-test/noarch/foobar-0.1.0-a.whl doesn't exist as a file. The only place that the file would exist is at the url (files.pypihosted.com/myexamplehash/cp310-cp310-macosx_10_9_x86_64.whl). Wouldn't the lockfile look like:

environments:
  default:
    channels:
      - url: conda-pypi
      - url: conda-forge
    packages:
      osx-arm64:
        - conda: https://files.pypihosted.com/myexamplehash/annotated_types-0.7.0-py3-none-any.whl
        - conda: https://files.pypihosted.com/myexamplehash/annotated_doc-0.0.4-py3-none-any.whl

[baszalmstra]

Let me try to clarify with a problem statement:

What happens if you have two wheel channels that have different metadata for the same url? Both could end up in the lock-file, but if we use the "download url" we cant uniquely identify them.

[jaimergp]

The way I read it, Bas wants a unique identifier of a package record entry in the repodata. So far, that happened to be the download URL as well, so that's what we used. This CEP breaks that happy accident and now we don't have a guaranteed unique identifier. Bas wants to keep using that URL as a URI (identifier instead of location).

I wonder though if we should elevate PURLs (with the changes proposed in Cheng's #159) as those unique identifiers.

[baszalmstra]

Exactly. 👍

[danyeaw]

Hi @baszalmstra and @jaimergp, I added a Lockfiles considerations section, please let me know if you think that captures the concern and the right proposal for how to deal with it going forward.

[baszalmstra]

This is not quite enough because it doesnt include the full url of the channel itself. A lockfile can easily contain numpy from conda-forge but from different sources. I dont know what the canonical form for that is?

[wolfv]

Agreed, and would argue that we should either stick to the "conda URL" or do something like:

{conda_url}@{resolved_url}

Otherwise, indirection via conda_url -> PackageRecord (in lockfile) -> URL sounds like it should work fine.

[hunger]

As I read #159, this should resolve to https://conda.anaconda.org/conda-forge/noarch/repodata.json. There is a repodata in that place, assuming numpy resolves in there with the data provided, this should specify the package OK.

[baszalmstra]

Yes but if the lockfile contains packages from http://prefix.dev/conda-forge and http://conda.anaconda.org/conda-forge we need to be able to distinguish them.

[adam-browning]

Hi @danyeaw — thanks for driving this CEP.

Before the vote, I want to flag a critical security gap that I think
warrants explicit treatment in the spec: organizations that consume
conda channels through an artifact proxy (Artifactory, Nexus,
Cloudsmith, Anaconda Server, internal mirrors, air-gapped relays)
rely on the channel URL being the single source of truth for every
artifact. They use that boundary for caching, vulnerability/license
scanning, egress allow-listing, and audit.

CEP #151's absolute url field breaks that boundary. A compromised
channel maintainer (or typosquat channel) can re-point a single
record at an attacker-controlled host. The artifact bypasses every
proxy and scanner the consumer trusts for the channel host. Hashes
don't help — the attacker controls them.

Two requests:

1. Add a Security Considerations section to the CEP that names this
   threat model and the recommended mitigations.
2. Require client implementations to expose a policy knob (e.g.
   allow / relative_only / disabled) so admins in regulated
   environments can keep fetches inside the trusted channel
   boundary. Without this, enterprise users have no opt-out.

Without an explicit opt-out on the client side, our mutual customers will expose their organizations to downloads circumventing their security gates.

Thanks in advance,

[danyeaw]

Hey @adam-browning, thanks for the feedback. I added a new security considerations section. I think a compromised channel maintainer normally could also modify packages on the channel, but I do mention that this adds a narrower attack vector if someone somehow only got access to change the repodata.json. I think adding a security knob on the client side is good. Do you think proxies should also be able to restrict usage of the URL field? Please let me know what you think.

[beckermr]

I want to make sure we track any security implications for url fields added to the index.json blobs at package build time. We don't want folks injecting random urls into artifacts this way. I think these should be stripped, if ever found, and never kept in the indexing process.

[wolfv]

I see some alternative possibilities for the mirroring use case:

• we could have a field similar to base_url that can be added into the metadata section of repodata json as the URL forwarder (url_base_url or something). There would be a API that we'd ping with the URL and then follow the redirect (e.g. https://foo.com/?url=bla... -> pypi.org. If the field is not set, then we just go to the URL indicated.
• Alternatively, we could scrap the proposal and Anaconda can run a reverse proxy themselves that maps from the conda URL to the PyPI wheel URL and forwards to it with a 301. I think that could be the best solution IMO.

[danyeaw]

If a channel operator wanted to do something malicious today, they already can. They control the whole channel and all the packages. Is there some other threat scenario that I'm missing? If a channel could reverse proxy a package URL today, that seems like it is even worse than if a channel could explicitly set a malicious channel URL with this CEP. At least the explicit case is auditable.

The only genuinely new thing security related about this CEP is that a proxy/mirror now needs to understand and rewrite the url field to keep downloads within its perimeter. This is no different than a PyPI mirror has to do today.

I agree that packages shouldn't be setting their own URLs, and that should be stripped if they do.