CEP XXXX: Conditional dependencies, extras and flags

[wolfv]

Checklist for submitter

• I am submitting tree new CEPs:
  • Conditional dependencies
  • Optional dependency groups
  • Flags
• I am using the CEP template by creating a copy cep-0000.md named cep-XXXX.md in the root level.
• I am submitting modifications to CEP XX.
• Something else: (add your description here).

Checklist for CEP approvals

• The vote period has ended and the vote has passed the necessary quorum and approval thresholds.
• A new CEP number has been minted. Usually, this is ${greatest-number-in-main} + 1.
• The cep-XXXX.md file has been renamed accordingly.
• The # CEP XXXX - header has been edited accordingly.
• The CEP status in the table has been changed to approved.
• The last modification date in the table has been updated accordingly.
• The table in the README has been updated with the new CEP entry.
• The pre-commit checks are passing.

This defines new repodata with two additional fields and a modfied matchspec syntax:

• extra (optional) dependency sets that can be activated by the user or other packages
• conditional dependencies to be activated on a specific platform, or when other package versions are also pulled in (e.g. require six; if python <3.8)
• flags to make it easier to select variants confidently (instead of globbing the build string)

Some initial discussion happened over the past days on:

• Zulip:
  • "Flags in conda packages", conda-forge
  • "Flags", conda
  • "Conditional dependencies syntax in v2 environments & recipes", conda
• PR
• HackMD.

TODO:

• specify that we need to understand full-blown matchspecs in the repodata.v2 (e.g. with bracket syntax), because we want other packages to be able to request extras or flags).
• Decide whether flag negation should be ~foo, !foo, or -foo.
• Decide if we need to add a default flags entry to the repodata record, or another way to mark the default variant (can also still use down-priorization via track records)
• Investigate ordering of flags and whether we should have some weighing e.g. for numerical flags (higher == better)? Or wether this should be decided by another order integer that the users can compute.

[wolfv]

Here are comments from the HackMD:

Should we consider a system like Spack has (https://spack.readthedocs.io/en/latest/basic_usage.html#specs-dependencies) or maybe Conan (https://docs.conan.io/2/reference/conanfile/attributes.html#options)?
Additionally, how would a package declare which flag is the default? (Edited)

For default flags I think we might want to deprecate the track_features field and instead add a priority: integer field that can hold any value. The user could mark the default this way. We could also use the inverse of priority, so that higher values are less desirable.

Or, maybe a better thought: we could also make it so that if priority: -1 we never enable this package, except if we request its flags directly.

Regarding Spack and Conan: I would be super hyped if you can distill this on how it could work for the conda ecosystem. I haven't had the time yet to read into it deeply. I am not a fan of super arbitrary code execution though.

[mariusvniekerk]

can you add an example for the string matching for say blas:mkl

[wolfv]

Would an exact match not work fine?

[wolfv]

Would an exact match not work fine?

[mariusvniekerk]

It would, we should just state how to do an exact match

[wolfv]

You coudl just ask for flags = ["blas:mkl"] for an exact match :)

[jaimergp]

You probably need NOT and parentheses for precedence overrides, right?

[AntoinePrv]

In regular MatchSepc, we have , and | used for versions for and and or. I think we should keep thing similar, even if it means supporting and and or in versions.

[baszalmstra]

If find this hard to distinguish when parsing the version. E.g.

python <3.8|>3.9 | numpy >=2.0
python <3.8|>3.9 or numpy >=2.0

I find the version with or easier to read than the one with pipes.

[jaimergp]

Environment markers have been doing fine with and and or, I don't think we need to introduce more character-based operators.

[jaimergp]

It would be nice if we didn't need the ; because somehow conda will happily ignore the if parts while parsing MatchSpecs.

>>> from conda.models.match_spec import MatchSpec as M
>>> M("python 3.8 * if python")
MatchSpec("python==3.8[build=*]")
>>> M("python 3.8 'if' if  __win")  # quote 'if' to force parse it as a build string
MatchSpec("python==3.8='if'")

[jaimergp]

It will also ignore parenthesised blocks:

>>> M("python 3.8 * (__win)")
MatchSpec("python==3.8[build=*]")
>>> M("python 3.8 (__win)")
MatchSpec("python==3.8")
>>> M("python 3.8 (__win and __osx)")
MatchSpec("python==3.8")
>>> M("python 3.8 (if __win and __osx)")
MatchSpec("python==3.8")

[jaimergp]

libmamba also ignores parentheses:

>>> from libmambapy.specs import MatchSpec as LibmambaMatchSpec
>>>print(LibmambaMatchSpec.parse("python 3.8 * (__win and __osx)"))
python==3.8
>>> print(LibmambaMatchSpec.parse("python 3.8 * (if __win and __osx)"))
python==3.8

[jaimergp]

My suggestion would be to design the syntax like this:

name [version [build]] ('if' condition)

The if literal could be omitted, or replaced with with, if folks feel it's clearer that way. See discussion in #conda-maintainers > Conditional dependencies syntax in v2 environments & recipes @ 💬.

[AntoinePrv]

Another idea (not sure if a good one) could be: rather than make MatchSpecs more complex than they already are, build on the idea of selectors from the new recipe format and allow depends to contain objects like so:

depends:
 - python >=3.8
 - if: python <3.8
    then: six

In the recipe format, this is expressed via a variation on the selector to avoid being process at build time (if(run): for instance).

This disallow basically disallow conditionals outside of recipes (or format with selector), but makes a more consistent narrative around conditionals.

[baszalmstra]

The idea was indeed also posted on zulip. One issue is that it then becomes harder for cli tools to adopt.

[baszalmstra]

I propose we use the when syntax without a semicolon to force an error on older versions of conda that dont support it and the distinguish between the already established if syntax in recipe v1. We can use the same keyword in a more expanded form as the "build spec"

foobar when python >=3.8

E.g. in a recipe:

- if: unix
  then: foobar
  when: python >=3.8
  
# OR
  
- when: python >=3.8
  then: foobar
 
# OR 
  
- "foobar when python >=3.8"

[jaimergp]

I like this when idea a lot, and combines well with the if/then syntax, but I'd like to discuss the serialized MatchSpec. foobar when python>=3.8 is unparsable because it could be understood as version=when, build=python>=3.8. Of course we could force parsing rules to "split on when first" and so on, but let me suggest something that wouldn't imply so many changes in the MatchSpec parsers: use a square bracket keyword.

foobar[when="python>=3.8"]
foobar[when="__win"]
foobar[when="python>=3.8 and __unix"]

[AntoinePrv]

I very much like this idea. An implementation note: while if __unix is reasonably easy to implement because it is "static", if python <3.8 is conceptually much harder as it is not something that can be decided ahead of solving. I requires to be able to adapt the dependencies of a package as partial candidates are investigated during solve.

[baszalmstra]

Yep, this is a form of boolean dependencies in rpm (already supported by libsolv I believe)

[baszalmstra]

This is the PR that implements this for resolvo: prefix-dev/resolvo#136

[AntoinePrv]

How do flags and extra mix and overlap? Wouldn't conditional dependencies on a flag be enough to generate the extra category?

[baszalmstra]

Flags are part of a variant. So there is no variation of flags for a single variant. E.g. flags could be used to say a particular variant is using cuda and another is used to target cpu. Extras are a way to select additional dependencies for a particular variant. If a variant also adds a CLI tool it provides the extra "cli". Only if that extra is requested by another package are particular dependencies also requested.

In technical terms extras can indeed be implemented as conditional dependencies. E.g. for a package my_package we could express it as typer when my_package[cli]. If there is a package that depends on my_package[cli] typer would also be required.

[baszalmstra]

If find this hard to distinguish when parsing the version. E.g.

python <3.8|>3.9 | numpy >=2.0
python <3.8|>3.9 or numpy >=2.0

I find the version with or easier to read than the one with pipes.

[baszalmstra]

I propose we use the when syntax without a semicolon to force an error on older versions of conda that dont support it and the distinguish between the already established if syntax in recipe v1. We can use the same keyword in a more expanded form as the "build spec"

foobar when python >=3.8

E.g. in a recipe:

- if: unix
  then: foobar
  when: python >=3.8
  
# OR
  
- when: python >=3.8
  then: foobar
 
# OR 
  
- "foobar when python >=3.8"

[baszalmstra]

Flags are part of a variant. So there is no variation of flags for a single variant. E.g. flags could be used to say a particular variant is using cuda and another is used to target cpu. Extras are a way to select additional dependencies for a particular variant. If a variant also adds a CLI tool it provides the extra "cli". Only if that extra is requested by another package are particular dependencies also requested.

In technical terms extras can indeed be implemented as conditional dependencies. E.g. for a package my_package we could express it as typer when my_package[cli]. If there is a package that depends on my_package[cli] typer would also be required.

[baszalmstra]

This is the PR that implements this for resolvo: prefix-dev/resolvo#136

[baszalmstra]

I think we should clarify what happens if an extra is requested for a package but the selected variant doesnt provide that extra. E.g. what happens if I depend on a foobar[extras=["doesntexist"]]

[baszalmstra]

Maybe I missed it but its also not defined how to depend on an extra?

[wolfv]

Yes, we discussed this today some more and there are a few options:

• An extra implicitly always exists (as empty) and so we would always choose the latest "base" package (whether it has the extra or not). We would then warn if it does not have the extra. I believe this is the behavior that pip implements, and makes it easier to be forward compatible even if other dependencies might ask for an extra that does not exist anymore.
• We could strictly require that the extra is there and otherwise fail (or select an older version of the package that has the extra).
• We could forward propagate extras (e.g. the first time we see an extra, we implicitly add it to all later versions, even if empty) which would prevent the solver from choosing a version before the extra existed, but might choose a later one after the extra was removed).

I think we should decide on a behavior, most likely choosing the one from pip. We should also read up on the idea of adding a "default" extra (default dependencies that can be deselected with no-default-extra or something along the lines which I recently saw being proposed somewhere in PyPI world as part of wheel-next I think.

[wolfv]

PEP for default extras: https://peps.python.org/pep-0771/

[jezdez]

Do we have a proposal for the matchspec syntax for these extras? Something like this?

conda install sqlalchemy[sqlite]

[wolfv]

What currently works is sqlalchemy[extras=sqlite].

Please note that [ ... ] is a syntax already in conda to specify key-value pairs, e.g. foobar[sha256="1234..."].

[wolfv]

Mental note that flags should be added to the variant hash to ensure that packages have unique names.

[wolfv]

This is what a recipe looks like - we have a branch where the tests are passing.

package:
  name: conditional
  version: 0.1.0

build:
  flags:
    - cuda
    - openmp:mpi

requirements:
  run:
    - rattler-build
    - bzip2 *; if rattler-build >=0.44
    - zstd *; if rattler-build <0.44

  extras:
    science:
      - numpy
      - matplotlib
    web:
      - flask
    cli:
      - click

tests:
  - script:
    - echo "Hello, world!"
    requirements:
      run:
        - rattler-build >=0.44

  - script:
    - echo "This is a test for Rattler < 0.44"
    requirements:
      run:
        - rattler-build <0.44

  - script:
    - echo "This is a test for Rattler < 0.44"
    requirements:
      run:
        - conditional [extras=[science, web]]

[wolfv]

Another comment to remind ourselves, to simplify the initial implementation:

• we will not allow conditions in run_constraints (for now)
• We will not allow extra groups to contain anything other than run dependencies (no run constraints or run exports)

[baszalmstra]

We just merged

• feat!: add support for conditional dependencies prefix-dev/resolvo#136

which adds support for all features mentioned in this CEP. We are working on an preliminary version of this CEP in rattler based on this.

[jezdez]

Let's split this CEP up between the various proposals here, each have sufficiently complex to implement constraints.

[baszalmstra]

Let's split this CEP up between the various proposals here, each have sufficiently complex to implement constraints.

The reason we combined these is they are require changes to the repodata format. We aimed to bundle these together in a single change.

But I do understand the desire to split the different features. Do you have a suggestion how we can avoid having to bump the repodata version in each CEP?

[jaimergp]

I think nothing prevents us from saying something like "This CEP is co-submitted with X, Y, Z. All of them propose to bump to repodata version v2, the implementation of which MUST happen simultaneously when all CEPs have been reviewed and accepted or rejected".

[wolfv]

I think I would be pro-splitting if we would see lots of engagement on the topics presented here. But so far it is not an unwieldy discussion so I don't exactly see the point of splitting :)

[jezdez]

The point is that repodata and matchspec are two different technical specifications and shouldn't be bunched into one CEP since that makes referring and evolving them in the future much harder. Imagine reading a future CEP saying "This refers to the matchspec portion of CEP 45, but not the repodata part."

[h-vetinari]

"This CEP is co-submitted with X, Y, Z. All of them propose to bump to repodata version v2, the implementation of which MUST happen simultaneously when all CEPs have been reviewed and accepted or rejected".

One of that CEPs could be about this proposed design for a better run-export infrastructure, which will likely (probably? certainly?) require repodata changes. So v2 would be the moment to pull that off, because I'm not assuming we'll be doing this very often.

[Hind-M]

Is there a reason negative values are allowed? (It seems that all the other integer criteria used for sorting are positive?)
Is it to support ordering when variant_priority is missing and defaults to 0?
If so, it might be helpful to mention this and maybe add more examples to clarify the use cases?

[wolfv]

I just removed the variant_priority field because there are too many open questions about it, especially with regards to where it should sit in the version ordering.

We can (continue) to workaround this by using the build number, or local version numbers.

We want to start voting on these 3 CEPs on Monday.

[jaimergp]

Superseded by #164, #165, #166.