feat: implement flags

[wolfv]

This is a new idea for Conda packages.

I imaging that for compiled packages, having flags can be quite useful. For example, a library like libarchive can be built with support for a number of different compression algorithms. It would be nice if we could express these variants with a more natural syntax vs. relying on globs on the build string. For example, for libarchive it would be nice to be able to say libarchive[flags=[zstd, bz2, zlib]] to denote the minimum enabled flags.

For pixi build we already discussed we might want to have release and debug flags which we could express in a similar way. We might also want to be able to negate flags (e.g. libarchive[flags=[~zstd]].

Different from extras, flags would select a different variant of the package (ie. completely different repodata record). This different package might also have different requirements (for example, if zstd is enabled, the zstd package will be a dependency of libarchive).

Conceptually, this is similar to matching packages based on their build string, but just a much better syntax vs trying to do that with globs.

I would like to implement two extensions to the current implementation:

• ?release or ?debug: if the release flag is available, select it (can be turned on for all packages). If the flag does not exist, that's ignored.
• ~zstd: negate the existence of zstd (could also be !zstd).

[h-vetinari]

I love the direction! I'm not sure about the name flags= though (I think the issue is much more general, or at least the name is misleading to me; perhaps features=[...]? 🤔), and I think rather than booleans that can be negated, we should ideally allow full string matching, à la:

pytorch[gpu="cuda", blas="mkl"] ffmpeg[license="lgpl"] faiss[gpu="None", microarch="v4"]

Obviously we could have a fixed set of keys and their values could be enums rather than free-form, but this would allow to select these things in an intuitive way, and it's strictly more powerful (and IMO expressive) than toggling flags on and off (which becomes difficult anyway if there are any choices with more than two states, which is easily possible for e.g. licenses, microarches, etc.)

For completely custom overrides per package (like your libarchive example), there's probably still some room to have some flags= / features= as a fully customizable escape hatch, but I'd love to move common toggles from the build string into more a user-friendly selection syntax.

I'm sorry for always referencing conda/conda#11053 in these discussions, but that's still the basic shape of the problem I'd love to solve (though for the purpose of this issue, I'd be fine to descope the *args part for the extras, and just focus on moving the build string matching to a **kwargs-style syntax).

[wolfv]

With the current implementation you would be free to do foobar[flags=["gpu:cuda", "openmp:true"]] which is close to your suggestion but less free form.

I don't think adding completely open key-value to the matchspec will be a good idea as we would have no way to evolve the matchspec syntax anymore without breaking someone.

Features is something that existed at some point in the conda universe which is why I avoided the name :)

[wolfv]

But I do like the idea of also having a numerical flag value, e.g. for microarch, so that you can say microarch>=3 and that would also naturally lend itself to sorting (e.g. prefer highest microarch).

[h-vetinari]

I don't think adding completely open key-value to the matchspec will be a good idea as we would have no way to evolve the matchspec syntax anymore without breaking someone.

That's what I meant with having a fixed set of keys, for example: gpu=, license=, microarch= (and other major ones if I missed something), but not arbitrary strings. Plus a more open flags=[...] for specialized needs that don't themselves deserve their own keyword.

Those keys will not change meaning, and we could make it an error if a key is not in the allowed set. That way you guard the possibility for evolution in the future, while providing an easier (less nested) syntax for a large majority of cases.

Sidenote: I've been quite heavily involved in the conda ecosystem for many years, and I never heard about features (outside of track_features of course). I think it's possible that enough time has passed that the name has become available again. 😉

[wolfv]

But license is already a key in the match spec. You can use it today with a string matcher in conda, e.g. conda search "zstd[license=bsd*]".

[h-vetinari]

But license is already a key in the match spec.

Great! One thing less to worry about. Some key dimensions (e.g. gpu) should also be on that level, and both should be possible to use in environment specs.

[baszalmstra]

I think from an implementation perspective this should be pretty trivial to add, should we maybe open a CEP where we can discuss the concept overall?

[wolfv]

Flags and more CEP here: conda/ceps#111

@h-vetinari I see where you are coming from but that's mostly syntactic sugar for conda / mamba / pixi or other implementations. I think we should not add this kind of extra stuff into the rigid repodata to ensure the highest flexibility to evolve it further as a standard and the highest value & flexibility for conda-forge and other distributions.

On the CLI we could totally allow to map gpu=cuda* to flags=["gpu:cuda*"]

We could also do that in rattler-build if that cristallizes as the standard, without touching CEPs.

[wolfv]

Closing in favor of #1550