Finalize the proposal for extras with optional dependencies (`dev-dependencies`)

[Hofer-Julian]

Description

Extras are part of a CEP (rendered) which is yet to be merged.
However, we already implemented it in rattler.

This issue proposes to implement extras support, so we will get capabilities to optional dependencies and dependency groups of PyPI.

Pixi packages could add extras like this:

[package]
name = "bas-package"

[package.dev-dependencies.test]
pytest = "*"
python = "*"

[package.extra-dependencies.arrow]
arrow = "*"

Note that there are two keys: dev-dependencies and extra-dependencies.

If the package is part of your workspace, there's no difference, you can just enable both kind of extras like this:

[dependencies]
bas-package = { path = "./bas-package" , extras = ["test", "arrow"]}

However, if the package comes from a channel, only extras specified in extra-dependencies are available:

[dependencies]
bas-package = { version = "*" , extras = ["arrow"]}

That means, if you run pixi build, dev-dependencies are not taken into account.

Implementation

The information, which extras are activated need to be added to the environments, since the package metadata is unaffected by which extras are enabled.

environments:
  defaults:
    packages:
      - conda: ./bas-package 
        extras: ["test", "arrow"]

In the first iteration, we should also only allow extras to be enabled for source packages, since there are no channels around yet that support extras.

[ruben-arts]

Thanks for this proposal, and good idea to keep these two concepts together. I have a few points I would like to discuss:

---

I would like to understand why we would call it extra-dependencies instead of optional-dependencies and dev-dependencies instead of dependency-groups

---

I'm missing the ability to inherit like you can with the pypi dependency-groups.

Some ideas that came to mind are, allowing includes as a special key. This is be cleanest Idea IMO, but it will conflict with a potential packages called includes in the future.

[package.dependency-groups.test]
pytest = "*"

[package.dependency-groups.dev]
includes = ["test"]  # Reserved key
ipython = "*"

[package.optional-dependencies.viz]
matplotlib = ">=3.0"

[package.optional-dependencies.all]
includes = ["viz", "cuda"]  # Same pattern as dependency groups

Other ideas could be:

• Separate table for putting together, like we do with environments:

   [package.optional-dependencies.viz]
   matplotlib = ">=3.0"
   seaborn = "*"
   
   [package.optional-dependencies.cuda]
   cupy = "*"
   cudatoolkit = "*"
   
   [package.optional-dependencies]  # Defines composition
   viz = ["viz"]  # Just itself, potentionally not required to define.
   all = ["viz", "cuda"]  # Combines multiple

  A con would be that you have to add extra tables, which doesn't help with readability.
• Special prefix to clarify that it's not a package:

  [package.optional-dependencies.test]
    pytest = "*"
    
    [package.optional-dependencies.dev]
    __inherits__ = ["test"]  # or __extends__, __includes__
    ipython = "*"

---

I worry that using the extras key in the dependency definition for both types of table might create confusion.
Adding a groups key would make sense to me.

[ruben-arts]

Dev groups

The package definition has a dev entry which can contain groups of (now only) dependencies.
The test group will be special and could be treated as test-dependencies.
We use the dev.<group-name>.dependencies format for future extension like tasks.

We use the inherits field to define groups that the group inherits.

[package.dev.group1.dependencies]
python = "*"

[package.dev.group2.dependencies]
ruff = { path = "src/ruff" }

[package.dev.test]
dependencies = { pytest = "*" }
inherits = ["group1"]

[package.dev.all]
inherits = ["test", "group2"]

[package.dev.all.dependencies]
black = "*"

The dev groups can be "depended on" through the extras entry.
Both the dev and dependencies table can except the package.dev extras.
This means that the dev.<group-name>.dependencies will be added to the dependencies of the environment.

[dev]
package-x = { path = "src/package", extras = ["all"] }

# Or

[dependencies]
package-x = { path = "src/package", extras = ["group1"]}

Optional dependencies

Optional dependencies will become part of the resulting repodata of a package. Similar to how the optional-dependencies work in the PyPI ecosystem.

The definition is also grouped like dev but with no extra dependencies field.

[package.optional-dependencies.arrow]
arrow = "*"

The user defines the use of optional-dependencies with the extras field.

[dev]
package-x = { path = "src/package", extras = ["arrow"] }

# Or

[dependencies]
package-x = { path = "src/package", extras = ["arrow"]}

There is no inheritance for optional-dependencies.

As optional-dependencies will become part of the conda repodata as extras we'll also make it possible in the future to add extra's to normal binary dependencies.

[lucascolley]

RE: #4764 (comment)

The dream for SciPy would be to have dev groups inheriting from pyproject.toml dependency-groups, while applying the conda-pypi map. E.g. now we have duplication at https://github.com/scipy/scipy/blob/62b32649c44a05441aeea5dff4133e5cb39cf574/pyproject.toml#L105-L111 and https://github.com/scipy/scipy/blob/62b32649c44a05441aeea5dff4133e5cb39cf574/pixi.toml#L291-L303. Would be awesome to have just a single source of truth in pyproject.toml, then we would only need to specify non-PyPI deps explicitly in the manifest.

[Hofer-Julian]

@lucascolley that sounds pretty doable to me