CEP XXXX: Improving dependency export infrastructure

[h-vetinari]

First draft after discussion in #77. Does not contain (much) specification yet, because I'm unsure how to go about changing the schema of v1 recipes (does it need a bump in the schema version, or do we specify build tools must translate between them?), and how to deal with the repodata side of things. This is my first CEP, please excuse my lack of experience with a lot of the underlying details.

Help on these questions would be much appreciated! I decided to write up the design in more comprehensive form than originally in this comment though, in order to hopefully facilitate more effective discussion of how to solve the transition issues posed by the new design.

Closes #77 (eventually)

[h-vetinari]

pre-commit.ci autofix

[h-vetinari]

pre-commit.ci autofix

[h-vetinari]

pre-commit.ci autofix

[h-vetinari]

because I'm unsure how to go about changing the schema of v1 recipes (does it need a bump in the schema version, or do we specify build tools must translate between them?)

Wolf mentioned in private that we don't necessarily have to go to a v2 schema over this, because despite being renamed semantically, the new keys would just be extending the v1 schema, not breaking it. Of course, we'd then have to mandate mutual exclusivity between exports: and run_exports: etc., but I think that's probably a gentler approach to this than taking this comparatively minor issue as cause for "recipe v2".

The same approach (consider the new keys if present, error if not mutually exclusive with the old way) could even by used by conda-build to support the CEP¹, which would be great because a lot of our compiler feedstocks that would need this the most are not necessarily ready to be ported to v1 yet. :)

If people are in agreement over this approach, I can try to write up specification for it.

In any case, I think this is in a good enough state to ask for a first round of feedback; I'd be very curious to hear the thoughts of
@chenghlee @isuruf @jezdez @msarahan @wolfv, people from mamba, prefix, cf/core, and anyone else interested in this!

Footnotes

1. somewhat informally perhaps, since it's currently formulated atop of the v1 format. ↩

[isuruf]

Nice that we are finding paths forward! Is this similar to the solution I proposed in #129 (comment), or does it differ in some way? If that’s the case, can you elaborate how it’s different?

[h-vetinari]

Nice that we are finding paths forward!

Glad to hear it. It's not for lack of wanting to solve the issue that I had descoped self-exports; now I'm beginning to see a path that allows the various pieces to work together in a non-hacky way (where before I couldn't see it at all).

Is this similar to the solution I proposed in #129 (comment), or does it differ in some way? If that’s the case, can you elaborate how it’s different?

It's different in that we do not have to add a condition like "build_to_build: has to match run:", but we still maintain the fact that it doesn't require multiple solves.

I've been getting down into the nitty gritty details at least one level deeper (e.g. @jaimergp opened another rabbit hole under my feet about the mechanics of ignore_exports:, and I had an illuminating chat with @baszalmstra about the steps that (can) happen between the metadata and the resolver); I'm planning to expand the CEP with the design conclusions from those discussions, which should hopefully tell the whole story better than yet another very long comment.

[h-vetinari]

I've written an update based on the various discussions. Since I'm trying to design things holistically (despite the fact that I want to scope this CEP to a manageable size without external dependencies), I've added a draft of the "self-exports" CEP for now. This goes into much more detail about how things should work in practice. I had a long discussion with @baszalmstra about the list of concrete steps, but I'll be the first to admit that I'm not very familiar with that part of the process, and any mistakes are almost certainly my own.

The procedure for self-exports is a superset of steps compared to the "regular" cross-exports, so describing the more complicated workflow should immediately show how the simpler procedure works (i.e. by dropping the steps marked [self]), and thus allow for unified discussions. Once the design has survived validation by various parties, I'll split off self-exports into a separate PR.

The key point relevant for self-exports that requires conditional exports (and avoids multiple solves) is that there is a necessary pre-processing step between fetching the package metadata and feeding it to the SAT solver. This preprocessing step is where we can apply self-exports (and any matching ignore_exports: rules).

In fact, conditional dependencies as implemented by resolvo are way more powerful than what we need here. Resolvo can even handle conditional dependencies that depend dynamically on other packages in the resolution, whereas everything we need here is statically known (i.e. "are we solving for the host: environment?" If so, all we need to do is turn host_to_host: exports into regular dependencies before handing to the SAT solver. This works even for transitive self-exports, i.e. X_to_X: chained with X_to_X: both get applied in the same solve).

Nice that we are finding paths forward! Is this similar to the solution I proposed in #129 (comment), or does it differ in some way? If that’s the case, can you elaborate how it’s different?

See 2f3d23f (though I recommend reading the updates to the main exports CEP first)

[h-vetinari]

OK, I actually needed to see a third verb related to exports (inherit: in #102) for it to click for me that it makes no sense to add ignore_exports: next to exports:. So the most recent commit here changes this to

requirements:
  exports:
    inherit:  # exports to be inherited from cache stage when the output is being *built*
    ignore:   # exports to be ignored (from its dependencies) when the output is being *built*
    attach:   # exports that the output itself will apply when *used* for building other packages

I'm not super attached (ha!) to the verb attach:, but as I try to explain in the updated introduction, we have too many actions that piggyback on the verb "export" currently (itself describing the cross-environment dependency injection; whereas we also need language to inherit/ignore/attach such exports). If you have a better idea (or any other feedback), I'm all ears!

[JeanChristopheMorinPerso]

attach: # exports that the output itself will apply when used for building other packages

@h-vetinari Maybe apply? If I look at what you suggest:

inherit: # exports to be inherited from cache stage when the output is being built
ignore: # exports to be ignored (from its dependencies) when the output is being built
attach: # exports that the output itself will apply when used for building other packages

The description you give says "apply", which kind of makes sense I think. Or maybe inject could work too, considering that the export will be injected into the current artifact (itself).

[h-vetinari]

Thanks for the comment!

The description you give says "apply", which kind of makes sense I think. Or maybe inject could work too, considering that the export will be injected into the current artifact (itself).

We "apply" exports at a very different time from when we're building the current artefact that will carry the export, so I think that would be a confusing verb to use for the act of imbuing the artefact with the export in the first place.

I think "inject" would be perfectly fine (though I still have a slight preference¹²³ for "attach").

Footnotes

1. I think it depends a bit on one's mental image of what a package looks like metaphorically; if you picture it as a closed container, then "inject" or "insert" makes more sense; if you picture it as a collection of things strapped together, "attach" makes more sense. ↩

2. Since it came to me while struggling to find a good formulation above: "imbue" would also work (though probably too obscure). ↩

3. I'll admit it's tempting to have all three verbs start with the same letter; my first thought was "instate", as "install" is too overloaded here. ↩