Spike: Design multi-target approach

[sgallagher]

We need to decide how to implement the multi-target approach for ELNBuildSync. Some questions that need to be answered:

1. Do we handle batches that contain packages intended for separate targets independently, or does a batch contain all packages that EBS understands and we just issue the tasks to different build targets and Bodhi updates?
2. If we handle them as separate batches, are they fully independent (once a batch for eln-extras completes, it immediately starts the next eln-extras batch) or do we have all targets start and stop together? (So if we have a package for eln and one for eln-extras, the next batches will only start once both eln and eln-extras are done?

As questions are asked, update this description.

@bhoy-troy @yselkowitz

[sgallagher]

Problem: Extras depending on skip-tagged ELN packages

For each target, we will need a separate build side-tag, since they will need to build from that target's buildroot. Side tags cannot (or are not configured to?) derive from other side-tags. How do we handle "skip-tag" situations like llvm if some packages are building for ELN and some for ELN Extras in the same batch?

To explain in more detail: LLVM on ELN builds with a different set of features and ELF symbols from Rawhide. As a result, we have to avoid pulling LLVM from Rawhide into the buildroot or else the packages it builds will fail at runtime when linked to the libllvm from ELN.

Now: what happens if we have llvm in the batch as well as an ELN Extras package that builds from llvm? We can't tag the Rawhide build into the ELN Extras target and it would be tremendously wasteful to rebuild llvm in the ELN Extras buildroot a second time.

Solution 1

We could simulate inheritance by running all targets' slices concurrently and then tagging the built results into each dependent side-tag before kicking off the next slice of builds.

For example, let's assume we have:

• llvm which builds in the eln buildroot at slice 0
• clang which builds in the eln buildroot at slice 1
• criticalapp, which BuildRequires: llvm and builds in the eln buildroot at slice 1000 (default)
• funapphelper, which BuildRequires: llvm and builds in the eln-extras buildroot at slice 1
• funapp, which BuildRequires: funapphelper and BuildRequires: clang and builds in the eln-extras buildroot at slice 1000 (default)

As ELNBuildSync proceeds, it would divide these up into:

{
  eln:
    0: [llvm],
    1: [clang],
    1000: [criticalapp]
  eln-extras:
    1: [funapphelper],
    1000: [funapp]
}

EBS creates two side-tags:

• eln buildroot (eln-build-side-1234)
• eln-extras buildroot (eln-extras-build-side-1235)

Slice 0

EBS would tag criticalapp (not a skip-tag package) into both the eln-build-side-1234 and eln-extras-build-side-1235 buildroots. It would not tag llvm into either, because it is skip-tag.

It would see that the eln buildroot (eln-build-side-1234) needs to build llvm and the eln-extras buildroot doesn't need to build anything, so it would just build llvm. When that concludes, it would tag the llvm build into both the eln buildroot and the eln-extras buildroot (eln-extras-build-side-1235).

Slice 1

EBS would tag the built llvm from slice 0 into eln-extras-build-side-1235. It would then concurrently build clang and funapphelper in the eln-build-side-1234 and eln-extras-build-side-1235 buildroots, respectively.

Slice 1000

EBS would tag the built clang from slice 1 into eln-extras-build-side-1235 and funapphelper from slice 1 into eln-build-side-1234 (I'll explain why we do this later).

Next, it will concurrently trigger the build of criticalapp in eln-build-side-1234 and the build of funapp in eln-extras-side-2345. After it waits for both of these, it would promote the llvm, clang, funnaphelper, criticalapp and funapp draft builds and create two new side-tags: eln-build-side-1236 and eln-extras-build-side-1237. It would tag llvm, clang and criticalapp into eln-build-side-1236 and submit that to Bodhi as an update and tag funapphelper and funapp into eln-extras-build-side-1237 and submit it to another Bodhi update.

Why the extra tags?

The idea is that at each slice, we would cross-tag every completed build from the prior slice into all of the buildroots. By doing this, we avoid needing to establish an "order" between all of the buildroots. The downside to this is that there is a very slight chance that one of the later slices might opportunistically pull in a package if it happens to be available (via Recommends: or some other weak dependency) and that could change the build results. I think the risk of that is very low and would likely result in that package getting pulled in by Content Resolver on the next iteration, surfacing it.

Solution 2

This is just Solution 1, except instead of all the cross-tagging, we establish a precedence order and only tag the builds into the higher-order buildroots. I feel like having two separate ordering mechanisms (one for slices, another for buildroots) will be confusing on the configuration side of things, though. I'm open to being convinced.

Solution 3

We could create an ordering of variants ( eln -> eln-extras -> hyperscale -> <...>) and serially execute them once each lower-order sub-batch had completed and was tagged into their destination tags (after Bodhi updates finish and push to stable). This is very slow, but it avoids a bunch of extra tagging as described above and is the most conservative option where the buildroot is concerned. It also means we only need to have one build side-tag in operation at a time.

Solution 4

Investigate whether side-tags can be made to inherit from other side-tags (is this a policy choice or a hard limitation in Koji?). If they can inherit from one another, we can do a mix of Solutions 1, 2 and 3 where instead of cross-tagging, we just use Koji inheritance to handle which builds are available in the buildroot.

Conclusion

Option 4 would probably be my first choice, if it's technically possible. After that, I prefer Options 1, 2 and 3 in that order.

[sgallagher]

In a meeting today, we discussed these solutions and we are leaning towards Solution 1 as the implementation.

Solution 4 will not work, since they would be inheriting from a common base tag, rather than the correct tag for the target buildroot. So our next-best option is the cross-tagging.