feat!: prefer packages from QLI repository - #576
feat!: prefer packages from QLI repository#576Christopher Obbard (obbardc) wants to merge 9 commits into
Conversation
The `qsc-deb-releases` Artifactory repository is no longer used; the package delta it carried is now available from the Qualcomm Linux (qli) Debusine archive. Remove the overlay and its keyring, the APT sources/preferences generated from it and switch the default value of the overlays option to `none`. Closes: qualcomm-linux#536 Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Packages which exist in both the Debian repository (incl. backports) as well as the Qualcomm Linux APT repository should be installed from the Qualcomm Linux APT repository. Write APT preferences configuration to match the above intent. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
The initial packages are bootstrapped from Debian before the Qualcomm Linux APT sources are configured. The subsequent full-upgrade applies the QLI APT repository and is therefore the first point at which installed Debian packages can be replaced by packages from the QLI APT repository. Where QLI provides a lower version than Debian (and the APT preferences are set to prefer QLI) this package replacement is considered a downgrade. APT refuses such downgrades when running with `-y` unless `--allow-downgrades` is specified. Add `--allow-downgrades` to the APT call to allow downgrading packages. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Now that the QLI repository is pinned above Debian, a package which is present in both the local repository and QLI would otherwise resolve to the QLI version, defeating the point of installing it from a local APT repository. Pin the local repository at 1002 so packages from the local repository are preferred over all other APT repositories. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
`qliaptrepo` was originally a boolean toggle to enable the QLI archive. Turn it into the name of the Debusine archive to configure so the staging archive can be selected without editing the recipe. Accept `qli` (the new default), `qli-staging` and `false` to configure no Qualcomm Linux sources at all; anything else causes the recipe to fail. Configure the QLI Staging APT repository (instead of the standard QLI repository) when `qliaptrepo` is set to `qli-staging`. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
`qliaptrepo` was originally a boolean toggle to enable the QLI APT repository. Turn it into the name of the repository to configure so the QLI staging repository can be optionally configured. Accept `qli` (the new default), `qli-staging` and `false` to configure no Qualcomm Linux sources at all; anything else causes the recipe to fail. The QLI staging repository is layered on top of the QLI release repository rather than replacing it, so `qli-staging` configures both: packages which have not been rebuilt in staging still have to come from `qli`. Derive the list of archives to configure from `qliaptrepo` up early and loop over it when writing the sources and the snapshot URL rewrite table. Note that the two archives cannot be given different pin priorities: the Debusine-generated Release files carry no Origin or Label and share a hostname, so no pin expression can tell them apart. Both sit at priority 1001, which makes the highest version the candidate; as staging is where newer builds land first, its packages are preferred in practice. TODO: squash previous commit into this one. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
The Qualcomm Linux archives are only published for trixie and forky, so fail the recipe early when they are enabled on any other suite instead of silently skipping the sources. Builds of other suites now have to pass `-t qliaptrepo:false` explicitly. With that check in place, the sources no longer need to be guarded against unstable. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
The QLI APT sources are only configured when `qliaptrepo` is not disabled, so gating the fastrpc-tests installation on the same single condition is enough; the extra suite check was redundant. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
The QLI APT sources are configured when `qliaptrepo` is not `false`; gate the multimedia variant properly. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
| # When both are enabled they sit at the same priority so the repository | ||
| # with the higher version wins. | ||
| Package: * | ||
| Pin: origin "deb.debusine.qualcomm.com" |
There was a problem hiding this comment.
Need to open an issue to modify the Debusine configuration to add Origin/Label fields to Release and rework this change.
There was a problem hiding this comment.
There was a problem hiding this comment.
The Origin/Label fields have been added to the QLI repositories now, so need to rework this.
|
To summarise my memory of a previous discussion: I believe there are exactly two options to choose from in terms of user visible behaviour when a new package is published into Debian for which we have a delta in our overlay:
In both cases, for security reasons, we should promptly update the delta package in our overlay if the new package in Debian is a security update. Either: A) we have a suitable process for doing that promptly, thinking about monitoring, assignments and rotas, acceptable lag, vacation, etc, or B) we don't. We make no public commitment for A this since our overlay is for reference use by product creators and not for direct use. Regardless we should still try to do the right thing. If A, then then option 2 is probably better, since by definition we'd be able to rebase promptly so the security vulnerability window would be small. If B, then I think 1 is better than attempting 2, but failing on updating promptly and leaving the window wide and indefinite. My opinion is therefore that we go for 1 for the time being, if/until we have defined A in some way and have monitoring in place, even if that doesn't translate to a public commitment. I think this would fit current expectations around providing reference packages. On the matter of pinning, I prefer to using versions to get the behaviour we want. I think to get 1 we can rely on that without needing pinning, and we only need pinning for 2. So for now, the above means I don't think we should pin for this purpose. |
|
(further discussion welcome; this is just my opinion!) |
|
Our current priorities meant #600 affected the images Robie Basak (@basak-qcom) there might be further options than the one you list; we could distinguish the different classes of Debian updates, e.g. -security is something we're worried of gating, but I'm less worried about holding back on a new backport or a new stable update for a little bit. So we could keep fast tracking security updates, abandoning our QLI changes, but not other updates. My personal preference remains to prioritize our QLI overlay all the time, even over security updates, and progressively adjust to integrate these faster. |
| - `qliaptrepo`: which Qualcomm Linux APT repositories to configure in the root | ||
| filesystem; either `qli` (the default) for the release archive alone, or | ||
| `qli-staging` which configures both the staging archive and the release | ||
| archive it is layered on top of. Set it to `false` to leave the image with |
|
We've been discussing special casing a set of packages (like we do for trixie-backports) and also how -backports might be a channel of security updates (of the backport). In any case, we should capture our conclusion on policy in a visible place in our docs too. |
|
We should probably pin QLI at the same priority as Debian backports (higher than debian non-backports?). That way, packages from QLI can be preferred when they are newer, but if we stop maintaining a package in QLI and a newer version appears in Debian backports, normal APT version ordering will allow Debian to take over rather than an older QLI package remaining preferred indefinitely. Also, should we have this config even in the "vanilla" debian images in case someone sets the repo up later? |
3 participants
Depends on #548
Policy change: packages from QLI now takes precedence over Debian, including security updates
Replacing the overlay is not a like-for-like swap of where packages come from; with this
PR we change the policy for which archive is used when the same package exists in more than
one. This needs wider discussion.
Before the overlay pinned a single source package:
Everything else came from Debian at the default priority. The pin was an
explicit allowlist and at 950 (< 1000) APT would never replace an
already installed Debian package with a lower version from the overlay.
This PR pins the entire archive:
Which results in:
trixie(500),trixie-updates(500),trixie-security(500)and
trixie-backports(900). Note that priority (not version) decides thecandidate: once QLI is above 500, a Debian security update loses to the QLI
version even when Debian's version is higher.
the QLI version. This is why 3cff730 has to add
--allow-downgradesto thefull-upgrade; the flag is a symptom of the pin, not an independent fix.qli,qli-stagingand any futurearchive served from
deb.debusine.qualcomm.com. Separately, we should addOrigin/Labelto the debusine-generated repository Release file and use that matching here. See debusine.qualcomm.com: AddOrigin/Labelfields to qli/qli-staging APT repositories debusine-action#115The effect of this is if QLI carries a package that Debian later issues a DSA for, images
keep shipping the unfixed QLI build until it is present in QLI repo.
A possible outcome could be that Debian security fixes should take priority over QLI changes, i.e. security updates should win.
Verification
To verify this, I built the rootfs recipe with the following cases: