ci: build & test images using an APT snapshot - #578
Open
Christopher Obbard (obbardc) wants to merge 7 commits into
Open
ci: build & test images using an APT snapshot#578Christopher Obbard (obbardc) wants to merge 7 commits into
Christopher Obbard (obbardc) wants to merge 7 commits into
Conversation
Christopher Obbard (obbardc)
marked this pull request as ready for review
August 15, 2026 00:53
Robie Basak (basak-qcom)
left a comment
Robie Basak (basak-qcom)
left a comment
Contributor
There was a problem hiding this comment.
I think this is a worthwhile improvement. Thank you! A couple of comments inline for discussion.
| debos_extra_args: -t snapshot:20260810T082104Z | ||
| # nothing consumes these images, so don't spend the time compressing | ||
| # them nor the storage keeping them around | ||
| upload_artifacts: false |
Contributor
There was a problem hiding this comment.
I'm unsure about this. If we get a test failure, debugging it would be assisted by having the build available. Theoretically it might be reproducible, but maybe that's the bug we'd be debugging. Does that make this a premature optimisation, or is there really serious enough pressure on the artifact store that we can't keep the artifacts around for a month or so?
Contributor
Author
There was a problem hiding this comment.
This was simply a premature optimisation and I am not attached to it. I agree with your thoughts about keeping the artifacts.
I will modify the PR to upload the artifacts, possibly under a separate directory build_snapshot or similar (to keep parity with the name).
I will keep the commit where I allowed disabling artifact upload separate for a later PR perhaps :-)
Contributor
Author
There was a problem hiding this comment.
Done! I am not sure what happens to the artifacts if a job fails to build though; I think nothing gets uploaded (pending verification). Do you think that will matter?
Contributor
|
Ah sorry I just saw the reference to #564 after clicking submit. If we're closing that issue, then should we add some snapshot-specific tests? I suggest:
|
Contributor
|
A third, harder test might be to ensure that we don't have packages that are installed that are newer than the snapshot selected. That might not be worth it right now though for the effort it would take. |
Contributor
Author
Thanks, I agree! I will do that and either extend the existing qemu test or just do something like check the files in the rootfs. Not decided yet. We could also check the sources.list contains the snapshot ID, it's another trivial check.
Agree.
I agree, but I think that check could be a bit too much effort to implement right now, but I will create a follow-up issue about it once I've done 1 & 2. |
Christopher Obbard (obbardc)
force-pushed
the
wip/obbardc/ci-snapshots
branch
from
August 21, 2026 20:37
b0c1521 to
e5020bd
Compare
Christopher Obbard (obbardc)
changed the title
Christopher Obbard (obbardc)
changed the title
Snapshot builds pin the APT sources to a dated archive (see docs/snapshot.md). They have their own code paths in the debos recipes which nothing in the CI currently exercises, so build an image with them regularly, in a separate workflow. It runs weekly and on demand rather than daily or on pull requests: these are full image builds and snapshot.debian.org is slower and more rate-limited than the regular Debian mirrors. Keeping it out of build.yml lets it have its own build cadence, gives its artifacts a separate destination and doesn't add any overhead to the existing nightly image builds. This workflow currently only tests that the images build with a snapshot: no LAVA job boots the images on hardware, the images are not tested other than booting in QEMU. Later commits will add CI checks to verify the snapshot-enabled images are built correctly. The built image artifacts are uploaded to S3. The workflow only builds the default rootfs variant for each supported suite, since the snapshot code paths are independent of the rootfs variant. The snapshot timestamp is derived from the timestamp of the commit being built, in the YYYYMMDDTHHMMSSZ form required. The timestamp is printed to the job log and the run summary along with the commit it came from to allow reproduction. The snapsnot timestamp can be overwritten when running the workflow manually. Any correctly formed timestamp resolves to the state of the APT repositories which were live at that moment, so the value only has to be a date the archives still serve; not one tied to a specific publication. Using the commit date to generate the timestamp means the workflow needs no maintenance and the timestamp moves forward on its own as the repository is worked on. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Currently each test function creates its own VM, which means paying for a fresh boot of an emulated aarch64 guest each time, which can be many minutes per test. Switch the pytest fixture to be session-scoped so one VM boot serves all of the tests. Logging in to the guest now has to happen first, before any test can run, so move it into a helper function called after the VM starts. The verification of a mandatory password reset at boot is still exercised, just when the test logs into the VM for the first time. The tests share a console from here on, so each of them has to leave it at a shell prompt; the prompt and the password become constants rather than literals repeated at every call site. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Matching the console output of a command has to work around the serial echo of the command itself, which is why the random-seed check matched "not found" twice and why it printed that string at all rather than just running grep. Switch to a helper run() which runs a shell command and returns the exit status the shell reports for it; which should be more resilient than parsing the VM's serial output. Rework the random-seed check to use this helper: dump the journal to a file first (so that both reading the journal and searching the file are checked) and assert the exact status grep returns so that "found no such line" is told apart from "grep could not read the file". Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
The QEMU tests run for every image debos.yml builds and must hold for all of them, so what differs between images has to be described to them rather than expressed by running different test files. Give debos.yml a qemu_test_env input for that: NAME=value lines, one per line, which a caller uses to say how it built the image it is asking to have tested. The whole set travels as one QEMU_TEST_ENV payload, set on the test step only -- it describes the image to these tests and has no business affecting any other step -- and image_environment() in ci/qemu_test.py splits it at import, before pytest collects anything in the module. Splitting it in Python rather than in the step itself: that step runs in a container with no bash, so POSIX sh would need a heredoc, the positional parameters as the only list it has and a case glob standing in for a pattern match -- thirty lines which could only ever be run in CI, to do what a dozen testable ones do here. Listing the variables in the step's env: instead would hardcode in debos.yml which variables its callers care about. Not a conftest.py either. That applies to every test in ci/, while these variables describe an image only the QEMU tests care about: ci/gen_flash_dirs_test.py, a pure unit test of gen-flash-dirs.py, would import it and have its process environment rewritten for nothing. A payload in a variable rather than a file or a CSV: the transport was never the problem. A file would have to be written, named and cleaned up by the same shell this avoids, and CSV or JSON would buy quoting rules for values which are timestamps and suite names. QEMU_TEST_ENV, when set at all, is the whole of what the tests are told, and the ambient environment is ignored; with no payload the variables are read from the environment directly, which is what running by hand does. In CI the payload is the caller's complete statement about the image it just built, so a stray EXPECTED_* inherited by a runner has no way to contribute to it. The names it may set are whitelisted in KNOWN, and a payload which sets anything else fails the run rather than being dropped: a name nothing reads is a mistake in the caller, and these variables are not optional detail -- one which never arrives does not mean "skip that check", it means "the image was built without that". Every name in KNOWN is reported before the tests run, set or not, and with the source it was read from, because a caller which meant to pass one and did not looks from the outside exactly like one which never meant to. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Only one thing has ever been passed through qemu_test_env, so the multi-line handling in debos.yml -- and the tests reading more than one variable -- has never actually run. Pass a second one, and give it something to do. EXPECTED_SUITE is the Debian suite the build asked for, checked against VERSION_CODENAME in /etc/os-release: what the image *is*, rather than how the recipe happens to name its APT sources, and it comes from the base-files the bootstrap pulled in, so a suite applied only to the later package installs is caught too. debos.yml sets it rather than its callers. It was told which suite to build and passes it to debos on every run, so it can describe the image without any caller having to remember to -- and the check then covers every image built here, not only those from the callers which happened to opt in. The caller's own lines follow it in the payload and win on a repeated name, a caller describing the image it asked for being the more specific statement of the two. There is nothing to assert when the variable is unset -- every image is some suite -- so the test skips, via a marker rather than a skip() in the body so that the VM fixture is not booted just to skip. sid and unstable track whichever codename is next and /etc/os-release records neither, so those must not be passed; this is documented next to the variable. docs/qemu-tests.md is new, and is where this is written down: the tests run for every image and are not a snapshot feature, so documenting them inside docs/snapshot.md would put them behind a heading no one reading about suites would think to open. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
A snapshot build which silently fell back to the live mirrors would still pass CI, as would one which ships the pinned sources it was built with which could tie every future APT update to a dated archive. Add a test_snapshot function to the existing QEMU unit tests and run it against all built images. Verify both of the above cases and ensure that each image was built with the expected snapshot timestamp, or without if the image was not built from a snapshot. Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Signed-off-by: Christopher Obbard <chris.obbard@oss.qualcomm.com>
Christopher Obbard (obbardc)
force-pushed
the
wip/obbardc/ci-snapshots
branch
from
August 26, 2026 08:46
45187ec to
5031d82
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Snapshot builds pin the APT sources to a dated archive (see
docs/snapshot.md). They have their own code paths in the rootfs and
image recipes which nothing else exercises, so build an image with them
regularly, in a workflow of their own.
It runs weekly and on demand rather than daily or on pull requests:
these are full image builds and snapshot.debian.org is slower and more
rate-limited than the regular mirrors. Keeping it out of build.yml is
what lets it have that cadence and gives its artifacts a destination of
their own and doesn't pollute the nightly builds with additional build jobs or artifacts.
This only ever answers "do the snapshot code paths still work?": no LAVA
job boots the images, the build going green and the QEMU tests ran by
debos.yml are the coverage we're after. The artifacts are uploaded all
the same, so that a failure can be investigated. The default image variant is
enough, as the snapshot code paths don't depend on it.
The snapshot timestamp is derived from the committer date of the commit being
built. Any well-formed timestamp resolves to the publication which was live at that
moment, so the value only has to be a date the archives still serve, not
one which coincides with a publication. The commit date is such a date
and needs no maintenance: it moves forward on its own as the repository
is worked on, while two runs of the same commit still pin the same
publication, so a failure points at a change in our recipes rather than
at a change in the archives.
Fixes: #564
TODO
flashbuild as it is a waste of time/space?test-snapshotjob:snapshot.txtinto artifactsqemu_snapshot_tests.py, add tests intoqemu_test.pyand only pass vars in debos job. this will allow us to run snapshot tests on non-snapshot enabled image too, and removeextra_testsfrom debos.yml. also, renametest_env.Verification
TODO: See sample build-snapshot.yml workflow run