ephemeral: Mask bootloader-update.service - #5
Closed
cgwalters-bot wants to merge 1 commit into
Closed
cgwalters-bot wants to merge 1 commit into
cgwalters-bot wants to merge 1 commit into
Conversation
cgwalters-bot
added a commit
that referenced
this pull request
Sep 24, 2026
bcvk bind-mounts its own binary into the container it launches. From a toolbox or distrobox, podman is often forwarded to the host, so that path is resolved on the host. With bcvk installed only in the toolbox this failed with the rather cryptic Podman command failed: Error: statfs /usr/bin/bcvk: no such file or directory Rather than trying to detect a toolbox or distrobox, which can't tell a forwarded podman from one installed in the container, key on the error itself: if podman reports a bind mount source missing that bcvk can see, podman must be looking at a different filesystem. That covers a remote podman client too. Wrap the error with an explanation and a pointer to new docs on how to run bcvk in those setups. This only improves the failure; actually supporting bcvk from a toolbox is the larger design question in the issue. `ephemeral run` execs podman directly, so its error stays podman's own. Related: #5 Generated-by: AI
cgwalters-bot
added a commit
that referenced
this pull request
Sep 24, 2026
bcvk bind-mounts its own binary into the container it launches. From a toolbox or distrobox, podman is often forwarded to the host, so that path is resolved on the host. With bcvk installed only in the toolbox this failed with the rather cryptic Podman command failed: Error: statfs /usr/bin/bcvk: no such file or directory Rather than trying to detect a toolbox or distrobox, which can't tell a forwarded podman from one installed in the container, key on the error itself: if podman reports a bind mount source missing that bcvk can see, podman must be looking at a different filesystem. That covers a remote podman client too. Wrap the error with an explanation and a pointer to new docs on how to run bcvk in those setups, with the guidance in color_eyre note and suggestion sections. This only improves the failure; actually supporting bcvk from a toolbox is the larger design question in the issue. `ephemeral run` execs podman directly, so its error stays podman's own. Related: #5 Generated-by: AI
cgwalters
approved these changes
Sep 25, 2026
bootupd's bootloader-update.service looks for the block device backing /boot or /sysroot, and in an ephemeral VM the root is virtiofs, so it fails with "Failed to find block device from /boot or /sysroot". The system then comes up "degraded", which breaks every test that boots an image with bcvk and checks systemctl is-system-running, forcing each consumer (e.g. bootc-dev/infra, bootupd CI) to mask it themselves. There is never a bootloader to update in an ephemeral VM, so mask it by default like systemd-journal-flush.service. coreos/bootupd#1072 makes bootupd skip this case itself, but existing images will carry older bootupd for a long time. The ephemeral system-command integration test now asserts the VM reaches "running", so a unit failing in every ephemeral boot is caught here rather than by bcvk's consumers. Generated-by: AI Signed-off-by: Colin Walters <walters@verbum.org>
Collaborator
Author
|
Signed off 1 commit(s) with |
cgwalters-bot
force-pushed
the
bot/mask-bootloader-update
branch
from
September 25, 2026 22:23
645fcab to
5280e78
Compare
Collaborator
Author
|
Opened upstream as bootc-dev#376. Closing this review draft. |
cgwalters-bot
added a commit
that referenced
this pull request
Sep 30, 2026
bcvk bind-mounts its own binary into the container it launches. From a toolbox or distrobox, podman is often forwarded to the host, so that path is resolved on the host. With bcvk installed only in the toolbox this failed with the rather cryptic Podman command failed: Error: statfs /usr/bin/bcvk: no such file or directory Rather than trying to detect a toolbox or distrobox, which can't tell a forwarded podman from one installed in the container, key on the error itself: if podman reports a bind mount source missing that bcvk can see, podman must be looking at a different filesystem. That covers a remote podman client too. Wrap the error with an explanation and a pointer to new docs on how to run bcvk in those setups, with the guidance in color_eyre note and suggestion sections. This only improves the failure; actually supporting bcvk from a toolbox is the larger design question in the issue. `ephemeral run` execs podman directly, so its error stays podman's own. Related: #5 Generated-by: AI
cgwalters-bot
added a commit
that referenced
this pull request
Sep 30, 2026
bcvk bind-mounts its own binary into the container it launches. From a toolbox or distrobox, podman is often forwarded to the host, so that path is resolved on the host. With bcvk installed only in the toolbox this failed with the rather cryptic Podman command failed: Error: statfs /usr/bin/bcvk: no such file or directory Rather than trying to detect a toolbox or distrobox, which can't tell a forwarded podman from one installed in the container, key on the error itself: if podman reports a bind mount source missing that bcvk can see, podman must be looking at a different filesystem. That covers a remote podman client too. Wrap the error with an explanation and a pointer to new docs on how to run bcvk in those setups, with the guidance in color_eyre note and suggestion sections. This only improves the failure; actually supporting bcvk from a toolbox is the larger design question in the issue. `ephemeral run` execs podman directly, so its error stays podman's own. Related: #5 Generated-by: AI Signed-off-by: Colin Walters <walters@verbum.org>
cgwalters-bot
added a commit
that referenced
this pull request
Sep 30, 2026
bcvk bind-mounts its own binary into the container it launches. From a toolbox or distrobox, podman is often forwarded to the host, so that path is resolved on the host. With bcvk installed only in the toolbox this failed with the rather cryptic Podman command failed: Error: statfs /usr/bin/bcvk: no such file or directory Rather than trying to detect a toolbox or distrobox, which can't tell a forwarded podman from one installed in the container, key on the error itself: if podman reports a bind mount source missing that bcvk can see, podman must be looking at a different filesystem. That covers a remote podman client too. Wrap the error with an explanation and a pointer to new docs on how to run bcvk in those setups, with the guidance in color_eyre note and suggestion sections. This only improves the failure; actually supporting bcvk from a toolbox is the larger design question in the issue. `ephemeral run` execs podman directly, so its error stays podman's own. Related: #5 Generated-by: AI Signed-off-by: Colin Walters <walters@verbum.org>
cgwalters-bot
added a commit
that referenced
this pull request
Oct 1, 2026
bcvk bind-mounts its own binary into the container it launches. From a toolbox or distrobox, podman is often forwarded to the host, so that path is resolved on the host. With bcvk installed only in the toolbox this failed with the rather cryptic Podman command failed: Error: statfs /usr/bin/bcvk: no such file or directory Rather than trying to detect a toolbox or distrobox, which can't tell a forwarded podman from one installed in the container, key on the error itself: if podman reports a bind mount source missing that bcvk can see, podman must be looking at a different filesystem. That covers a remote podman client too. Wrap the error with an explanation and a pointer to new docs on how to run bcvk in those setups, with the guidance in color_eyre note and suggestion sections. This only improves the failure; actually supporting bcvk from a toolbox is the larger design question in the issue. `ephemeral run` execs podman directly, so its error stays podman's own. Related: #5 Generated-by: AI Signed-off-by: Colin Walters <walters@verbum.org>
cgwalters-bot
added a commit
that referenced
this pull request
Oct 3, 2026
bcvk bind-mounts its own binary into the container it launches. From a toolbox or distrobox, podman is often forwarded to the host, so that path is resolved on the host. With bcvk installed only in the toolbox this failed with the rather cryptic Podman command failed: Error: statfs /usr/bin/bcvk: no such file or directory Rather than trying to detect a toolbox or distrobox, which can't tell a forwarded podman from one installed in the container, key on the error itself: if podman reports a bind mount source missing that bcvk can see, podman must be looking at a different filesystem. That covers a remote podman client too. Wrap the error with an explanation and a pointer to new docs on how to run bcvk in those setups, with the guidance in color_eyre note and suggestion sections. This only improves the failure; actually supporting bcvk from a toolbox is the larger design question in the issue. `ephemeral run` execs podman directly, so its error stays podman's own. Related: #5 Generated-by: AI Signed-off-by: Colin Walters <walters@verbum.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Decision needed before this goes anywhere: the same problem can be fixed in bootupd instead, and masking the unit here breaks bootupd's own proposed CI check (see "Alternatives" below).
bootupd's
bootloader-update.servicelooks for the block device backing/bootor/sysroot. In an ephemeral VM the root is virtiofs, so it fails with "Failed to find block device from /boot or /sysroot" and the system comes updegraded. That breaks any test that boots an image withbcvk ephemeraland checkssystemctl is-system-running.There is never a bootloader to update in an ephemeral VM, so this masks the unit by default on the kernel command line, like
systemd-journal-flush.servicealready is. Masking a unit that doesn't exist is harmless, so images without bootupd are unaffected. Images already in use carry the current bootupd, so a fix there only helps once they're rebuilt with it.test_run_ephemeral_ssh_system_commandused to runsystemctl is-system-running || true, which checked nothing. It now waits for boot to finish and asserts the state isrunning(printing the failed units otherwise), so a unit that fails in every ephemeral boot is caught here.Alternatives, and a conflict
bootupctl updateskip the update when no block device backs the root.ExecConditionthat skips live environments ([[ ! $(findmnt -n -o FSTYPE /sysroot) =~ ^(erofs|squashfs)$ ]], see the unit). Addingvirtiofsto that regex would make the unit skip cleanly instead of failing.systemctl is-active bootloader-update.serviceunderbcvk ephemeral run-ssh. With this mask the unit never runs, so that assertion would fail, and this PR adds no kernel argument or flag to opt out of the mask.So the options are: fix it in bootupd (#1072 or the
ExecConditionchange) and drop this PR, or keep the bcvk mask and add an opt-out (e.g. a flag that skips the default masks).Testing
On a 16-core RHEL 10.2 devspace with KVM:
make validateandcargo testpass (82 + 22 unit tests).degraded,bootloader-update.service loaded failed) and passes with this change (twice in a row, ~19s each) on centos-bootc:stream10.degradedwith bootloader-update.service failed on main,runningwith this change.One caveat: an earlier run of the new test, while two other VM tests and a toolbox experiment were running on the same machine, timed out waiting for the VM to come up (240s). It passed quickly when rerun, so I believe that was contention, not this change.
CI note (2026-09-24): the
integration-tests (3)failure is unrelated to this change:test_run_ephemeral_dns_resolutionfailed with "Connection timed out during banner exchange" fromephemeral run-ssh, theConnectTimeoutflake targeted by #8. Reran the failed jobs once.Generated-by: https://github.com/cgwalters/#llms
Review draft in cgwalters-forge, not upstream yet. This section is removed when the PR is opened upstream.
bootc-dev/bcvk, basemainPVTI_lAHOAQ_SPs4Bj2Gizg8Pl94To review:
/promoteon a line of its own, to open it upstream, ready for review. Either covers only the commits pushed so far.Signed-off-by: Colin Walters <walters@verbum.org>to the commits lacking it (the bot's and yours; anyone else's only if you ask), with you as committer./draftline (in the same comment or before) to open it upstream as a draft (/readyundoes that).