Skip to content

ephemeral: Mask bootloader-update.service - #5

Closed
cgwalters-bot wants to merge 1 commit into
mainfrom
bot/mask-bootloader-update
Closed

cgwalters-bot wants to merge 1 commit into
mainfrom
bot/mask-bootloader-update

Conversation

@cgwalters-bot

@cgwalters-bot cgwalters-bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Decision needed before this goes anywhere: the same problem can be fixed in bootupd instead, and masking the unit here breaks bootupd's own proposed CI check (see "Alternatives" below).

bootupd's bootloader-update.service looks for the block device backing /boot or /sysroot. In an ephemeral VM the root is virtiofs, so it fails with "Failed to find block device from /boot or /sysroot" and the system comes up degraded. That breaks any test that boots an image with bcvk ephemeral and checks systemctl is-system-running.

There is never a bootloader to update in an ephemeral VM, so this masks the unit by default on the kernel command line, like systemd-journal-flush.service already is. Masking a unit that doesn't exist is harmless, so images without bootupd are unaffected. Images already in use carry the current bootupd, so a fix there only helps once they're rebuilt with it.

test_run_ephemeral_ssh_system_command used to run systemctl is-system-running || true, which checked nothing. It now waits for boot to finish and asserts the state is running (printing the failed units otherwise), so a unit that fails in every ephemeral boot is caught here.

Alternatives, and a conflict

  • update: Skip bootloader update when no block devices back the root coreos/bootupd#1072 (open) proposes making bootupctl update skip the update when no block device backs the root.
  • Smaller: the unit already has an ExecCondition that skips live environments ([[ ! $(findmnt -n -o FSTYPE /sysroot) =~ ^(erofs|squashfs)$ ]], see the unit). Adding virtiofs to that regex would make the unit skip cleanly instead of failing.
  • bootupd#1072 also adds a CI smoke test that runs systemctl is-active bootloader-update.service under bcvk ephemeral run-ssh. With this mask the unit never runs, so that assertion would fail, and this PR adds no kernel argument or flag to opt out of the mask.

So the options are: fix it in bootupd (#1072 or the ExecCondition change) and drop this PR, or keep the bcvk mask and add an opt-out (e.g. a flag that skips the default masks).

Testing

On a 16-core RHEL 10.2 devspace with KVM:

  • make validate and cargo test pass (82 + 22 unit tests).
  • The tightened integration test fails with bcvk from main (degraded, bootloader-update.service loaded failed) and passes with this change (twice in a row, ~19s each) on centos-bootc:stream10.
  • By hand on fedora-bootc:44: degraded with bootloader-update.service failed on main, running with this change.

One caveat: an earlier run of the new test, while two other VM tests and a toolbox experiment were running on the same machine, timed out waiting for the VM to come up (240s). It passed quickly when rerun, so I believe that was contention, not this change.

CI note (2026-09-24): the integration-tests (3) failure is unrelated to this change: test_run_ephemeral_dns_resolution failed with "Connection timed out during banner exchange" from ephemeral run-ssh, the ConnectTimeout flake targeted by #8. Reran the failed jobs once.

Generated-by: https://github.com/cgwalters/#llms


Review draft in cgwalters-forge, not upstream yet. This section is removed when the PR is opened upstream.

  • Upstream: bootc-dev/bcvk, base main
  • Board item: PVTI_lAHOAQ_SPs4Bj2Gizg8Pl94
  • Fork CI: off; the devspace testing described above is this PR's CI, and upstream CI runs once it is opened there

To review:

  • Approve, or comment /promote on a line of its own, to open it upstream, ready for review. Either covers only the commits pushed so far.
  • If upstream requires DCO, approving also signs off: promote adds Signed-off-by: Colin Walters <walters@verbum.org> to the commits lacking it (the bot's and yours; anyone else's only if you ask), with you as committer.
  • Add a /draft line (in the same comment or before) to open it upstream as a draft (/ready undoes that).
  • Close to drop it.
  • Edit the title and description freely: they become the upstream PR's. Review comments are squashed into the commits they concern, with a reply here.

cgwalters-bot added a commit that referenced this pull request Sep 24, 2026
bcvk bind-mounts its own binary into the container it launches. From a
toolbox or distrobox, podman is often forwarded to the host, so that
path is resolved on the host. With bcvk installed only in the toolbox
this failed with the rather cryptic

  Podman command failed: Error: statfs /usr/bin/bcvk: no such file or directory

Rather than trying to detect a toolbox or distrobox, which can't tell a
forwarded podman from one installed in the container, key on the error
itself: if podman reports a bind mount source missing that bcvk can
see, podman must be looking at a different filesystem. That covers a
remote podman client too. Wrap the error with an explanation and a
pointer to new docs on how to run bcvk in those setups.

This only improves the failure; actually supporting bcvk from a toolbox
is the larger design question in the issue. `ephemeral run` execs
podman directly, so its error stays podman's own.

Related: #5
Generated-by: AI
cgwalters-bot added a commit that referenced this pull request Sep 24, 2026
bcvk bind-mounts its own binary into the container it launches. From a
toolbox or distrobox, podman is often forwarded to the host, so that
path is resolved on the host. With bcvk installed only in the toolbox
this failed with the rather cryptic

  Podman command failed: Error: statfs /usr/bin/bcvk: no such file or directory

Rather than trying to detect a toolbox or distrobox, which can't tell a
forwarded podman from one installed in the container, key on the error
itself: if podman reports a bind mount source missing that bcvk can
see, podman must be looking at a different filesystem. That covers a
remote podman client too. Wrap the error with an explanation and a
pointer to new docs on how to run bcvk in those setups, with the
guidance in color_eyre note and suggestion sections.

This only improves the failure; actually supporting bcvk from a toolbox
is the larger design question in the issue. `ephemeral run` execs
podman directly, so its error stays podman's own.

Related: #5
Generated-by: AI
bootupd's bootloader-update.service looks for the block device backing
/boot or /sysroot, and in an ephemeral VM the root is virtiofs, so it
fails with "Failed to find block device from /boot or /sysroot". The
system then comes up "degraded", which breaks every test that boots an
image with bcvk and checks systemctl is-system-running, forcing each
consumer (e.g. bootc-dev/infra, bootupd CI) to mask it themselves.

There is never a bootloader to update in an ephemeral VM, so mask it
by default like systemd-journal-flush.service. coreos/bootupd#1072
makes bootupd skip this case itself, but existing images will carry
older bootupd for a long time.

The ephemeral system-command integration test now asserts the VM
reaches "running", so a unit failing in every ephemeral boot is caught
here rather than by bcvk's consumers.

Generated-by: AI
Signed-off-by: Colin Walters <walters@verbum.org>
@cgwalters-bot

Copy link
Copy Markdown
Collaborator Author

Signed off 1 commit(s) with Signed-off-by: Colin Walters <walters@verbum.org> for bootc-dev/bcvk's DCO check, on cgwalters's approval #5 (review) of 645fcab3ffe8. Same trees; the head is now 5280e78a35da.

@cgwalters-bot
cgwalters-bot force-pushed the bot/mask-bootloader-update branch from 645fcab to 5280e78 Compare September 25, 2026 22:23
@cgwalters-bot

Copy link
Copy Markdown
Collaborator Author

Opened upstream as bootc-dev#376. Closing this review draft.

cgwalters-bot added a commit that referenced this pull request Sep 30, 2026
bcvk bind-mounts its own binary into the container it launches. From a
toolbox or distrobox, podman is often forwarded to the host, so that
path is resolved on the host. With bcvk installed only in the toolbox
this failed with the rather cryptic

  Podman command failed: Error: statfs /usr/bin/bcvk: no such file or directory

Rather than trying to detect a toolbox or distrobox, which can't tell a
forwarded podman from one installed in the container, key on the error
itself: if podman reports a bind mount source missing that bcvk can
see, podman must be looking at a different filesystem. That covers a
remote podman client too. Wrap the error with an explanation and a
pointer to new docs on how to run bcvk in those setups, with the
guidance in color_eyre note and suggestion sections.

This only improves the failure; actually supporting bcvk from a toolbox
is the larger design question in the issue. `ephemeral run` execs
podman directly, so its error stays podman's own.

Related: #5
Generated-by: AI
cgwalters-bot added a commit that referenced this pull request Sep 30, 2026
bcvk bind-mounts its own binary into the container it launches. From a
toolbox or distrobox, podman is often forwarded to the host, so that
path is resolved on the host. With bcvk installed only in the toolbox
this failed with the rather cryptic

  Podman command failed: Error: statfs /usr/bin/bcvk: no such file or directory

Rather than trying to detect a toolbox or distrobox, which can't tell a
forwarded podman from one installed in the container, key on the error
itself: if podman reports a bind mount source missing that bcvk can
see, podman must be looking at a different filesystem. That covers a
remote podman client too. Wrap the error with an explanation and a
pointer to new docs on how to run bcvk in those setups, with the
guidance in color_eyre note and suggestion sections.

This only improves the failure; actually supporting bcvk from a toolbox
is the larger design question in the issue. `ephemeral run` execs
podman directly, so its error stays podman's own.

Related: #5
Generated-by: AI
Signed-off-by: Colin Walters <walters@verbum.org>
cgwalters-bot added a commit that referenced this pull request Sep 30, 2026
bcvk bind-mounts its own binary into the container it launches. From a
toolbox or distrobox, podman is often forwarded to the host, so that
path is resolved on the host. With bcvk installed only in the toolbox
this failed with the rather cryptic

  Podman command failed: Error: statfs /usr/bin/bcvk: no such file or directory

Rather than trying to detect a toolbox or distrobox, which can't tell a
forwarded podman from one installed in the container, key on the error
itself: if podman reports a bind mount source missing that bcvk can
see, podman must be looking at a different filesystem. That covers a
remote podman client too. Wrap the error with an explanation and a
pointer to new docs on how to run bcvk in those setups, with the
guidance in color_eyre note and suggestion sections.

This only improves the failure; actually supporting bcvk from a toolbox
is the larger design question in the issue. `ephemeral run` execs
podman directly, so its error stays podman's own.

Related: #5
Generated-by: AI
Signed-off-by: Colin Walters <walters@verbum.org>
cgwalters-bot added a commit that referenced this pull request Oct 1, 2026
bcvk bind-mounts its own binary into the container it launches. From a
toolbox or distrobox, podman is often forwarded to the host, so that
path is resolved on the host. With bcvk installed only in the toolbox
this failed with the rather cryptic

  Podman command failed: Error: statfs /usr/bin/bcvk: no such file or directory

Rather than trying to detect a toolbox or distrobox, which can't tell a
forwarded podman from one installed in the container, key on the error
itself: if podman reports a bind mount source missing that bcvk can
see, podman must be looking at a different filesystem. That covers a
remote podman client too. Wrap the error with an explanation and a
pointer to new docs on how to run bcvk in those setups, with the
guidance in color_eyre note and suggestion sections.

This only improves the failure; actually supporting bcvk from a toolbox
is the larger design question in the issue. `ephemeral run` execs
podman directly, so its error stays podman's own.

Related: #5
Generated-by: AI
Signed-off-by: Colin Walters <walters@verbum.org>
cgwalters-bot added a commit that referenced this pull request Oct 3, 2026
bcvk bind-mounts its own binary into the container it launches. From a
toolbox or distrobox, podman is often forwarded to the host, so that
path is resolved on the host. With bcvk installed only in the toolbox
this failed with the rather cryptic

  Podman command failed: Error: statfs /usr/bin/bcvk: no such file or directory

Rather than trying to detect a toolbox or distrobox, which can't tell a
forwarded podman from one installed in the container, key on the error
itself: if podman reports a bind mount source missing that bcvk can
see, podman must be looking at a different filesystem. That covers a
remote podman client too. Wrap the error with an explanation and a
pointer to new docs on how to run bcvk in those setups, with the
guidance in color_eyre note and suggestion sections.

This only improves the failure; actually supporting bcvk from a toolbox
is the larger design question in the issue. `ephemeral run` execs
podman directly, so its error stays podman's own.

Related: #5
Generated-by: AI
Signed-off-by: Colin Walters <walters@verbum.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants