cleanup: Mount the previous system's /var before removing its packages by cgwalters-bot · Pull Request #45 · cgwalters-forge/bootc

cgwalters-bot · 2026-10-10T09:20:21Z

Step 3 of the plan in bootc-dev#2559, stacked on #44 (which is stacked on #43): the fix for bootc-destructive-cleanup.service failing after an in-place reinstall.

Root cause

fedora-bootc-destructive-cleanup runs rpm -e --root=/sysroot on every package of the previous OS. rpm runs the erase scriptlets chrooted in /sysroot, so they need the previous OS's /var. But /sysroot is only the previous root filesystem (btrfs subvol=/root); on Fedora 44 cloud images /var is a separate var subvolume that nothing mounts after the reinstall. netcat's %preun (alternatives --remove) then fails with failed to create admindir, xargs exits 123 and the unit fails.

The script is unchanged since f567f7e (January 2026), so this is not a regression. It went unseen because nothing checked the unit until test 014 was added in 41049cc (2026-09-18), and the bcvk flows never boot a reinstalled system.

The fix

In contrib/scripts/fedora-bootc-destructive-cleanup (plus its man page): if /sysroot/etc/fstab has a /var entry whose source resolves to a block device, mount it at /sysroot/var with mount --fstab /sysroot/etc/fstab --target-prefix /sysroot /var before removing the packages. The unit already has PrivateMounts=true, so the mount stays private to the service.

The mount is skipped, and the script behaves as before, when:

/sysroot/etc/fstab does not exist or has no /var entry;
/sysroot/var is already a mount point;
the source is not a block device, e.g. a bind mount (warning);
the mount fails (warning);
the mounted /var turns out to be on the same filesystem as /sysroot or as the booted /var: it is unmounted again (warning).

The podman prune now runs only if /sysroot/var/lib/containers exists.

Not handled: nested mounts under /var, a /var defined by a .mount unit, and LVM or LUKS devices that are not active in the new system. Those give a warning and then the old behaviour. The script only uses /sysroot and /var, so it is the same for ostree and composefs; but on composefs --cleanup never writes the stamp file (only ostree_install does), so the service never runs there. That is a separate gap and is not fixed here.

Alternatives the run listed

The run called this "not a clear-cut choice":

(a) rpm -e --noscripts, bootc-dev/bootc#2561. Smaller, but it skips every scriptlet on every system (cgwalters called it a big hammer on that PR), and the old /var stays unreachable for the podman prune.
(b) Record the needed mounts at install time in a state file (cgwalters' general idea). More general, but bigger and riskier.
(c) Delete btrfs subvolumes instead of using rpm.

It chose the targeted mount because it matches cgwalters' stated preference ("special case mount preservation").

Questions for the maintainer (from the run)

Fix design: I implemented a guarded mount of the previous /var from its fstab. The alternative is rpm -e --noscripts (bootc-dev/bootc#2561), which is smaller but skips all scriptlets. I recommend keeping the mount and closing that PR or rebasing it onto this; a maintainer should decide.
Not run here (no KVM): just test-tmt-readonly. Test 014 must now pass on the reinstall path, and the cleanup unit's journal should show mount --fstab /sysroot/etc/fstab --target-prefix /sysroot /var succeeding.
Should install to-existing-root --cleanup with the composefs backend write the cleanup stamp? Today it is silently ignored. I left it alone as a separate change.
bootloader-update.service failure after the reinstall on a BIOS-booted guest: file a separate issue (bootc removes EFI/ while bootupd --auto installs only BIOS)? I recommend yes, after confirming the guest's boot mode. (The run inferred this cause; it did not verify it.)

What the run did and did not run

Ran: shellcheck and bash -n on the script; cargo xtask update-generated direct --check; and the new function in an unprivileged user and mount namespace (unshare -rm) against a scratch fstab with /sysroot and /var relocated, 8 scenarios (a tmpfs /var entry mounts; no /var entry, no fstab, an already-mounted /sysroot/var, a non-block source and a bad mount option are each skipped; a /var on the same filesystem as the old root or as the live /var is unmounted again). The block-device check was stubbed for the cases that need a mount.

Not run: a real boot, a btrfs subvolume mount, just test-tmt-readonly, and the full just validate container build. There are no unit tests because the logic is bash. So as opened, this change has never executed on a reinstalled system; a devspace verification will follow as a comment.

Run: https://github.com/bootc-dev/cgwalters-devspace-sandbox/actions/runs/38040104249
Base PR: #44 (on #43)

Generated-by: https://github.com/cgwalters/#llms

cgwalters-bot · 2026-10-10T10:10:19Z

Verification run on a devspace (16 cores, Intel Xeon Platinum 8573C, RHEL 10, KVM, bcvk 0.19.0, tmt 1.79.0 with testcloud; host set up as in tmt/README.md). LLM-run; nothing here was checked by a human. Head is ce3cd6d, unchanged: no fix was needed.

Result: the fix works on the case it was written for. Both paths pass, exit 0, and test 014 passes on the reinstall path. The cleanup unit mounts the previous /var subvolume, removes all packages and prunes the old container storage.

What ran

BOOTC_base=quay.io/fedora/fedora-bootc:44 just test-tmt-readonly (build, then cargo xtask run-tmt ... --provision=bcvk,reinstall readonly). Summary, verbatim:

========================================
Test Summary
========================================
/tmt/plans/integration/plan-01-readonly (bcvk): PASSED
/tmt/plans/integration/plan-01-readonly (reinstall): PASSED
========================================

Exit code 0.

BOOTC_base=quay.io/fedora/fedora-bootc:44 just test-tmt-nobuild --provision=reinstall --preserve-vm readonly on the same build, to look at the guest: PASSED, exit 0. From that run's test output:
```
Running booted/readonly/014-test-no-failed-units.nu...
TAP version 14
verify no failed systemd/bootc units
No failed systemd-*/bootc-* units
ok
```
cargo fmt --check: clean. bash -n on the script: clean. cargo xtask update-generated direct --check (the first half of just validate): exit 0.

The reinstalled guest (Fedora 44 cloud image, btrfs, booted image version `202610100919.gce3cd6d578`)

$ systemctl status --no-pager -l bootc-destructive-cleanup.service
○ bootc-destructive-cleanup.service - Cleanup previous the installation after an alongside installation
     Loaded: loaded (/usr/lib/systemd/system/bootc-destructive-cleanup.service; enabled-runtime; preset: disabled)
     Active: inactive (dead) since Sat 2026-10-10 10:07:22 UTC; 1min 47s ago
   Main PID: 826 (code=exited, status=0/SUCCESS)
   Mem peak: 378.1M
        CPU: 23.867s

$ journalctl -b --no-pager -u bootc-destructive-cleanup.service   (65 lines; trimmed)
fedora-bootc-destructive-cleanup[826]: + mount -o remount,rw /sysroot
fedora-bootc-destructive-cleanup[826]: + mount_old_var
fedora-bootc-destructive-cleanup[854]: ++ findmnt --fstab --tab-file /sysroot/etc/fstab --mountpoint /var --evaluate --noheadings --output SOURCE
fedora-bootc-destructive-cleanup[826]: + source=/dev/vda3
fedora-bootc-destructive-cleanup[826]: + '[' '!' -b /dev/vda3 ']'
fedora-bootc-destructive-cleanup[826]: + mount --fstab /sysroot/etc/fstab --target-prefix /sysroot /var
fedora-bootc-destructive-cleanup[858]: ++ stat -c %d /sysroot/var
fedora-bootc-destructive-cleanup[826]: + dev=61
fedora-bootc-destructive-cleanup[862]: ++ stat -c %d /sysroot
fedora-bootc-destructive-cleanup[826]: + '[' 61 = 36 ']'
fedora-bootc-destructive-cleanup[863]: ++ stat -c %d /var
fedora-bootc-destructive-cleanup[826]: + '[' 61 = 36 ']'
fedora-bootc-destructive-cleanup[864]: + rpm -qa --root=/sysroot --dbpath=/usr/lib/sysimage/rpm
fedora-bootc-destructive-cleanup[865]: + xargs -r rpm -e --root=/sysroot --dbpath=/usr/lib/sysimage/rpm
fedora-bootc-destructive-cleanup[931]: warning: erasing gpg-pubkey packages is deprecated; use rpmkeys --delete 36f612dcf27f7d1a48a835e4dbfcf71c6d9f90a6
fedora-bootc-destructive-cleanup[931]: warning: file /run/cloud-init: remove failed: No such file or directory
  (... 9 more "remove failed: No such file or directory" warnings, for /run/*, /var/run/*, /boot/grub2 and /boot/efi/EFI; 7 "/etc/... saved as ....rpmsave" ...)
fedora-bootc-destructive-cleanup[1264]: realpath: /etc/resolv.conf: No such file or directory
fedora-bootc-destructive-cleanup[931]: warning: directory /var: remove failed: Device or resource busy
fedora-bootc-destructive-cleanup[931]: warning: directory //: remove failed: Device or resource busy
fedora-bootc-destructive-cleanup[826]: + '[' -d /sysroot/var/lib/containers ']'
fedora-bootc-destructive-cleanup[826]: + mount --bind -o rw /sysroot/var/lib/containers /var/lib/containers
fedora-bootc-destructive-cleanup[826]: + podman system prune --all -f
fedora-bootc-destructive-cleanup[1406]: Total reclaimed space: 2.345GB
systemd[1]: bootc-destructive-cleanup.service: Deactivated successfully.
systemd[1]: Finished bootc-destructive-cleanup.service - Cleanup previous the installation after an alongside installation.

$ rpm -qa --root=/sysroot --dbpath=/usr/lib/sysimage/rpm | wc -l
0
$ grep /var /sysroot/etc/fstab
UUID=15c26993-ac30-424a-9c4b-faec4434d234 /var btrfs compress=zstd:1,subvol=var 0 0

Compared with the failing journal in #43 (comment) : there is no failed to create admindir, no scriptlet error, and the roughly 35 /var/...: remove failed warnings are gone. realpath: /etc/resolv.conf is still printed and is harmless here. The podman prune now reaches the previous system's storage, which it could not before (it removed the install image and three others, 2.345GB). Outside the service /sysroot/var is not a mount point, as intended with PrivateMounts=true.

$ systemctl --failed --no-pager
  UNIT                      LOAD   ACTIVE SUB    DESCRIPTION
● bootloader-update.service loaded failed failed Update bootloader on boot
1 loaded units listed.

bootupctl[866]: error: Failed adopt and update: opening EFI dir: No such file or directory (os error 2)

bootloader-update.service still fails, and I did not touch it. The guest is BIOS-booted (/sys/firmware/efi does not exist), which is consistent with the run's guess about its cause but does not prove it. Test 014 does not look at this unit.

Not run

The skip conditions on a real system (only the happy path ran: a btrfs subvolume /var; the non-block, failed-mount and same-filesystem branches have only the run's stubbed namespace tests). CentOS Stream, the composefs variant, plans other than readonly, a previous OS with /var on the root filesystem, LVM or LUKS. shellcheck (not installed on the devspace; the run reported it clean) and the container half of just validate (clippy, man pages).

Generated-by: https://github.com/cgwalters/#llms

cgwalters-bot · 2026-10-10T10:22:36Z

Review by an LLM (run 38044072404), requested by the coordinator. Not a human review.

VERDICT: APPROVE (safe to merge as is; the items below can follow). This review is LLM-generated, by Claude Opus 5.5 (claude-opus-5-5), running unattended. No human checked it.

Scope: git diff origin/bot/reinstall-tmt-2559-step2...HEAD is one commit, ce3cd6d. It changes contrib/scripts/fedora-bootc-destructive-cleanup (+40/-2) and docs/src/man/bootc-destructive-cleanup.service.5.md. It changes no Rust, no workflows and no permissions.

Findings (ranked; none block the merge)

The container prune is now skipped without any message. This can follow, but it is cheap to fix now. contrib/scripts/fedora-bootc-destructive-cleanup:48-52. Before this PR, a missing /sysroot/var/lib/containers made the mount --bind fail, so the unit failed. Now the prune is skipped and nothing is printed. Cases where mount_old_var gives up (inactive LVM/LUKS, an unresolvable UUID, a .mount-unit /var) will now leave the install image's storage behind (2.3 GB in the verification run), and the journal will not say so. In practice rpm -e will usually fail first in those cases, but there is no guarantee. Fix: add else echo "warning: /sysroot/var/lib/containers not found, not pruning container storage" 1>&2.
The tests do not check the intent. This can follow. The only automated check is tmt/tests/booted/readonly/014-test-no-failed-units.nu, and it only checks that no unit is in the failed state. It still passes if the unit never ran (stamp absent, composefs) or if every step was skipped. Fix: in the reinstall plan (running_env=reinstall), assert that bootc-destructive-cleanup.service ran with Result=success, and that rpm -qa --root=/sysroot --dbpath=/usr/lib/sysimage/rpm is empty. The verification comment checked this by hand, but nothing in CI keeps it from regressing.
The old fstab's mount options are used as written. Low priority. mount --fstab "${fstab}" ... /var (line 27) applies the previous OS's options, for example ro. With ro the mount succeeds, passes the st_dev checks, and rpm -e then fails exactly as before. This is not a privilege boundary: the previous OS's root already owned the disk, and helper=/uhelper= resolve against the new root's binaries. Optional fix: append -o rw, or skip when the resulting mount is read-only.
Nit: when an fstab entry's UUID does not resolve, findmnt --evaluate exits 0 and prints nothing. The warning then reads not mounting previous /var from '': not a block device. It is harmless, but a [ -z "${source}" ] branch would print a clearer message.

Claims in the PR description checked against the code

The skip conditions listed (no fstab or no /var entry; /sysroot/var already a mount point; source not a block device; mount fails; result on the same st_dev as /sysroot or the booted /var, which is then unmounted) match lines 11-37 exactly. I probed findmnt --fstab --tab-file X --mountpoint /var --evaluate (util-linux 2.40.2) on three fstabs. No /var entry gives rc=1, so the function returns. An unresolvable UUID gives rc=0 with empty output, which fails -b, so it warns and returns. Two /var entries give two lines, which fail -b, so it warns and returns. Every one of these falls back to the old behaviour.
PrivateMounts=true is present in systemd/bootc-destructive-cleanup.service, so the /sysroot/var mount (and the containers bind mount) stay private to the unit.
The st_dev comparisons are sound for ext4/xfs and for btrfs (each subvolume gets its own anonymous dev). Comparing against the booted /var is the important guard: it stops scriptlets from running against live state if the new image itself mounts the same device or subvolume. Under set -e, a failed umount aborts before rpm -e, so this guard fails closed. local source is declared separately from the assignment, so if ! source=$(...) sees the real exit code.
"Fixes the Fedora 44 case": the runtime evidence is in the bot's verification comment (mount from /dev/vda3, rpm -qa count 0, prune reclaimed 2.345 GB). I could not reproduce it here (no KVM).
The commit message is accurate, carries Generated-by: AI, and has no Signed-off-by.

Security posture

The change runs only inside the existing root oneshot unit, after the install. It adds no new input surface beyond the previous OS's /etc/fstab, which comes from the same root that already controlled the disk. It does not touch CI, workflows, tokens, the sandbox, or the check and apply jobs. It writes nothing outside the unit's private mount namespace.

Checks run here (rustc 1.99.0)

shellcheck contrib/scripts/fedora-bootc-destructive-cleanup: exit 0.
cargo fmt --check: exit 0.
cargo clippy --workspace --all-targets -- -D warnings: exit 101. The error is clippy::collapsible_if at crates/mount/src/mount.rs:160. This failure was already there: crates/ is identical to step2, and this PR changes no Rust. The Makefile's CI gate config (-A clippy::all -D correctness -D suspicious -D disallowed-methods ...) with --workspace --all-targets also fails with exit 101, on str::len at crates/ostree-ext/src/fixture.rs:274. That failure is also unrelated to this PR.
cargo test --workspace --no-fail-fast: exit 101. 23 test binaries pass. ostree-ext --test it has 15 failures (Parsing layer blob ...: No such file or directory), which come from this sandbox's environment and not from this PR (no Rust is changed).

Not checked

I did not run a real reinstall boot, just test-tmt-readonly, the reinstall tmt plan, LVM/LUKS, nested mounts under /var, CentOS Stream, or composefs (where the service never runs, as the description says).
Open maintainer decision: upstream contrib: Skip RPM scriptlets during cleanup bootc-dev/bootc#2561 (rpm -e --noscripts) addresses the same failure. If it lands, this PR's remaining value is that scriptlets still run and the prune reaches the old /var. The two changes are compatible, but someone needs to choose deliberately.

Generated-by: https://github.com/cgwalters/#llms

After an alongside install, fedora-bootc-destructive-cleanup runs `rpm -e --root=/sysroot` on every package of the previous OS. rpm runs the erase scriptlets chrooted in /sysroot, so they need the previous OS's /var. But /sysroot is only the previous root filesystem; on Fedora 44 cloud images /var is a separate btrfs subvolume that nothing mounts after the reinstall. netcat's %preun (`alternatives --remove`) then fails with "failed to create admindir", xargs exits 123 and bootc-destructive-cleanup.service fails. So mount the previous /var at /sysroot/var from the previous system's fstab when its source is a block device. The unit has PrivateMounts=true so the mount stays private to the service. Anything that looks wrong (already a mount point, not a block device, the mount fails, or the result is the same filesystem as /sysroot or the booted /var) gets a warning and the old behaviour, since deleting from the wrong /var would be much worse than a failed unit. Each of those, and a skipped container prune, prints a line saying why, so the journal shows what was not cleaned up. Add a readonly test for the in-place reinstall path that checks the unit ran to completion and left no packages of the previous system: "no failed units" alone also passes when the cleanup never ran. The alternative is `rpm -e --noscripts`, which is smaller but skips every scriptlet on every system and leaves the old /var unreachable for the podman prune. bootc-dev#2559 Generated-by: AI Agent-run: https://github.com/bootc-dev/cgwalters-devspace-sandbox/actions/runs/38040104249/attempts/1

cgwalters-bot · 2026-10-10T10:54:37Z

Addressed review findings 1, 2 and 4 from the review above, squashed into the one commit. New head: 47b1a70 (was ce3cd6d); still one commit on top of step 2. LLM-written and LLM-run; nothing here was checked by a human.

What changed

Finding 1 (silent skips). fedora-bootc-destructive-cleanup now prints a line to stderr (so the unit's journal) when it skips the container prune: warning: not pruning container images: /sysroot/var/lib/containers does not exist (the previous system had no container storage, or its /var could not be mounted). Every way out of mount_old_var that leaves the previous /var unmounted now prints its own line as well: no /etc/fstab, /sysroot/var already a mount point, no /var entry in the fstab, source not a block device, mount failed, not a separate filesystem. The last three name the source.

Finding 4 (unresolvable source). An fstab entry whose source findmnt --evaluate cannot resolve now gives warning: source of /var in the previous fstab could not be resolved, not mounting it instead of from '': not a block device.

Finding 2 (tests check intent). New tmt/tests/booted/readonly/018-test-destructive-cleanup.nu, a separate numbered test like the others there (they are picked up by glob, so no generated file changes). On the reinstall path, detected by BOOTC_REINSTALL_IMAGE, which only cargo xtask run-tmt --provision=reinstall passes to the guest, it waits up to 5 minutes for bootc-destructive-cleanup.service to have run to completion (a main-process exit timestamp, which tells "finished" from "never started"), asserts Result=success and ExecMainStatus=0, and asserts that rpm -qa --root=/sysroot --dbpath=/usr/lib/sysimage/rpm lists nothing (or that the rpmdb is gone). On every other path it prints a skip line and passes.

The man page says that these lines are printed and where to find them, and the commit message has two more sentences for the warnings and the test.

Finding 3 (the old fstab's mount options, e.g. ro) is not addressed, on purpose; it is left for the maintainer to decide.

Verification

On a devspace (16 cores, AMD EPYC 9V45, RHEL 10, /dev/kvm, bcvk 0.19.0, tmt 1.79.0 with testcloud, requests-cache 1.2.1 and the genisoimage wrapper from tmt/README.md), at 47b1a70:

BOOTC_base=quay.io/fedora/fedora-bootc:44 just test-tmt-readonly, exit code 0:

========================================
Test Summary
========================================
/tmt/plans/integration/plan-01-readonly (bcvk): PASSED
/tmt/plans/integration/plan-01-readonly (reinstall): PASSED
========================================

The new test, from each guest's output.txt. Reinstall path:

Running booted/readonly/018-test-destructive-cleanup.nu...
TAP version 14
verify destructive cleanup after an in-place reinstall
bootc-destructive-cleanup.service: Result=success ExecMainStatus=0
Packages of the previous system left in /sysroot: 0
ok

bcvk path:

Running booted/readonly/018-test-destructive-cleanup.nu...
TAP version 14
verify destructive cleanup after an in-place reinstall
# skip: not provisioned via in-place reinstall (BOOTC_REINSTALL_IMAGE not set)
ok

Test 014 passed on both paths too. bash -n contrib/scripts/fedora-bootc-destructive-cleanup, cargo fmt --check and cargo xtask update-generated direct --check all exit 0 there, and the tree was clean afterwards. shellcheck on the script is clean (run locally; it is not on the devspace).

Not run

The new warning lines themselves: the run only took the happy path (a btrfs var subvolume that mounts), so none of the skip branches, nor the prune-skipped line, executed on a real system. I did not capture the unit's journal from this run. The new test's failure cases (unit never ran, packages left) were not exercised. The container half of just validate (clippy, man page build) was not run, nor CentOS Stream, composefs, or plans other than readonly.

Generated-by: https://github.com/cgwalters/#llms

cgwalters-bot added this to Workstream Oct 10, 2026

cgwalters-bot moved this to In Review in Workstream Oct 10, 2026

cgwalters-bot mentioned this pull request Oct 10, 2026

bootc#2559: guarded /var mount or rpm --noscripts for the cleanup service? cgwalters-forge/tracker#481

Closed

cgwalters-bot force-pushed the bot/reinstall-tmt-2559-step3 branch from ce3cd6d to 47b1a70 Compare October 10, 2026 10:54

cgwalters-bot mentioned this pull request Oct 13, 2026

Working agreement: priorities and in-progress state live on the board, operator direction lives on issues cgwalters-forge/tracker#523

Open

Navigation Menu

Repository navigation

cleanup: Mount the previous system's /var before removing its packages - #45

cleanup: Mount the previous system's /var before removing its packages#45
cgwalters-bot wants to merge 1 commit into
bot/reinstall-tmt-2559-step2from
bot/reinstall-tmt-2559-step3

cgwalters-bot commented Oct 10, 2026

Uh oh!

cgwalters-bot commented Oct 10, 2026

Uh oh!

cgwalters-bot commented Oct 10, 2026

Uh oh!

cgwalters-bot commented Oct 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

cgwalters-bot commented Oct 10, 2026

Root cause

The fix

Alternatives the run listed

Questions for the maintainer (from the run)

What the run did and did not run

Uh oh!

cgwalters-bot commented Oct 10, 2026

What ran

The reinstalled guest (Fedora 44 cloud image, btrfs, booted image version 202610100919.gce3cd6d578)

Not run

Uh oh!

cgwalters-bot commented Oct 10, 2026

Findings (ranked; none block the merge)

Claims in the PR description checked against the code

Security posture

Checks run here (rustc 1.99.0)

Not checked

Uh oh!

cgwalters-bot commented Oct 10, 2026

What changed

Verification

Not run

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

The reinstalled guest (Fedora 44 cloud image, btrfs, booted image version `202610100919.gce3cd6d578`)