tmt: Print journals for failed units

[ckyrouac]

Include each failed unit's journal in test output to make failures easier to diagnose.

Assisted-by: AI

[jeckersb]

Ok I built packages locally and threw them at testing-farm, here's the result from https://artifacts.dev.testing-farm.io/b17252b9-2eee-41f7-b0b8-fd4481df829d/:

Running booted/readonly/014-test-no-failed-units.nu...
TAP version 14
verify no failed systemd/bootc units
Failed units:
bootc-destructive-cleanup.service loaded failed failed Cleanup previous the installation after an alongside installation
Journal for bootc-destructive-cleanup.service:
Oct 08 21:02:28 fedora systemd[1]: Starting bootc-destructive-cleanup.service - Cleanup previous the installation after an alongside installation...
Oct 08 21:02:28 fedora fedora-bootc-destructive-cleanup[800]: + mount -o remount,rw /sysroot
Oct 08 21:02:28 fedora fedora-bootc-destructive-cleanup[832]: + rpm -qa --root=/sysroot --dbpath=/usr/lib/sysimage/rpm
Oct 08 21:02:28 fedora fedora-bootc-destructive-cleanup[833]: + xargs -r rpm -e --root=/sysroot --dbpath=/usr/lib/sysimage/rpm
Oct 08 21:02:29 fedora fedora-bootc-destructive-cleanup[911]: warning: erasing gpg-pubkey packages is deprecated; use rpmkeys --delete 36f612dcf27f7d1a48a835e4dbfcf71c6d9f90a6
Oct 08 21:02:32 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /run/cloud-init: remove failed: No such file or directory
Oct 08 21:02:37 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/NetworkManager: remove failed: No such file or directory
Oct 08 21:02:38 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[1278]: /var/tmp/rpm-tmp.qtexls: line 44: /var/lib/rpm-state/nfs-server.cleanup: No such file or directory
Oct 08 21:02:38 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/nfs: remove failed: No such file or directory
Oct 08 21:02:38 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/lvm: remove failed: No such file or directory
Oct 08 21:02:38 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /run/lvm: remove failed: No such file or directory
Oct 08 21:02:40 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: /etc/chrony.conf saved as /etc/chrony.conf.rpmsave
Oct 08 21:02:43 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/dhcpcd: remove failed: No such file or directory
Oct 08 21:02:43 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/logrotate: remove failed: No such file or directory
Oct 08 21:02:44 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/sss: remove failed: No such file or directory
Oct 08 21:02:44 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /run/sssd: remove failed: No such file or directory
Oct 08 21:02:45 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: error: %preun(netcat-1.238-1.fc44.x86_64) scriptlet failed, exit status 2
Oct 08 21:02:45 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: error: netcat-1.238-1.fc44.x86_64: erase failed
Oct 08 21:02:46 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/unbound: remove failed: No such file or directory
Oct 08 21:02:50 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/samba: remove failed: No such file or directory
Oct 08 21:02:51 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/selinux/targeted/active/users_extra: remove failed: No such file or directory
Oct 08 21:02:51 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/selinux/targeted/active/seusers: remove failed: No such file or directory
Oct 08 21:02:51 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/selinux/targeted/active/policy.kern: remove failed: No such file or directory
Oct 08 21:02:51 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/selinux/targeted/active/modules_checksum: remove failed: No such file or directory
Oct 08 21:02:51 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/selinux/targeted/active/homedir_template: remove failed: No such file or directory
Oct 08 21:02:51 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/selinux/targeted/active/file_contexts.homedirs: remove failed: No such file or directory
Oct 08 21:02:51 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/selinux/targeted/active/file_contexts: remove failed: No such file or directory
Oct 08 21:02:51 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/selinux/targeted/active/commit_num: remove failed: No such file or directory
Oct 08 21:02:51 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /boot/efi/EFI: remove failed: No such file or directory
Oct 08 21:02:51 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/dnf: remove failed: No such file or directory
Oct 08 21:02:52 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/initramfs: remove failed: No such file or directory
Oct 08 21:02:52 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/os-prober: remove failed: No such file or directory
Oct 08 21:02:53 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /run/rpcbind: remove failed: No such file or directory
Oct 08 21:02:54 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/selinux/tmp: remove failed: No such file or directory
Oct 08 21:02:54 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/gssproxy: remove failed: No such file or directory
Oct 08 21:02:55 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /run/criu: remove failed: No such file or directory
Oct 08 21:02:56 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /boot/grub2: remove failed: No such file or directory
Oct 08 21:02:57 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/systemd: remove failed: No such file or directory
Oct 08 21:02:58 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/authselect: remove failed: No such file or directory
Oct 08 21:02:58 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/lastlog: remove failed: No such file or directory
Oct 08 21:02:58 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/kerberos: remove failed: No such file or directory
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: %postun(p11-kit-trust-0.26.5-1.fc44.x86_64) scriptlet failed, exit status 2
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /run/setrans: remove failed: No such file or directory
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/yp: remove failed: No such file or directory
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/spool: remove failed: No such file or directory
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/preserve: remove failed: No such file or directory
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/opt: remove failed: No such file or directory
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/nis: remove failed: No such file or directory
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/mail: remove failed: No such file or directory
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/log: remove failed: No such file or directory
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/local: remove failed: No such file or directory
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/rpm-state: remove failed: No such file or directory
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/misc: remove failed: No such file or directory
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/lib/games: remove failed: No such file or directory
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/games: remove failed: No such file or directory
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/ftp: remove failed: No such file or directory
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/empty: remove failed: No such file or directory
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/db: remove failed: No such file or directory
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/cache: remove failed: No such file or directory
Oct 08 21:03:03 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /var/adm: remove failed: No such file or directory
Oct 08 21:03:04 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: directory //: remove failed: Device or resource busy
Oct 08 21:03:04 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: file /run/motd.d: remove failed: No such file or directory
Oct 08 21:03:04 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: /etc/subuid saved as /etc/subuid.rpmsave
Oct 08 21:03:04 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: /etc/subgid saved as /etc/subgid.rpmsave
Oct 08 21:03:04 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: /etc/shadow saved as /etc/shadow.rpmsave
Oct 08 21:03:04 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: /etc/passwd saved as /etc/passwd.rpmsave
Oct 08 21:03:04 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: /etc/gshadow saved as /etc/gshadow.rpmsave
Oct 08 21:03:04 ip-172-31-24-105.us-east-2.compute.internal fedora-bootc-destructive-cleanup[911]: warning: /etc/group saved as /etc/group.rpmsave
Oct 08 21:03:04 ip-172-31-24-105.us-east-2.compute.internal systemd[1]: bootc-destructive-cleanup.service: Main process exited, code=exited, status=123/n/a
Oct 08 21:03:04 ip-172-31-24-105.us-east-2.compute.internal systemd[1]: bootc-destructive-cleanup.service: Failed with result 'exit-code'.
Oct 08 21:03:04 ip-172-31-24-105.us-east-2.compute.internal systemd[1]: Failed to start bootc-destructive-cleanup.service - Cleanup previous the installation after an alongside installation.
Oct 08 21:03:04 ip-172-31-24-105.us-east-2.compute.internal systemd[1]: bootc-destructive-cleanup.service: Consumed 30.041s CPU time over 36.133s wall clock time, 137.8M memory peak.

Error:   × Expected zero failed systemd-*/bootc-* units
    ╭─[/var/ARTIFACTS/work-plan-01-readonlys4q33ho4/tmt/plans/integration/plan-01-readonly/discover/default-0/tests/tmt/tests/booted/readonly/014-test-no-failed-units.nu:60:18]
 59 │     }
 60 │     assert equal ($failed | length) 0 "Expected zero failed systemd-*/bootc-* units"
    ·                  ──────────┬─────────
    ·                            ╰─┤ These are not equal.
    ·                              │         Left  : '1'
    ·                              │         Right : '0'
 61 │ }
    ╰────

Looks like %preun for netcat is the problem. %postun for p11-kit-trust throws a warning but doesn't appear to actually cause a problem.

[jeckersb]

%preun for netcat is just:

%preun
if [ $1 -eq 0 ]; then
  alternatives --remove %{link_bin} %{_bindir}/%{name}
fi

[cgwalters]

Let’s just change our read only test to skip failures in that unit for now.

Probably simplest is a quick workaround to do a noscripts force removal of netcat if it exists in our cleanup. With a queued patch fixing it in Fedora

[cgwalters]

I don’t think we should block on upstream release cycles for the underlying bug here. It’s not new

[jeckersb]

I don't think scriptlets are going to work correctly at all for this case given that we're operating on --root=/sysroot right? In this case alternatives is still going to be running "outside" against /?

Probably should just add --noscripts to that rpm -e invocation and call it a day?

[jeckersb]

And yeah I think for the moment in Fedora we can just merge all of the dist-git PRs over red, we at least understand what's happening now and it's not a regression. We'll fix it upstream and it can naturally land with the next release.