test flakes tracker

CI Flakes Tracker

This tracks CI failures, usually in the merge queue.

COPR repo unavailable

Error: Failed to download metadata for repo
'copr:copr.fedorainfracloud.org:group_CoreOS:continuous':
Cannot download repomd.xml: All mirrors were tried

The COPR continuous repo is transiently unavailable during the container build,
causing dnf to fail. Happens specifically on centos-10 jobs that pull from
the CoreOS COPR. Retry would likely succeed.

quay.io 502

Error:

error: Installing to filesystem: Creating ostree deployment: Pulling:
Importing: Unencapsulating base: failed to invoke method GetBlob:
fetching blob: received unexpected HTTP status: 502 Bad Gateway

The in-VM install test hit a transient 502 from the container registry while
pulling an image. Pure network flake.

GHA runner killed mid-TMT

Symptom: Job ran unit tests successfully, TMT step started but produced no
conclusion after 2h15m. No artifact uploaded. Run marked failed with no logged
error. Runner appears to have been killed externally (OOM or host issue).

`test_container_write_derive` assertion failure

Frequency: Once (fedora-44, ostree, xfs, grub, bls, unsealed — Unit and container integration tests)
Error:

assertion `left == right` failed
  left: "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"
 right: "newderivedfile v0"
at crates/ostree-ext/tests/it/main.rs:1713

File written into a derived container was read back as all null bytes. The same
test passed in fedora-43, ostree in the same queue batch. Likely a race
condition or tmpfs/storage timing issue on the fedora-44 runner; not seen
before or since.

Docs

Basically cargo binstall tries to fetch binaries from GitHub releases, but because we aren't passing in a GH_TOKEN into that inner job we can get rate limited. When that happens, cargo binstall falls back to trying to compile from source, which can then fail due to missing dependencies.

Package expiration

Our workflow splits package jobs from integration tests, if a PR sticks around too long the package artifacts expire, meaning all workflows need to be retried if a flake happens. I think we could extend package expiry to a week or potentially even longer.

https://github.com/bootc-dev/bootc/actions/runs/27350791739/job/80817797747 we got:

Error: No space left on device : '/home/runner/actions-runner/cached/2.335.1/_diag/pages/c1de0eb0-4981-4a5f-8a5d-c197f4a11e80_760c16b0-69cb-4987-9094-c1d6fd2a21fd_1.log'

in the test-integration (centos-10, ostree, xfs, grub, bls, unsealed) job

Error: No space left on device

Yes, one issue here turns out to be that some GHA runners have 70G disks, whereas others have 140G and the assignment is random. I think #2293 will help reduce used disk space during CI runs, but there's more to do here for sure.

quay.io 502

I think as of recently, we've mostly mitigated these flakes with anything that runs podman, but we don't have retries on the bootc/composefs side, which is composefs/composefs-rs#348

Error: No space left on device

Yes, one issue here turns out to be that some GHA runners have 70G disks, whereas others have 140G and the assignment is random. I think #2293 will help reduce used disk space during CI runs, but there's more to do here for sure.

Also very hostile to efforts to parallelize tests, which is by nature going to require more peak disk usage and blur the lines on when it may or may not be safe to throw data out to save space.

OK i just forced in a few PRs, especially #2293 but also a few other non-risky ones.

Error: No space left on device

#2306

Over in https://github.com/bootc-dev/bootc/actions/runs/30287090523/job/90057479263 we have:

[11/19] STEP 2/2: RUN --network=none --mount=type=tmpfs,target=/run --mount=type=tmpfs,target=/tmp     --mount=type=secret,id=secureboot_key     --mount=type=secret,id=secureboot_cert <<EORUN (set -xeuo pipefail...)
+ cd /tmp
+ rpm2cpio /out/systemd-boot-unsigned-259.8-1.fc44.x86_64.rpm
+ cpio -idmv
./usr/lib/systemd/boot/efi
./usr/lib/systemd/boot/efi/addonx64.efi.stub
./usr/lib/systemd/boot/efi/linuxx64.efi.stub
./usr/lib/systemd/boot/efi/systemd-bootx64.efi
./usr/share/man/man5/loader.conf.5.gz
./usr/share/man/man7/sd-boot.7.gz
./usr/share/man/man7/systemd-boot.7.gz
526 blocks
Error: building at STEP "RUN --network=none --mount=type=tmpfs,target=/run --mount=type=tmpfs,target=/tmp --mount=type=secret,id=secureboot_key --mount=type=secret,id=secureboot_cert <<EORUN": while running runtime: exit status 141
error: Recipe `build` failed with exit code 141

Which although rare I'm sure I've seen before. AI suggests it might be a race between rpm2cpio and cpio through a pipe.

Edit: ah found it - #1896

Agent investigation on the reboot failures points at teemtee/tmt#5068

Generated-by: https://github.com/cgwalters/cgwalters#llms

Merge queue stall, 2026-07-31

Snapshot of https://github.com/bootc-dev/bootc/queue/main today: 4 PRs queued (#2354, #2357, #2342, #2315), all effectively stuck for 2-8h. Traced through the actual merge_group CI runs:

ci: Pin tmt version (to older release) instead of always installing latest #2354 ("ci: Pin tmt version…", position 1): failed twice in the queue on the known quay.io flake — bootc install to-filesystem timed out pulling quay.io/centos-bootc/centos-bootc:stream10 (dial tcp ...: i/o timeout). Same class as the quay.io 502 already logged above. This PR needs to be fixed/retried or manually dequeued; it's blocking everything behind it.
Workaround SIGPIPE with cpio #2357 ("Workaround SIGPIPE with cpio", position 2): its own PR checks are all green, but the merge queue builds each entry cumulatively on top of the (unverified) entries ahead of it, so its merge_group run inherited ci: Pin tmt version (to older release) instead of always installing latest #2354. It failed on something new: bootc container lint --fatal-warnings reports var-tmpfiles: Found content in /var missing tmpfiles.d entries: d /var/roothome/.ssh on the composefs/uki/systemd legs. Not caused by Workaround SIGPIPE with cpio #2357's diff — looks like a new, previously-untracked regression (base image drift is my guess).
UKI print dump diff #2315 ("UKI print dump diff", position 4): hits the same roothome/UKI lint failure on two legs, and three more legs simply hung for the full default 6h GitHub Actions job timeout before being force-cancelled (no timeout-minutes set on test-integration). It's also now reporting a real merge conflict against ci: Pin tmt version (to older release) instead of always installing latest #2354 in tmt/plans/integration.fmf / tmt/tests/tests.fmf, since both touch those files and ci: Pin tmt version (to older release) instead of always installing latest #2354 is ahead of it in queue.
Switch download only #2342: fine on its own, just stuck in FIFO behind the above.

Why failures take hours to surface, and whether we need "fail fast"

All the heavy jobs (test-integration, test-upgrade, test-baseconfigs, package) run with fail-fast: false across ~35+ matrix legs, which is good for full PR diagnostic signal. But the actual required gate for the queue, required-checks-merge, does needs: [...] if: always() on all of those matrix jobs — needs: on a matrix job blocks until every leg reaches a terminal state. Combined with no timeout-minutes on test-integration (defaults to GitHub's 6h), a failure that's detected in the first 20-40 minutes (e.g. the roothome lint error above, which fails during container build before any boot/TMT test even runs) isn't visible to the merge queue until the slowest unrelated leg finishes or times out hours later.

Suggested fixes, roughly by impact/effort:

Set timeout-minutes on test-integration/test-upgrade/test-baseconfigs (TMT plan durations are mostly 30m; 60-90m should be plenty). Would have turned UKI print dump diff #2315's 6h stall into ~1.5h.
Don't blanket-flip the matrix to fail-fast: true — that would cancel ~35 healthy legs on a single flake, which given how flaky these jobs already are would waste more CI than it saves.
Instead, add a cheap fail-fast gate specifically for merge_group: a lint-only container build (bootc container lint --fatal-warnings, no TMT boot) that runs in minutes ahead of the expensive matrix, so deterministic build/lint failures get caught once instead of being silently rediscovered on N legs hours later.
The quay.io pull inside bootc install to-filesystem in the TMT tests isn't covered by the existing BOOTC_CI_RETRIES/BOOTC_CI_DELAY wrapper (that only wraps the pre-build podman pull in the Justfile) — extending retry coverage there would directly address the ci: Pin tmt version (to older release) instead of always installing latest #2354 flake class.

Immediate action needed: fix/dequeue #2354, rebase #2315 onto a resolved queue tip, and track down the /var/roothome/.ssh var-tmpfiles regression on the UKI+systemd variant (new, not seen before today).

lint: Ignore `/var/roothome/buildinfo/content-sets.json`

lint: Ignore `/var/roothome/.ssh`

OK we discussed having a ci/skip-integration style label which would bypass all the integration testing in the merge queue. Some research says this is somewhat nontrivial to do because the binding between the originating PR (and its labels) requires some complex backreferencing from the merge queue.

I guess I lean towards going back to merging individual PRs until we can make the queue much more reliable. And there a ci/skip-integration label would bypass all the integration testing jobs.

Another case where the GitHub runner just gives up https://github.com/bootc-dev/bootc/actions/runs/30875522062/job/91891077870?pr=2342

This probably has to do with the runner itself

I guess I lean towards going back to merging individual PRs until we can make the queue much more reliable. And there a ci/skip-integration label would bypass all the integration testing jobs.

FTR this is done now, however we are also now I realize not gating on integration tests at all for merging.

deploy: Retry transient registry blob pulls

install: Use retrying pull for OSTree installs

deploy: Retry transient registry blob pulls

install: Use retrying pull for OSTree installs

ci: Keep package artifacts for a week instead of a day

From https://github.com/bootc-dev/bootc/actions/runs/37623963296/job/112817153092?pr=2553:

                        content: umount: /var/mnt/esp_check: target is busy.
                        content: Error:   × Single ESP test failed: {msg: External command had a non-zero exit code,
                        content:   │ debug: NonZeroExitCode { exit_code: 32, span: Span { start: 100428, end:
                        content:   │ 100434 } }, raw: NonZeroExitCode { exit_code: 32, span: Span { start:
                        content:   │ 100428, end: 100434 } }, exit_code: 32}
                        content:      ╭─[/var/tmp/tmt/bootc-tmt-q8t5zrgu-tmt-plans-integration-plan-32-multi-device-esp/tmt/plans/integration/plan-32-multi-device-esp/discover/default-0/tests/tmt/tests/booted/test-multi-device-esp.nu:238:9]
                        content:  237 │         rm -f $disk1 $disk2
                        content:  238 │         error make {msg: $"Single ESP test failed: ($e)"}
                        content:      ·         ─────┬────
                        content:      ·              ╰── originates from here
                        content:  239 │     }
                        content:      ╰────

Looks like we need to do a better job of waiting/retrying/killing so the unmount succeeds

umount -l is the right default for cases like this

test: Unmount using `umount -l`

Navigation Menu

Repository navigation

test flakes tracker #2177

CI Flakes Tracker

COPR repo unavailable

quay.io 502

GHA runner killed mid-TMT

`test_container_write_derive` assertion failure

Docs

Package expiration

Merge queue stall, 2026-07-31

Why failures take hours to surface, and whether we need "fail fast"

16 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

test flakes tracker #2177

Description

CI Flakes Tracker

COPR repo unavailable

quay.io 502

GHA runner killed mid-TMT

test_container_write_derive assertion failure

Docs

Package expiration

Activity

jeckersb commented on Jun 11, 2026

cgwalters commented on Jul 9, 2026

jeckersb commented on Jul 10, 2026

cgwalters commented on Jul 10, 2026

cgwalters commented on Jul 10, 2026

jeckersb commented on Jul 28, 2026

cgwalters commented on Jul 30, 2026

cgwalters commented on Jul 31, 2026

Merge queue stall, 2026-07-31

Why failures take hours to surface, and whether we need "fail fast"

cgwalters commented on Aug 3, 2026

Johan-Liebert1 commented on Aug 4, 2026

cgwalters commented on Aug 4, 2026

16 remaining items

jeckersb commented on Oct 7, 2026

cgwalters commented on Oct 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`test_container_write_derive` assertion failure