tmt: Generate one file per test and plan, drop test numbers by cgwalters-bot · Pull Request #2581 · bootc-dev/bootc

cgwalters-bot · 2026-10-19T18:59:56Z

tmt: Generate one file per test and plan, drop test numbers

Every PR that adds a tmt test currently touches the same two generated files (tmt/tests/tests.fmf and the generated block of tmt/plans/integration.fmf) and picks the next free # number: N in its test header. Two PRs open at once therefore collide twice: textually in the generated files and logically on the number (there are already duplicates on main, and several open PRs claim 50).

This series removes both:

xtask/tmt: Generate one fmf file per test and per plan: cargo xtask update-generated writes tmt/tests/tests/test-*.fmf and tmt/plans/integration/plan-*.fmf, one each. fmf merges these directories into the nodes of tests.fmf and integration.fmf, which stay as the hand-written common settings, so names, and what just test-tmt FILTER selects, are unchanged. Removed tests have their files removed, and --check fails on stale ones.
tmt: Identify tests and plans by name, not by number: tests and plans are test-<name> / plan-<name> from the file name, the # number: header is gone, and test names must be unique (checked). The commit is mechanical: header lines dropped, three scripts and three legacy fmf files renamed, files regenerated, docs updated. Plans run in separate VMs, so nothing depends on their order and no replacement ordering field was needed.

A new test is now one tests/booted/test-<name>.nu file with its # tmt: header (documented in tmt/tests/booted/README.md) plus the two generated files for it.

Tested on a devspace at 43fae7a, rebased onto bootc-dev/bootc main 1a04bf2 (AMD EPYC 9V45):

just validate passes; cargo test -p xtask passes at both commits, and cargo xtask update-generated direct --check is clean at both.
tmt plan ls --filter enabled:true lists 40 plans at the first commit, and the second generates the same 40 tests and plans under their new names.
just test-tmt readonly switch-zstd-chunked with BOOTC_variant=composefs BOOTC_bootloader=systemd: plan-readonly and plan-switch-zstd-chunked (the Rust test upstream added since) PASSED.
just test-tmt readonly usroverlay on the ostree variant: plan-readonly and plan-usroverlay PASSED.

For the VM runs, the image build needed a local, uncommitted workaround: the Dockerfile's grub-cc-download stage currently fails on main because dnf download grub2-efi-x64-cc finds no such package in Fedora ELN, so it was allowed to fail there. Neither bootloader tested uses grub-cc.

Open PRs that add or change tests will need to drop their # number: line, rename if their file has a numeric prefix, and rerun just update-generated after rebasing. The Rust booted tests in crates/tests-integration/src/booted/ follow the same rule: their // number: header is gone too.

The Signed-off-by: Colin Walters <walters@verbum.org> on these commits was added on cgwalters's approval of the review draft: cgwalters-forge#41 (review)

Generated-by: https://github.com/cgwalters/#llms

The generated tests all lived in tmt/tests/tests.fmf and the plans in a block of tmt/plans/integration.fmf. Every pull request adding a test therefore touched the same two files, next to each other in a sorted list, so they conflict whenever two are open at once. Generate tmt/tests/tests/test-*.fmf and tmt/plans/integration/plan-*.fmf instead, one each. fmf merges the directory into the node of the same name, so tests.fmf and integration.fmf stay as the hand-written places for the settings common to all tests and plans, and the test and plan names (and so what `just test-tmt FILTER` selects) are unchanged; `tmt plan export` and `tmt test export` are identical before and after, modulo the file list. The tests need an explicit `path` now, since by default it follows the directory of the file defining them. `update-generated` also removes files of tests that went away, and the check fails on stale ones. Prep for dropping the test numbers from the names. Generated-by: AI Signed-off-by: Colin Walters <walters@verbum.org>

Each test file carried a `# number: N` header that became part of the test and plan names (test-50-foo, plan-50-foo) and the sort order. The number had to be unique by convention only, so every new test picked the next free one, and parallel pull requests regularly picked the same one; duplicates already exist on main. It carries no meaning: each plan runs in its own VM, so nothing depends on the order. Name tests and plans after the slug in the file name instead (test-<name>, plan-<name>) and drop the header. Test names must now be unique, which the generator checks. This is the mechanical part: the header lines are removed, the three scripts and three hand-written fmf files that still had a number in their name are renamed, the generated files are regenerated, and the docs that mention a number are updated. `just test-tmt <filter>` matches substrings of the plan name as before, so existing invocations such as `just test-tmt readonly` keep working. Generated-by: AI Signed-off-by: Colin Walters <walters@verbum.org>

Copilot

🟡 Changes recommended

Overlapping regex selectors can run unintended tests and plans in the same VM and bypass plan-specific exclusions.

2 open issues

What changed in this PR

Reduces merge conflicts in bootc’s TMT integration suite by generating separate test and plan files and replacing numeric identifiers with names.

Changes:

Generates one file per test and plan, with stale-file cleanup and validation.
Removes test numbers and checks name uniqueness.
Updates shared settings, metadata parsing, and contributor documentation.

File	Description
tmt/tests/tests/test-usroverlay.fmf	Separate usroverlay test metadata.
tmt/tests/tests/test-user-agent.fmf	Separate User-Agent test metadata.
tmt/tests/tests/test-upgrade-tag.fmf	Separate upgrade-tag test metadata.
tmt/tests/tests/test-upgrade-preflight-disk-check.fmf	Separate disk-check test metadata.
tmt/tests/tests/test-upgrade-check-status.fmf	Separate upgrade-status test metadata.
tmt/tests/tests/test-switch-zstd-chunked.fmf	Separate Rust test metadata.
tmt/tests/tests/test-switch-to-unified.fmf	Separate unified-switch test metadata.
tmt/tests/tests/test-switch-same-digest.fmf	Separate same-digest test metadata.
tmt/tests/tests/test-switch-mutate-in-place.fmf	Separate in-place switch metadata.
tmt/tests/tests/test-soft-reboot.fmf	Separate soft-reboot test metadata.
tmt/tests/tests/test-soft-reboot-selinux-policy.fmf	Separate SELinux reboot metadata.
tmt/tests/tests/test-shadow-fixup.fmf	Separate shadow-fixup test metadata.
tmt/tests/tests/test-rollback.fmf	Separate rollback test metadata.
tmt/tests/tests/test-readonly.fmf	Separate readonly test metadata.
tmt/tests/tests/test-multi-device-esp.fmf	Separate multi-device ESP metadata.
tmt/tests/tests/test-logically-bound-switch.fmf	Separate bound-image switch metadata.
tmt/tests/tests/test-logically-bound-install.fmf	Separate bound-image install metadata.
tmt/tests/tests/test-loader-entries-source.fmf	Separate loader-entry test metadata.
tmt/tests/tests/test-install-unified-flag.fmf	Separate unified-install test metadata.
tmt/tests/tests/test-install-to-filesystem-var-mount.fmf	Separate filesystem-install test metadata.
tmt/tests/tests/test-install-repart.fmf	Separate repart test metadata.
tmt/tests/tests/test-install-outside-container.fmf	Separate outside-container install metadata.
tmt/tests/tests/test-install-no-boot-dir.fmf	Separate missing-boot-directory test metadata.
tmt/tests/tests/test-install-karg-delete.fmf	Separate kernel-argument test metadata.
tmt/tests/tests/test-install-composefs-native.fmf	Separate native-composefs install metadata.
tmt/tests/tests/test-install-bootloader-none.fmf	Separate bootloader-none test metadata.
tmt/tests/tests/test-image-upgrade-reboot.fmf	Separate upgrade-reboot test metadata.
tmt/tests/tests/test-image-pushpull-upgrade.fmf	Separate push/pull upgrade metadata.
tmt/tests/tests/test-factory-reset.fmf	Separate factory-reset test metadata.
tmt/tests/tests/test-etc-merge-conflict.fmf	Separate merge-conflict test metadata.
tmt/tests/tests/test-download-only-upgrade.fmf	Separate download-only upgrade metadata.
tmt/tests/tests/test-download-only-switch.fmf	Separate download-only switch metadata.
tmt/tests/tests/test-custom-selinux-policy.fmf	Separate custom-policy test metadata.
tmt/tests/tests/test-composefs-uki-dumpfile.fmf	Separate UKI dumpfile metadata.
tmt/tests/tests/test-composefs-separate-boot.fmf	Separate boot-partition test metadata.
tmt/tests/tests/test-composefs-gc.fmf	Separate composefs GC metadata.
tmt/tests/tests/test-composefs-gc-uki.fmf	Separate UKI GC metadata.
tmt/tests/tests/test-composefs-corrupted-state-resilience.fmf	Separate corruption-resilience test metadata.
tmt/tests/tests/test-composefs-1-16-bridge.fmf	Separate compatibility-bridge test metadata.
tmt/tests/tests/test-bib-build.fmf	Separate image-builder test metadata.
tmt/tests/tests.fmf	Retains shared settings and sets working directory.
tmt/tests/test-switch-to-unified.fmf	Removes number from legacy metadata filename.
tmt/tests/test-multi-device-esp.fmf	Removes number from legacy metadata filename.
tmt/tests/test-install-unified-flag.fmf	Removes number from legacy metadata filename.
tmt/tests/install/test-on-ostree/test-install-on-ostree.sh	Removes numeric header.
tmt/tests/booted/test-usroverlay.nu	Removes numeric header.
tmt/tests/booted/test-user-agent.py	Removes numeric header.
tmt/tests/booted/test-upgrade-tag.nu	Removes numeric header.
tmt/tests/booted/test-upgrade-preflight-disk-check.nu	Removes numeric header.
tmt/tests/booted/test-upgrade-check-status.nu	Removes numeric header.
tmt/tests/booted/test-switch-to-unified.nu	Removes numeric header.
tmt/tests/booted/test-switch-same-digest.nu	Removes numeric header.
tmt/tests/booted/test-switch-mutate-in-place.nu	Removes numeric header.
tmt/tests/booted/test-soft-reboot.nu	Removes numeric header.
tmt/tests/booted/test-soft-reboot-selinux-policy.nu	Removes numeric header.
tmt/tests/booted/test-shadow-fixup.nu	Removes filename number and numeric header.
tmt/tests/booted/test-rollback.nu	Removes numeric header.
tmt/tests/booted/test-readonly.nu	Removes filename number and numeric header.
tmt/tests/booted/test-multi-device-esp.nu	Removes numeric header.
tmt/tests/booted/test-logically-bound-switch.nu	Removes numeric header.
tmt/tests/booted/test-logically-bound-install.nu	Removes numeric header.
tmt/tests/booted/test-loader-entries-source.nu	Removes numeric header.
tmt/tests/booted/test-install-unified-flag.nu	Removes numeric header.
tmt/tests/booted/test-install-to-filesystem-var-mount.sh	Removes numeric header.
tmt/tests/booted/test-install-repart.nu	Removes numeric header.
tmt/tests/booted/test-install-outside-container.nu	Removes numeric header.
tmt/tests/booted/test-install-no-boot-dir.nu	Removes numeric header.
tmt/tests/booted/test-install-karg-delete.nu	Removes numeric header.
tmt/tests/booted/test-install-composefs-native.nu	Removes numeric header.
tmt/tests/booted/test-install-bootloader-none.nu	Removes numeric header.
tmt/tests/booted/test-image-upgrade-reboot.nu	Removes numeric header.
tmt/tests/booted/test-image-pushpull-upgrade.nu	Removes numeric header.
tmt/tests/booted/test-factory-reset.nu	Removes numeric header; updates example comment.
tmt/tests/booted/test-etc-merge-conflict.nu	Removes numeric header.
tmt/tests/booted/test-download-only-upgrade.nu	Removes numeric header.
tmt/tests/booted/test-download-only-switch.nu	Removes numeric header.
tmt/tests/booted/test-custom-selinux-policy.nu	Removes numeric header.
tmt/tests/booted/test-composefs-uki-dumpfile.nu	Removes numeric header.
tmt/tests/booted/test-composefs-separate-boot.nu	Removes numeric header.
tmt/tests/booted/test-composefs-gc.nu	Removes numeric header.
tmt/tests/booted/test-composefs-gc-uki.nu	Removes numeric header.
tmt/tests/booted/test-composefs-corrupted-state-resilience.nu	Removes numeric header.
tmt/tests/booted/test-composefs-1-16-bridge.nu	Removes filename number and numeric header.
tmt/tests/booted/test-bib-build.nu	Removes numeric header.
tmt/tests/booted/README.md	Documents name-based test creation.
tmt/plans/integration/plan-usroverlay.fmf	Separate usroverlay plan.
tmt/plans/integration/plan-user-agent.fmf	Separate User-Agent plan.
tmt/plans/integration/plan-upgrade-tag.fmf	Separate upgrade-tag plan.
tmt/plans/integration/plan-upgrade-preflight-disk-check.fmf	Separate disk-check plan.
tmt/plans/integration/plan-upgrade-check-status.fmf	Separate upgrade-status plan.
tmt/plans/integration/plan-switch-zstd-chunked.fmf	Separate Rust test plan.
tmt/plans/integration/plan-switch-to-unified.fmf	Separate unified-switch plan.
tmt/plans/integration/plan-switch-same-digest.fmf	Separate same-digest plan.
tmt/plans/integration/plan-switch-mutate-in-place.fmf	Separate in-place switch plan.
tmt/plans/integration/plan-soft-reboot.fmf	Separate soft-reboot plan.
tmt/plans/integration/plan-soft-reboot-selinux-policy.fmf	Separate SELinux reboot plan.
tmt/plans/integration/plan-shadow-fixup.fmf	Separate shadow-fixup plan.
tmt/plans/integration/plan-rollback.fmf	Separate rollback plan.
tmt/plans/integration/plan-readonly.fmf	Separate readonly plan.
tmt/plans/integration/plan-multi-device-esp.fmf	Separate multi-device ESP plan.
tmt/plans/integration/plan-logically-bound-switch.fmf	Separate bound-image switch plan.
tmt/plans/integration/plan-logically-bound-install.fmf	Separate bound-image install plan.
tmt/plans/integration/plan-loader-entries-source.fmf	Separate loader-entry plan.
tmt/plans/integration/plan-install-unified-flag.fmf	Separate unified-install plan.
tmt/plans/integration/plan-install-to-filesystem-var-mount.fmf	Separate filesystem-install plan.
tmt/plans/integration/plan-install-repart.fmf	Separate repart plan.
tmt/plans/integration/plan-install-outside-container.fmf	Separate outside-container install plan.
tmt/plans/integration/plan-install-no-boot-dir.fmf	Separate missing-boot-directory plan.
tmt/plans/integration/plan-install-karg-delete.fmf	Separate kernel-argument plan.
tmt/plans/integration/plan-install-composefs-native.fmf	Separate native-composefs install plan.
tmt/plans/integration/plan-install-bootloader-none.fmf	Separate bootloader-none plan.
tmt/plans/integration/plan-image-upgrade-reboot.fmf	Separate upgrade-reboot plan.
tmt/plans/integration/plan-image-pushpull-upgrade.fmf	Separate push/pull upgrade plan.
tmt/plans/integration/plan-factory-reset.fmf	Separate factory-reset plan.
tmt/plans/integration/plan-etc-merge-conflict.fmf	Separate merge-conflict plan.
tmt/plans/integration/plan-download-only-upgrade.fmf	Separate download-only upgrade plan.
tmt/plans/integration/plan-download-only-switch.fmf	Separate download-only switch plan.
tmt/plans/integration/plan-custom-selinux-policy.fmf	Separate custom-policy plan.
tmt/plans/integration/plan-composefs-uki-dumpfile.fmf	Separate UKI dumpfile plan.
tmt/plans/integration/plan-composefs-separate-boot.fmf	Separate boot-partition plan.
tmt/plans/integration/plan-composefs-gc.fmf	Separate composefs GC plan.
tmt/plans/integration/plan-composefs-gc-uki.fmf	Separate UKI GC plan.
tmt/plans/integration/plan-composefs-corrupted-state-resilience.fmf	Separate corruption-resilience plan.
tmt/plans/integration/plan-composefs-1-16-bridge.fmf	Separate compatibility-bridge plan.
tmt/plans/integration/plan-bib-build.fmf	Separate image-builder plan.
tmt/plans/integration.fmf	Retains shared plan settings.
tmt/bug-soft-reboot.md	Points documentation to source metadata.
hack/provision-configure.sh	Updates test name in a comment.
docs/src/bootc-composefs.7.md	Updates documented bridge-test name.
crates/xtask/src/tmt.rs	Implements per-file generation and name-based metadata.
crates/tests-integration/src/booted/switch_zstd_chunked.rs	Removes numeric header.
crates/tests-integration/src/booted.rs	Updates header documentation.

_{Review effort: Balanced}

    for test in &tests {
-        let test_key = format!("/test-{:02}-{}", test.number, test.name);
+        let test_id = format!("test-{}", test.name);
+        let plan_id = format!("plan-{}", test.name);


cgwalters · 2026-10-21T14:46:25Z

            serde_yaml::Value::String("fmf".to_string()),
        );
-        let test_path = format!("/tmt/tests/tests/test-{:02}-{}", test.number, test.name);
+        let test_path = format!("/{TMT_TESTS_DIR}/{test_id}");


Wow, very insightful if true and clearly a huge problem

cgwalters-bot · 2026-10-21T14:53:48Z

Fix round 1 of 2 for 2acec7492dfa, on cgwalters's comments: claimed.

Generated-by: https://github.com/cgwalters/#llms

cgwalters-bot · 2026-10-21T14:53:49Z

Fix round 1 of 2 for 2acec7492dfa, on cgwalters's comments: run.

Generated-by: https://github.com/cgwalters/#llms

cgwalters-bot · 2026-10-21T14:54:58Z

✅ dispatch ended well and handed back no result. Its reason:

The operator's quoted comment expresses agreement with a review finding but requests no change. Per the brief's no-change rule, left the branch unchanged and produced no replacement series.

cgwalters-bot added 2 commits October 19, 2026 14:59

cgwalters-bot force-pushed the bot/tmt-no-numbers branch from e60876b to 2acec74 Compare October 19, 2026 18:59

cgwalters-bot mentioned this pull request Oct 19, 2026

tmt: Generate one file per test and plan, drop test numbers cgwalters-forge/bootc#41

Closed

github-actions Bot added the area/documentation Updates to the documentation label Oct 19, 2026

bootc-bot Bot requested a review from henrywang October 19, 2026 19:00

cgwalters requested a balanced review from Copilot October 19, 2026 19:38

Copilot started reviewing on behalf of cgwalters October 19, 2026 19:39

View session

Copilot AI reviewed Oct 19, 2026

View reviewed changes

Navigation Menu

Repository navigation

tmt: Generate one file per test and plan, drop test numbers - #2581

tmt: Generate one file per test and plan, drop test numbers#2581
cgwalters-bot wants to merge 2 commits into
bootc-dev:mainfrom
cgwalters-forge:bot/tmt-no-numbers

cgwalters-bot commented Oct 19, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

cgwalters Oct 21, 2026

Uh oh!

cgwalters-bot commented Oct 21, 2026

Uh oh!

cgwalters-bot commented Oct 21, 2026 •

edited

Loading

Uh oh!

cgwalters-bot commented Oct 21, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

cgwalters-bot commented Oct 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

cgwalters Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

cgwalters-bot commented Oct 21, 2026

Uh oh!

cgwalters-bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters-bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cgwalters-bot commented Oct 21, 2026 •

edited

Loading

cgwalters-bot commented Oct 21, 2026 •

edited

Loading