tests: Port integration tests to barrage

barrage has no equivalent to pytest's `parametrize`. Open-code that with
`do_test_*(params...)` helpers and individual tests calling that.

Stop passing the explicit `--distribution` to the test. The new
`ImageConfigManager` singleton (which replaces the old `config` pytest
fixture) reads the distribution from `mkosi.local.conf`. This avoids
having to specify it twice, and leaves the configuration to
`integration-test-setup.sh`.

Drop the `log_setup()` call. The integration tests themselves don't use
`logging`, this mostly just affects the called `mkosi` subprocesses
which already do it. This is very prone to interfere with barrage's own
stderr capturing.

Replace the `--debug-shell` pytest option with a `TEST_DEBUG_SHELL` env
variable, as barrage doesn't have test-defined CLI options.

Make the mkosi invocations in tests/__init__.py async, so that they can
actually run in parallel.

Remove installation and remaining traces of pytest.

Co-developed-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: martinpitt

.github/workflows/ci.yml

@@ -144,14 +144,14 @@ jobs:
       - name: Run integration tests
         run: |
           # Without KVM the tests are way too slow and time out
+          # barrage currently has no dynamic runtime throttling to resources
+          # each test runs a ~ 1.5 GiB VM, and GitHub runners have 4 GiB
           if [[ -e /dev/kvm ]]; then
             sudo mkosi box -- \
               timeout -k 30 1h \
-              python3 -m pytest \
-              --tb=no \
-              --capture=no \
+              python3 -m barrage \
               --verbose \
-              -m integration \
-              --distribution ${{ matrix.distro }} \
+              --max-concurrency=2 \
+              --pattern 'integration_*.py' \
               tests/
           fi

.gitignore

@@ -5,7 +5,6 @@
 .mypy_cache/
 .project
 .pydevproject
-.pytest_cache/
 /.mkosi-*
 /SHA256SUMS
 /SHA256SUMS.gpg

AGENTS.md

@@ -17,9 +17,8 @@ Always consult these files as needed:
 - select a subset by name, e.g. `bin/mkosi box -- python3 -m barrage test_mypy` to run a specific check
 - `bin/mkosi box -- ruff format mkosi tests kernel-install/*.install` to format code
 - `bin/mkosi box -- ruff check --fix mkosi tests kernel-install/*.install` to fix ruff issues
-- `python3 -m pytest -m integration ...` to run integration tests. No need to run these by default.
+- `bin/mkosi box -- python3 -m barrage --pattern 'integration_*.py' tests/` to run integration tests (after `tools/integration-test-setup.sh`). No need to run these by default.
 - `bin/mkosi box -- python3 -m barrage --pattern 'install_*.py'` to run installation tests (venv/pip/zipapp). Skipped by default as they install from the network. No need to run these by default.
-
 - Never invent your own build commands or try to optimize the build process.
 - Never use `head`, `tail`, or pipe (`|`) the output of build or test commands. Always let the full output
 display. This is critical for diagnosing build and test failures.

README.md

@@ -161,9 +161,9 @@ When a tool that mkosi runs inside its sandbox fails, see
 
 # Integration tests
 
-Integration tests build and boot full images. They are marked with the
-`integration` marker, and are skipped by default. They need a tools tree and an
-image to be built first. `tools/integration-test-setup.sh` writes a local
+Integration tests build and boot full images. They live in `integration_*.py`
+files and are not run as part of the normal test suite. They need a tools tree
+and an image to be built first; `tools/integration-test-setup.sh` writes a local
 configuration for the given image and tools tree distribution and builds both:
 
 ```sh
@@ -176,13 +176,37 @@ single integration test:
 ```sh
 tools/integration-test-setup.sh arch fedora
 
-bin/mkosi box -- pytest -m integration --distribution arch --capture=no --verbose \
-    'tests/test_boot.py::test_bootloader[systemd-boot]'
+bin/mkosi box -- python3 -m barrage -v \
+    'tests/integration_boot.py::test_bootloader_systemd_boot'
+```
+
+To run all integration tests, select them by file name pattern:
+
+```sh
+bin/mkosi box -- python3 -m barrage --pattern 'integration_*.py' tests/
 ```
 
 The integration tests require KVM and are skipped (or very slow) without
 `/dev/kvm`.
 
+To debug a failing build, set `TEST_DEBUG_SHELL=1` to pass `--debug-shell` to
+mkosi, which drops into an interactive shell when a build step fails. This only
+makes sense when running a single test interactively (`barrage -i`):
+
+```sh
+TEST_DEBUG_SHELL=1 bin/mkosi box -- python3 -m barrage -i \
+    'tests/integration_boot.py::test_bootloader_systemd_boot'
+```
+
+barrage runs tests concurrently with no limit by default. Pass
+`--max-concurrency N` to restrict the parallelism.
+
+When running unprivileged (i.e. not as root), building several images in the
+same session tends to exhaust systemd-nsresourced's pool of dynamic UID ranges,
+which makes builds fail with `io.systemd.NamespaceResource.NoDynamicRange`. Run
+the tests one at a time (or with `--max-concurrency 1`) to avoid this; it does
+not happen when the tests run as root, as they do in CI.
+
 # References
 
 * [Primary mkosi git repository on GitHub](https://github.com/systemd/mkosi/)

mkosi.tools.conf/mkosi.conf.d/arch.conf

@@ -11,7 +11,6 @@ Packages=
         mypy
         pyright
         python-pip
-        python-pytest
         reuse
         ruff
         sequoia-sop

mkosi.tools.conf/mkosi.conf.d/azure-centos-fedora.conf

@@ -12,5 +12,4 @@ Packages=
         python3-mypy
         python3-pip
         npm
-        python3-pytest
         shellcheck

mkosi.tools.conf/mkosi.conf.d/debian-kali-ubuntu.conf

@@ -13,7 +13,6 @@ Packages=
         mypy
         npm
         python3-pip
-        python3-pytest
         reuse
         sqop
         shellcheck

mkosi.tools.conf/mkosi.conf.d/opensuse.conf

@@ -11,7 +11,6 @@ Packages=
         mypy
         npm
         python3-pip
-        python3-pytest
         reuse
         ruff
         ShellCheck

pyproject.toml

@@ -110,13 +110,3 @@ lint.select = [
 
 [tool.ruff.lint.flake8-bugbear]
 extend-immutable-calls = ["mkosi.run.nosandbox"]
-
-[tool.pytest.ini_options]
-markers = [
-    "integration: mark a test as an integration test.",
-    "install: mark a test as an installation test (venv/pip/zipapp).",
-]
-testpaths = ["tests"]
-# The unit and install tests run under barrage now; pytest only runs the integration tests, which live
-# in integration_*.py files (not pytest's default test_*.py pattern).
-python_files = ["integration_*.py"]

tests/__init__.py

@@ -1,23 +1,25 @@
 # SPDX-License-Identifier: LGPL-2.1-or-later
 
-import contextlib
+import asyncio
 import dataclasses
 import os
 import subprocess
 import sys
 import uuid
-from collections.abc import Iterator, Mapping, Sequence
+from collections.abc import Mapping, Sequence
 from pathlib import Path
 from types import TracebackType
-from typing import Any, Optional
+from typing import Optional
 
-import pytest
+from barrage import Singleton
 
+import mkosi.resources
+from mkosi.config import parse_config
 from mkosi.distribution import Distribution
 from mkosi.run import CompletedProcess, run
 from mkosi.tree import rmtree
 from mkosi.user import INVOKING_USER
-from mkosi.util import _FILE, PathString
+from mkosi.util import _FILE, PathString, resource_path
 
 
 @dataclasses.dataclass(frozen=True)
@@ -28,6 +30,36 @@ class ImageConfig:
     snapshot: Optional[str] = None
 
 
+class ImageConfigManager(Singleton):
+    """Provide the integration test ImageConfig
+
+    The distribution and release are read from mkosi.local.conf as written by
+    tools/integration-test-setup.sh.
+    """
+
+    config: ImageConfig
+
+    async def __aenter__(self) -> "ImageConfigManager":
+        if not Path("mkosi.local.conf").exists():
+            raise RuntimeError(
+                "mkosi.local.conf not found: run 'tools/integration-test-setup.sh "
+                "<distribution> <tools-tree-distribution>' to configure and build the image before "
+                "running the integration tests."
+            )
+
+        with resource_path(mkosi.resources) as resources:
+            config = parse_config(resources=resources)[2][0]
+        self.config = ImageConfig(
+            distribution=config.distribution,
+            release=config.release,
+            debug_shell=bool(os.getenv("TEST_DEBUG_SHELL")),
+            # Pin the same snapshot as the main image so builds that don't read mkosi.local.conf still
+            # use it (e.g. the extension build, which passes --directory '').
+            snapshot=config.snapshot,
+        )
+        return self
+
+
 class Image:
     def __init__(self, config: ImageConfig) -> None:
         self.config = config
@@ -53,7 +85,7 @@ def __exit__(
     ) -> None:
         rmtree(self.output_dir)
 
-    def mkosi(
+    async def mkosi(
         self,
         verb: str,
         options: Sequence[PathString] = (),
@@ -62,7 +94,11 @@ def mkosi(
         check: bool = True,
         env: Mapping[str, str] = {},
     ) -> CompletedProcess:
-        return run(
+        # mkosi.run.run() is synchronous, so run it in a worker thread.
+        # Safe because the test-level invocation uses no sandbox (and thus no preexec_fn) and installs no
+        # signal handlers; all sandboxing happens inside the spawned mkosi subprocess.
+        return await asyncio.to_thread(
+            run,
             [
                 "python3", "-m", "mkosi",
                 "--debug",
@@ -73,10 +109,10 @@ def mkosi(
             check=check,
             stdin=stdin,
             stdout=sys.stdout,
-            env=os.environ | env,
+            env={**os.environ, **env},
         )  # fmt: skip
 
-    def build(
+    async def build(
         self,
         options: Sequence[PathString] = (),
         args: Sequence[str] = (),
@@ -107,18 +143,18 @@ def build(
             *options,
         ]  # fmt: skip
 
-        self.mkosi("summary", opt, env=env)
+        await self.mkosi("summary", opt, env=env)
 
-        return self.mkosi(
+        return await self.mkosi(
             "build",
             opt,
             args,
             stdin=sys.stdin if sys.stdin.isatty() else None,
             env=env,
         )
 
-    def boot(self, options: Sequence[str] = (), args: Sequence[str] = ()) -> CompletedProcess:
-        result = self.mkosi(
+    async def boot(self, options: Sequence[str] = (), args: Sequence[str] = ()) -> CompletedProcess:
+        result = await self.mkosi(
             "boot",
             [
                 "--runtime-build-sources=no",
@@ -139,15 +175,15 @@ def boot(self, options: Sequence[str] = (), args: Sequence[str] = ()) -> Complet
 
         return result
 
-    def vm(
+    async def vm(
         self,
         options: Sequence[str] = (),
         args: Sequence[str] = (),
         ram: str = "1536M",
     ) -> CompletedProcess:
         need_hyperv_workaround = os.uname().machine == "x86_64"
 
-        result = self.mkosi(
+        result = await self.mkosi(
             "vm",
             [
                 "--runtime-build-sources=no",
@@ -171,35 +207,3 @@ def vm(
             raise subprocess.CalledProcessError(result.returncode, result.args, result.stdout, result.stderr)
 
         return result
-
-
-@pytest.fixture(scope="session", autouse=True)
-def suspend_capture_stdin(pytestconfig: Any) -> Iterator[None]:
-    """
-    When --capture=no (or -s) is specified, pytest will still intercept
-    stdin. Let's explicitly make it not capture stdin when --capture=no is
-    specified so we can debug image boot failures by logging into the emergency
-    shell.
-    """
-
-    capmanager: Any = pytestconfig.pluginmanager.getplugin("capturemanager")
-
-    if pytestconfig.getoption("capture") == "no":
-        capmanager.suspend_global_capture(in_=True)
-
-    yield
-
-    if pytestconfig.getoption("capture") == "no":
-        capmanager.resume_global_capture()
-
-
-@contextlib.contextmanager
-def ci_group(s: str) -> Iterator[None]:
-    github_actions = os.getenv("GITHUB_ACTIONS")
-    if github_actions:
-        print(f"\n::group::{s}", flush=True)
-    try:
-        yield
-    finally:
-        if github_actions:
-            print("\n::endgroup::", flush=True)