Add a `BuildahChrootProvisioner` to run nested "containers"

[comps]

When running in an existing podman/docker container, SELinux + seccomp prevents effective nesting (UID mapping, etc.), but old-school chroots can still be used to somewhat isolate execution.

buildah supports both puling OCI images and using them via chroots, even for arbitrary command execution (not just building), allowing tests to run in chroots in nicely prepared container images (no need to custom-prepare images).

Basically:

• buildah pull to fetch the image
• buildah from to prepare a new rootfs ("start a container")
• buildah run to run commands in that rootfs as chroot ("podman exec")
• buildah rm to delete it

There's likely no PID / network / etc. isolation, so this can never be made to support systemd.

$ dnf -y install buildah

$ buildah --storage-driver vfs containers
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME

$ buildah --storage-driver vfs pull fedora:44
Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.fedoraproject.org/fedora:44...
Getting image source signatures
Copying blob b1ad493156b5 done   | 
Copying config 338bec00ee done   | 
Writing manifest to image destination
338bec00ee4404d6ba6624d577c6bd3a47e10469e4658ac9c3d756fc1b0fb683

$ buildah --storage-driver vfs from --isolation chroot --pull=never 338bec00ee44
fedora-working-container

$ buildah --storage-driver vfs run --isolation chroot fedora-working-container ls -a /root
.  ..  .bash_logout  .bash_profile  .bashrc  .cshrc  .tcshrc

$ buildah --storage-driver vfs run --isolation chroot fedora-working-container touch /root/here

$ buildah --storage-driver vfs run --isolation chroot fedora-working-container ls -a /root
.  ..  .bash_logout  .bash_profile  .bashrc  .cshrc  .tcshrc  here

# buildah --storage-driver vfs rm fedora-working-container

While buildah generates unique names automatically (fedora-working-container-1, -2, etc.), we should not rely on it - we can just generate an UUID:

uuid=$(uuid -v4)
buildah --storage-driver vfs from --isolation chroot --name atex-$uuid --pull=never 338bec00ee44

A custom chrooting logic could be implemented, but buildah automates image fetching and is established for running chroots for building already, and if the environment changes (ie. podman prohibits bind mounts), it would adjust to that, we don't need to modify anything.

[comps]

That said, it's questionable what the benefit would be over LocalProvisioner - a chroot doesn't isolate processes, network or permissions, just the basic rootfs visibility, so the only realistic use would be avoiding dnf install pollution in the parent container or something like that, and that's a questionable use case.

[comps]

Maybe UML would be a better bet - for RHEL/Fedora, we'd need to rebuild the kernel image, but with the extremely minimal HW that UML needs, that takes only a few minutes. Coupled with the container image in one tarball, it could be very quickly deployed as a full systemd-based OS.

Networking / ssh can be taken care of by slirp4netns, or (more modern) gvproxy, part of the podman/buildah ecosystem.

Some AI-assisted notes follow.

---

Kernel-building and tarballing script

#!/bin/bash
set -euo pipefail

image="${1:-registry.fedoraproject.org/fedora:43}"
output="${2:-uml-image.tar}"

# Build the UML kernel from the distro's kernel SRPM
build=$(buildah --storage-driver vfs from --quiet "$image")

buildah --storage-driver vfs run --isolation chroot "$build" -- bash <<'BUILD'
set -euo pipefail

# Download and prep the kernel source (applies all distro patches)
dnf download --source kernel
dnf builddep -y kernel-*.src.rpm
rpmbuild -rp --define "_topdir /root/rpmbuild" kernel-*.src.rpm  # unpack + patch

cd /root/rpmbuild/BUILD/kernel-*-build/kernel-*/linux-*/

# Use the distro's own kernel config as the base
arch=$(uname -m)
if [ -f /root/rpmbuild/SOURCES/kernel-${arch}-fedora.config ]; then
    cp /root/rpmbuild/SOURCES/kernel-${arch}-fedora.config .config
elif [ -f /root/rpmbuild/SOURCES/kernel-${arch}-rhel.config ]; then
    cp /root/rpmbuild/SOURCES/kernel-${arch}-rhel.config .config
else
    echo "error: no kernel config found for ${arch}" >&2; exit 1
fi

# Adapt the distro config for ARCH=um, then add UML-specific options
make ARCH=um olddefconfig
scripts/config \
    --enable HOSTFS \
    --enable UML_NET_VECTOR \
    --disable DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
    --enable DEBUG_INFO_NONE
make ARCH=um olddefconfig  # re-resolve after overrides

make ARCH=um -j"$(nproc)" linux
install -m755 linux /kernel-uml
BUILD

# Extract the kernel binary from the build container
build_mnt=$(buildah --storage-driver vfs mount "$build")
cp "$build_mnt/kernel-uml" kernel-uml
buildah --storage-driver vfs rm "$build" >/dev/null

# Build the runtime image via Containerfile
cat > Containerfile <<'CONTAINERFILE'
ARG BASE_IMAGE
FROM ${BASE_IMAGE}
RUN dnf install -y --setopt=install_weak_deps=False --nodocs \
        systemd NetworkManager openssh-server passwd iproute
RUN dnf clean all
RUN echo "PermitRootLogin yes" >> /etc/ssh/sshd_config
RUN ssh-keygen -A
RUN systemctl enable sshd NetworkManager
COPY kernel-uml /kernel-uml
CONTAINERFILE

runtime_image=$(buildah --storage-driver vfs build --isolation chroot \
    --quiet --build-arg "BASE_IMAGE=$image" -f Containerfile .)
runtime=$(buildah --storage-driver vfs from --quiet "$runtime_image")
runtime_mnt=$(buildah --storage-driver vfs mount "$runtime")
tar --numeric-owner -cf "$output" -C "$runtime_mnt" .
buildah --storage-driver vfs rm "$runtime" >/dev/null
rm -f kernel-uml Containerfile

echo "wrote $output ($(du -h "$output" | cut -f1))"

With the procedure being, more or less, this:

GUIDE

UML in a Podman Container

Run a full Fedora/RHEL kernel as an unprivileged process inside a default
Podman container, using User-mode Linux (UML). No --privileged, no extra
capabilities, no /dev/kvm.

Building

build-uml.sh downloads the kernel SRPM, applies all distro patches,
builds a UML kernel using the distro's own config, installs systemd +
NetworkManager + sshd into a runtime image, and packages everything as
a flat tarball.

./build-uml.sh                                      # Fedora 43 (default)
./build-uml.sh quay.io/centos/centos:stream9        # CentOS Stream 9

Requires buildah in the container. x86_64 only (UML does not support
other architectures).

Unpacking

The tarball is the output of build-uml.sh:

mkdir rootfs
tar --numeric-owner -xpf uml-image.tar -C rootfs

Setting up access

The build script already installs and enables sshd with root login
permitted. You just need to provide credentials.

Set a root password:

chroot rootfs bash -c 'echo "root:changeme" | chpasswd'

Or inject an SSH key:

mkdir -p rootfs/root/.ssh
cp ~/.ssh/id_ed25519.pub rootfs/root/.ssh/authorized_keys
chmod 700 rootfs/root/.ssh
chmod 600 rootfs/root/.ssh/authorized_keys

Booting

# UML needs a tmpdir without noexec — default Podman mounts /dev/shm
# as noexec, so point TMPDIR at /tmp which allows exec
TMPDIR=/tmp ./rootfs/kernel-uml \
    root=/dev/root rootfstype=hostfs "rootflags=$(realpath rootfs)" rw \
    mem=512M init=/usr/lib/systemd/systemd \
    con0=fd:0,fd:1 con=null

Note: rw means the guest writes directly to the rootfs/ directory
on the host. Changes persist after shutdown. To start fresh, re-extract
the tarball.

halt -f or poweroff -f inside the guest shuts down cleanly.

Networking

gvproxy (shipped with Podman at /usr/libexec/podman/gvproxy) provides
unprivileged networking via the BESS socket transport. It includes a DHCP
server, a DNS forwarder, and automatic SSH port forwarding.

/usr/libexec/podman/gvproxy \
    --listen-bess unixpacket:///tmp/bess.sock \
    -ssh-port 2222 &

TMPDIR=/tmp ./rootfs/kernel-uml \
    root=/dev/root rootfstype=hostfs "rootflags=$(realpath rootfs)" rw \
    mem=512M init=/usr/lib/systemd/systemd \
    con0=fd:0,fd:1 con=null \
    vec0:transport=bess,dst=/tmp/bess.sock,mac=5a:94:ef:e4:0c:ee
    # MAC must match gvproxy's static DHCP lease so the VM gets
    # 192.168.127.2 — the IP that -ssh-port forwards to

NetworkManager (included in the image) auto-configures the network via
gvproxy's DHCP. No config files, no systemd service masking needed.

SSH

gvproxy forwards localhost:2222 to port 22 inside the VM by default:

ssh -p 2222 root@localhost

Port forwarding (for extra non-ssh ports)

Start gvproxy with a control socket:

/usr/libexec/podman/gvproxy \
    --listen-bess unixpacket:///tmp/bess.sock \
    --listen unix:///tmp/gvproxy.sock &

Expose ports via the HTTP API:

# TCP port
curl --unix-socket /tmp/gvproxy.sock \
    http:/unix/services/forwarder/expose \
    -X POST -d '{"local":":8080","remote":"192.168.127.2:80"}'

# Unix socket
curl --unix-socket /tmp/gvproxy.sock \
    http:/unix/services/forwarder/expose \
    -X POST -d '{"local":"unix:///tmp/vm.sock","remote":"192.168.127.2:80"}'

# List / remove
curl --unix-socket /tmp/gvproxy.sock http:/unix/services/forwarder/all
curl --unix-socket /tmp/gvproxy.sock \
    http:/unix/services/forwarder/unexpose \
    -X POST -d '{"local":":8080"}'

Extra kernel parameters

Extra arguments go at the end of the kernel command line:

... con0=fd:0,fd:1 con=null ncpus=2 mem=1G

Notable parameters:

• ncpus=N — SMP (requires kernel SECCOMP mode; default ptrace mode
  only supports ncpus=1)
• mem=N — guest memory (UML default is 64M; examples above use 512M)
• con=null con0=null,fd:2 con1=fd:0,fd:1 — boot messages to stderr,
  login prompt on stdout (useful for systemd, where getty runs on tty1)
• hostfs=/path — confine additional hostfs mounts to a directory
  (security hardening; the root mount is already confined by rootflags)