BYOL0 — bring your own layer zero
A build tool that assembles a Linux distribution onto ZFS root from the vendor's own package repositories — and the artifact OS that falls out of it.
Boot environments, air-gapped installs, and a snapshot before every package transaction.
The family: kldload — the substrate · zxplore — the ZFS console · wgxplore — the WireGuard console · vmxplore — the VM console
There are two ways to use this.
1 — As a build tool, against your own Linux. It replaces the
"download the .iso" step. dnf --installroot, debootstrap or pacstrap
against the vendor's own CDN, then ZFS and NVIDIA compiled against the kernel
that install just laid down, signed and MOK-enrolled so they load with Secure
Boot left on. Nothing forked, nothing patched. The build stops wherever you
decide — a plain install you finish by hand, or a fully baked image that gets
deployed rather than installed.
2 — As the finished article. The ISO is what that build leaves behind: ZFS on root with boot environments, WireGuard, KVM, Kubernetes, an eBPF observability plane and a local AI stack, with the kernel and its out-of-tree modules pinned as one matched set so a routine update cannot separate them.
The thing it adds that a vendor image does not have: apt, dnf and
pacman snapshot the root before every transaction, so reversing a failed
upgrade is one command rather than a rescue USB and an evening.
Two substrates in the installer menu — Fedora and Debian — and two package managers underneath. Both install with the network unplugged, from complete mirrors baked into the ISO.
Website: kldload.com · Download: full ISO / net installer · Release notes: 1.5.0 · Discord: discord.gg/QX8wf38N3V
The install, in two screens. Boot the USB and this opens by itself — no
terminal, no wiki. It is also reachable at https://<host>:8443 from another
machine, for a box with no screen.
Pick what to build — the distribution, the profile, and what goes in:
Say where it goes, then start it:
First boot. Nobody drives this part. The machine bootstraps Kubernetes, pulls the AI model and builds its golden images on its own, then tells you whether anything was flagged.
zxplore — the ZFS console. Every dataset, every property, and the snapshot
list that makes apt rollback possible.
Docker's layers are ZFS datasets. The storage driver is zfs, so a pull is
a clone and every layer inherits the pool's compression.
wgxplore — the WireGuard estate. Four planes across the fleet, joined into one view, with every peer no host declares called out.
vmxplore — the VM estate. Grouped by what the machines are for rather than by name, with the guest's own screen rendered in the console — no separate viewer. Disks are ZFS zvols, so a clone is instant and a rollback is one command.
ztxplore — the OpenZFS test lab. Six distributions on zvols, each running the distro's own ZFS packages against its own kernel. Goldens are built once and cloned per run, and it reports what did not build as plainly as what did.
Kubernetes, HA by default. Three control planes behind a kube-vip VIP; adding a node reconciles the mesh, etcd and the firewall everywhere else.
Metrics, grouped. 29 Grafana dashboards: the estate, eBPF, the pool, and the OpenZFS test lab kept separate from it.
| A 64-bit x86 machine | UEFI required. Legacy BIOS boot is not supported. |
| A USB stick, 32 GB or larger | The full image is ~15 GB. The 2.2 GB net installer fits a 4 GB stick. |
| A target disk | It will be erased. |
| Network | Optional — both substrates install fully offline. |
Legacy BIOS is absent by design, not untested: the installer writes GPT with an EFI System Partition and a ZFS pool, and Secure Boot, ZFSBootMenu, boot repair and rollback all operate on the ESP. See docs/INSTALL.md for the full detail.
# Download and burn (USB target)
curl -L -o kldload.iso https://dl.kldload.com/kldload-free-latest.iso # full: offline mirrors, ~15 GB
curl -L -o kldload.iso https://dl.kldload.com/kldload-free-net-latest.iso # or the net installer, ~2.2 GB
sudo wipefs -af /dev/sdX
sudo dd if=kldload.iso of=/dev/sdX bs=4M oflag=direct conv=fsync status=progress && sync
# Or build from source
git clone https://github.com/kldload/kldload.git && cd kldload
PROFILE=desktop ./deploy.sh build
sudo ./deploy.sh burn /dev/sdX # names the device, shows it, asks before writingThe full image carries the offline mirrors, the Kubernetes images and the AI stack, about 15 GB. Two smaller ways to build it:
EDITION=net ./deploy.sh build # tools only, ~2.2 GB — installs from the distros' mirrors
DARKSITES=fedora OLLAMA=no ./deploy.sh build # one mirror, no AI stack — a Fedora-only offline desktop
./deploy.sh menu # a checklist that writes kldload.env and shows the sizePAYLOAD, DARKSITES (debian, fedora, el), K8S_IMAGES and OLLAMA are
the knobs; ./deploy.sh help lists them with sizes. The ISO name says what
it carries: -net, or -fedora for a single mirror.
Boot the USB → the web UI opens over TLS at https://<host>:8443 → pick distro + profile + disk → install.
Building saturates every core by default. To keep the machine usable while it runs, cap the compress step — it is the long one:
KLDLOAD_BUILD_PROCESSORS=$(( $(nproc) - 4 )) PROFILE=desktop ./deploy.sh buildThe same USB that installs one machine is the network server for a whole rack. It serves its own kernel and root image; nothing is installed on the machine running it, and nothing is fetched from the internet.
# 1. Make the master USB (the full image, 16.9 GB: it carries the offline mirrors)
curl -L -o kldload.iso https://dl.kldload.com/kldload-free-latest.iso
sudo dd if=kldload.iso of=/dev/sdX bs=4M oflag=direct conv=fsync status=progress && sync
# 2. Boot any machine on the rack's switch from it and press p at the boot menu:
# "provision the rack from this USB". Log in as live / live and check:
sudo kldload-netboot-server status # payload: the live medium, armed: any
# 3. Only if the switch has NO DHCP server of its own (air-gapped): give the
# master an address and let it hand them out
sudo ip addr add 10.99.0.1/24 dev enp3s0
printf 'NETBOOT_IFACE=enp3s0\nNETBOOT_DHCP=standalone\n' | sudo tee /etc/kldload/netboot.env
sudo systemctl restart kldload-netboot-live- Network-boot each target. It shows the kldload menu and counts down. Left alone it boots its own disk, so nothing is erased by accident; any key opens the install, and the password and the disk are asked for on that machine's own screen.
- Or arm a whole rack unattended: one answers file per MAC in a directory,
then
sudo kldload-netboot-server arm-all ./rack/(add--dry-runfirst to check every file). Each armed machine installs exactly its file.
Before you start: each target needs UEFI network boot and more RAM than the image — the root image is loaded into memory, and an 8 GB machine panics part way. The full procedure, the two network cases and the known issues in 1.5.0: docs/NETBOOT.md.
Full-disk ZFS encryption is off unless you ask for it — that changed in 1.5.0 — and asking for it is recommended. Three ways to ask:
- Web UI: on the install page set ZFS encryption to Encrypted — AES-256-GCM, passphrase at boot and enter a passphrase. The selector starts on None.
- Answers file:
KLDLOAD_ZFS_ENCRYPT=1andKLDLOAD_ZFS_PASSPHRASE=.... A netbooted or seeded machine whose file sets the flag but not the passphrase is asked for one on its own screen before anything is erased. - Netboot menu: press any key during the countdown, tick ZFS encryption under options, and type the passphrase on the console when asked.
The flow is then short:
- Download & burn the ISO to a USB stick (see Quickstart above).
- Boot the USB. The installer opens automatically in the browser at
https://<host>:8443— no login prompt. - Choose your distribution, profile, and target disk, turn encryption on as above, set your disk encryption passphrase, and start the install.
- When it finishes the machine reboots. Remove the USB stick.
- At the ZFSBootMenu prompt, enter your encryption passphrase. With Secure Boot off the pool key is embedded in the initramfs, so that one prompt is the only one. With Secure Boot on it deliberately is not (the initramfs sits on the unencrypted ESP), so the initramfs asks a second time.
- The desktop loads and the console opens at
https://<host>:8443— no certificate warning, no login prompt. Done.
An unencrypted install asks for nothing at boot. Turning Secure Boot on or off
does not change whether the pool is encrypted. TPM unlock is in 1.5.0 as
kldload-tpm-seal plus a dracut module that unlocks the encrypted root from
the TPM, bound to Secure Boot through PCR 7; it is inert until you seal it,
and has not been tested on real hardware.
What "default" means depends on how you install, and the code paths do not agree with each other, so say it explicitly every time:
- Web UI: the Secure Boot card starts unticked and the page sends
KLDLOAD_ENABLE_SECURE_BOOT=0. Off unless you tick it. - Netboot menu, a key pressed: the Secure Boot tick starts from the
answers file; an absent key shows unticked and is sent as
0. Off unless you tick it. - Answers file with the key absent (a USB seed,
--config, or a netboot left to its countdown): the boot-chain code treats absent as on — it generates a MOK, signs the modules, installs shim and makes GRUB'sdirectentry the default — while the install manifest records0and the machine reboots instead of powering off for the enrollment. WriteKLDLOAD_ENABLE_SECURE_BOOT=0or=1; never leave it out.
With it off the firmware boots ZFSBootMenu directly: no shim, no GRUB stage, no MOK enrollment, and nothing to miss at a ten-second prompt. That is the right default for a lab machine, and it is the path above.
Turn it on when the machine's threat model wants a verified boot chain, by
ticking the card or setting KLDLOAD_ENABLE_SECURE_BOOT=1. The install then
generates a per-install MOK, signs the out-of-tree modules with it, and boots
firmware → shim → GRUB → kernel. ZFSBootMenu stays in GRUB's
menu but is not the default there: shim 15.8's SBAT rule refuses to chainload
it. That path needs three extra steps:
- When the install finishes it powers the machine off rather than rebooting — so you control the enrollment boot instead of racing an auto-reboot. (An unattended netboot or seed install reboots regardless.) Remove the USB stick.
- Power on and enter firmware setup (usually
Del,F2, orF10). Enable Secure Boot, then save and exit. - On the next boot the blue MokManager screen appears — it waits
only ~10 seconds, so press any key immediately, then:
Enroll MOK → Continue → Yes → password
kldload→ Reboot. The password is literallykldload— not your admin or encryption password.
Missed the MokManager screen? Just reboot — kldload re-offers enrollment on every boot until the key is actually enrolled. No reinstall. If you end up at a "Secure Boot validation failed" screen instead, the app grid's Secure Boot Repair tool (or
sudo kldload-mok-repairfrom any terminal — including the live USB) diagnoses and queues the fix in one step.
Full install walkthrough, including what each Secure Boot failure looks like and how to decide whether to run with it on at all: docs/INSTALL.md.
Secure Boot is off in a default web UI or netboot-menu install, and none of
the below applies unless you turned it on — or left the key out of an
answers file, which turns the boot-chain preparation on (see above). These are
the failure modes of that path (KLDLOAD_ENABLE_SECURE_BOOT=1).
| Symptom | Fix |
|---|---|
| Missed the blue MokManager screen | Reboot — enrollment is re-offered automatically. Or sudo kldload-mok-repair repair, then reboot. |
| "Secure Boot validation failed" / "Verification failed" at boot | The install's MOK isn't enrolled. Run sudo kldload-mok-repair (installed system or live USB) — it shows whether the boot chain's key is enrolled and repair queues the fix; then reboot, press a key at the 10-second blue screen, Reset → Enroll, password kldload. Or temporarily disable Secure Boot in firmware to boot and repair from the OS. |
| Reinstalled several times / MOK operations start failing | Stale keys accumulate in NVRAM (one per install). sudo kldload-mok-repair repair queues a Reset MOK list + enrollment of the current key in one pass. |
| Check enrollment / signing state | sudo kldload-mok-repair (or kldload-mok-repair status, mokutil --list-enrolled). |
| NVIDIA or ZFS module won't load under SB | Same cause — enroll the MOK. sudo kldload-mok-repair status shows the module signer. |
| Forgot the MOK password | It's kldload (set a different one at install with KLDLOAD_MOK_PASSWORD). |
| Boots to emergency mode after skipping enrollment | The MOK is not enrolled, so the kernel refuses the DKMS-signed ZFS module and non-root datasets never mount. Confirm with modprobe zfs — Key was rejected by service is conclusive. Fix: sudo kldload-mok-repair repair then reboot and enroll, or disable Secure Boot in firmware if this is a lab box. Note this can appear weeks later, at the first kernel update after the missed prompt. |
| Boots fine with SB off, fails with SB on (installed before 1.4.0-rc3) | Older builds re-signed the staged kernel with the per-install MOK key, discarding the distro's own signature. Restore it: sudo cp /boot/vmlinuz-$(uname -r) /boot/efi/EFI/BOOT/vmlinuz then enable Secure Boot. Fixed at install time from 1.4.0-rc3 on. |
Shouldn't happen on a fresh install — the console cert is issued by the kldload CA, which is trusted in the browser automatically. If a warning appears, re-import the CA root (clearing any stale entry first):
kldload-trust-cert # re-import the CA root
# stubborn? drop stale entries first, then re-import:
certutil -d sql:"$HOME/.pki/nssdb" -D -n kldload-webui 2>/dev/null
certutil -d sql:"$HOME/.pki/nssdb" -D -n kldload-ca 2>/dev/null
kldload-trust-certOnly remote browsers do — sign in with your admin account (a wheel/
sudo user). On the machine itself the console never prompts.
| Distribution | Install method | Offline |
|---|---|---|
| Fedora 44 | dnf --installroot |
Yes (RPM darksite) |
| Debian 13 (Trixie) | debootstrap |
Yes (APT darksite) |
Those two are what the installer offers, because they are the two that get tested — and the two that deliver the offline promise. Everything else in the RPM and APT families is a network install, which is a different product from the one this README describes.
The method underneath is not distro-specific: kldload installs by calling
dnf --installroot or debootstrap against a distribution's own repositories.
Nothing is forked, nothing is patched, and no image is pre-baked for a given
distro.
Which means the reachable set is wider than the tested one. RHEL, CentOS
Stream, Rocky, Ubuntu and Arch all still work with KLDLOAD_DISTRO=<name> —
the code paths are there and maintained — but they are not on the web UI's
menu and not mirrored offline. RHEL is the partial exception: added to the
netboot menu (NETBOOT_DISTROS) it is sent the 2.2 GB net image and installs
from Red Hat's CDN with a subscription, and the 1.5.0 sweep installed a RHEL
desktop that way — from the net image only, never offline. The rest are not
tested this release, and the menu is the honest statement of what is. Adding
one back is repository configuration, not new machinery.
Fedora and Debian are deliberately the widest useful pair rather than a narrow one: two package managers, two firmware-splitting conventions, two initramfs generators, and one leading-edge substrate against one stable one. Most bugs worth finding show up as a difference between them.
The honest bound: a new distribution needs its repos and keys declared, and its kernel paired with a version of OpenZFS that builds against it. That is a morning's work, not a port.
Live environment is Fedora 44 (kernel 7.0.x — currently 7.0.12 — with OpenZFS 2.4.3 on root).
Fedora 44 + ZFS: OpenZFS ships a native
fc44build (2.4.3), so there is nofc43bridge. The kernel is not taken as whatever Fedora ships today:builder/kernel-pin.shreads the ceiling OpenZFS itself declares (zfs-dkmsConflicts — 2.4.3 caps at kernel ≤ 7.0.999) and resolves the newest matching build, pulling kernel,-core,-modules,-develand-headersas one set so they cannot be split. On the installed system that set plus NVIDIA is versionlocked at first boot, so a routinednf updatecannot pull a kernel ZFS has no build for.
A GUI-first workstation that looks like stock RHEL 10: expert operations — ZFS replication, KVM, Kubernetes, eBPF observability — exposed as point-and-shoot desktop apps, not CLI rituals.
- Install-time Platform Options. Checkboxes for NVIDIA drivers, KVM, Kubernetes, eBPF tooling, and golden-image building. Desktop-only, default-clean — you opt into the heavy stuff.
- Native app windows. Each tool (VMs, Kubernetes, ZFS, Metrics, the model, …) opens as its own chromeless GTK/WebKit window — no browser chrome, no left menu — backed by the same web console the server edition serves.
- Console as its own app. The tmux F-key operator cockpit (k9s, ZFS internals, eBPF panels, VM/log streams) is a single Console application — not embedded inside every tool window.
- Local model. Ollama with Open WebUI (plus RAG and voice) as a desktop app. No cloud, no telemetry.
Seven profiles, the seven the netboot menu offers. A profile is a starting point, not a cage: every one of them is an answers file, and any key it sets can be changed per machine.
| Profile | What you get |
|---|---|
| Core | ZFS on root and nothing else. Stock distro, no k* tools, no web UI, no darksites — about 200 MB beyond the vendor's base install |
| Server | Headless: SSH, ZFS root, the full k* tool suite, the web console, sanoid snapshots, WireGuard, eBPF tooling, offline darksites |
| Desktop | GNOME on ZFS root, GPU drivers, Firefox, native app windows for every tool, the Console cockpit, Ollama, offline darksites |
| KVM host | A hypervisor: libvirt + qemu-kvm, every VM on a ZFS zvol, ~100 ms copy-on-write clones, golden images, zfs send replication |
| Kubernetes | KVM host plus a highly available cluster in VMs — 3 control planes and 3 workers, Cilium + Hubble + Tetragon |
| Storage | A ZFS file server: NFS, SMB and iSCSI, shares as datasets, snapshots and replication underneath |
| AI | KVM host plus Ollama and Open WebUI on the local GPU, RAG and voice, no cloud |
KVM is on for every profile except Core. Two templates build on the KVM host rather than being profiles of their own:
| Template | What it adds |
|---|---|
| klab | Golden VMs for every supported distro, blue/green sites from ZFS instant clones, fault injection, the distro matrix runner, a live Hubble traffic map |
| OpenZFS suite | Dedicated test goldens wired into ztest/zloop for upstream OpenZFS regression hunting |
kube-cluster bootstrap --workers 3 # golden image, then 3 control planes + 3 workers
kube-demo # PetClinic + ArgoCD smoke test
klab golden centos # build the CentOS golden VM
klab run-playbook ./my-test.sh # run a script on every VM of the blue siteThe USB installs one machine. The same USB, booted with its provision the
rack entry (p at the boot menu), is the network server for every other
machine on the wire: it serves the kernel, initramfs and root image straight
off the stick, nothing is installed or copied on the machine running it, and
any box that network-boots gets the install menu on its own screen. A key
there installs; left alone, the countdown boots that machine's own disk, so a
stranger that happens to network-boot first loses nothing. The menu asks for
the password, and the disk, on the target itself — nothing secret is ever on
the wire. Fedora and Debian are offered from the USB (RHEL needs the separate
net image).
An installed machine can be the permanent server instead — install it with
KLDLOAD_KEEP_NETBOOT=1, systemctl enable --now kldload-netboot — and arm
machines one at a time, which is how a rack is rebuilt from a directory of
answers files:
kldload-netboot-server arm-install <mac> <answers>.env # one machine
kldload-netboot-server arm-all ./rack/ # every .env in a directory
kldload-netboot-server status # what is armed right now
A target needs nothing but a network port. It pulls the whole system — every package, the container images, the lot — from one image staged once on your LAN. Nothing else leaves the network: no package mirrors, no registries, no vendor endpoints. A rack can be built in a room with no uplink, which is the entire reason the darksite payload exists. The fiftieth machine costs the same to prepare as the second, because nothing is fetched per machine.
It cannot wipe a machine by accident. Installing requires a per-MAC consent
token. A machine that network-boots without one prints that it is continuing the
boot order and boots from its own disk. arm-all refuses duplicate MACs or
hostnames outright and exits non-zero unless every machine armed, so a
half-armed rack is an error you hear about rather than discover.
An armed machine shows a summary of what it was armed with and counts down
(NETBOOT_MENU_TIMEOUT seconds, 10 unless set) before fetching anything large.
Left alone, it installs exactly the armed answers file. Any key opens the
manual override: pick a profile (core, server, desktop, kvm, k8s, storage, ai
-- each the armed file with that profile's settings), a distro (from
NETBOOT_DISTROS, default fedora; list only the ones you have verified --
Fedora and Debian are the offline, tested pair; RHEL and Arch are sent the
net image instead (NETBOOT_NET_DISTROS), and Arch stays off the list: an
encrypted Arch install panics at boot, and a rolling release cannot be
version-locked the way the rest of the substrate is),
then tick options -- ZFS encryption, Secure Boot, KVM, golden images,
Kubernetes, the ZFS lab, AI -- and change the hostname, user, time zone or
keyboard. Golden images, Kubernetes and the ZFS lab need KVM, so ticking one
ticks it. Secrets never go through the menu: a login password or encryption
passphrase the answers file lacks is asked for on the machine's own screen,
before anything is erased. Encryption asks for the passphrase at every boot
and Secure Boot asks for the key to be enrolled on the first reboot. Esc goes
back, then boots the local disk, so a machine netbooted by mistake can be sent
back to its own disk without it being wiped or pulling the image.
Per-machine differences live in the answers file — hostname, disk, profile, how many control planes and workers. Same image, different answers.
Boot on one NIC, download on a faster one. Many 10G cards carry no UEFI
PXE code, so only the onboard port can start the boot. --netdev <mac> (or
KLDLOAD_NETBOOT_NETDEV=<mac> in the answers file, which is how arm-all
takes it) hands the multi-gigabyte image to the fast card instead. After the
install, the host bridge goes on the fastest linked NIC and the others stay up
as fallbacks.
Your own workloads land with the cluster, not after it. Drop a Helm chart in
/root/darksite/helm-charts/workloads/ or plain YAML in
/root/darksite/manifests/, and first boot installs them once the cluster is up
and before it reports ready. Manifests apply in sorted order, so 10-namespace
lands before 20-deploy. Add your container images to
build/darksite/k8s-images.txt and they are baked in too — so your application
is on every node with nothing pulled from a registry.
Ansible runs inverted here: ansible-playbook executes locally on each machine
at first boot rather than being pushed from a control node. No inventory to keep
current, no SSH fan-out, no credentials held centrally. Each machine builds
itself, and a hundred do it at once without coordinating.
Measured end to end on one target: fifteen minutes from power-on to a
six-node HA Kubernetes cluster with all nodes Ready, landing on a usable
desktop, with the six node clones taking 0 bytes against one 2.32 GB
golden image.
Step by step — make the master USB, the air-gapped case, arming a rack, and the known issues: docs/NETBOOT.md · Arming, the failure modes, and what to check →
- OpenZFS on root — checksummed, compressed, snapshotted, self-healing on mirrors. lz4 default. Native AES-256-GCM encryption recommended, off unless selected; passphrase at ZFSBootMenu on every boot. TPM unlock (
kldload-tpm-seal, bound to PCR 7) ships inert and untested on hardware; dedup optional. - ZFSBootMenu — UEFI bootloader that understands ZFS. Boot environments. Seconds-fast rollback. It is what the firmware boots with Secure Boot off; with Secure Boot on the chain is shim → GRUB → kernel, because shim's SBAT rule refuses to chainload ZFSBootMenu, which then stays as a GRUB menu entry.
- WireGuard — kernel-level encrypted networking. One UDP port at the firewall.
- eBPF observability — BCC tools + bpftrace + an F-key tmux cockpit on the host; Cilium + Hubble + Tetragon inside the K8s profile (no kube-proxy, no iptables, no sidecars).
- KVM hypervisor — libvirt + qemu-kvm with every VM on a ZFS zvol.
~100ms clones via COW. Atomic snapshots. fs-freeze app-consistency. Incrementalzfs sendreplication. - Docker & podman on ZFS — every image layer is a real dataset, not a directory inside an overlay. A
pullis a clone, layers inherit compression, and the whole container estate — layers, the engine's database and the volumes — snapshots and replicates as one recursivezfs send. Measured: a running container cloned and started in 328 ms with its state intact, and 24.7 GB of estate in a single stream. Docker on the apt distros, podman on the RPM ones. The full list of what this replaces → - NVIDIA + CUDA — drivers and CUDA optional at install. Time-sliced GPU sharing across the model and guest VMs. No PCIe passthrough required.
- Ollama + Open WebUI — local model: RAG over the codebase + voice + tmux awareness + ReAct agent loop + eBPF-aware tool registry. No cloud, no telemetry.
- Observability — Prometheus + Grafana + Loki + Alertmanager, Go + bash exporters, pre-wired dashboards,
zedZFS events bridged to Loki. - Secure Boot + MOK — per-machine key generation, automatic module signing, DKMS auto-sign on kernel upgrades. Off in the web UI and the netboot menu unless ticked; an answers file must state it.
- Image export —
kexportproduces qcow2 / VMDK / VHD / OVA / raw, auto-sealed with cloud-init multi-datasource config. Ready for Packer or direct hypervisor import. - Offline + Air-gap — RPM and APT mirrors baked in. The USB is the deployment, the recovery, and the air gap.
kldload invents almost nothing. It is an opinionated assembly of software you already know, installed from the vendors' own repos and wired together so the pieces actually meet. If you recognise a name below, that is the point — you already know how to operate it, and nothing here is a bespoke reimplementation you would have to learn.
Nothing is forked and nothing is patched. The tree carries zero .patch
files and zero vendored third-party source; every component arrives from its
upstream package repo, its official release artifact, or its own git remote.
Two shipped components are not open source, and it would be dishonest to bury them in a list like this: Google Chrome (the default desktop browser and the renderer for the kldload GUI apps — its open-source upstream Chromium is what Debian targets get) and the NVIDIA driver + CUDA, which is opt-in at install. Everything else below is open source under its own licence.
| Project | What it does here |
|---|---|
| OpenZFS | root filesystem, snapshots, clones, send/recv, native encryption |
| ZFSBootMenu (2.3.0) | UEFI boot environments, rollback from the boot screen |
| sanoid / syncoid | snapshot retention policy and replication |
| dracut, GRUB2, shim, mokutil, sbsigntools, pesign | initramfs, UEFI boot chain, Secure Boot module signing |
| cryptsetup, LVM2, mdadm, e2fsprogs, xfsprogs, btrfs-progs | non-ZFS storage the installer must still read |
| Project | What it does here |
|---|---|
| libvirt + QEMU/KVM | every VM, each backed by its own ZFS zvol |
virt-install, qemu-img, qemu-guest-agent |
provisioning, image conversion, in-guest control |
| swtpm + edk2/OVMF | emulated TPM 2.0 and UEFI firmware for guests |
| cloud-init | first-boot configuration of golden-image clones |
| Project | What it does here |
|---|---|
| Kubernetes 1.32 (kubeadm/kubelet/kubectl) | the cluster itself |
| containerd | container runtime |
| Cilium 1.16.5 + Hubble | eBPF CNI, kube-proxy replacement, flow visibility |
| MetalLB 0.14.9 | bare-metal LoadBalancer services |
| kube-vip 0.8.9 | control-plane VIP for HA |
| OpenEBS ZFS LocalPV | CSI storage on ZFS |
| local-path-provisioner (Rancher) | fallback StorageClass where a node has no ZFS |
| Gateway API 1.2.1, metrics-server | ingress API, resource metrics |
| Helm, k9s, Headlamp | chart installs, terminal cluster UI, web cluster UI |
| WireGuard, nftables, NetworkManager, chrony | encrypted backplane, firewall, networking, time |
| nginx | one TLS reverse proxy on :8443 for every browser-facing service |
| Project | What it does here |
|---|---|
| Prometheus + Alertmanager | metrics and alerting |
| Grafana | pre-wired dashboards |
| Loki + Promtail | log aggregation, with ZFS zed events bridged in |
| node_exporter, ebpf_exporter, process-exporter, smartctl_exporter, zfs_exporter, libvirt-exporter | the metric sources |
| Project | What it does here |
|---|---|
| BCC tools + bpftrace | the F-key tracing cockpit (execsnoop, biosnoop, tcplife, …) |
| Tetragon | runtime security observability |
| Secure Boot + MOK toolchain | per-machine keys, DKMS auto-signing on kernel upgrade |
| Project | What it does here |
|---|---|
| GNOME — Shell, GDM, Nautilus, Terminal/Ptyxis, Control Center | the workstation session (LightDM on Debian Trixie) |
| PipeWire + WirePlumber | audio |
| Google Chrome | the default browser on RPM desktops, from Google's own repo, and what the kldload GUI apps render in |
| Firefox | also installed on RPM desktops, and the browser on the GhostBSD posture |
| Chromium | the browser on Debian targets |
| NVIDIA driver + CUDA | optional at install, via RPM Fusion akmod-nvidia |
| Steam | optional, via Flathub (Fedora) |
| eza, bat, fd, ripgrep, zoxide, fzf, fastfetch, htop | the modern CLI set, pre-wired into the shell |
| ttyd + tmux | browser terminal, and the session everything attaches to |
| Project | What it does here |
|---|---|
| Ollama | the LLM runtime |
| Llama 3.1 / 3.2 & Qwen2.5 | the models, chosen automatically by detected VRAM (incl. Llama 3.2-Vision, Qwen2.5-Coder) |
ChromaDB + nomic-embed-text |
the RAG vector store and embeddings over your own docs |
| whisper.cpp | speech to text (voice input) |
| Piper | text to speech (voice output) |
No cloud, no telemetry, no API key — the models and the index live on the machine.
| Project | What it does here |
|---|---|
| Ansible (ansible-core) | golden-VM provisioning and the web UI's Ansible tab |
| Argo CD | GitOps engine behind the demo app stack |
| osbuild-composer | Red Hat's own toolchain, used to build the RHEL golden image |
GParted, TestDisk/PhotoRec, ddrescue,
fsarchiver, smartmontools, nvme-cli, p7zip,
ntfs-3g/exfatprogs, fio, stress-ng, memtest86+ — the install USB doubles
as the recovery USB.
The installer bootstraps each target with that distro's own tool —
debootstrap for Debian and
dnf --installroot for Fedora. The same two paths reach the rest of their
families when driven directly, but only these two are on the menu and tested.
Packages come from the vendors' own
CDNs; the ISO itself is built with Red Hat's
lorax, dracut, squashfs-tools and
xorriso inside a Fedora 44 container.
Three of the consoles are their own BSD-3 projects by the same author, with their own repos and release cadence. kldload builds each from its own upstream at ISO-build time and records the exact commit it shipped, so an installed system can say precisely what it is running:
| Console | Upstream | Commit recorded in |
|---|---|---|
| zxplore — ZFS | its own repo | /etc/kldload/zxplore-commit |
| vmxplore — KVM | its own repo | /etc/kldload/vmxplore-commit |
| wgxplore — WireGuard | this repo's wg/ |
/etc/kldload/wgxplore-commit |
wgxplore now has its own repo, but the copy kldload builds and ships is the
in-tree wg/ — folded in on 2026-08-10 as a read-only estate lens, which is
why its recorded commit is kldload's HEAD rather than a separate upstream. The
two have since diverged, with the in-tree copy carrying work the standalone
repo does not, so treat wg/ as the source of what is on the ISO.
They run on any Linux or BSD box — kldload is their first-party distribution, not their owner.
Licences are each project's own; kldload ships them unmodified and adds no licence terms of its own to them. See License for kldload's.
apt, dnf and pacman are symlinked to kldload-pkg-wrapper at install, so
these are the normal commands — a script, unattended-upgrades or the GUI
updater get the same protection.
| Command | What it does |
|---|---|
apt upgrade / dnf update |
Snapshots the root, then runs the real transaction |
apt rollback |
Stages a return to the pre-transaction snapshot; reboot to apply |
apt rollback list |
Shows which transactions you could go back to |
apt rollback cancel |
Un-stages it — nothing has changed until you reboot |
kldload-rollback |
The same machinery directly, with boot-environment control |
kpkg |
Package operations with pre-install snapshots |
kupgrade |
Guided upgrade with automatic rollback on failure |
Rollback clones the snapshot into a new boot environment rather than running
zfs rollback, which cannot touch a mounted root and would destroy every newer
snapshot. Nothing is overwritten.
| Command | What it does |
|---|---|
kldload-overview |
Unified host status — ZFS, VMs, K8s, GPU, eBPF, services |
kst |
System health dashboard |
kldload-console |
tmux F-key cockpit with live eBPF panels |
| Command | What it does |
|---|---|
ksnap |
Snapshot manager |
kclone |
Clone datasets / zvols |
kbe |
Boot environment manager |
kdf |
ZFS-aware disk usage |
kpkg |
Package manager with pre-install snapshots |
kupgrade |
Safe upgrade with automatic rollback |
krecovery |
Disaster recovery |
kexport |
Export golden images (qcow2 / VMDK / VHD / OVA / raw) |
| Command | What it does |
|---|---|
kvm-create |
Create VM on a ZFS zvol |
kvm-clone |
ZFS instant clone (~100 ms) |
kvm-snap |
Snapshot a VM |
kvm-list |
List all VMs |
kvm-delete |
Destroy VM + zvol |
| Command | What it does |
|---|---|
kube-cluster bootstrap |
Build the golden image and bring up the cluster: 3 control planes by default, --workers N (--cps 1 only for a host that cannot fit three) |
kube-cluster destroy |
Tear it down (golden preserved) |
kube-demo |
Deploy PetClinic + ArgoCD smoke test |
kube-smoke-test |
Automated cluster verification |
| Command | What it does |
|---|---|
klab golden <distro> |
Build / refresh a golden VM image |
klab run-playbook <script> [blue|green] |
Run a script on every VM of a site (blue by default) |
klab-vm-debug-bundle |
Auto-fires on test failure — OpenZFS-ready debug tarball |
| Subcommand | What it does |
|---|---|
build |
Build the ISO (uses cached darksites) |
full |
Rebuild the builder image + all darksites, then build the ISO |
clean |
Remove build artifacts |
burn [/dev/sdX] [--yes] |
Write the ISO to a USB device. Names the target; falls back to USB_DEVICE, then auto-detects a single removable drive. Confirms interactively (prints the device's model and size, and asks you to type the name back); --yes skips the prompt for scripts. |
builder-image |
Rebuild the Fedora 44 builder container |
smoke-build |
Static checks on the built ISO (size, freshness, content) |
zfs-pin |
Derive the kernel pin from the newest OpenZFS release's declared Linux-Maximum and report drift against build-iso.sh (--check for CI, --json for scripts) |
smoke-test <distro> <profile> |
Full install lifecycle in KVM, then smoke-test the installed target |
build-debian-darksite |
Build / refresh the Debian APT offline mirror |
build-ubuntu-darksite |
Ubuntu mirror — retired 2026-08, opt in with KLDLOAD_INCLUDE_UBUNTU_DARKSITE=1 |
build-fedora-darksite |
Build / refresh the RPM offline mirror |
build-ollama-darksite |
Cache the Bob/Ollama model bundle |
kvm-deploy / kvm-deploy-bob |
Deploy the ISO to local KVM via virt-install |
proxmox-deploy |
Deploy to a remote Proxmox host via the qm API |
deploy-all |
Build + deploy across the configured targets |
Live environment: Fedora 44 (kernel 7.0.x, OpenZFS 2.4.3)
Builder: Fedora 44 container (lorax + squashfs-tools + xorriso + dracut)
Bootstrap paths: dnf --installroot (Fedora)
debootstrap (Debian)
Installer: Python web UI + ~10 bash libraries (lib/) + backend/bin tools
Web UI: single HTML file per edition + WebSocket install-log stream
Single-port TLS: kldload-proxy fronts the web UI, Grafana, Prometheus, Headlamp,
Bob, k9s/ttyd, and the libvirt console on one URL (:8443) with one cert
The user picks the target distro at install time. After install the system runs upstream packages from the vendor's public repos. There is no kldload package repository and no kldload-specific runtime updates — dnf update / apt upgrade / pacman -Syu just work.
Current release: 1.5.0 (26 September 2026). 582 commits since 1.4.2, 455 files changed, 47,421 lines added. In operator terms:
-
One key provisions the rack. An installed machine can keep the netboot payload it was built from and serve the next one: proxyDHCP beside your existing DHCP, one answers file per MAC,
arm-allon a directory. -
A netboot menu. An armed machine shows what it is about to do and counts down; any key opens the override — profile, distro, encryption, Secure Boot, components, identity — with secrets typed on the console only.
-
Rebuild a node from its replica, restamped for the machine it lands on and proved by a canary boot rather than by the receive finishing.
-
The first boot explains itself. The install show holds the screen through the reboot and into first boot, with the real build log beside it.
-
Firecracker microVMs on the same zvols, jailed and unprivileged; and blue/green for whole clusters, not just workloads.
-
Two defaults changed. ZFS encryption is off unless asked for, and the shipped answer templates carry no default password. KVM is on for every profile but core.
-
GitHub release: v1.5.0
-
Full changelog:
CHANGELOG.md -
Release notes, with screenshots: kldload.com/releases/1.5.0.html
-
History back to 1.0: kldload.com/release-notes.html
Every release is tagged (git tag -l 'v*'), so checking out the tag gives the
exact tree an ISO was built from. Cutting one: docs/RELEASING.md.
BSD-3-Clause. See LICENSE.











