Skip to content
View kldload's full-sized avatar

Block or report kldload

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
kldload/README.md

 kldload

BYOL0 — bring your own layer zero

A build tool that assembles a Linux distribution onto ZFS root from the vendor's own package repositories — and the artifact OS that falls out of it.

Boot environments, air-gapped installs, and a snapshot before every package transaction.

License: BSD-3 Platform Substrates Root Boot

The family: kldload — the substrate · zxplore — the ZFS console · wgxplore — the WireGuard console · vmxplore — the VM console

kldload desktop — the installed system, with the console tools on the dock

There are two ways to use this.

1 — As a build tool, against your own Linux. It replaces the "download the .iso" step. dnf --installroot, debootstrap or pacstrap against the vendor's own CDN, then ZFS and NVIDIA compiled against the kernel that install just laid down, signed and MOK-enrolled so they load with Secure Boot left on. Nothing forked, nothing patched. The build stops wherever you decide — a plain install you finish by hand, or a fully baked image that gets deployed rather than installed.

2 — As the finished article. The ISO is what that build leaves behind: ZFS on root with boot environments, WireGuard, KVM, Kubernetes, an eBPF observability plane and a local AI stack, with the kernel and its out-of-tree modules pinned as one matched set so a routine update cannot separate them.

The thing it adds that a vendor image does not have: apt, dnf and pacman snapshot the root before every transaction, so reversing a failed upgrade is one command rather than a rescue USB and an evening.

Two substrates in the installer menu — Fedora and Debian — and two package managers underneath. Both install with the network unplugged, from complete mirrors baked into the ISO.

Website: kldload.com · Download: full ISO / net installer · Release notes: 1.5.0 · Discord: discord.gg/QX8wf38N3V

The install, in two screens. Boot the USB and this opens by itself — no terminal, no wiki. It is also reachable at https://<host>:8443 from another machine, for a box with no screen.

Pick what to build — the distribution, the profile, and what goes in:

Installer: pick what to build

Say where it goes, then start it:

Installer: target disk and identity

First boot. Nobody drives this part. The machine bootstraps Kubernetes, pulls the AI model and builds its golden images on its own, then tells you whether anything was flagged.

First boot: build phases complete, nothing flagged

What it looks like once it is up

zxplore — the ZFS console. Every dataset, every property, and the snapshot list that makes apt rollback possible.

zxplore datasets and snapshots

Docker's layers are ZFS datasets. The storage driver is zfs, so a pull is a clone and every layer inherits the pool's compression.

Docker on ZFS in zxplore

wgxplore — the WireGuard estate. Four planes across the fleet, joined into one view, with every peer no host declares called out.

wgxplore estate view

vmxplore — the VM estate. Grouped by what the machines are for rather than by name, with the guest's own screen rendered in the console — no separate viewer. Disks are ZFS zvols, so a clone is instant and a rollback is one command.

vmxplore: the VM estate with a live guest screen

ztxplore — the OpenZFS test lab. Six distributions on zvols, each running the distro's own ZFS packages against its own kernel. Goldens are built once and cloned per run, and it reports what did not build as plainly as what did.

ztxplore: the OpenZFS test lab

Kubernetes, HA by default. Three control planes behind a kube-vip VIP; adding a node reconciles the mesh, etcd and the firewall everywhere else.

Kubernetes in the web console

Metrics, grouped. 29 Grafana dashboards: the estate, eBPF, the pool, and the OpenZFS test lab kept separate from it.

The metrics section


Requirements

A 64-bit x86 machine UEFI required. Legacy BIOS boot is not supported.
A USB stick, 32 GB or larger The full image is ~15 GB. The 2.2 GB net installer fits a 4 GB stick.
A target disk It will be erased.
Network Optional — both substrates install fully offline.

Legacy BIOS is absent by design, not untested: the installer writes GPT with an EFI System Partition and a ZFS pool, and Secure Boot, ZFSBootMenu, boot repair and rollback all operate on the ESP. See docs/INSTALL.md for the full detail.

Quickstart

# Download and burn (USB target)
curl -L -o kldload.iso https://dl.kldload.com/kldload-free-latest.iso        # full: offline mirrors, ~15 GB
curl -L -o kldload.iso https://dl.kldload.com/kldload-free-net-latest.iso    # or the net installer, ~2.2 GB
sudo wipefs -af /dev/sdX
sudo dd if=kldload.iso of=/dev/sdX bs=4M oflag=direct conv=fsync status=progress && sync

# Or build from source
git clone https://github.com/kldload/kldload.git && cd kldload
PROFILE=desktop ./deploy.sh build
sudo ./deploy.sh burn /dev/sdX      # names the device, shows it, asks before writing

The full image carries the offline mirrors, the Kubernetes images and the AI stack, about 15 GB. Two smaller ways to build it:

EDITION=net ./deploy.sh build            # tools only, ~2.2 GB — installs from the distros' mirrors
DARKSITES=fedora OLLAMA=no ./deploy.sh build   # one mirror, no AI stack — a Fedora-only offline desktop
./deploy.sh menu                         # a checklist that writes kldload.env and shows the size

PAYLOAD, DARKSITES (debian, fedora, el), K8S_IMAGES and OLLAMA are the knobs; ./deploy.sh help lists them with sizes. The ISO name says what it carries: -net, or -fedora for a single mirror.

Boot the USB → the web UI opens over TLS at https://<host>:8443 → pick distro + profile + disk → install.

Building saturates every core by default. To keep the machine usable while it runs, cap the compress step — it is the long one:

KLDLOAD_BUILD_PROCESSORS=$(( $(nproc) - 4 )) PROFILE=desktop ./deploy.sh build

Provision a rack from one USB

The same USB that installs one machine is the network server for a whole rack. It serves its own kernel and root image; nothing is installed on the machine running it, and nothing is fetched from the internet.

# 1. Make the master USB (the full image, 16.9 GB: it carries the offline mirrors)
curl -L -o kldload.iso https://dl.kldload.com/kldload-free-latest.iso
sudo dd if=kldload.iso of=/dev/sdX bs=4M oflag=direct conv=fsync status=progress && sync

# 2. Boot any machine on the rack's switch from it and press  p  at the boot menu:
#    "provision the rack from this USB". Log in as live / live and check:
sudo kldload-netboot-server status        # payload: the live medium, armed: any

# 3. Only if the switch has NO DHCP server of its own (air-gapped): give the
#    master an address and let it hand them out
sudo ip addr add 10.99.0.1/24 dev enp3s0
printf 'NETBOOT_IFACE=enp3s0\nNETBOOT_DHCP=standalone\n' | sudo tee /etc/kldload/netboot.env
sudo systemctl restart kldload-netboot-live
  1. Network-boot each target. It shows the kldload menu and counts down. Left alone it boots its own disk, so nothing is erased by accident; any key opens the install, and the password and the disk are asked for on that machine's own screen.
  2. Or arm a whole rack unattended: one answers file per MAC in a directory, then sudo kldload-netboot-server arm-all ./rack/ (add --dry-run first to check every file). Each armed machine installs exactly its file.

Before you start: each target needs UEFI network boot and more RAM than the image — the root image is loaded into memory, and an 8 GB machine panics part way. The full procedure, the two network cases and the known issues in 1.5.0: docs/NETBOOT.md.

Installing with encryption

Full-disk ZFS encryption is off unless you ask for it — that changed in 1.5.0 — and asking for it is recommended. Three ways to ask:

  • Web UI: on the install page set ZFS encryption to Encrypted — AES-256-GCM, passphrase at boot and enter a passphrase. The selector starts on None.
  • Answers file: KLDLOAD_ZFS_ENCRYPT=1 and KLDLOAD_ZFS_PASSPHRASE=.... A netbooted or seeded machine whose file sets the flag but not the passphrase is asked for one on its own screen before anything is erased.
  • Netboot menu: press any key during the countdown, tick ZFS encryption under options, and type the passphrase on the console when asked.

The flow is then short:

  1. Download & burn the ISO to a USB stick (see Quickstart above).
  2. Boot the USB. The installer opens automatically in the browser at https://<host>:8443 — no login prompt.
  3. Choose your distribution, profile, and target disk, turn encryption on as above, set your disk encryption passphrase, and start the install.
  4. When it finishes the machine reboots. Remove the USB stick.
  5. At the ZFSBootMenu prompt, enter your encryption passphrase. With Secure Boot off the pool key is embedded in the initramfs, so that one prompt is the only one. With Secure Boot on it deliberately is not (the initramfs sits on the unencrypted ESP), so the initramfs asks a second time.
  6. The desktop loads and the console opens at https://<host>:8443 — no certificate warning, no login prompt. Done.

An unencrypted install asks for nothing at boot. Turning Secure Boot on or off does not change whether the pool is encrypted. TPM unlock is in 1.5.0 as kldload-tpm-seal plus a dracut module that unlocks the encrypted root from the TPM, bound to Secure Boot through PCR 7; it is inert until you seal it, and has not been tested on real hardware.

Secure Boot — opt in, and state it

What "default" means depends on how you install, and the code paths do not agree with each other, so say it explicitly every time:

  • Web UI: the Secure Boot card starts unticked and the page sends KLDLOAD_ENABLE_SECURE_BOOT=0. Off unless you tick it.
  • Netboot menu, a key pressed: the Secure Boot tick starts from the answers file; an absent key shows unticked and is sent as 0. Off unless you tick it.
  • Answers file with the key absent (a USB seed, --config, or a netboot left to its countdown): the boot-chain code treats absent as on — it generates a MOK, signs the modules, installs shim and makes GRUB's direct entry the default — while the install manifest records 0 and the machine reboots instead of powering off for the enrollment. Write KLDLOAD_ENABLE_SECURE_BOOT=0 or =1; never leave it out.

With it off the firmware boots ZFSBootMenu directly: no shim, no GRUB stage, no MOK enrollment, and nothing to miss at a ten-second prompt. That is the right default for a lab machine, and it is the path above.

Turn it on when the machine's threat model wants a verified boot chain, by ticking the card or setting KLDLOAD_ENABLE_SECURE_BOOT=1. The install then generates a per-install MOK, signs the out-of-tree modules with it, and boots firmware → shim → GRUB → kernel. ZFSBootMenu stays in GRUB's menu but is not the default there: shim 15.8's SBAT rule refuses to chainload it. That path needs three extra steps:

  1. When the install finishes it powers the machine off rather than rebooting — so you control the enrollment boot instead of racing an auto-reboot. (An unattended netboot or seed install reboots regardless.) Remove the USB stick.
  2. Power on and enter firmware setup (usually Del, F2, or F10). Enable Secure Boot, then save and exit.
  3. On the next boot the blue MokManager screen appears — it waits only ~10 seconds, so press any key immediately, then: Enroll MOK → Continue → Yes → password kldload → Reboot. The password is literally kldload — not your admin or encryption password.

Missed the MokManager screen? Just reboot — kldload re-offers enrollment on every boot until the key is actually enrolled. No reinstall. If you end up at a "Secure Boot validation failed" screen instead, the app grid's Secure Boot Repair tool (or sudo kldload-mok-repair from any terminal — including the live USB) diagnoses and queues the fix in one step.


Troubleshooting

Full install walkthrough, including what each Secure Boot failure looks like and how to decide whether to run with it on at all: docs/INSTALL.md.

Secure Boot / MOK — only if you enabled it

Secure Boot is off in a default web UI or netboot-menu install, and none of the below applies unless you turned it on — or left the key out of an answers file, which turns the boot-chain preparation on (see above). These are the failure modes of that path (KLDLOAD_ENABLE_SECURE_BOOT=1).

Symptom Fix
Missed the blue MokManager screen Reboot — enrollment is re-offered automatically. Or sudo kldload-mok-repair repair, then reboot.
"Secure Boot validation failed" / "Verification failed" at boot The install's MOK isn't enrolled. Run sudo kldload-mok-repair (installed system or live USB) — it shows whether the boot chain's key is enrolled and repair queues the fix; then reboot, press a key at the 10-second blue screen, Reset → Enroll, password kldload. Or temporarily disable Secure Boot in firmware to boot and repair from the OS.
Reinstalled several times / MOK operations start failing Stale keys accumulate in NVRAM (one per install). sudo kldload-mok-repair repair queues a Reset MOK list + enrollment of the current key in one pass.
Check enrollment / signing state sudo kldload-mok-repair (or kldload-mok-repair status, mokutil --list-enrolled).
NVIDIA or ZFS module won't load under SB Same cause — enroll the MOK. sudo kldload-mok-repair status shows the module signer.
Forgot the MOK password It's kldload (set a different one at install with KLDLOAD_MOK_PASSWORD).
Boots to emergency mode after skipping enrollment The MOK is not enrolled, so the kernel refuses the DKMS-signed ZFS module and non-root datasets never mount. Confirm with modprobe zfs — Key was rejected by service is conclusive. Fix: sudo kldload-mok-repair repair then reboot and enroll, or disable Secure Boot in firmware if this is a lab box. Note this can appear weeks later, at the first kernel update after the missed prompt.
Boots fine with SB off, fails with SB on (installed before 1.4.0-rc3) Older builds re-signed the staged kernel with the per-install MOK key, discarding the distro's own signature. Restore it: sudo cp /boot/vmlinuz-$(uname -r) /boot/efi/EFI/BOOT/vmlinuz then enable Secure Boot. Fixed at install time from 1.4.0-rc3 on.

Console certificate warning

Shouldn't happen on a fresh install — the console cert is issued by the kldload CA, which is trusted in the browser automatically. If a warning appears, re-import the CA root (clearing any stale entry first):

kldload-trust-cert                                              # re-import the CA root
# stubborn? drop stale entries first, then re-import:
certutil -d sql:"$HOME/.pki/nssdb" -D -n kldload-webui 2>/dev/null
certutil -d sql:"$HOME/.pki/nssdb" -D -n kldload-ca     2>/dev/null
kldload-trust-cert

Console asks for a password

Only remote browsers do — sign in with your admin account (a wheel/ sudo user). On the machine itself the console never prompts.


Two in the menu, two package managers underneath

Distribution Install method Offline
Fedora 44 dnf --installroot Yes (RPM darksite)
Debian 13 (Trixie) debootstrap Yes (APT darksite)

Those two are what the installer offers, because they are the two that get tested — and the two that deliver the offline promise. Everything else in the RPM and APT families is a network install, which is a different product from the one this README describes.

The method underneath is not distro-specific: kldload installs by calling dnf --installroot or debootstrap against a distribution's own repositories. Nothing is forked, nothing is patched, and no image is pre-baked for a given distro.

Which means the reachable set is wider than the tested one. RHEL, CentOS Stream, Rocky, Ubuntu and Arch all still work with KLDLOAD_DISTRO=<name> — the code paths are there and maintained — but they are not on the web UI's menu and not mirrored offline. RHEL is the partial exception: added to the netboot menu (NETBOOT_DISTROS) it is sent the 2.2 GB net image and installs from Red Hat's CDN with a subscription, and the 1.5.0 sweep installed a RHEL desktop that way — from the net image only, never offline. The rest are not tested this release, and the menu is the honest statement of what is. Adding one back is repository configuration, not new machinery.

Fedora and Debian are deliberately the widest useful pair rather than a narrow one: two package managers, two firmware-splitting conventions, two initramfs generators, and one leading-edge substrate against one stable one. Most bugs worth finding show up as a difference between them.

The honest bound: a new distribution needs its repos and keys declared, and its kernel paired with a version of OpenZFS that builds against it. That is a morning's work, not a port.

Live environment is Fedora 44 (kernel 7.0.x — currently 7.0.12 — with OpenZFS 2.4.3 on root).

Fedora 44 + ZFS: OpenZFS ships a native fc44 build (2.4.3), so there is no fc43 bridge. The kernel is not taken as whatever Fedora ships today: builder/kernel-pin.sh reads the ceiling OpenZFS itself declares (zfs-dkms Conflicts — 2.4.3 caps at kernel ≤ 7.0.999) and resolves the newest matching build, pulling kernel, -core, -modules, -devel and -headers as one set so they cannot be split. On the installed system that set plus NVIDIA is versionlocked at first boot, so a routine dnf update cannot pull a kernel ZFS has no build for.


The desktop profile

A GUI-first workstation that looks like stock RHEL 10: expert operations — ZFS replication, KVM, Kubernetes, eBPF observability — exposed as point-and-shoot desktop apps, not CLI rituals.

  • Install-time Platform Options. Checkboxes for NVIDIA drivers, KVM, Kubernetes, eBPF tooling, and golden-image building. Desktop-only, default-clean — you opt into the heavy stuff.
  • Native app windows. Each tool (VMs, Kubernetes, ZFS, Metrics, the model, …) opens as its own chromeless GTK/WebKit window — no browser chrome, no left menu — backed by the same web console the server edition serves.
  • Console as its own app. The tmux F-key operator cockpit (k9s, ZFS internals, eBPF panels, VM/log streams) is a single Console application — not embedded inside every tool window.
  • Local model. Ollama with Open WebUI (plus RAG and voice) as a desktop app. No cloud, no telemetry.

Profiles

Seven profiles, the seven the netboot menu offers. A profile is a starting point, not a cage: every one of them is an answers file, and any key it sets can be changed per machine.

Profile What you get
Core ZFS on root and nothing else. Stock distro, no k* tools, no web UI, no darksites — about 200 MB beyond the vendor's base install
Server Headless: SSH, ZFS root, the full k* tool suite, the web console, sanoid snapshots, WireGuard, eBPF tooling, offline darksites
Desktop GNOME on ZFS root, GPU drivers, Firefox, native app windows for every tool, the Console cockpit, Ollama, offline darksites
KVM host A hypervisor: libvirt + qemu-kvm, every VM on a ZFS zvol, ~100 ms copy-on-write clones, golden images, zfs send replication
Kubernetes KVM host plus a highly available cluster in VMs — 3 control planes and 3 workers, Cilium + Hubble + Tetragon
Storage A ZFS file server: NFS, SMB and iSCSI, shares as datasets, snapshots and replication underneath
AI KVM host plus Ollama and Open WebUI on the local GPU, RAG and voice, no cloud

KVM is on for every profile except Core. Two templates build on the KVM host rather than being profiles of their own:

Template What it adds
klab Golden VMs for every supported distro, blue/green sites from ZFS instant clones, fault injection, the distro matrix runner, a live Hubble traffic map
OpenZFS suite Dedicated test goldens wired into ztest/zloop for upstream OpenZFS regression hunting
kube-cluster bootstrap --workers 3   # golden image, then 3 control planes + 3 workers
kube-demo                            # PetClinic + ArgoCD smoke test
klab golden centos                   # build the CentOS golden VM
klab run-playbook ./my-test.sh       # run a script on every VM of the blue site

One machine, or the whole rack

The USB installs one machine. The same USB, booted with its provision the rack entry (p at the boot menu), is the network server for every other machine on the wire: it serves the kernel, initramfs and root image straight off the stick, nothing is installed or copied on the machine running it, and any box that network-boots gets the install menu on its own screen. A key there installs; left alone, the countdown boots that machine's own disk, so a stranger that happens to network-boot first loses nothing. The menu asks for the password, and the disk, on the target itself — nothing secret is ever on the wire. Fedora and Debian are offered from the USB (RHEL needs the separate net image).

An installed machine can be the permanent server instead — install it with KLDLOAD_KEEP_NETBOOT=1, systemctl enable --now kldload-netboot — and arm machines one at a time, which is how a rack is rebuilt from a directory of answers files:

kldload-netboot-server arm-install <mac> <answers>.env   # one machine
kldload-netboot-server arm-all     ./rack/               # every .env in a directory
kldload-netboot-server status                            # what is armed right now

A target needs nothing but a network port. It pulls the whole system — every package, the container images, the lot — from one image staged once on your LAN. Nothing else leaves the network: no package mirrors, no registries, no vendor endpoints. A rack can be built in a room with no uplink, which is the entire reason the darksite payload exists. The fiftieth machine costs the same to prepare as the second, because nothing is fetched per machine.

It cannot wipe a machine by accident. Installing requires a per-MAC consent token. A machine that network-boots without one prints that it is continuing the boot order and boots from its own disk. arm-all refuses duplicate MACs or hostnames outright and exits non-zero unless every machine armed, so a half-armed rack is an error you hear about rather than discover.

An armed machine shows a summary of what it was armed with and counts down (NETBOOT_MENU_TIMEOUT seconds, 10 unless set) before fetching anything large. Left alone, it installs exactly the armed answers file. Any key opens the manual override: pick a profile (core, server, desktop, kvm, k8s, storage, ai -- each the armed file with that profile's settings), a distro (from NETBOOT_DISTROS, default fedora; list only the ones you have verified -- Fedora and Debian are the offline, tested pair; RHEL and Arch are sent the net image instead (NETBOOT_NET_DISTROS), and Arch stays off the list: an encrypted Arch install panics at boot, and a rolling release cannot be version-locked the way the rest of the substrate is), then tick options -- ZFS encryption, Secure Boot, KVM, golden images, Kubernetes, the ZFS lab, AI -- and change the hostname, user, time zone or keyboard. Golden images, Kubernetes and the ZFS lab need KVM, so ticking one ticks it. Secrets never go through the menu: a login password or encryption passphrase the answers file lacks is asked for on the machine's own screen, before anything is erased. Encryption asks for the passphrase at every boot and Secure Boot asks for the key to be enrolled on the first reboot. Esc goes back, then boots the local disk, so a machine netbooted by mistake can be sent back to its own disk without it being wiped or pulling the image.

Per-machine differences live in the answers file — hostname, disk, profile, how many control planes and workers. Same image, different answers.

Boot on one NIC, download on a faster one. Many 10G cards carry no UEFI PXE code, so only the onboard port can start the boot. --netdev <mac> (or KLDLOAD_NETBOOT_NETDEV=<mac> in the answers file, which is how arm-all takes it) hands the multi-gigabyte image to the fast card instead. After the install, the host bridge goes on the fastest linked NIC and the others stay up as fallbacks.

Your own workloads land with the cluster, not after it. Drop a Helm chart in /root/darksite/helm-charts/workloads/ or plain YAML in /root/darksite/manifests/, and first boot installs them once the cluster is up and before it reports ready. Manifests apply in sorted order, so 10-namespace lands before 20-deploy. Add your container images to build/darksite/k8s-images.txt and they are baked in too — so your application is on every node with nothing pulled from a registry.

Ansible runs inverted here: ansible-playbook executes locally on each machine at first boot rather than being pushed from a control node. No inventory to keep current, no SSH fan-out, no credentials held centrally. Each machine builds itself, and a hundred do it at once without coordinating.

Measured end to end on one target: fifteen minutes from power-on to a six-node HA Kubernetes cluster with all nodes Ready, landing on a usable desktop, with the six node clones taking 0 bytes against one 2.32 GB golden image.

Step by step — make the master USB, the air-gapped case, arming a rack, and the known issues: docs/NETBOOT.md · Arming, the failure modes, and what to check →


What's wired into the image

  • OpenZFS on root — checksummed, compressed, snapshotted, self-healing on mirrors. lz4 default. Native AES-256-GCM encryption recommended, off unless selected; passphrase at ZFSBootMenu on every boot. TPM unlock (kldload-tpm-seal, bound to PCR 7) ships inert and untested on hardware; dedup optional.
  • ZFSBootMenu — UEFI bootloader that understands ZFS. Boot environments. Seconds-fast rollback. It is what the firmware boots with Secure Boot off; with Secure Boot on the chain is shim → GRUB → kernel, because shim's SBAT rule refuses to chainload ZFSBootMenu, which then stays as a GRUB menu entry.
  • WireGuard — kernel-level encrypted networking. One UDP port at the firewall.
  • eBPF observability — BCC tools + bpftrace + an F-key tmux cockpit on the host; Cilium + Hubble + Tetragon inside the K8s profile (no kube-proxy, no iptables, no sidecars).
  • KVM hypervisor — libvirt + qemu-kvm with every VM on a ZFS zvol. ~100 ms clones via COW. Atomic snapshots. fs-freeze app-consistency. Incremental zfs send replication.
  • Docker & podman on ZFS — every image layer is a real dataset, not a directory inside an overlay. A pull is a clone, layers inherit compression, and the whole container estate — layers, the engine's database and the volumes — snapshots and replicates as one recursive zfs send. Measured: a running container cloned and started in 328 ms with its state intact, and 24.7 GB of estate in a single stream. Docker on the apt distros, podman on the RPM ones. The full list of what this replaces →
  • NVIDIA + CUDA — drivers and CUDA optional at install. Time-sliced GPU sharing across the model and guest VMs. No PCIe passthrough required.
  • Ollama + Open WebUI — local model: RAG over the codebase + voice + tmux awareness + ReAct agent loop + eBPF-aware tool registry. No cloud, no telemetry.
  • Observability — Prometheus + Grafana + Loki + Alertmanager, Go + bash exporters, pre-wired dashboards, zed ZFS events bridged to Loki.
  • Secure Boot + MOK — per-machine key generation, automatic module signing, DKMS auto-sign on kernel upgrades. Off in the web UI and the netboot menu unless ticked; an answers file must state it.
  • Image export — kexport produces qcow2 / VMDK / VHD / OVA / raw, auto-sealed with cloud-init multi-datasource config. Ready for Packer or direct hypervisor import.
  • Offline + Air-gap — RPM and APT mirrors baked in. The USB is the deployment, the recovery, and the air gap.

What's inside — the open source it's made of

kldload invents almost nothing. It is an opinionated assembly of software you already know, installed from the vendors' own repos and wired together so the pieces actually meet. If you recognise a name below, that is the point — you already know how to operate it, and nothing here is a bespoke reimplementation you would have to learn.

Nothing is forked and nothing is patched. The tree carries zero .patch files and zero vendored third-party source; every component arrives from its upstream package repo, its official release artifact, or its own git remote.

Two shipped components are not open source, and it would be dishonest to bury them in a list like this: Google Chrome (the default desktop browser and the renderer for the kldload GUI apps — its open-source upstream Chromium is what Debian targets get) and the NVIDIA driver + CUDA, which is opt-in at install. Everything else below is open source under its own licence.

Storage & boot

Project What it does here
OpenZFS root filesystem, snapshots, clones, send/recv, native encryption
ZFSBootMenu (2.3.0) UEFI boot environments, rollback from the boot screen
sanoid / syncoid snapshot retention policy and replication
dracut, GRUB2, shim, mokutil, sbsigntools, pesign initramfs, UEFI boot chain, Secure Boot module signing
cryptsetup, LVM2, mdadm, e2fsprogs, xfsprogs, btrfs-progs non-ZFS storage the installer must still read

Virtualization

Project What it does here
libvirt + QEMU/KVM every VM, each backed by its own ZFS zvol
virt-install, qemu-img, qemu-guest-agent provisioning, image conversion, in-guest control
swtpm + edk2/OVMF emulated TPM 2.0 and UEFI firmware for guests
cloud-init first-boot configuration of golden-image clones

Kubernetes & networking

Project What it does here
Kubernetes 1.32 (kubeadm/kubelet/kubectl) the cluster itself
containerd container runtime
Cilium 1.16.5 + Hubble eBPF CNI, kube-proxy replacement, flow visibility
MetalLB 0.14.9 bare-metal LoadBalancer services
kube-vip 0.8.9 control-plane VIP for HA
OpenEBS ZFS LocalPV CSI storage on ZFS
local-path-provisioner (Rancher) fallback StorageClass where a node has no ZFS
Gateway API 1.2.1, metrics-server ingress API, resource metrics
Helm, k9s, Headlamp chart installs, terminal cluster UI, web cluster UI
WireGuard, nftables, NetworkManager, chrony encrypted backplane, firewall, networking, time
nginx one TLS reverse proxy on :8443 for every browser-facing service

Observability

Project What it does here
Prometheus + Alertmanager metrics and alerting
Grafana pre-wired dashboards
Loki + Promtail log aggregation, with ZFS zed events bridged in
node_exporter, ebpf_exporter, process-exporter, smartctl_exporter, zfs_exporter, libvirt-exporter the metric sources

eBPF & security

Project What it does here
BCC tools + bpftrace the F-key tracing cockpit (execsnoop, biosnoop, tcplife, …)
Tetragon runtime security observability
Secure Boot + MOK toolchain per-machine keys, DKMS auto-signing on kernel upgrade

Desktop

Project What it does here
GNOME — Shell, GDM, Nautilus, Terminal/Ptyxis, Control Center the workstation session (LightDM on Debian Trixie)
PipeWire + WirePlumber audio
Google Chrome the default browser on RPM desktops, from Google's own repo, and what the kldload GUI apps render in
Firefox also installed on RPM desktops, and the browser on the GhostBSD posture
Chromium the browser on Debian targets
NVIDIA driver + CUDA optional at install, via RPM Fusion akmod-nvidia
Steam optional, via Flathub (Fedora)
eza, bat, fd, ripgrep, zoxide, fzf, fastfetch, htop the modern CLI set, pre-wired into the shell
ttyd + tmux browser terminal, and the session everything attaches to

AI — Ollama and Open WebUI, entirely local

Project What it does here
Ollama the LLM runtime
Llama 3.1 / 3.2 & Qwen2.5 the models, chosen automatically by detected VRAM (incl. Llama 3.2-Vision, Qwen2.5-Coder)
ChromaDB + nomic-embed-text the RAG vector store and embeddings over your own docs
whisper.cpp speech to text (voice input)
Piper text to speech (voice output)

No cloud, no telemetry, no API key — the models and the index live on the machine.

Automation

Project What it does here
Ansible (ansible-core) golden-VM provisioning and the web UI's Ansible tab
Argo CD GitOps engine behind the demo app stack
osbuild-composer Red Hat's own toolchain, used to build the RHEL golden image

Rescue toolkit (on the live USB)

GParted, TestDisk/PhotoRec, ddrescue, fsarchiver, smartmontools, nvme-cli, p7zip, ntfs-3g/exfatprogs, fio, stress-ng, memtest86+ — the install USB doubles as the recovery USB.

How a distro gets built

The installer bootstraps each target with that distro's own tool — debootstrap for Debian and dnf --installroot for Fedora. The same two paths reach the rest of their families when driven directly, but only these two are on the menu and tested. Packages come from the vendors' own CDNs; the ISO itself is built with Red Hat's lorax, dracut, squashfs-tools and xorriso inside a Fedora 44 container.

The sister consoles

Three of the consoles are their own BSD-3 projects by the same author, with their own repos and release cadence. kldload builds each from its own upstream at ISO-build time and records the exact commit it shipped, so an installed system can say precisely what it is running:

Console Upstream Commit recorded in
zxplore — ZFS its own repo /etc/kldload/zxplore-commit
vmxplore — KVM its own repo /etc/kldload/vmxplore-commit
wgxplore — WireGuard this repo's wg/ /etc/kldload/wgxplore-commit

wgxplore now has its own repo, but the copy kldload builds and ships is the in-tree wg/ — folded in on 2026-08-10 as a read-only estate lens, which is why its recorded commit is kldload's HEAD rather than a separate upstream. The two have since diverged, with the in-tree copy carrying work the standalone repo does not, so treat wg/ as the source of what is on the ISO.

They run on any Linux or BSD box — kldload is their first-party distribution, not their owner.

Licences are each project's own; kldload ships them unmodified and adds no licence terms of its own to them. See License for kldload's.


CLI tools

Packages and rollback

apt, dnf and pacman are symlinked to kldload-pkg-wrapper at install, so these are the normal commands — a script, unattended-upgrades or the GUI updater get the same protection.

Command What it does
apt upgrade / dnf update Snapshots the root, then runs the real transaction
apt rollback Stages a return to the pre-transaction snapshot; reboot to apply
apt rollback list Shows which transactions you could go back to
apt rollback cancel Un-stages it — nothing has changed until you reboot
kldload-rollback The same machinery directly, with boot-environment control
kpkg Package operations with pre-install snapshots
kupgrade Guided upgrade with automatic rollback on failure

Rollback clones the snapshot into a new boot environment rather than running zfs rollback, which cannot touch a mounted root and would destroy every newer snapshot. Nothing is overwritten.

Host

Command What it does
kldload-overview Unified host status — ZFS, VMs, K8s, GPU, eBPF, services
kst System health dashboard
kldload-console tmux F-key cockpit with live eBPF panels

ZFS

Command What it does
ksnap Snapshot manager
kclone Clone datasets / zvols
kbe Boot environment manager
kdf ZFS-aware disk usage
kpkg Package manager with pre-install snapshots
kupgrade Safe upgrade with automatic rollback
krecovery Disaster recovery
kexport Export golden images (qcow2 / VMDK / VHD / OVA / raw)

KVM

Command What it does
kvm-create Create VM on a ZFS zvol
kvm-clone ZFS instant clone (~100 ms)
kvm-snap Snapshot a VM
kvm-list List all VMs
kvm-delete Destroy VM + zvol

Kubernetes

Command What it does
kube-cluster bootstrap Build the golden image and bring up the cluster: 3 control planes by default, --workers N (--cps 1 only for a host that cannot fit three)
kube-cluster destroy Tear it down (golden preserved)
kube-demo Deploy PetClinic + ArgoCD smoke test
kube-smoke-test Automated cluster verification

klab

Command What it does
klab golden <distro> Build / refresh a golden VM image
klab run-playbook <script> [blue|green] Run a script on every VM of a site (blue by default)
klab-vm-debug-bundle Auto-fires on test failure — OpenZFS-ready debug tarball

deploy.sh

Subcommand What it does
build Build the ISO (uses cached darksites)
full Rebuild the builder image + all darksites, then build the ISO
clean Remove build artifacts
burn [/dev/sdX] [--yes] Write the ISO to a USB device. Names the target; falls back to USB_DEVICE, then auto-detects a single removable drive. Confirms interactively (prints the device's model and size, and asks you to type the name back); --yes skips the prompt for scripts.
builder-image Rebuild the Fedora 44 builder container
smoke-build Static checks on the built ISO (size, freshness, content)
zfs-pin Derive the kernel pin from the newest OpenZFS release's declared Linux-Maximum and report drift against build-iso.sh (--check for CI, --json for scripts)
smoke-test <distro> <profile> Full install lifecycle in KVM, then smoke-test the installed target
build-debian-darksite Build / refresh the Debian APT offline mirror
build-ubuntu-darksite Ubuntu mirror — retired 2026-08, opt in with KLDLOAD_INCLUDE_UBUNTU_DARKSITE=1
build-fedora-darksite Build / refresh the RPM offline mirror
build-ollama-darksite Cache the Bob/Ollama model bundle
kvm-deploy / kvm-deploy-bob Deploy the ISO to local KVM via virt-install
proxmox-deploy Deploy to a remote Proxmox host via the qm API
deploy-all Build + deploy across the configured targets

Architecture

Live environment:  Fedora 44 (kernel 7.0.x, OpenZFS 2.4.3)
Builder:           Fedora 44 container (lorax + squashfs-tools + xorriso + dracut)
Bootstrap paths:   dnf --installroot  (Fedora)
                   debootstrap        (Debian)

Installer:         Python web UI + ~10 bash libraries (lib/) + backend/bin tools
Web UI:            single HTML file per edition + WebSocket install-log stream
Single-port TLS:   kldload-proxy fronts the web UI, Grafana, Prometheus, Headlamp,
                   Bob, k9s/ttyd, and the libvirt console on one URL (:8443) with one cert

The user picks the target distro at install time. After install the system runs upstream packages from the vendor's public repos. There is no kldload package repository and no kldload-specific runtime updates — dnf update / apt upgrade / pacman -Syu just work.


Releases

Current release: 1.5.0 (26 September 2026). 582 commits since 1.4.2, 455 files changed, 47,421 lines added. In operator terms:

  • One key provisions the rack. An installed machine can keep the netboot payload it was built from and serve the next one: proxyDHCP beside your existing DHCP, one answers file per MAC, arm-all on a directory.

  • A netboot menu. An armed machine shows what it is about to do and counts down; any key opens the override — profile, distro, encryption, Secure Boot, components, identity — with secrets typed on the console only.

  • Rebuild a node from its replica, restamped for the machine it lands on and proved by a canary boot rather than by the receive finishing.

  • The first boot explains itself. The install show holds the screen through the reboot and into first boot, with the real build log beside it.

  • Firecracker microVMs on the same zvols, jailed and unprivileged; and blue/green for whole clusters, not just workloads.

  • Two defaults changed. ZFS encryption is off unless asked for, and the shipped answer templates carry no default password. KVM is on for every profile but core.

  • GitHub release: v1.5.0

  • Full changelog: CHANGELOG.md

  • Release notes, with screenshots: kldload.com/releases/1.5.0.html

  • History back to 1.0: kldload.com/release-notes.html

Every release is tagged (git tag -l 'v*'), so checking out the tag gives the exact tree an ISO was built from. Cutting one: docs/RELEASING.md.

License

BSD-3-Clause. See LICENSE.

Popular repositories Loading

  1. kldload kldload Public

    4 distros, one USB, ZFS on root. Debian, Fedora, RHEL and Arch. Offline install, boot environments, WireGuard, eBPF. Free.

    Shell 39 6