kalmia

Kalmia is the Lentago Labs provisioning system for workstations, VMs, and containers. It has three layers, split by what they own:

• Ansible (repo root) — what's inside a machine's OS: idempotent, role-based workstation provisioning that turns a fresh Linux box into a configured dev workstation across five targets.
• Terraform (terraform/) — which Proxmox guests exist on the on-prem Proxmox cluster, the machines in our own building. Terraform owns every VM and LXC on the cluster (bar one released to claytonia), applied on merge from a LAN self-hosted runner.
• Forge (forge/) — the versioned machine images those guests are cut from.

graph TB
    subgraph github ["GitHub Actions (cloud)"]
        PR("pull request")
        PUSH("push to main")
        VALIDATE["validate / ubuntu-latest\nfmt · init · validate"]
    end

    subgraph aws ["AWS S3 (cloud)"]
        STATE[("state backend + lock table")]
    end

    subgraph runner ["LAN self-hosted runner (LXC 115)"]
        PLAN["plan — posts diff as PR comment"]
        APPLY["apply — serialised, no cancel-in-progress"]
    end

    subgraph pve ["homelab-cluster (Proxmox)"]
        GUESTS["every VM + LXC\nexcept claytonia's runner pool"]
    end

    FORGE["forge/ — substrate.sh\nbuilds versioned LXC templates"]
    ANSIBLE["Ansible — separate plane\nsite.yml / pub.yml / brasenia.yml / pe-publish.yml\nidempotent roles provision inside each guest"]

    PR --> VALIDATE
    VALIDATE -->|"same-repo PRs only — fork PRs\nnever reach the LAN runner"| PLAN
    PUSH --> APPLY
    PLAN <-->|"GitHub OIDC — no long-lived AWS keys"| STATE
    APPLY <-->|"GitHub OIDC"| STATE
    APPLY -->|"owns existence + shape"| GUESTS
    FORGE -.->|"images consumed by Terraform"| GUESTS
    GUESTS -.->|"Ansible self-provisions inside"| ANSIBLE

Loading

Terraform plane: merge to main is the deploy button. plan and apply run on the LAN self-hosted runner (LXC 115) because the Proxmox API is unreachable from GitHub-hosted runners; apply is serialised so two quick merges can't race on the same state. GitHub OIDC eliminates long-lived AWS keys for the S3 state backend. Fork PRs are blocked from the LAN runner by a same-repo guard in the workflow, reinforced by the repo's require-approval-for-outside-collaborators Actions setting. Ansible plane: independent — Terraform owns guest existence and shape; Ansible provisions inside them. Forge: substrate.sh produces versioned LXC templates that Terraform resources consume.

It supersedes the shell scripts in workstation-bootstrap: same toolchain, prompt, and workflow, expressed as Ansible roles instead of ~1,000-line bash scripts.

Authorship: The Ansible code and documentation in this repo are co-written with Claude (Anthropic). I direct the work and review the output; Claude writes the code. I'm an infrastructure operator, not a software engineer — please don't read this repo as a portfolio of coding ability.

🧭 What this repo demonstrates

kalmia proves the same PR-gated, apply-on-merge model works for real infrastructure — Proxmox VMs/LXCs and base OS images — that config repos use for files. The patterns an IT-ops reader can lift wholesale:

Pattern	How it shows up here
Apply-on-merge Terraform via a LAN self-hosted runner — merge is the deploy button, even for infra GitHub-hosted runners can't reach	terraform.yml: plan posts the diff as a PR comment; apply runs on push to main from a LAN runner (LXC 115) because the Proxmox API is LAN-only
Deadlock-safe required check via a synthetic gate job — a required check whose workflow is path-filtered hangs PRs forever ("Expected" never resolves)	The gate job runs unconditionally and is the only context the ruleset requires; heavy jobs skip on docs-only PRs (skipped → green). See lentago/.github#27
OIDC for the Terraform state backend — no long-lived AWS keys in secrets	terraform.yml assumes role/kalmia-github-actions-terraform with id-token: write; least-privilege to this repo's S3 state key + lock table only
Operational safety rails codified in IaC — irreplaceable guests can't be destroyed by a bad plan	lifecycle { prevent_destroy = true } on every guest the API-token pipeline can't recreate — HAOS (100), imported VMs, n8n/pub/grafana/k3s LXCs; classification in terraform/README.md
Two-layer image build pipeline (forge) — versioned base images with a clear ownership boundary	forge/: kalmia owns the OS substrate (runner/substrate.sh); the software layer is referenced from claytonia, not copied — script + pct capture over Packer, rationale documented
Shared, org-wide reusable workflows — one fix propagates fleet-wide	docs-check.yml and shellcheck.yml are thin wrappers over lentago/shared-workflows/.github/workflows/*.yml@main
Branch protection as policy, not habit — change can't land without green CI, merge method constrained	The main ruleset requires squash-only merges + four checks (gate, ansible-lint, shellcheck, docs-check); see Workflow
Idempotent, role-based provisioning across heterogeneous targets — the everyday config-management pattern	Five profiles share one role set; ansible-lint.yml runs --syntax-check + ansible-lint on every PR

Architecture decisions: The design choices behind these patterns — tool selection, profile routing, Terraform provider, apply-on-merge, capacity ownership, image forge tooling — are recorded in docs/adr/.

🛠️ Make a change yourself

These systems are real, and nothing critical rides on them. That makes this a safe place to try a change before you make the same kind of change in your own shop. Pick one:

Stand up a purpose-built container via Terraform. Add a proxmox_virtual_environment_container resource to terraform/containers.tf (and, if it needs configuring, a matching Ansible role/playbook), then open a PR. plan runs on the LAN runner and posts the diff as a PR comment; once you merge to main, apply runs automatically and the guest exists on the cluster — no manual pct/qm commands. Proof this works: #59 — lunaria: wall-display compositor LXC (terraform guest + provisioning role).

Build and ship a purpose-built machine image (forge). Add or edit a forge recipe under forge/<class>/ — the OS-level substrate.sh — document the image contract in its README, and run build.sh to produce a versioned LXC template that Terraform consumes elsewhere. The PR captures both the recipe and the rationale. (Forge ships exactly one image class today — runner, for claytonia; the other classes in the forge README are roadmap, not yet built.) Proof this works: #42 — Image forge + first artifact: the claytonia-runner LXC template and #43 — Docs: the runner image is consumed, not a pending follow-up.

Harden the CI pipeline against a gating deadlock. Edit .github/workflows/terraform.yml to add an unconditional gate job so the required status check always reports, then point the branch ruleset at gate instead of the path-filtered jobs directly. The merge activates the new gate immediately as policy. (Editing the ruleset needs org-admin permission; the workflow edit itself is an ordinary PR.) Proof this works: #41 — Always run terraform CI on PRs, gate merges on a gate check.

Move capacity ownership across repos cleanly. Remove a resource block from kalmia's terraform/ in one PR, coordinated with the receiving repo importing the same guests into its own state in a paired PR. Apply-on-merge in each repo takes over exactly the resources it now owns — no live guest is ever recreated. Proof this works: #40 — Release the bullpen runner pool from the guest layer — capacity moves to claytonia.

---

The rest of this README documents the Ansible workstation layer — the oldest and most-validated part of kalmia.

Why Ansible

The bash scripts worked, but every "is it already installed?" guard, the marker-bounded .bashrc block, and the ((count++)) || true traps were hand-rolled idempotency. Ansible makes idempotency the module contract, models the "80% shared / 20% per-platform" split as roles + variables, and self-provisions a fresh box with one command.

Targets (profiles)

Profile	Target	Package manager
xubuntu	Xubuntu 24.04 (Proxmox VM)	apt
ubuntu_laptop	Ubuntu Desktop LTS (bare-metal laptop)	apt
crostini	Chromebook Crostini, legacy LXC container	apt
baguette	Chromebook ChromeOS M147+, containerless VM	apt
fedora	Fedora KDE (Proxmox VM)	dnf

Both Chromebook profiles keep the penguin hostname, so autodetection splits them on the LXC guest facts: legacy Crostini is an LXD container (Docker CLI only), while Baguette is a crosvm KVM guest with systemd, cgroups v2, and /dev/kvm — enough to run the full Docker Engine. ChromeOS does not auto-upgrade existing installs, so both targets coexist.

Quick start

What you're about to do: run one command against a fresh Linux box and get back a fully configured dev workstation — Docker, cloud tooling, your shell, editors, the works.

Why bother: it's the same toolchain every time, on any of the five target machine types above, without babysitting a thousand-line shell script by hand.

Time: no fixed number — it's mostly unattended package installs and downloads, so it depends on your network and what's already on the box. You don't have to sit and watch it.

Heads up. The one-liner needs curl, and the clone path needs git. Both are present on a typical desktop install, but a minimal image may ship neither — sudo apt install -y curl (Debian/Ubuntu) or sudo dnf install -y curl (Fedora) first if so. bootstrap.sh installs everything else.

Self-provision a fresh box (installs Ansible, then runs the play):

curl -fsSL https://raw.githubusercontent.com/lentago/kalmia/main/bootstrap.sh \
  | WORKSTATION_PROFILE=xubuntu bash

Or from a clone:

git clone https://github.com/lentago/kalmia.git
cd kalmia
WORKSTATION_PROFILE=xubuntu ./bootstrap.sh
# …or, with Ansible already installed:
ansible-galaxy collection install -r requirements.yml
ansible-playbook site.yml -e workstation_profile=xubuntu

The profile autodetects from host facts when unset. Run a subset with tags, e.g. ansible-playbook site.yml --tags cli,shell.

How you know it worked: the run ends with a play recap showing failed=0. Open a new shell (or source ~/.bashrc) and the tools from the Roles table below — docker, code, the CLI utilities — should all be on your PATH.

How it works

• Self-provisioning against localhost (connection: local).
• Profile + facts model: workstation_profile selects profiles/<name>.yml (feature toggles like docker_daemon, enable_tlp); ansible_os_family selects vars/<family>.yml (package-name differences such as batcat↔bat). The toggles exist because the split isn't purely by distro — Crostini is Debian but daemonless; the Ubuntu laptop is Debian but the lone TLP power-management target.
• Privilege model: user context by default; system installs escalate with become: true; user config (nvm, .bashrc, Starship, VS Code settings) runs as you.

Roles

Role	Installs / configures
common	base packages, base services, global git config
languages	Python, Node.js (nvm), Go
cloud_tools	AWS CLI v2, Granted, Terraform (tfswitch), kubectl, eksctl, Helm
containers	Docker Engine + Compose (daemon optional)
power	TLP power management, ThinkPad charge thresholds, fwupd (opt-in)
editors	VS Code (+ extensions), Claude Code
cli_tools	jq, yq, bat, fd, ripgrep, fzf, tmux, direnv, Starship, …
shell	marker-managed .bashrc block + Starship config + the pull-all / clean-all / gh-issue / standing helpers
repos	GitHub CLI + clone your personal and org repos into ~/repos/<owner>/<name> (on by default)

Status

xubuntu and fedora are validated end-to-end on clean Xubuntu 26.04 and Fedora 44 VMs: a pristine box provisions with no failures and re-runs idempotently, with the profile autodetected. Fedora installs the shared toolchain plus Docker + VS Code from their dnf repos.

crostini is validated end-to-end on a Chromebook running Debian 13 trixie: same bar as the VMs. It runs Docker CLI-only (no daemon in the container), with hostname-resolution and ~/.local/bin fixes from the common role.

baguette is validated end-to-end on a Chromebook running the containerless VM under Debian 13 trixie: same bar as the others. ChromeOS shipped the containerless VM in M143 behind chrome://flags#containerless-crostini, then made it the default for new Linux installs in M147 — which invalidates the crostini profile's daemonless premise on upgraded Chromebooks. The hostname still matches, so the box would silently get a Docker CLI with no daemon behind it. This profile runs the full engine instead, and the first pristine run confirmed it: the profile autodetected off the LXC guest facts and Docker Engine came up enabled and active rather than CLI-only.

ubuntu_laptop is proven idempotent on a provisioned host — a pristine first-run is still unproven (#14). It was run on a ThinkPad T14 Gen 2i (Ubuntu 26.04, profile autodetected), converging from a workstation-bootstrap-provisioned state rather than a clean image, not from a pristine snapshot like the other profiles. Re-verified against merged main, a live run reaches ok=61 changed=0 failed=0, and --check completes (ok=60 changed=2 failed=0; the two changes are non-destructive module artifacts tracked in #80). All nine roles ran, including power (TLP, ThinkPad charge thresholds, fwupd) — exercised for the first time on any profile.

Beyond workstations

The first non-workstation target: pub.yml + the pub role codify the Morning Brief publisher (rclone + systemd timer) running inside the pub LXC (114). It's a standalone playbook, not a site.yml profile — see docs/pub.md for how to run it and seed its secret.

The second: brasenia.yml + the brasenia role codify the wall display's runtime (LXC 118) for brasenia (named lunaria until its runtime rename, #63; see docs/brasenia.md): the viewport's panes, rendered ahead by a sandboxed Chromium → ffmpeg 4K HEVC (H.265 video) → HLS (HTTP Live Streaming, short segments over plain HTTP) for the office Roku. Credential-free by design; the guest itself is defined in terraform/containers.tf.

The third targets the hypervisors themselves: pve-host.yml + the pve_host role set lvmthin autoextend and a nightly LXC fstrim on every Proxmox node, and install the mount hook that lets the Terraform pipeline attach allowlisted bind mounts (#190). Unlike the others it is pushed over SSH, since PVE nodes carry no Ansible — see docs/pve-host.md.

The fourth: pe-publish.yml + the pe_publish role provision LXC 123, the self-hosted Actions runner that publishes cpitzi/professional-endeavors to pub.lan on every merge, the first guest to get its NAS bind mounts from the pipeline — see docs/pe-publish.md.

The Terraform layer that declares these guests carries its own naming-debt footnote: kalmia's state backend migrated to solidago-tfstate-* keys (2026-07-08), so this repo's own state sits under a solidago prefix — an artifact worth knowing about, not worth churning to fix.

---

🌱 Lentago Labs is a pro-bono operations practice for organizations that run on volunteers, donations, and one overworked tech person. Everything here is free to take, and we practice what we publish: our own estate runs this way, in the open. Start at the org profile.

Part of the Lentago Labs portfolio.