Skip to content

tsilenzio/homelab

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

79 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Homelab

Infrastructure as code for a single node Proxmox VE homelab running on a 7950X3D with 256GB RAM.

Everything lives in LXC containers provisioned by Terraform. Secrets are managed by Infisical. Persistent state sits on ZFS datasets that outlive the containers they're mounted into.

What's Here

terraform/       Proxmox LXC definitions, ZFS datasets, firewall rules
pyinfra/         Service provisioning and configuration (pyinfra deploys)
scripts/         Operational scripts (LXC upgrades, IRQ affinity, etc.)

Stack

Role Tool
Hypervisor Proxmox VE 9.1
IaC Terraform (bpg/proxmox provider)
Config mgmt pyinfra
Secrets Infisical (self hosted)
Reverse proxy Traefik (via Dokploy), plus Caddy sidecar on Forgejo
DNS / ad blocking AdGuard Home
VPN Tailscale (subnet router)
Metrics VictoriaMetrics + vmagent
Logs Loki + Promtail
Dashboards Grafana
Alerting Alertmanager → ntfy + email
Git Forgejo (private) + GitHub (public mirror)
CI GitHub Actions (self hosted runner)
Deployments Dokploy
Game servers AMP (CubeCoders)

Hardware

Component Spec
CPU AMD Ryzen 9 7950X3D (16C/32T)
RAM 256GB DDR5 (4x64GB Crucial Pro)
Primary storage 4TB WD SN850X NVMe (ZFS, rpool)
Backup storage 4TB WD SN850X NVMe (ZFS, bpool: vzdump archives + syncoid replica of state datasets)
Motherboard MSI MEG X870E GODLIKE
Network Marvell 10GbE (internal) + Realtek 5GbE (public facing)
Router ASUS GT-BE98 Pro (WiFi 7, AiMesh)
UPS CyberPower CP1500PFCLCD

Containers

CTID Name Purpose
100 infisical-db Postgres for secret management
101 infisical Secret management (app + Redis)
110 adguard DNS + ad blocking
111 tailscale VPN subnet router
112 victoriametrics Metrics storage + alerting
113 grafana Dashboards + log aggregation
114 uptime-kuma External uptime monitoring
121 forgejo Private git server (public, own IP, Caddy sidecar)
122 ntfy Push notifications
123 woodpecker Self hosted CI for private repos
124 github-runner Self hosted CI for public GitHub repos
125 forgejo-runner Forgejo Actions runner for private repos
130 amp Game servers (public)
131 dokploy App deployment platform

All containers run Debian 13 (Trixie). Persistent state lives on per service ZFS datasets, not inside container rootfs.

Architecture

A few patterns shape the rest of the repo and aren't obvious from the setup steps.

Disposable LXC rootfs, persistent ZFS state. Long-lived data (databases, game worlds, configs) lives on bind-mounted ZFS datasets, not inside the container rootfs. Destroying a container and recreating it from Terraform is a non-event because the state survives. lifecycle { prevent_destroy = true } guards the categorically critical datasets. Validated end-to-end by a deliberate full destroy-and-rebuild that successfully restored a real Minecraft world onto fresh containers.

Per-container Terraform module. Each container's resources live behind a module boundary (terraform/infra/modules/container/) so the dependency graph is contained. Targeted destroys don't cascade across the fleet.

Hybrid external access per service. Dokploy's built-in Traefik serves internal *.home.tsilenz.io hostnames (wildcard Let's Encrypt via Cloudflare DNS-01) and Dokploy-deployed apps on the public IP. Vendor-native integrations where they make sense: AMP's nginx + Let's Encrypt for the game-server admin panel, a Caddy sidecar inside the Forgejo container for git.tsilenz.io HTTPS preserving real client IPs on SSH. Each choice has documented trade-offs.

Defense-in-depth fail2ban. Every LXC protects its own sshd via a shared base_packages helper. Service-specific jails layered on top (Forgejo web auth, AMP nginx 4xx floods, Traefik 4xx, Proxmox pvedaemon). Escalating bantime.increment globally: first offenses get short bans, repeat attackers escalate up to 24h.

Layered backups. Sanoid snapshots every 5 to 15 minutes on state datasets, syncoid incremental replication to a second physical NVMe, nightly vzdump archives. Three independent recovery paths.

Secrets via get_or_generate. On first deploy, strong passwords are generated and persisted to a gitignored pyinfra/secrets/homelab.json. Subsequent deploys reuse existing values. Never checked into git, never regenerated surprising a working service.

Setup

Prerequisites

  • A Proxmox VE 9.x host with ZFS, network bridges (vmbr0 internal, vmbr1 public), and SSH access from your workstation
  • Terraform >= 1.5
  • uv for the pyinfra Python project
  • SSH agent running with the key that's authorized on the Proxmox host

Phase 1: Provision infrastructure (Terraform)

Creates 12 LXC containers, ZFS datasets, firewall rules, and pushes your SSH key into each container so pyinfra can connect.

cd terraform/infra
cp terraform.tfvars.example terraform.tfvars
# Fill in proxmox_host, proxmox_password, public IPs
terraform init
terraform apply

terraform.tfvars is gitignored. It holds the Proxmox root password and your public IP block. Do not commit this file.

Phase 2: Provision services (pyinfra)

Installs and configures each service inside its container. Run by tier in order. Within a tier, services can be run in any order unless noted.

cd pyinfra
uv sync

Run everything at once:

uv run python provision.py

Or run tier-by-tier:

Tier 0 (common to every container):

uv run pyinfra inventories/homelab.py deploys/node_exporter.py

Tier 1 (foundation, sequential, db before app):

uv run pyinfra inventories/homelab.py deploys/infisical_db.py --limit 10.0.1.100
uv run pyinfra inventories/homelab.py deploys/infisical.py    --limit 10.0.1.101

After this, open http://10.0.1.101:8080 in a browser and complete Infisical's first-run wizard.

Tier 2 (core infrastructure):

uv run pyinfra inventories/homelab.py deploys/adguard.py          --limit 10.0.1.110
uv run pyinfra inventories/homelab.py deploys/tailscale.py        --limit 10.0.1.111
uv run pyinfra inventories/homelab.py deploys/victoriametrics.py  --limit 10.0.1.112
uv run pyinfra inventories/homelab.py deploys/grafana.py          --limit 10.0.1.113

Tier 3 (services and CI runners):

uv run pyinfra inventories/homelab.py deploys/forgejo.py          --limit 10.0.1.121
uv run pyinfra inventories/homelab.py deploys/ntfy.py             --limit 10.0.1.122
uv run pyinfra inventories/homelab.py deploys/woodpecker.py       --limit 10.0.1.123
uv run pyinfra inventories/homelab.py deploys/github_runner.py    --limit 10.0.1.124
uv run pyinfra inventories/homelab.py deploys/forgejo_runner.py   --limit 10.0.1.125

Tier 4 (applications):

AMP's vendor installer (getamp.sh) holds open a PTY that pyinfra can't reliably reap, so the AMP install is a three-step flow. Prep + gate first, then the vendor installer runs in your terminal via a Mac-side runner, then post-install finishes.

# Prep (expected to fail at the "Verify AMP installed" gate)
uv run pyinfra inventories/homelab.py deploys/amp.py              --limit 10.0.1.130

# Vendor installer, live output in your terminal, Ctrl+C when you've finished first-launch in the AMP UI
../scripts/install-amp.sh 10.0.1.130

# Post-install (gate passes, systemd units + fail2ban amp-nginx jail land)
uv run pyinfra inventories/homelab.py deploys/amp.py              --limit 10.0.1.130

# Dokploy (standard single deploy)
uv run pyinfra inventories/homelab.py deploys/dokploy.py          --limit 10.0.1.131

The amp, forgejo, and dokploy deploys read public IPs from shared-config.json, which is generated by terraform apply in Phase 1. They will fail with a clear error if you skip Phase 1 first.

Manual steps

A few services need interactive setup that can't be automated.

  • AdGuard Home first-run: visit http://10.0.1.110:3000, set admin credentials, configure listen interfaces. Don't activate on the router yet.
  • Tailscale auth + subnet route:
    ssh root@10.0.1.111 "tailscale up --advertise-routes=10.0.0.0/16 --accept-routes"
    Open the URL it prints to authenticate the node, then approve the 10.0.0.0/16 subnet route in the Tailscale admin console.
  • Grafana first login: visit http://10.0.1.113:3000, log in as admin / admin, change the password. VictoriaMetrics and Loki datasources are pre-provisioned.
  • AMP first-run: visit https://gaming.tsilenz.io (or http://10.0.1.130:8080 from LAN) to complete the AMP installer's first-launch wizard (license acceptance, license key entry, choose Standalone mode). The admin username, admin password, sysadmin password, and Let's Encrypt email all live in pyinfra/secrets/homelab.json (amp_admin_user, amp_admin_password, amp_sys_password, amp_letsencrypt_email) and are passed to the vendor installer via the install-amp.sh script. Create your first game instance from the AMP web UI after first-launch completes.
  • Dokploy first-run: visit http://10.0.1.131:3000 over Tailscale and complete the first-run wizard to create the admin user. The Postgres password and other secrets are managed entirely by the upstream installer inside the Docker swarm.

Notes

  • Container UIDs are remapped (container root = host UID 100000) on unprivileged LXCs. Mount point ownership is fixed automatically by Terraform after container creation.
  • Tailscale needs /dev/net/tun passthrough on the container, which Terraform handles via tailscale_tun.tf.
  • terraform apply generates a shared-config.json artifact at the project root via the hashicorp/local provider. This file holds non-sensitive infrastructure metadata (public IPs, gateway, proxmox host) that pyinfra deploys read at module load. It is gitignored, with terraform.tfvars as the source of truth.
  • The amp, dokploy, woodpecker, github-runner, and forgejo-runner LXCs are privileged with keyctl=1 because all of them run Docker (AMP for game instances, Dokploy for app deployment, and the runners for container-based CI actions). Docker inside an LXC needs access to kernel namespaces, cgroups, and overlay filesystems, which unprivileged containers restrict. Making it work unprivileged requires individually granting specific kernel capabilities and is fragile. The security tradeoff is acceptable for a single-operator homelab where Docker itself is already a trust boundary.
  • Forgejo is exposed externally on its own public IP (107.209.36.133 / git.tsilenz.io) with a Caddy sidecar for TLS. Git-over-SSH works at port 22 on that IP alongside HTTPS. AMP is exposed on its own public IP with vendor nginx (installed by getamp.sh with HTTPS=y) for the web UI and SFTP for the FileManagerPlugin (port range 2224-2250).
  • Fail2ban runs on every LXC and the Proxmox host with service-specific jails (sshd everywhere, plus caddy-web, forgejo, amp-nginx, and proxmox jails on the relevant hosts).
  • Sanoid snapshots critical datasets (amp/data, infisical-db/data, forgejo/data, forgejo/db) every 5-15 minutes. Syncoid replicates those snapshots to bpool every 5 minutes, and nightly vzdump archives to bpool/vzdump for container-level backups.
  • The Proxmox API token approach was attempted but bind mounts require root@pam direct authentication. The provider uses username/password instead.

About

Single-node Proxmox VE homelab as IaC: Terraform + pyinfra across a dozen+ LXC services, disposable rootfs with persistent ZFS state

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors