Homelab Kubernetes Cluster

Production-ready Kubernetes homelab infrastructure built with Talos Linux, Proxmox, GitOps (ArgoCD), Istio service mesh, and comprehensive observability stack. Features auto-scaling with Karpenter, zero-trust security, and full cloud-native tooling.

Component versions are defined in Terraform variables (terraform.tfvars, variables.tf) and ArgoCD application targetRevision / Helm values.

A comprehensive, production-grade Kubernetes homelab infrastructure built on Talos Linux and Proxmox. This repository provides a complete cloud-native platform featuring GitOps workflows, service mesh capabilities, full observability (LGTM stack), auto-scaling, and enterprise-grade security. Perfect for learning Kubernetes, running personal services, or building a production-like environment at home.

📑 Table of Contents

🏗️ Architecture Overview
🛠️ Technology Stack
📊 Services Dashboard
🚀 Deployment Categories
🛠️ Setup Instructions
🔑 Access Credentials
🔄 Maintenance & Upgrades
🌐 Network Configuration
🎯 Key Features
💡 Use Cases
📁 Repository Structure
🔧 Configuration Highlights
🤝 Contributing
📚 Documentation & Resources
🚨 Important Notes
🏷️ Topics & Tags

⚠️ Temporary Warning

Known Issue: When using Cilium with eBPF host routing (kubeproxyreplacement) and Istio ambient mode enabled, there is a known health probe issue that causes readiness and liveness probes to fail.

Fix: Apply the fix following the instructions at Istio Issue #57911. The fix involves patching the istio-cni-node DaemonSet to set HOST_PROBE_SNAT_IP environment variable to use status.hostIP:

spec:
  template:
    spec:
      containers:
      - name: install-cni
        env:
        - name: HOST_PROBE_SNAT_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.hostIP

🏗️ Architecture Overview

This homelab deploys a production-ready Kubernetes cluster using modern cloud-native technologies:

Infrastructure: Talos Linux on Proxmox VMs with Terraform for immutable Kubernetes deployment; Ubuntu bastion hosts for jump access
Cloud Provider: Proxmox integration with Cloud Controller Manager (CCM), Container Storage Interface (CSI), and Karpenter for automation
Networking: Cilium CNI with eBPF host routing (kubeProxyReplacement), BGP integration, hybrid load balancing with native acceleration, and dual-stack IPv4/IPv6 support
Auto-scaling: Karpenter with Proxmox provider for dynamic node provisioning and workload scaling
GitOps: ArgoCD for declarative application management and continuous delivery
Service Mesh: Istio Ambient Mode for zero-trust traffic management, security, and observability
Ingress: Custom domain (local.xuhuisun.com) with Let's Encrypt certificates via cert-manager
Storage: Multiple storage solutions (AIStor S3-compatible with MinKMS, NFS CSI, CloudNativePG PostgreSQL, Proxmox CSI)
Observability: Complete LGTM stack (Loki, Grafana, Tempo, Mimir) plus ELK stack (Elasticsearch, Kibana)
Authentication: Keycloak for identity and access management with OIDC integration
Automation: Renovate for dependency updates, KEDA for event-driven auto-scaling

🛠️ Technology Stack

Core Infrastructure

Kubernetes: Container orchestration platform
Talos Linux: Immutable, API-driven Linux distribution for Kubernetes
Proxmox VE: Virtualization platform and hypervisor
Terraform: Infrastructure as Code (IaC) tool

Networking & Service Mesh

Cilium: eBPF-based CNI plugin with advanced networking features
Istio: Service mesh with ambient mode (zero-trust networking)
BGP: Border Gateway Protocol for advanced routing

GitOps & CI/CD

ArgoCD: GitOps continuous delivery tool
Renovate: Automated dependency updates

Observability & Monitoring

LGTM Stack: Loki (logs), Grafana (visualization), Tempo (traces), Mimir (metrics)
ELK Stack: Elasticsearch, Logstash, Kibana
OpenTelemetry Kube Stack: Unified observability framework with automated instrumentation and metrics collection
Kiali: Service mesh observability

Storage Solutions

AIStor: S3-compatible object storage with MinKMS key management (MinIO-compatible)
CloudNativePG: PostgreSQL operator for Kubernetes
NFS CSI: Network File System support
Proxmox CSI: Native Proxmox block storage

Security & Authentication

Keycloak: Identity and Access Management (IAM)
cert-manager: Automated certificate lifecycle management
mTLS: Mutual TLS for service-to-service encryption

Auto-scaling & Orchestration

Karpenter: Dynamic node provisioning and scaling
KEDA: Event-driven autoscaling for workloads

📊 Services Dashboard

Access your services at Homepage Dashboard

🔐 Management & Security

Service	URL	Purpose
ArgoCD	https://argocd.local.xuhuisun.com	GitOps continuous delivery
Keycloak	https://keycloak.local.xuhuisun.com	Identity & access management
IAM (OIDC)	https://iam.local.xuhuisun.com	OIDC issuer / SSO (Keycloak)
Homepage	https://homepage.local.xuhuisun.com	Service dashboard
Registry Mirror (Harbor)	https://registry-mirror.local.xuhuisun.com	Pull-through cache for docker.io (Talos nodes use Harbor path-based proxy)

📈 Observability & Monitoring

Service	URL	Purpose
Grafana	https://grafana.local.xuhuisun.com	Metrics visualization
Mimir	https://mimir.local.xuhuisun.com	Long-term metrics storage
Kiali	https://kiali.local.xuhuisun.com	Istio service mesh console
Kibana	https://kibana.local.xuhuisun.com	Elasticsearch visualization

💾 Storage & Data

Service	URL	Purpose
AIStor Console	https://aistor-console.local.xuhuisun.com	Object storage management (MinKMS)
AIStor S3 API	https://aistor.local.xuhuisun.com	S3-compatible API endpoint

🤖 AI & Productivity

Service	URL	Purpose
Open-WebUI	https://open-webui.local.xuhuisun.com	AI interface (LLM frontend)
OpenClaw	https://openclaw-cluster-admin-agent.local.xuhuisun.com	AI agent Control UI (Telegram, Discord, in-cluster admin; openclaw-rocks/k8s-operator)
Immich	https://immich.local.xuhuisun.com	Photo management & backup
n8n	https://n8n.local.xuhuisun.com	Workflow automation

🏠 Infrastructure

Service	URL	Purpose
Proxmox Datacenter Manager	https://pdm.local.xuhuisun.com	Proxmox cluster management
Proxmox VE	https://pve2.local.xuhuisun.com	Hypervisor management
Proxmox Backup	https://pbs.local.xuhuisun.com	Backup management
Scrypted	https://scrypted.local.xuhuisun.com	Home automation

🚀 Deployment Categories

This homelab includes a comprehensive set of cloud-native applications organized by category:

API Gateway & Ingress

Istio Ingress Gateway: HTTP/HTTPS traffic routing with TLS termination and load balancing
Certificate Management: Automated Let's Encrypt certificates via cert-manager with Route53 DNS validation

Cloud Native Storage

AIStor: S3-compatible object storage with MinKMS key management for backups, artifacts, and data lake
CloudNativePG: PostgreSQL operator for Kubernetes-native database management
NFS CSI Driver: Network File System support for shared storage
Proxmox CSI: Native Proxmox block storage provisioner for persistent volumes

Continuous Integration/Delivery (CI/CD)

ArgoCD: GitOps platform for declarative Kubernetes deployments
Self-healing: Automated drift detection and correction with continuous sync
Renovate: Automated dependency and Helm chart updates

DNS & Network Management

External DNS: Automated DNS record management with Unifi integration
Istio Service Mesh: Advanced traffic management, security policies, and observability

Observability & Monitoring Stack

LGTM Stack: Complete observability with Loki (distributed logs), Grafana (visualization), Tempo (distributed traces), Mimir (distributed metrics)
OpenTelemetry Kube Stack: Unified observability framework with automated instrumentation and metrics collection - replaces Prometheus stack
Elastic Stack: Elasticsearch and Kibana for advanced log analytics and search
Kiali: Service mesh visualization and traffic flow analysis
Metrics Server: Kubernetes metrics API implementation for HPA and VPA

Security & Compliance

Cert-Manager: Automated certificate lifecycle management with Let's Encrypt integration
Keycloak: Enterprise identity and access management (IAM) with OIDC and OAuth2
OIDC Integration: Single sign-on (SSO) across all services
mTLS: Mutual TLS encryption for service-to-service communication via Istio

Service Mesh

Istio Ambient Mode: Zero-trust service mesh without sidecars using ztunnel for secure overlay
Istio Base: Core service mesh functionality and control plane
Istio CNI: Network plugin for pod-to-pod encryption and traffic policies
Istio Ztunnel: Secure overlay network for ambient mode with L4 encryption
Traffic Policies: Advanced routing, load balancing, circuit breaking, and retry logic

Streaming & Messaging

Strimzi Kafka: Apache Kafka operator for Kubernetes with high availability
OAuth Authentication: Secured Kafka access via Keycloak integration

Auto-scaling & Orchestration

KEDA: Event-driven autoscaling for workloads based on metrics, queues, and external events
Karpenter: Dynamic node provisioning and scaling with Proxmox integration

🛠️ Setup Instructions

This guide will help you deploy a production-ready Kubernetes cluster on Proxmox using Talos Linux and Terraform. The setup follows Infrastructure as Code best practices and uses GitOps for application management.

Prerequisites

Before starting, ensure you have:

A Proxmox VE cluster with appropriate storage and network configuration
Network access to your Proxmox cluster
Basic knowledge of Kubernetes, Terraform, and Linux

Install required tools:

# Install Terraform
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install terraform

# Install Talos CLI
curl -sL https://talos.dev/install | sh
sudo mv talosctl /usr/local/bin/

# Install kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Note: This setup requires a Proxmox VE cluster with appropriate storage and network configuration.

1. Initial Kubernetes Cluster Setup

Deploy Kubernetes using Talos Linux with Terraform on Proxmox:

# Navigate to terraform directory
cd terraform

# Initialize Terraform
terraform init

# Configure Proxmox credentials. This project uses Terraform Cloud for the backend (see terraform/terraform.tf).
# Set credentials either in the Terraform Cloud workspace or via a local file:
cat > credentials.auto.tfvars << EOF
virtual_environment_endpoint = "https://<pve-ip>:8006"
virtual_environment_api_token = "your-proxmox-api-token"
virtual_environment_ssh_username = "root"
EOF

# Review and customize terraform.tfvars for your environment
# Key parameters:
# - region: Proxmox cluster name
# - nodes: Proxmox nodes with storage and network config
# - controlplane: Control plane VM specifications
# - cluster_name: Kubernetes cluster name
# - cluster_endpoint: API server endpoint
# - kubernetes_version: Kubernetes version (e.g. v1.35.2)
# - release: Talos version (e.g. v1.12.6) or "latest" for auto-detect
# - registry_mirror_endpoint: Harbor pull-through URL for docker.io

# Plan the deployment
terraform plan

# Deploy the infrastructure (creates control plane VMs and bootstraps cluster)
terraform apply

# Get the talosconfig
terraform output -raw talosconfig > talosconfig

# Configure talosctl
export TALOSCONFIG="$(pwd)/talosconfig"

# Get the kubeconfig
terraform output -raw kubeconfig > kubeconfig

# Configure kubectl
export KUBECONFIG="$(pwd)/kubeconfig"

# Verify cluster is running
kubectl get nodes
kubectl get pods --all-namespaces

# kubeconfig and talosconfig are written under terraform/; keep them local (do not commit)

2. Configure Cluster Networking and Cloud Providers

Deploy Cilium CNI with BGP integration and Proxmox cloud integrations. Run all commands in this step from the terraform directory (paths like files/ are relative to it).

# Add Helm repositories
helm repo add cilium https://helm.cilium.io/

# Install Talos Cloud Controller Manager
# Note: CCM credentials are automatically configured via inline manifests
helm upgrade -i talos-cloud-controller-manager oci://ghcr.io/siderolabs/charts/talos-cloud-controller-manager --namespace kube-system --values files/talos-ccm.yaml

# Install Cilium CNI with advanced networking features
helm upgrade -i cilium cilium/cilium --namespace kube-system --values files/cilium.yaml

# Apply Cilium BGP Configuration (for routing and load balancer IP pools)
kubectl apply -f files/cilium-bgp.yaml

# Install Proxmox Cloud Controller Manager
# Note: CCM credentials are automatically configured via inline manifests
helm upgrade -i proxmox-cloud-controller-manager oci://ghcr.io/sergelogvinov/charts/proxmox-cloud-controller-manager --namespace kube-system --values files/proxmox-ccm.yaml

# Install Proxmox CSI Plugin (block storage provisioner)
# Note: CSI credentials are automatically configured via inline manifests
helm upgrade -i proxmox-csi-plugin oci://ghcr.io/sergelogvinov/charts/proxmox-csi-plugin --namespace kube-system --values files/proxmox-csi.yaml 

# Install Karpenter Provider Proxmox (dynamic node provisioning)
# Note: Karpenter credentials and worker template are configured via inline manifests
helm upgrade -i karpenter-provider-proxmox oci://ghcr.io/sergelogvinov/charts/karpenter-provider-proxmox --namespace kube-system --values files/proxmox-karpenter.yaml 

# Configure Karpenter NodePool for system and user workloads
kubectl apply -f files/karpenter-node.yaml

# Add custom priority class
kubectl apply -f files/priority-class.yaml

3. Bootstrap ArgoCD

From the repository root, deploy ArgoCD manually for the first time (it will self-manage afterwards):

helm repo add argo https://argoproj.github.io/argo-helm
helm upgrade --install argocd argo/argo-cd \
  --values=argocd/values.yaml \
  --namespace=argocd \
  --create-namespace

4. Required Secrets Setup

NFS CSI Driver Configuration

kubectl create secret generic nfs-mount-options \
  --from-literal mountOptions="nolock" \
  --namespace kube-system

Cert-Manager Route53 Credentials

 export aws_secret=<your_aws_secret_access_key>
kubectl create secret generic route53-credentials-secret \
  --from-literal="secret-access-key=$aws_secret" \
  --namespace cert-manager

External DNS Unifi Integration

 export api_key=<your_unifi_api_key>
kubectl create secret generic external-dns-unifi-secret \
  --from-literal="api-key=$api_key" \
  --namespace external-dns

Elasticsearch MinIO Snapshots

 export YOUR_ACCESS_KEY=<minio_access_key>
 export YOUR_SECRET_ACCESS_KEY=<minio_secret_key>
kubectl create secret generic snapshot-settings \
  --from-literal=s3.client.default.access_key=$YOUR_ACCESS_KEY \
  --from-literal=s3.client.default.secret_key=$YOUR_SECRET_ACCESS_KEY \
  --namespace elastic

OpenClaw agents

Create the required secrets in the openclaw namespace before deploying OpenClaw agents (e.g. via Argo CD). Agents reference these via spec.envFrom; without them, pods may fail to start or integrations (Discord, etc.) will not work.

Environment and integration tokens (e.g. Discord bot token, AI provider keys if not in env-vars). Create openclaw-env-vars:

export DISCORD_BOT_TOKEN=<discord_bot_token>
kubectl create secret generic openclaw-env-vars \
  --from-literal="DISCORD_BOT_TOKEN=$DISCORD_BOT_TOKEN" \
  --namespace openclaw

Add more --from-literal entries for other integrations or API keys as needed (e.g. ANTHROPIC_API_KEY, OPENAI_API_KEY). Ensure the openclaw namespace exists, or create it first: kubectl create namespace openclaw. Agent definitions are in openclaw/agents/ (one OpenClawInstance YAML per agent); Istio ingress is in openclaw/ingress/.

MinKMS Operator MinIO License

If you use the MinKMS operator (e.g. for AIStor) with a commercial MinIO license, create a secret from your license file so the operator can use it:

kubectl create secret generic minio-license -n minkms-operator --from-file minio.license

Ensure the minio.license file is in your current directory, or pass the full path: --from-file /path/to/minio.license. The namespace minkms-operator must exist (it is created when the minkms-operator Argo CD application is deployed).

AIStor Operator MinIO License

If you use the AIStor operator with a commercial MinIO license, create a secret from your license file in the aistor namespace:

kubectl create secret generic minio-license -n aistor --from-file minio.license

Ensure the minio.license file is in your current directory, or pass the full path: --from-file /path/to/minio.license. The namespace aistor must exist (it is created when the aistor-operator Argo CD application is deployed).

MinKMS Enclave for AIStor

Before deploying AIStor (e.g. the aistor-objectstore object store), create the MinKMS enclave so AIStor can use it for key management. Run this before deploying the AIStor operator or object store:

minkms add-enclave aistor-objectstore -k

The enclave name aistor-objectstore must match the object store name used in your AIStor configuration. The -k flag creates the necessary keys for the enclave. Ensure the minkms CLI is installed and you are authenticated to your MinKMS instance.

CNPG database backup to internal AIStor (MinIO)

CloudNativePG clusters that back up to the internal AIStor/MinIO endpoint (e.g. https://minio.primary-object-store.svc.cluster.local) need the cluster root CA so the Postgres pods can verify TLS. Create the minio-ca-bundle secret in each namespace where a CNPG cluster uses backups.endpointURL pointing at that internal host and has endpointCA.name: minio-ca-bundle in its values (e.g. Keycloak, or n8n if using the internal endpoint).

Example for the keycloak namespace (repeat for other namespaces that use internal AIStor for backups, e.g. n8n):

 export ca_crt=$(kubectl get cm kube-root-ca.crt -o jsonpath="{['data']['ca\\.crt']}")
kubectl create secret generic minio-ca-bundle --from-literal="ca-bundle.crt=$ca_crt" --namespace keycloak

Ensure the cluster’s backup config references this secret, for example in keycloak/cnpg-cluster-values.yaml:

backups:
  enabled: true
  endpointURL: https://minio.primary-object-store.svc.cluster.local
  endpointCA:
    create: false
    name: minio-ca-bundle
    key: ca-bundle.crt
  provider: s3
  s3:
    bucket: cloudnative-pg-backups
    # accessKey / secretKey from your object store credentials

Create the secret before or right after deploying the CNPG cluster so scheduled and manual backups can run without TLS verification errors.

Proxmox Cloud Provider Credentials

The Proxmox credentials for CCM, CSI, and Karpenter are automatically configured during cluster bootstrap via inline manifests in Talos. No manual secret creation is needed for Proxmox integrations.

5. Deploy All Applications

From the repository root, deploy the entire stack using GitOps:

kubectl apply -f deployment.yaml

This deploys the root ArgoCD Application that recursively syncs argocd/applications, installing all applications in the correct order via sync waves.

6. Apply Istio CNI Health Probe Fix (Required for Cilium with eBPF Host Routing)

If you're using Cilium with kubeProxyReplacement enabled and Istio ambient mode, you must apply the health probe fix to prevent probe failures:

# Patch the istio-cni-node DaemonSet to set HOST_PROBE_SNAT_IP
kubectl patch daemonset istio-cni-node -n istio-system --type='json' \
  -p='[{"op": "add", "path": "/spec/template/spec/containers/0/env/-", "value": {"name": "HOST_PROBE_SNAT_IP", "valueFrom": {"fieldRef": {"apiVersion": "v1", "fieldPath": "status.hostIP"}}}}]'

Alternatively, you can manually edit the DaemonSet:

kubectl edit daemonset istio-cni-node -n istio-system

Add the following environment variable to the install-cni container:

env:
- name: HOST_PROBE_SNAT_IP
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: status.hostIP

See the warning section at the top of this README for more details.

7. Apply Mimir Record/Alert Rules and Alertmanager Config

mimirtool rules list --address=https://mimir.local.xuhuisun.com --id=homelab-k8s

mimirtool rules load ./lgtm/mimir-control-plane-rules.yaml --address=https://mimir.local.xuhuisun.com --id=homelab-k8s
mimirtool rules load ./lgtm/mimir-kube-prometheus-rules.yaml --address=https://mimir.local.xuhuisun.com --id=homelab-k8s
mimirtool rules load ./lgtm/mimir-kube-state-metrics-rules.yaml --address=https://mimir.local.xuhuisun.com --id=homelab-k8s
mimirtool rules load ./lgtm/mimir-node-exporter-rules.yaml --address=https://mimir.local.xuhuisun.com --id=homelab-k8s

mimirtool rules load ./strimzi/kafka/strimzi-rule.yaml --address=https://mimir.local.xuhuisun.com --id=homelab-k8s

# Alertmanager config (routing, receivers, etc.)
mimirtool alertmanager load ./lgtm/mimir-alertmanager.yaml --address=https://mimir.local.xuhuisun.com --id=homelab-k8s

8. Registry Mirror (Harbor)

Harbor provides a pull-through cache at https://registry-mirror.local.xuhuisun.com. Talos machine config (control plane and worker templates) mirrors docker.io only via a path-based Harbor proxy (${registry_mirror_endpoint}/v2/docker.io with overridePath: true). Other registries (e.g. ghcr.io, registry.k8s.io) pull directly from upstream unless you add mirrors manually.

Change the mirror URL: Set registry_mirror_endpoint in terraform.tfvars or via -var, then re-apply Terraform and roll nodes or re-apply Talos machine config so nodes pick up the new endpoint.
Harbor proxy cache project: Create a Harbor proxy cache project named docker.io so path-based pulls match Talos config. You can add more proxy projects in Harbor for other registries, but they are not wired in Terraform templates by default. See Talos: Harbor as a caching registry and Harbor: Configure Proxy Cache.
Existing nodes: After changing Terraform/Talos config, apply the updated machine config to existing nodes (e.g. talosctl apply-config or roll nodes). New Karpenter nodes receive config from the inline template secret automatically.

🔑 Access Credentials

Talos Cluster Management

# Set talosconfig
export TALOSCONFIG="$(pwd)/terraform/talosconfig"

# Get cluster status
talosctl get nodes
talosctl get services
talosctl get pods

# View cluster configuration
talosctl get config
talosctl get machineconfig

# Restart services
talosctl restart kubelet
talosctl restart containerd

ArgoCD Admin Password

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d | xclip

Keycloak Admin Password

kubectl -n keycloak get secret keycloak \
  -o jsonpath="{.data.admin-password}" | base64 -d | xclip

Kiali Service Token

kubectl -n istio-system create token kiali-service-account | xclip

Elasticsearch/Kibana Password

kubectl -n elastic get secret elasticsearch-es-elastic-user \
  -o jsonpath="{.data.elastic}" | base64 --decode | xclip

AIStor / Object Storage Credentials

# AIStor (object store) credentials are configured in aistor/aistor-objectstore-values.yaml
# For legacy MinIO tenant, check minio/tenant-values.yaml for access key and secret

🔄 Maintenance & Upgrades

Kubernetes Cluster Upgrade

Upgrade Talos Linux and Kubernetes to the latest version:

Automatic Upgrade to Latest Version

Pin or bump versions in terraform/terraform.tfvars before you start:

release — Talos version (e.g. v1.12.6) or "latest" (auto-detect via terraform/talos-image-factory.tf)
kubernetes_version — target Kubernetes version (e.g. v1.35.2)

Steps:

Upgrade talosctl CLI to latest version:

# Upgrade talosctl to latest version
curl -sL https://talos.dev/install | sh

Upgrade Talos on control plane nodes (one node at a time, before terraform apply):

With release / kubernetes_version already set in terraform.tfvars, resolve the installer image URL (same schematic as terraform/talos-image-factory.tf):

cd terraform

# Option A: terraform output (refresh state first; does not recreate VMs)
terraform apply -refresh-only
terraform output -raw talos_installer_image

# Option B: terraform console (after terraform plan; no apply required)
terraform plan
terraform console
# > data.talos_image_factory_urls.talos_image.urls.installer

Upgrade each control plane node with that URL:

export TALOSCONFIG="$(pwd)/terraform/talosconfig"

IMAGE=$(terraform output -raw talos_installer_image)
talosctl upgrade --nodes <NODE_IP> --image "$IMAGE"

# Example with explicit image:
# talosctl upgrade --nodes 10.101.70.30 --image factory.talos.dev/nocloud-installer/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515:v1.12.6

You can also look up the image manually at Talos Image Factory. Wait for each node to rejoin before upgrading the next.

Review Terraform plan — confirm Proxmox ISO/template and inline manifest changes for the versions set in step 2:
```
cd terraform
terraform plan
```
Apply Terraform — refresh Proxmox Talos ISOs, worker/GPU templates, and Karpenter inline manifests:
```
terraform apply
```
Note: Step 4 updates Proxmox templates and secrets for future Karpenter nodes only. Existing workers stay on the old image until step 6.

Upgrade Kubernetes and resync inline manifests on a control plane node:

If kubernetes_version changed in step 2, upgrade the control plane to the new version. If only release changed, run upgrade-k8s with the current cluster version to push updated inline manifests (Karpenter template secret) after step 4.

export TALOSCONFIG="$(pwd)/terraform/talosconfig"

# Strip leading "v" from terraform.tfvars kubernetes_version (e.g. v1.35.2 -> 1.35.2)
K8S_VERSION=1.35.2

talosctl upgrade-k8s --nodes <NODE_IP> --to $K8S_VERSION --upgrade-kubelet=false

# Example:
# talosctl upgrade-k8s --nodes 10.101.70.30 --to 1.35.2 --upgrade-kubelet=false

Replace Karpenter worker nodes so new nodes pick up the Talos image and machine config from step 4:

# Option 1: Drift nodes (mark for replacement - Karpenter will gracefully replace them)
kubectl annotate node <NODE_NAME> karpenter.sh/do-not-consolidate=true
kubectl delete node <NODE_NAME> --grace-period=0

# Option 2: Directly delete old nodes (replace <NODE_NAME> with actual node name)
kubectl delete node <NODE_NAME>

# Karpenter will automatically provision new nodes with the updated Talos image and machine config
# Monitor node provisioning
kubectl get nodes -w

Verify cluster health:

# Verify Talos nodes
talosctl get nodes

# Verify Kubernetes nodes
kubectl get nodes
kubectl get pods --all-namespaces

# Verify Proxmox integrations are working
kubectl get pods -n kube-system | grep -E "(proxmox|karpenter|cilium)"

Proxmox Node Management

Worker nodes are automatically managed by Karpenter. Control plane nodes are managed by Terraform. To scale worker nodes:

System workloads: Karpenter automatically provisions based on Pod requirements
User workloads: Karpenter automatically provisions based on Pod requirements
Manual scaling: Edit NodePool limits in terraform/files/karpenter-node.yaml
Pool isolation: system and memory-optimized nodes are tainted (Talos CCM); workloads use nodeSelector/tolerations so only one pool can satisfy each pod. Underutilized consolidation runs 24/7, rate-limited by NodePool disruption budgets (20%, max 1 node) and workload PDBs (cert-manager, Argo CD, Istio, Strimzi, LGTM).

Application Updates

Automated: Renovate automatically creates PRs for Helm chart updates
Manual: Update targetRevision in ArgoCD application manifests
Self-healing: ArgoCD automatically syncs any configuration drift

🌐 Network Configuration

Domain Structure

Primary Domain: local.xuhuisun.com
Wildcard Certificate: *.local.xuhuisun.com (Let's Encrypt)
DNS Provider: Unifi with External DNS automation
TLS Termination: Istio Ingress Gateway
BGP Integration: Cilium BGP for LoadBalancer IP advertisement
Load Balancer IPs: Managed via CiliumLoadBalancerIPPool resources

Sync Wave Deployment Order

Wave 0: Cloud providers and CNI (Cilium, Proxmox CCM, Talos CCM, Proxmox CSI, Karpenter provider, priority classes)
Wave 10: Cilium BGP, NFS CSI driver, Karpenter NodePools, Metrics Server, ArgoCD ingress
Wave 20: Cert-Manager (certificate management)
Wave 21: Vertical Pod Autoscaler
Wave 30: Istio (service mesh) and OpenTelemetry Kube Stack
Wave 40: Harbor (container registry)
Wave 41: KEDA (event-driven autoscaling)
Wave 42: ArgoCD (GitOps)
Wave 50: External DNS (DNS automation)
Wave 60: Storage and observability operators (AIStor, CloudNativePG, ECK, Kiali)
Wave 70: Keycloak and LGTM stack
Wave 80: Strimzi (Kafka)
Wave 200: User applications (Homepage, Open-WebUI, OpenClaw, Immich, n8n)

🎯 Key Features

🔄 GitOps Workflow: Everything managed as code with ArgoCD - declarative, version-controlled infrastructure
🔒 Security First: mTLS, OAuth2/OIDC, automated certificate management with cert-manager and Let's Encrypt
📊 Full Observability: Complete metrics, logs, traces collection with LGTM stack, OpenTelemetry Kube Stack, and service mesh visibility via Kiali
🚀 Auto-scaling: KEDA for event-driven scaling and Karpenter for dynamic node provisioning
💾 Multiple Storage: Block storage (Proxmox CSI), object storage (AIStor S3 / MinKMS), file storage (NFS), and database storage (CloudNativePG)
🤖 AI Ready: Open-WebUI for LLM interactions and AI workloads
🏠 Home Integration: Proxmox virtualization, Scrypted home automation, and network infrastructure
⚡ Immutable Infrastructure: Talos Linux provides immutable, API-driven operating system with read-only root filesystem
🔧 Infrastructure as Code: Complete cluster lifecycle managed with Terraform - from VM creation to Kubernetes bootstrap
🛡️ Enhanced Security: Minimal attack surface with Talos Linux, zero-trust networking via Istio Ambient Mode
📦 Container-Optimized: Built specifically for Kubernetes workloads with optimized resource usage
🔄 Dynamic Scaling: Karpenter auto-scales nodes based on workload demand with Proxmox integration
🏷️ Node Pool Management: Separate pools for system and user workloads with resource isolation
🌐 Advanced Networking: Cilium eBPF datapath, BGP routing, dual-stack IPv4/IPv6, and hybrid load balancer mode with native acceleration
🔍 Service Mesh: Istio Ambient Mode for zero-trust security without sidecar overhead

💡 Use Cases

This Kubernetes homelab is perfect for:

Learning Kubernetes: Hands-on experience with production-grade Kubernetes tooling
Personal Cloud Services: Self-hosted alternatives to cloud services (photo management, AI tools, storage)
Development Environment: Production-like environment for testing and development
Home Automation: Centralized platform for smart home services and automation
DevOps Practice: Real-world experience with GitOps, service mesh, observability, and infrastructure automation
Cost Optimization: Run cloud-native workloads on-premises with enterprise features
Security Research: Zero-trust networking, mTLS, and advanced security configurations
CNCF Technology Exploration: Hands-on experience with CNCF projects (Istio, ArgoCD, Cilium, etc.)

📁 Repository Structure

├── terraform/                   # Talos Linux infrastructure as code
│   ├── files/                   # Kubernetes manifests and Helm values
│   │   ├── cilium.yaml         # Cilium CNI configuration (BGP, hybrid LB, dual-stack)
│   │   ├── cilium-bgp.yaml     # Cilium BGP routing configuration
│   │   ├── proxmox-ccm.yaml    # Proxmox Cloud Controller Manager
│   │   ├── proxmox-csi.yaml    # Proxmox CSI Plugin
│   │   ├── proxmox-karpenter.yaml # Karpenter Proxmox provider
│   │   ├── karpenter-node.yaml # Karpenter NodePool definitions
│   │   ├── priority-class.yaml # Kubernetes priority classes
│   │   └── talos-ccm.yaml      # Talos Cloud Controller Manager
│   ├── templates/              # Talos configuration templates
│   │   ├── controlplane.yaml.tmpl # Control plane configuration (Proxmox integration)
│   │   ├── metadata.yaml.tmpl  # VM metadata template
│   │   ├── worker.yaml.tmpl    # Worker node template for Karpenter
│   │   └── gpu-worker.yaml.tmpl # GPU worker node template
│   ├── terraform.tfvars        # Terraform variables (customize for your Proxmox)
│   ├── terraform.tf            # Terraform backend (Terraform Cloud) and required providers
│   ├── variables.tf            # Variable definitions
│   ├── outputs.tf              # Terraform outputs (kubeconfig, talosconfig, installer images, credentials)
│   ├── providers.tf            # Terraform provider configuration
│   ├── network.tf              # Network configuration and helpers
│   ├── talos-bootstrap.tf      # Talos cluster bootstrap and kubeconfig generation
│   ├── proxmox-kubenetes-token.tf # Proxmox API tokens for cloud providers
│   ├── proxmox-vm-cloud-image.tf  # Talos and Ubuntu cloud image downloads in Proxmox
│   ├── proxmox-vm-control-plane.tf # Control plane VM definitions
│   ├── proxmox-vm-worker-template.tf # Worker template for Karpenter
│   ├── proxmox-vm-gpu-worker-template.tf # GPU worker template
│   ├── proxmox-vm-bastion.tf   # Bastion host VM (Ubuntu cloud image)
│   ├── proxmox-pci-device.tf   # PCI device passthrough configuration
│   ├── proxmox-backup.tf     # Proxmox Backup Server jobs (bastion, control plane, templates)
│   ├── talos-image-factory.tf  # Talos image factory (version, schematics, URLs)
│   └── ubuntu-image-factory.tf # Ubuntu cloud image version locals (bastion)
├── argocd/                     # ArgoCD configuration and applications
│   ├── applications/           # Application definitions by category
│   │   ├── cloud-controller-manager/ # Proxmox & Talos CCM
│   │   ├── cloud-native-network/ # Cilium CNI and BGP
│   │   ├── cloud-native-storage/ # Storage solutions (AIStor, NFS CSI, Proxmox CSI)
│   │   ├── continuous-integration-delivery/ # ArgoCD
│   │   ├── container-registry/ # Harbor (pull-through cache)
│   │   ├── continuous-optimization/ # Karpenter, VPA, priority classes
│   │   ├── database/           # CloudNativePG operator
│   │   ├── dns/                # External DNS
│   │   ├── observability/      # LGTM, ECK, Kiali, Metrics Server (mertics-server.yaml), OpenTelemetry
│   │   ├── scheduling-orchestration/ # KEDA
│   │   ├── security-compliance/ # Cert-Manager, Keycloak
│   │   ├── service-mesh/       # Istio
│   │   ├── streaming-messaging/ # Strimzi Kafka
│   │   └── user-defined-apps/  # Homepage, Immich, n8n, Open-WebUI, OpenClaw
│   └── values.yaml            # ArgoCD Helm values
├── aistor/                    # AIStor object storage (MinKMS operator, AIStor operator, object store, ingress)
├── cert-manager/              # Certificate management
├── cloudnative-pg/            # CloudNativePG operator configuration
├── csi-driver-nfs/            # NFS CSI driver configuration
├── eck/                       # Elastic Cloud on Kubernetes
├── external-dns/              # External DNS configuration
├── harbor/                    # Harbor registry mirror (values, ArgoCD app, ingress)
├── homepage/                  # Service dashboard
├── immich/                    # Photo management application
├── istio/                     # Service mesh configuration
├── keda/                      # Event-driven autoscaling
├── keycloak/                  # Identity and access management
├── kiali/                     # Istio service mesh visualization
├── lgtm/                      # LGTM observability stack (Loki, Grafana, Tempo, Mimir)
├── metrics-server/            # Kubernetes metrics server
├── open-webui/                # AI interface application
├── openclaw/                  # OpenClaw AI agents: values.yaml, argocd/ (operator, agents, ingress), agents/ (one CR per agent), ingress/ (Istio VS)
├── opentelemetry-kube-stack/  # OpenTelemetry configuration
├── strimzi/                   # Kafka operator
├── deployment.yaml            # Root ArgoCD application (deploys all applications)
├── renovate.json             # Renovate configuration for automated dependency updates
└── */values.yaml              # Helm values for each service

🔧 Configuration Highlights

Cluster Architecture

Control Plane: 3-node control plane with stacked etcd on Proxmox VMs
Talos Linux: Immutable, API-driven operating system
Cloud Provider: Proxmox with CCM, CSI, and Karpenter integration
Auto-scaling: Karpenter with dynamic node provisioning from Proxmox templates
Dual Stack: IPv6/IPv4 support with native routing
CNI: Cilium with eBPF host routing (kubeProxyReplacement), BGP integration, hybrid load balancer mode (acceleration: native), and advanced BPF features (netkit datapath)
Networking: Native routing with BGP, load balancer IP pools, pod CIDR management, and eBPF datapath (netkit)

Storage Strategy

Proxmox CSI: Native Proxmox block storage provisioner
AIStor: S3-compatible object storage with MinKMS key management
CloudNativePG: PostgreSQL databases
NFS CSI: Network file system support

Security Features

mTLS: Service-to-service encryption via Istio Ambient Mode
Zero-Trust: Istio ambient mode provides security without sidecars
OIDC: Single sign-on with Keycloak
Certificate Automation: Let's Encrypt with Route53
RBAC: Role-based access control

Observability Stack

LGTM: Loki (distributed logs), Grafana (visualization), Tempo (distributed traces), Mimir (distributed metrics)
OpenTelemetry Kube Stack: Standardized telemetry collection with automated instrumentation, metrics collection, and export to LGTM stack (replaces Prometheus stack)
ELK: Elasticsearch, Kibana for advanced log analytics
Kiali: Service mesh visualization
Metrics Server: Kubernetes metrics API for HPA and VPA

🤝 Contributing

All changes should be made via pull requests
ArgoCD automatically syncs approved changes
Renovate handles dependency updates
Test changes in staging environment first

📚 Documentation & Resources

Official Documentation

Talos Linux Documentation - Immutable Kubernetes OS
Istio Service Mesh Guide - Service mesh documentation
ArgoCD User Guide - GitOps continuous delivery
Cilium Documentation - eBPF-based networking
Karpenter Documentation - Dynamic node provisioning
Proxmox VE Documentation - Virtualization platform

Learning Resources

CNCF Landscape - Cloud Native technology overview
Kubernetes Documentation - Official Kubernetes docs
Terraform Documentation - Infrastructure as Code

Related Projects

Talos Linux - Immutable Kubernetes OS
Istio - Service mesh
ArgoCD - GitOps tool
Cilium - eBPF networking

🚨 Important Notes

Talos Linux Benefits

Immutable OS: Read-only root filesystem prevents configuration drift
API-Driven: All configuration managed through gRPC API
Minimal Attack Surface: No SSH, package managers, or shell access
Atomic Updates: Rolling updates with automatic rollback on failure
Declarative Configuration: Infrastructure as code with Terraform
Proxmox Integration: Seamless VM lifecycle management via Terraform
Automated Bootstrap: Proxmox credentials automatically injected via inline manifests
Cloud-Ready: Built-in cloud controller integration for node management

Proxmox Integration

Automated Credentials: CCM, CSI, and Karpenter credentials are generated by Terraform
CCM Role: Proxmox CCM role includes VM.GuestAgent.Audit (with Sys.Audit and VM.Audit) for guest agent visibility
Secrets Management: All Proxmox API tokens stored securely via inline manifests
Dynamic Provisioning: Karpenter uses Proxmox worker and GPU worker templates for on-demand node scaling
Storage Integration: Proxmox CSI provides native block storage provisioning
Network Integration: Cilium BGP enables advanced routing with Proxmox infrastructure
Backups: Daily PBS snapshots cover bastion, control plane, and Karpenter VM templates (proxmox-backup.tf)

Bastion Host

Ubuntu cloud image: Version is centralized in terraform/ubuntu-image-factory.tf (default: Ubuntu 26.04 Resolute). Change ubuntu_release_version there and re-apply to roll bastion VMs.
Purpose: Jump host with QEMU guest agent; one bastion per Proxmox zone defined in bastion in terraform.tfvars.

Backup Strategy

Proxmox Backup Server: Daily snapshots of bastion, control plane, and Karpenter VM templates (terraform/proxmox-backup.tf)
AIStor / S3: Object versioning and lifecycle policies (S3-compatible)
CloudNativePG: Automated backups to object storage (e.g. AIStor S3)
Elasticsearch: Snapshot backups to object storage (e.g. AIStor S3)
Proxmox CSI: Native Proxmox storage snapshots

Monitoring Alerts

OpenTelemetry Kube Stack: Cluster and application metrics collection
Mimir: Long-term metrics storage and alerting
Grafana: Custom dashboards and alerting
Kiali: Service mesh health monitoring

Disaster Recovery

GitOps: All configuration in version control
ArgoCD: Self-healing and drift detection
Storage: Distributed and replicated storage
Backups: Automated backup strategies

🏷️ Topics & Tags

This project uses and demonstrates:

Kubernetes Ecosystem: kubernetes talos-linux proxmox terraform infrastructure-as-code gitops argocd

Service Mesh & Networking: istio cilium ebpf bgp service-mesh zero-trust mtls

Observability: grafana loki tempo mimir opentelemetry kiali elasticsearch kibana lgtm-stack

Storage: aistor minkms minio s3 cloudnative-pg postgresql nfs csi proxmox-csi

Security & Authentication: keycloak oidc oauth2 cert-manager lets-encrypt rbac

Auto-scaling: karpenter keda autoscaling dynamic-scaling

CI/CD & Automation: argocd gitops renovate continuous-delivery

Applications: immich open-webui openclaw homepage n8n kafka strimzi

Infrastructure: homelab self-hosted cloud-native cncf production-ready

Built with ❤️ using Cloud Native technologies

Name		Name	Last commit message	Last commit date
aistor		aistor
argocd		argocd
cert-manager		cert-manager
cloudnative-pg		cloudnative-pg
csi-driver-nfs		csi-driver-nfs
eck		eck
external-dns		external-dns
harbor		harbor
homepage		homepage
immich		immich
istio		istio
keda		keda
keycloak		keycloak
kiali		kiali
lgtm		lgtm
metrics-server		metrics-server
n8n		n8n
open-webui		open-webui
openclaw		openclaw
opentelemetry-kube-stack		opentelemetry-kube-stack
strimzi		strimzi
terraform		terraform
vertical-pod-autoscaler		vertical-pod-autoscaler
.gitmodules		.gitmodules
README.md		README.md
deployment.yaml		deployment.yaml
k8s-bgp-frr.conf		k8s-bgp-frr.conf
kcat.conf		kcat.conf
renovate.json		renovate.json

Latest commit

History

Folders and files

Repository files navigation

Homelab Kubernetes Cluster

📑 Table of Contents

⚠️ Temporary Warning

🏗️ Architecture Overview

🛠️ Technology Stack

Core Infrastructure

Networking & Service Mesh

GitOps & CI/CD

Observability & Monitoring

Storage Solutions

Security & Authentication

Auto-scaling & Orchestration

📊 Services Dashboard

🔐 Management & Security

📈 Observability & Monitoring

💾 Storage & Data

🤖 AI & Productivity

🏠 Infrastructure

🚀 Deployment Categories

API Gateway & Ingress

Cloud Native Storage

Continuous Integration/Delivery (CI/CD)

DNS & Network Management

Observability & Monitoring Stack

Security & Compliance

Service Mesh

Streaming & Messaging

Auto-scaling & Orchestration

🛠️ Setup Instructions

Prerequisites

1. Initial Kubernetes Cluster Setup

2. Configure Cluster Networking and Cloud Providers

3. Bootstrap ArgoCD

4. Required Secrets Setup

NFS CSI Driver Configuration

Cert-Manager Route53 Credentials

External DNS Unifi Integration

Elasticsearch MinIO Snapshots

OpenClaw agents

MinKMS Operator MinIO License

AIStor Operator MinIO License

MinKMS Enclave for AIStor

CNPG database backup to internal AIStor (MinIO)

Proxmox Cloud Provider Credentials

5. Deploy All Applications

6. Apply Istio CNI Health Probe Fix (Required for Cilium with eBPF Host Routing)

7. Apply Mimir Record/Alert Rules and Alertmanager Config

8. Registry Mirror (Harbor)

🔑 Access Credentials

Talos Cluster Management

ArgoCD Admin Password

Keycloak Admin Password

Kiali Service Token

Elasticsearch/Kibana Password

AIStor / Object Storage Credentials

🔄 Maintenance & Upgrades

Kubernetes Cluster Upgrade

Automatic Upgrade to Latest Version

Proxmox Node Management

Application Updates

🌐 Network Configuration

Domain Structure

Sync Wave Deployment Order

🎯 Key Features

💡 Use Cases

📁 Repository Structure

🔧 Configuration Highlights

Cluster Architecture

Storage Strategy

Security Features

Observability Stack

🤝 Contributing

📚 Documentation & Resources

Official Documentation

Learning Resources

Related Projects