My Homelab

... Powered by 📦

... Running on k3s, Truenas

... Managed by ArgoCD (GitOps), Renovate

... And a little bit of ☁️ AWS, and GitHub Actions_ 🤖

Status

Setup

Today is automated via Jailmaker. Bootstrap script can be found in truenas/k3s-jail-config.

Dependency - bootstrapped "External Secrets" secret

Create an IAM user with the following policy attached:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Get",
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:*:<AWS account id>:secret:k8s*"
    },
    {
      "Sid": "List",
      "Effect": "Allow",
      "Action": "secretsmanager:ListSecrets",
      "Resource": "*"
    }
  ]
}

This gives access to secrets prefixed with k8s. Your secrets can now be stored in AWS Secrets Manager.

# To bootstrap, we add AWS credentials via one secret:
printf "%s" "Enter ACCESS_KEY: "
read ACCESS_KEY

printf "%s" "Enter SECRET_KEY: "
read -s SECRET_KEY

kubectl create ns external-secrets
kubectl create secret generic awssm-secret -n external-secrets \
  --from-literal=access-key=$ACCESS_KEY --from-literal=secret-access-key=$SECRET_KEY

Multiple services like Cert-Manager and DDNS rely on external secrets that are bootstrapped via the above.

Backup & Restore

Backups are managed by Kopia.

If a restore is needed, before Kubernetes cluster is live, use the below script/instructions:

# Requires awscli, kopia & jq to be installed on the host.
# 1. Pull creds straight from Secrets Manager
SECRET=$(aws secretsmanager get-secret-value \
  --secret-id k8s-homelab-secrets \
  --region eu-west-1 \
  --query SecretString --output text)

KOPIA_PASSWORD=$(echo "$SECRET" | jq -r '.kopia.KOPIA_PASSWORD')
ACCESS_KEY=$(echo "$SECRET" | jq -r '.kopia.KOPIA_BUCKET_ACCESS_KEY_ID')
SECRET_KEY=$(echo "$SECRET" | jq -r '.kopia.KOPIA_BUCKET_SECRET_ACCESS_KEY')
BUCKET=$(echo "$SECRET" | jq -r '.kopia.KOPIA_BUCKET')

# 2. Export env var to child-process
export KOPIA_PASSWORD

# 3. Connect to the repo
kopia repository connect s3 \
  --bucket="$BUCKET" \
  --prefix=kopia/ \
  --endpoint=s3.eu-west-1.amazonaws.com \
  --access-key="$ACCESS_KEY" \
  --secret-access-key="$SECRET_KEY"

# 4. List snapshots, then restore
kopia snapshot list --all
kopia restore <snapshot-id> /path/to/restore/target