Mora

My self hosted cloud, available at cocointhe.cloud.

mò|ra
s.f., "frutto del gelso o del rovo"
English translation: blackberry

Table of Contents

• Software
  • Requirements
  • OS
  • Creating the cluster
  • Deploying the stack
• Other bits and pieces
  • Public facing services through a VPS
  • SOPS setup
  • OIDC-based ssh access w/ opkssh
  • Backup strategy
    • restic setup
  • Traefik WAF + geoblocking

Hardware and Software

Here is a top view diagram of the main components:

The Kubernetes cluster is deployed with k3s.

The cluster state is handled by fluxcd, based on what's in this repo.

In k8s/ there are 3 main folders:

• flux, the entrypoint used by the flux controller for synchronizing the cluster. Main 'apps' are declared here. The interval for the source GitRepo is set to 1m, so changes will be picked up within a minute or so.

• infra represents what's needed for the cluster to function:

  • a Local Path Provisioner for handling persistent storage
  • an IngressController with Traefik, one private (listens on local lan), one "public" routed through a VPS,
  • cert-manager for certificates management of my domain with cert-manager-webhook-ovh
  • kube-vip for managing the cluster's VIP.
  • rathole for exposing part of my services to the outside world (see here),
  • tailscale-operator for accessing my private services from wherever (using a subnet route) and for my cluster services to access my offsite backup server
  • system-upgrade-controller for managing k8s upgrades directly in the cluster using CRDs.
  • a renovate cronjob to create PR for components updates (w/ auto merging when it's a patch level update and other rules)
  • a restic cronjob that create a local backup of the cluster data (if an app borks itself) and rsync it to a remote location (in case the server catches on fire)
  • kyverno for enforcing policies in the cluster,
  • goldilocks and vpa for automatic adjustments of limits/requests,
  • a vcluster where I do all my pre production testing, see here.
  • falco for eBPF based behavior detection
  • owasp/modsecurity-csr as a 'WAF' for traefik

• apps, the actual services running on the cluster: Some 'core' services:

  • adguard for DNS/DHCP
  • pocketid as an OIDC provider
  • atuin for a centralized shell history
  • uptime kuma as a simple availability check system
  • n8n for basic automation workflows
  • grafana + prometheus + loki for monitoring the cluster and workloads (metrics + logs)

  And more user-facing services:

  • grist for a modern take on spreadsheets
  • jellyfin for local VOD + music (using explo, musicgrabber and AudioMuse)
  • gitea for local git and CI/CD
  • kaneo for project tracking
  • paperless-ngx for my important files
  • immich for photos backups and sync
  • vaultwarden as my passwords manager
  • filebrowser for file sharing
  • pairdrop for Airdrop like across all devices
  • glance as my internet homepage (see news.cocointhe.cloud)
  • and some self-built stuff like:
    • a blog
    • a groceries list
    • a pushup tracker
    • a notifications hub using WebPush
    • custom fetchers for podcasts, exporters for my solar panel , various public sources..

I try to adhere to gitops/automation principles. Some things aren't automated but it's mainly toil (one-time-things during setup etc..). 99% of the infrastructure should be deployable by following these instructions (assuming data and encryption keys are known).

Requirements

• ansible: infrastructure automation
• flux: cluster state mgmt
• sops + age: encryption
• git: change management
• gitleaks: secret detection as a pre-commit hook
• vcluster cli: virtual staging cluster management

# install everything needed
brew install git ansible fluxcd/tap/flux sops age gitleaks opkssh vcluster

# tell git where to find its hooks
git config core.hooksPath .githooks

Creating the cluster

The bootstrapping is done using ansible. The playbook will simply install k3s on all the nodes.

It is assumed that a ssh key auth is configured on the nodes (ssh-copy-id ), with passwordless sudo (<user> ALL=(ALL) NOPASSWD: ALL in visudo). Otherwise install sshpass and add -k to the ansible command.

cd ansible
ansible-galaxy collection install ansible.posix
ansible-playbook -i inventory.yaml -l lampone cluster-install.yaml

Deploying the stack

1. Get a github token and set an env var:

   export GITHUB_TOKEN=xxx

2. Enter some commands

   # pre create the decryption key
   kubectl create ns flux-system
   kubectl create secret generic sops-age --namespace=flux-system --from-file=age.agekey
   
   # bootstrap flux
   flux bootstrap github \
                 --owner=k0rventen \
                 --repository=lampone \
                 --branch=main \
                 --path=./k8s/flux

3. From here, Flux will create everything that is declared in k8s/, decrypt what's secret using the private key, and keep the stack in sync.

Other bits and pieces

Public facing services through a VPS

I previously used Cloudflare Tunnels to expose some apps to WAN without opening ports on my ISP router (and also being behind CGNAT). Due to concerns regarding their ability to decrypt traffic at the edge (as they provision their own cert for the domain), I switched to rathole + a VPS. The 'server' side on the VPS forwards all the traffic from port 80/443 to the client running in my cluster, which then forwards everything to my dedicated traefik instance, which handles SSL and routing. A traefik is in front of rathole to properly pass the client's IP in the headers down the chain, and requires an https upgrade on port 80.

Here is the compose for the VPS:

services:
  traefik:
    image: traefik
    restart: always
    ports:
    - 80:80
    - 443:443
    volumes:
      - ./traefik.yaml:/etc/traefik/traefik.yml
  rathole:
    image: rapiz1/rathole:v0.5.0
    restart: always
    ports:
      - 2333:2333
    volumes:
      - ./server.toml:/app/config.toml
    command: --server /app/config.toml

the traefik.yaml looks like this:

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: "websecure"
          scheme: "https"
  websecure:
    address: ":443"

providers:
  file:
    directory: /etc/traefik/

tcp:
  routers:
    to-k8s-tcp:
      entryPoints:
        - websecure
      rule: "HostSNI(`*`)"
      tls:
        passthrough: true
      service: rathole-proxy-tcp

  services:
    rathole-proxy-tcp:
      loadBalancer:
        serversTransport: tcp-transport
        servers:
          - address: "rathole:443"

  serversTransports:
    tcp-transport:
      proxyProtocol:
        version: 2

And an example server config file for rathole:

[server]
bind_addr = "0.0.0.0:2333" # `2333` specifies the port that rathole listens for clients
default_token = "verylongtoken"

[server.services.traefik_https]
bind_addr = "0.0.0.0:443"

SOPS setup

This assume you have the decryption key age.agekey, and the env var configured:

SOPS_AGE_KEY_FILE=age.agekey

If you want to encrypt an already created file (eg a k8s Secret spec):

sops encrypt -i <file.yaml>

If you want to edit inline a encrypted file (eg modify a value in a encrypted Secret/Configmap) using $EDITOR:

sops <file.yaml>

OIDC-based ssh access w/ opkssh

Whenever possible, authentification is managed through my OIDC provider (pocketID). That's true for most of the services in the cluster, but also for accessing infrastructure-level stuff like my servers, using opkssh.

1. On the OIDC provider, create a new app w/ a Public Client ID (no client secret).

2. On the client, install opkssh and in .opk/config.yaml, add the provider w/ the public client id as per the doc.

3. On the servers, install using the script:

   wget -qO- "https://raw.githubusercontent.com/openpubkey/opkssh/main/scripts/install-linux.sh" | sudo bash

   Add a provider in /etc/opk/providers, then add a user to the policy with sudo opkssh add local_user oidc_email https://oidc-provider. This means that the user with oidc_email will be able to log in as local_user.

4. On the client, do a opkssh login then ssh should be seamless.

Backup strategy

I try to follow a 3-2-1 backup rule. The 'live' data is on the nfs ssd. It's backed up daily onto the same ssd (mainly for rollbacks and potential local re-deployments). For disaster-recovery situations, it's also backed up daily onto a HDD offsite, which can be accessed through my tailnet.

restic setup

The backup tool is restic. It's deployed as a cronjob in the cluster. It launches a simple script as such:

# First restic backup command
restic backup -r $RESTIC_REPO_PATH $CLUSTER_DATA

# cleanup old backups
restic -r $RESTIC_REPO_PATH forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12

# remote location backup using rsync incremental backup
rsync -avz --delete -e "ssh -i sshkey -o StrictHostKeyChecking=off" $RESTIC_REPO_PATH $RESTIC_REMOTE_REPO

To create the restic repo:

restic init $RESTIC_REPO_PATH

2. remote repo

Create a mnt-backup.mount systemd service on the remote server to mount/umount the backup disk

coco@remote_server:~ $ cat /etc/systemd/system/mnt-backup.mount
[Unit]
Description=Restic Backup External Disk mount

[Mount]
What=/dev/disk/by-label/backup
Where=/mnt/backup
Type=ext4
Options=defaults

[Install]
WantedBy=multi-user.target

Traefik WAF + geoblocking

Since one of my traefik is exposed publicly, it's always a good idea to add basic security hygiene. This is done using 2 plugins for traefik:

• geoblock, for whitelisting countries that are allowed to connect. By default the plugins makes a request to a public API, which is not ideal. I developed/deployed a small local service, which pulls a geoDB from IPinfo once a day, and the plugin hit it, benefiting from local resilience, and faster responses.
• modsecurity forwards requests to a owasp/modsecurity container, which detects bad behavior (SQLi, paths, ..) and stops generic attack through detection rules.