diff --git a/.github/workflows/version-tag.yml b/.github/workflows/version-tag.yml index 7f807cca..216ebe35 100644 --- a/.github/workflows/version-tag.yml +++ b/.github/workflows/version-tag.yml @@ -86,6 +86,27 @@ jobs: echo "No proxy-domain changes detected." fi + # --- chutes-cvm domain --- + # src/chutes-cvm/* → src/chutes-cvm/VERSION + needs_cvm_bump=false + while IFS= read -r f; do + [ -z "$f" ] && continue + case "$f" in + src/chutes-cvm/*) needs_cvm_bump=true; break ;; + esac + done <<< "$changed_files" + + if [ "$needs_cvm_bump" = true ]; then + if ! echo "$changed_files" | grep -qx "src/chutes-cvm/VERSION"; then + echo "::error::Changes under src/chutes-cvm/ require an src/chutes-cvm/VERSION bump." + errors=$((errors + 1)) + else + echo "chutes-cvm domain: src/chutes-cvm/VERSION bumped." + fi + else + echo "No chutes-cvm-domain changes detected." + fi + if [ "$errors" -gt 0 ]; then exit 1 fi @@ -178,3 +199,4 @@ jobs: tag_if_ready "src/sek8s/VERSION" "sek8s-v" "changelogs/sek8s" tag_if_ready "src/sek8s-common/VERSION" "sek8s-common-v" "" tag_if_ready "src/attestation-proxy/VERSION" "attestation-proxy-v" "changelogs/attestation-proxy" + tag_if_ready "src/chutes-cvm/VERSION" "chutes-cvm-v" "changelogs/chutes-cvm" diff --git a/AGENT.md b/AGENT.md index 46698948..400208a9 100644 --- a/AGENT.md +++ b/AGENT.md @@ -24,8 +24,8 @@ Do not introduce alternate frameworks (e.g., Prisma, NextAuth, Firebase). Stay w - **Never install a new dependency** without discussion first - **Never modify database schemas** without showing the migration plan (sek8s has no DB; this applies if one is added) -- **Python services**: Poetry packages under `src/sek8s/` (import name `sek8s`), `src/sek8s-common/` (`sek8s_common`), and `src/attestation-proxy/` (`attestation_proxy`); tests under `tests/` -- **Shell scripts** in `host-tools/scripts/` and `guest-tools/` +- **Python services**: Poetry packages under `src/sek8s/` (import name `sek8s`), `src/sek8s-common/` (`sek8s_common`), `src/attestation-proxy/` (`attestation_proxy`), and `src/chutes-cvm/` (`chutes_cvm`, the host CLI/toolkit); tests under `tests/` +- **Shell scripts** in `host-tools/scripts/`, `guest-tools/`, and the bundled `src/chutes-cvm/chutes_cvm/scripts/` - **Ansible roles** in `ansible/guest/roles/` - **OPA policies** in `ansible/guest/roles/admission-controller/files/policies/` - **Environment variables** go in config files (pydantic-settings, Ansible vars) — never hardcoded @@ -35,7 +35,7 @@ Do not introduce alternate frameworks (e.g., Prisma, NextAuth, Firebase). Stay w - **Never commit or alter git history** without explicit human approval for that specific action — including `git commit`, `git commit --amend`, rebase, history-changing `reset`, `cherry-pick`, branch delete, or force-push. Leave changes for the author to review and commit unless they clearly asked you to perform a named git operation. - **Never modify Ansible roles** without understanding the guest image build pipeline - **Never hardcode attestation keys or measurements** -- **Version bumps** — Three domains; see [docs/versioning.md](docs/versioning.md) for the full policy. **VM domain** (`ansible/guest/*`, `src/sek8s/*`, `src/sek8s-common/*`, `nvevidence/*`, root `pyproject.toml`/`poetry.lock`): bump `ansible/guest/VERSION`. Changes under **`ansible/host/`** do not bump the guest image version. **Proxy domain** (`src/attestation-proxy/*`): bump `src/attestation-proxy/VERSION`. **Ops domain** (`ansible/host/*`, `host-tools/*`, `.github/workflows/*`): bump `changelogs/ops/VERSION` using CalVer `YYYY.MM.PATCH` (e.g. `2026.05.0`; increment PATCH for a second release in the same month). Per-package `VERSION` files are the source of truth for `[tool.poetry] version` — keep them in sync via `scripts/sync_pyproject_versions.py`. Version bumps happen at release time, not during feature development. +- **Version bumps** — Four domains; see [docs/versioning.md](docs/versioning.md) for the full policy. **VM domain** (`ansible/guest/*`, `src/sek8s/*`, `src/sek8s-common/*`, `nvevidence/*`, root `pyproject.toml`/`poetry.lock`): bump `ansible/guest/VERSION`. Changes under **`ansible/host/`** do not bump the guest image version. **Proxy domain** (`src/attestation-proxy/*`): bump `src/attestation-proxy/VERSION`. **chutes-cvm domain** (`src/chutes-cvm/*`, the independently installable host CLI): bump `src/chutes-cvm/VERSION` (SemVer); changelog fragments go in `changelogs/chutes-cvm/`. **Ops domain** (`ansible/host/*`, `host-tools/*`, `.github/workflows/*`): bump `changelogs/ops/VERSION` using CalVer `YYYY.MM.PATCH` (e.g. `2026.05.0`; increment PATCH for a second release in the same month). Per-package `VERSION` files are the source of truth for `[tool.poetry] version` — keep them in sync via `scripts/sync_pyproject_versions.py`. Version bumps happen at release time, not during feature development. - **Changelog fragments** — As you make changes, keep `changelogs//unreleased/.md` up to date using [Keep a Changelog](https://keepachangelog.com/) category headers (`### Added`, `### Changed`, `### Fixed`, `### Removed`). This is the only changelog file you should ever touch during development. **Never write `## [x.y.z]` version headings or edit `CHANGELOG.md` directly** — that is done by `make promote-changelogs` (or CI) at release time. PRs to `main` must have no unreleased fragments remaining. ## Patterns @@ -50,6 +50,30 @@ Do not introduce alternate frameworks (e.g., Prisma, NextAuth, Firebase). Stay w - **One concern per module** — keep files focused; split when they grow large - **Follow existing naming** — check neighboring files and packages for conventions +### chutes-cvm: bash vs Python + +**Python owns decisions and data; bash owns privileged, linear sequences of external-tool calls.** +The language boundary must fall at a **data handoff**: Python resolves the values, then hands them +to a bash step that performs the root-level system mutation. Never split a decision from its +execution across the boundary (bash deciding *and* executing while Python builds args downstream is +the anti-pattern). + +- **Put it in Python** when it makes decisions (precedence, validation, branching on parsed data), + models/validates structured data (config schema, manifest, GPU/topology profiles, measurements), + constructs commands from data (e.g. `guest/qemu.py` building the QEMU cmdline), is measurement- or + security-critical (must be unit-tested), or is the dispatch/UX surface (argparse, help, exit codes). +- **Put it in bash** (bundled under `chutes_cvm/scripts/`) when it is a thin, mostly-linear sequence + of privileged system mutations via external tools (cryptsetup/qemu-nbd/mkfs/losetup, ip/iptables, + aria2c, lspci/nvidia-smi) where the logic *is* the tool invocations, branching is shallow, there is + no structured data to model, and a reviewer benefits from reading the literal root commands. These + need root + real devices, so they are untestable in unit tests regardless — porting them to Python + buys indirection, not testability. +- **Smell tests**: a bash script carrying real precedence/parsing/validation logic → that logic + belongs in Python (the script shrinks to its system-mutation steps). A Python module that is only + `subprocess.run([...])` calls with no data modeling → fine to keep, but don't Pythonize a + cryptsetup sequence for purity. Standalone operator/diagnostic tools meant to be read and run + directly (e.g. `discover-profile.sh`) legitimately stay bash. + ## Architecture Overview | Component | Purpose | @@ -61,12 +85,13 @@ Do not introduce alternate frameworks (e.g., Prisma, NextAuth, Firebase). Stay w | **src/sek8s-common/sek8s_common/** | Shared config, server, auth, and constants for all sek8s packages | | **src/attestation-proxy/attestation_proxy/** | Dual-port attestation proxy (separate lean Docker image) | | **nvevidence/** | NVIDIA attestation SDK wrapper (separate Poetry package) | -| **host-tools/** | Host setup (`chutes.host`), GPU binding/VM launch (`chutes.guest`), networking, orchestration (`quick-launch.sh`) | +| **src/chutes-cvm/** | The `chutes-cvm` CLI + toolkit. `install.sh` is the **single source of truth for install** (fetch + venv + shims; repo-present=editable, standalone curl\|bash=non-editable). Under `chutes_cvm/` (import `chutes_cvm`, console script `chutes-cvm`): host setup (`host/`), GPU binding & VM launch (`guest/`), offline measurement generation (`measurement/`), and bundled data under `scripts/` — the privileged volume/network/teardown shell helpers the Python launch orchestrator (`guest/launch.py`) drives, config schema/template, and the nvidia-gpu-tools wheel (`scripts/gpu-tools/`). The GPU-tools **build recipe** is `tools/gpu-tools/` (`make bundle-gpu-tools`), outside the shipped package. | +| **host-tools/** | Operator config examples (`scripts/config/`) and docs. The install script, VM-management scripts, and the GPU-tools wheel all moved into the `chutes-cvm` package; only the guest firmware (`firmware/`, MRTD-measured, image-bound) stays external to it. | | **guest-tools/** | TDX VM image builder, boot measurement extraction | | **ansible/guest/** | Ansible roles for guest image build (k3s, GPU drivers, attestation services, LUKS) | | **ansible/host/** | Operational Ansible (setup / launch / upgrade) for bare-metal TDX hosts over SSH | | **opa/** | OPA policy files for admission controller | -| **guest-tools/** | Boot measurement extraction tools (`extract-acpi.sh`, `extract-vm-measurements.sh`) | +| **guest-tools/** | Guest measurement & verification tooling (`measurement/`), image build output (`image/`), and R2 publish (`publish-image.sh`) | ## Environment Setup diff --git a/Makefile b/Makefile index 7063ec56..bb9af815 100644 --- a/Makefile +++ b/Makefile @@ -10,17 +10,28 @@ DC=docker compose -p ${PROJECT} -f ${COMPOSE_FILE} -f ${COMPOSE_BASE_FILE} POETRY ?= "poetry" SRC_DIR := src -PACKAGES := $(shell ls $(SRC_DIR)) +# Python packages only: the Python tooling below derives src/// paths and +# -p/--cov import names from this list, so non-Python packages under src/ (e.g. the +# sr25519-signer Rust crate) must not appear in it. +PACKAGES := $(patsubst $(SRC_DIR)/%/pyproject.toml,%,$(wildcard $(SRC_DIR)/*/pyproject.toml)) VERSION := $(shell head ansible/guest/VERSION | grep -Eo "\d+.\d+.\d+") # Package filter: "make sek8s" selects one package PKG_FILTER := $(filter $(PACKAGES),$(MAKECMDGOALS)) SELECTED_PKGS := $(or $(PKG_FILTER),$(PACKAGES)) -# Wire package goal into PROJECT for docker targets (build/tag/push/sign) +# Standalone docker images: docker/ with a Dockerfile and no matching src/ package. +# Image filter: "make images busybox" selects one image; bare "make images" builds all. +STANDALONE_IMAGES := $(shell for d in docker/*/; do n=$$(basename "$$d"); [ -f "$$d/Dockerfile" ] && [ ! -d "src/$$n" ] && echo $$n; done) +IMG_FILTER := $(filter $(STANDALONE_IMAGES),$(MAKECMDGOALS)) +SELECTED_IMGS := $(or $(IMG_FILTER),$(STANDALONE_IMAGES)) + +# Wire package/image goal into PROJECT for docker targets (build/tag/push/sign) ifeq ($(PROJECT),) ifneq ($(PKG_FILTER),) override PROJECT := $(firstword $(PKG_FILTER)) +else ifneq ($(IMG_FILTER),) +override PROJECT := $(firstword $(IMG_FILTER)) endif endif @@ -45,6 +56,12 @@ $(PKG_FILTER): @: endif +# Allow standalone image names as make goals (no-op targets) +ifneq ($(IMG_FILTER),) +$(IMG_FILTER): + @: +endif + .DEFAULT_GOAL := help .EXPORT_ALL_VARIABLES: diff --git a/README.md b/README.md index bff12f35..4a1c23a5 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ Confidential GPU infrastructure for Chutes miners and zero-trust workloads. This | `**docs/**` | Integration guide with [chutes-miner](https://github.com/chutesai/chutes-miner) and system-status service documentation | | `ansible/guest/` | Ansible roles for guest image build automation | | `sek8s/`, `nvevidence/` | Python services running inside the guest (attestation, evidence verification, system status) | -| `guest-tools/` | Boot measurement extraction tools (`extract-acpi.sh`, `extract-vm-measurements.sh`) | +| `guest-tools/` | Guest measurement & verification tooling (`measurement/`), image build output (`image/`), R2 publish (`publish-image.sh`) | --- @@ -22,8 +22,8 @@ Confidential GPU infrastructure for Chutes miners and zero-trust workloads. This ## Quick start roadmap 1. **Set up the host** — Use `[host-tools/](host-tools/)` to prepare your TDX-capable machine with the required kernel, PCCS, and networking. -2. **Download the VM image** — Run `./quick-launch.sh --download` from `host-tools/scripts/` to fetch the prebuilt guest image (requires `aria2`). -3. **Configure and launch** — Run `./quick-launch.sh --template` to generate a `config.yaml`, fill in your miner credentials and network settings, then `./quick-launch.sh config.yaml` to create volumes, configure GPUs, and boot the VM in one command. +2. **Download the VM image** — Run `chutes-cvm image download` to fetch + verify the prebuilt guest image set (requires `aria2`). +3. **Configure and launch** — Run `chutes-cvm config init` to generate a `config.yaml`, fill in your miner credentials and network settings, then `chutes-cvm guest launch config.yaml` to create volumes, configure GPUs, and boot the VM in one command. 4. **Understand the integration** — Read `[docs/end-to-end-miner.md](docs/end-to-end-miner.md)` to see how this repo integrates with the [chutes-miner](https://github.com/chutesai/chutes-miner) control plane. 5. **Build the guest image** (optional) — Use `[guest-tools/](guest-tools/)` and `[ansible/guest/](ansible/guest/)` to customize or rebuild the encrypted VM image. 6. **Monitor VM status** — See `[docs/system-status.md](docs/system-status.md)` for using the system-status API to inspect service health and GPU telemetry inside the VM. @@ -46,7 +46,7 @@ The `config.yaml` defines your deployment: VM identity, miner credentials, netwo ## Key Documentation - `**[host-tools/README.md](host-tools/README.md)`** — Setting up the TDX host and launching VMs -- `**[guest-tools/README.md](guest-tools/README.md)**` — Building and measuring the encrypted VM image +- `**[docs/specs/tdx-measurement-verification.md](docs/specs/tdx-measurement-verification.md)**` — How the guest image's TDX measurements are structured, reproduced, and independently verified (on-host capture in `guest-tools/measurement/`; offline replay/generation in `chutes_cvm.measurement`) - `**[docs/end-to-end-miner.md](docs/end-to-end-miner.md)**` — Complete integration workflow with chutes-miner - `**[docs/system-status.md](docs/system-status.md)**` — System status API for monitoring service health and GPU telemetry diff --git a/ansible/guest/README.md b/ansible/guest/README.md index 61fb5309..ccee84c4 100644 --- a/ansible/guest/README.md +++ b/ansible/guest/README.md @@ -66,7 +66,7 @@ The build process: 3. Applies security hardening and admission policies 4. Encrypts root filesystem with LUKS 5. Configures initramfs for TDX-based boot unlock -6. Outputs final encrypted image under `guest-tools/image//.qcow2` (see `playbooks/group_vars/host.yml` and inventory `build_env`; `vm_version` comes from `ansible/guest/VERSION`; append `-debug` when `debug_build` is true) +6. Outputs the final encrypted image SET under `guest-tools/image///` — the `.qcow2`, its direct-boot `.vmlinuz`/`.initrd`/`.cmdline` sidecars, and `manifest.json` (see `playbooks/group_vars/host.yml` and inventory `build_env`; `vm_version` comes from `ansible/guest/VERSION`; a debug build appends `-debug` to both the directory and the image name). The directory is a ready-to-use image set: copy it into `/var/lib/chutes/base-images//` to boot it with `chutes-cvm guest launch`. At the **start** of `chutes-miner-vm.yml` (before the build VM is launched), the playbook prints the build configuration and **pauses for confirmation** (press Enter to continue, Ctrl+C to abort). @@ -91,10 +91,10 @@ Host tools automatically configure iptables rules for k3s API (port 6443) and No ### Configuration Volumes -Production VMs require three attached volumes (created by `quick-launch.sh`): +Production VMs require three attached volumes (created by `chutes-cvm guest launch`): #### Config Volume (`tdx-config`) -- **Created by**: `host-tools/scripts/volumes/create-config.sh` +- **Created by**: `src/chutes-cvm/chutes_cvm/scripts/volumes/create-config.sh` - **Filesystem**: ext4 with label `tdx-config` - **Mount point**: `/var/config` - **Contents**: @@ -106,14 +106,14 @@ Production VMs require three attached volumes (created by `quick-launch.sh`): - `docker-hub-token` - (optional) Docker Hub PAT for authenticated pulls and cosign #### Cache Volume (`tdx-cache`) -- **Created by**: `host-tools/scripts/volumes/create-cache.sh` +- **Created by**: `src/chutes-cvm/chutes_cvm/scripts/volumes/create-cache.sh` - **Filesystem**: XFS with label `tdx-cache` - **Mount point**: `/var/snap` - **Purpose**: Persistent storage for HF/model caches (e.g., `/var/snap/cache` for model weights) - **Size**: Configurable (default 5000G) #### Storage Volume (`storage`) -- **Created by**: `host-tools/scripts/volumes/create-cache.sh` (with label `storage`) +- **Created by**: `src/chutes-cvm/chutes_cvm/scripts/volumes/create-cache.sh` (with label `storage`) - **Filesystem**: XFS with label `storage` - **Mount point**: `/cache/storage` (contents bind-mounted into standard paths) - **Purpose**: Persistent k3s state, containerd data, kubelet pods, admission controller certs, and chutes agent state @@ -168,11 +168,11 @@ See role-specific defaults for component configuration. This Ansible playbook builds the VM image only. The following are handled by host-tools: -- ❌ TDX-enabled host system setup → See `host-tools/scripts/chutes/host/` -- ❌ GPU passthrough configuration → Handled automatically by `run-td` -- ❌ Network infrastructure → See `host-tools/scripts/network/setup-bridge.sh` -- ❌ Config/cache/storage volume creation → See `host-tools/scripts/volumes/create-*.sh` -- ❌ VM launch and orchestration → See `host-tools/scripts/quick-launch.sh` +- ❌ TDX-enabled host system setup → See `src/chutes-cvm/chutes_cvm/host/` +- ❌ GPU passthrough configuration → Handled automatically by `chutes-cvm guest launch` +- ❌ Network infrastructure → See `src/chutes-cvm/chutes_cvm/scripts/network/setup-bridge.sh` +- ❌ Config/cache/storage volume creation → See `src/chutes-cvm/chutes_cvm/scripts/volumes/create-*.sh` +- ❌ VM launch and orchestration → Handled by `chutes-cvm guest launch` - ✅ Guest OS and k3s installation - ✅ GPU drivers and attestation services - ✅ Security hardening and admission control diff --git a/ansible/guest/VERSION b/ansible/guest/VERSION index 3a3cd8cc..88c5fb89 100644 --- a/ansible/guest/VERSION +++ b/ansible/guest/VERSION @@ -1 +1 @@ -1.3.1 +1.4.0 diff --git a/ansible/guest/inventory-reproduce.yml b/ansible/guest/inventory-reproduce.yml new file mode 100644 index 00000000..d35a70fb --- /dev/null +++ b/ansible/guest/inventory-reproduce.yml @@ -0,0 +1,35 @@ +--- +# inventory-reproduce.yml +# +# Reproduce the Chutes production guest image and its measurements as an independent third party. +# Identical to inventory.yml except it needs NONE of our secrets: +# * luks_passphrase is a hardcoded, published constant (below) instead of the LUKS_PASSPHRASE env. +# It does NOT affect the measured registers — MRTD/RTMR0-3 cover the firmware, kernel, initrd, +# cmdline and the baked configs (incl. the root key), not the LUKS container — so any value +# yields the same measurements; first-boot rotation replaces it anyway. +# * The root RSA PUBLIC key is fetched from R2 (root_signing_key_url), same as our own builds. +# Everything else is the standard prod build. Build, then compare your measurements to the published +# ones at GET https://api.chutes.ai/servers/tee/measurements. +all: + children: + vm: + hosts: # Dynamically populated by playbook + host: + hosts: + localhost: + ansible_connection: local + + vars: + ansible_user: "{{ lookup('env', 'USER') }}" + # Root RSA PUBLIC key — fetched from R2 (public), baked into the image, measured into RTMR3. + root_signing_key_url: "https://vm.chutes.ai/root-signing-key.pem" + root_signing_key_path: "/tmp/chutes-root-signing-key.pem" + # Published constant — intentionally NOT a secret (does not affect measurements; see header). + luks_passphrase: "chutes" + tdx_base_url: "https://cvm.chutes.ai" + validator_base_url: "https://api.chutes.ai" + build_env: "prod" + prime_wait_timeout: 300 + + debug_build: false + guest_ssh_keys: [] diff --git a/ansible/guest/inventory.yml b/ansible/guest/inventory.yml index 7706bc63..553a9a9b 100644 --- a/ansible/guest/inventory.yml +++ b/ansible/guest/inventory.yml @@ -11,11 +11,15 @@ all: vars: ansible_user: "{{ lookup('env', 'USER') }}" - cosign_public_key_path: "~/.cosign/cosign.pub" - # Helm chart signing PGP public key (required) - helm_chart_public_key_path: "~/.chutes/helm-pubkey.gpg" + # Root RSA PUBLIC key — baked into the image and measured into RTMR3. Fetched at build time + # from R2 (not user-supplied). It is published from KMS out-of-band in chutes-ops + # (`make export-signing-keys && make publish-signing-key`) — a manual prerequisite, NOT part of + # the build (the build depends on it, so it can never upload it). root_signing_key_path is just + # the build-local cache the fetch fills. + root_signing_key_url: "https://vm.chutes.ai/root-signing-key.pem" + root_signing_key_path: "/tmp/chutes-root-signing-key.pem" luks_passphrase: "{{ lookup('env', 'LUKS_PASSPHRASE') | mandatory }}" - tdx_base_url: "https://tdx-attestation.example.com:8443" + tdx_base_url: "https://cvm.chutes.ai" validator_base_url: "https://api.chutes.ai" build_env: "prod" prime_wait_timeout: 300 @@ -23,6 +27,5 @@ all: debug_build: false # List of SSH public key strings for guest access (required when using tee-gpu-vm.yml) guest_ssh_keys: [] - validator: "5Dt7HZ7Zpw4DppPxFM7Ke3Cm7sDAWhsZXmM5ZAmE7dSVJbcQ" # Optional: Override sek8s image tag for testing (default: "latest") # sek8s_image_tag: "dev-latest" diff --git a/ansible/guest/playbooks/chutes-miner-vm.yml b/ansible/guest/playbooks/chutes-miner-vm.yml index eab8e774..31e88ec4 100644 --- a/ansible/guest/playbooks/chutes-miner-vm.yml +++ b/ansible/guest/playbooks/chutes-miner-vm.yml @@ -19,6 +19,22 @@ Is this the intended build? The next step launches the TDX guest VM toward this output path. Press Enter to continue, or Ctrl+C to abort. + # The root RSA PUBLIC key is baked into the image and measured into RTMR3, so every build + # (ours and third-party reproductions) must use the SAME key. Fetch it from R2 once here, before + # any consumer (chutes-gpu verifies the helm key against it; signing-keys bakes it). Published + # to R2 from KMS out-of-band in chutes-ops (make export-signing-keys && make publish-signing-key). + # A wrong/missing key fails the build here rather than as an opaque RTMR3 mismatch later. + - name: Fetch root signing public key from R2 + ansible.builtin.get_url: + url: "{{ root_signing_key_url }}" + dest: "{{ root_signing_key_path }}" + mode: "0644" + force: true + + - name: Verify the fetched root key is a valid RSA public key + ansible.builtin.command: "openssl pkey -pubin -in {{ root_signing_key_path }} -noout" + changed_when: false + - name: Run TDX guest image hosts: host become: true @@ -29,6 +45,16 @@ - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" tasks: + # Install the chutes-cvm CLI from THIS checkout (editable venv + PATH shim) so the + # build's `chutes-cvm` calls (measurements/image, here and in capture-ccel) + # run the code being built and get the package's deps. Runs once; the console script + # is then on PATH for every later play. + - name: Install the chutes-cvm CLI from this checkout + ansible.builtin.command: + cmd: "{{ repo_root }}/src/chutes-cvm/install.sh" + register: _cvm_install + changed_when: "'Installed chutes-cvm' in _cvm_install.stdout" + - name: Include run TDX vm role ansible.builtin.include_role: name: run-vm @@ -38,6 +64,7 @@ become: true tags: - common + - build handlers: - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" @@ -51,6 +78,7 @@ become: true tags: - gpu + - build handlers: - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" @@ -64,6 +92,7 @@ become: true tags: - checkpoint-gpu + - build tasks: - name: Save GPU checkpoint ansible.builtin.include_role: @@ -76,6 +105,7 @@ become: true tags: - k3s + - build handlers: - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" @@ -89,6 +119,7 @@ become: true tags: - gpu-verify + - build handlers: - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" @@ -102,6 +133,7 @@ become: true tags: - chutes-gpu + - build handlers: - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" @@ -117,6 +149,7 @@ become: true tags: - sek8s + - build handlers: - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" @@ -132,6 +165,7 @@ become: true tags: - attestation-service + - build handlers: - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" @@ -147,6 +181,7 @@ become: true tags: - admission-controller + - build handlers: - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" @@ -157,11 +192,44 @@ apply: tags: admission-controller +- name: Install signing key trust anchor and initramfs fetch scripts + hosts: vm + become: true + tags: + - signing-keys + - build + handlers: + - name: Global handlers + ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" + tasks: + - name: Signing keys + ansible.builtin.include_role: + name: signing-keys + apply: + tags: signing-keys + +- name: Install VM mTLS initramfs cert lifecycle + hosts: vm + become: true + tags: + - vm-tls + - build + handlers: + - name: Global handlers + ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" + tasks: + - name: VM TLS + ansible.builtin.include_role: + name: vm-tls + apply: + tags: vm-tls + - name: Setup system manager service (status + cache) hosts: vm become: true tags: - system-manager + - build handlers: - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" @@ -172,11 +240,28 @@ apply: tags: system-manager +- name: Setup chute log shipper service + hosts: vm + become: true + tags: + - chute-log-shipper + - build + handlers: + - name: Global handlers + ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" + tasks: + - name: Chute log shipper + ansible.builtin.include_role: + name: chute-log-shipper + apply: + tags: chute-log-shipper + - name: Setup cache volume service hosts: vm become: true tags: - cache-volume + - build handlers: - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" @@ -187,12 +272,29 @@ apply: tags: cache-volume +- name: AppArmor hardening + hosts: vm + become: true + tags: + - apparmor-hardening + - build + handlers: + - name: Global handlers + ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" + tasks: + - name: AppArmor hardening + ansible.builtin.include_role: + name: apparmor-hardening + apply: + tags: apparmor-hardening + - name: Add dyanmic config services hosts: vm become: true any_errors_fatal: false tags: - config + - build handlers: - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" @@ -208,6 +310,7 @@ become: true tags: - lock-accounts + - build tasks: - name: Lock accounts ansible.builtin.include_role: @@ -219,6 +322,7 @@ become: true tags: - disable-console + - build handlers: - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" @@ -233,6 +337,7 @@ become: true tags: - debug-credentials + - build tasks: - name: Set known root password for console access (debug builds only) ansible.builtin.user: @@ -240,15 +345,27 @@ password: "{{ 'debug' | password_hash('sha512') }}" when: debug_build | default(false) - - name: Allow root password SSH login (debug builds only) - ansible.builtin.lineinfile: - path: /etc/ssh/sshd_config - regexp: "{{ item.regexp }}" - line: "{{ item.line }}" - state: present - loop: - - { regexp: '^#?PermitRootLogin', line: 'PermitRootLogin yes' } - - { regexp: '^#?PasswordAuthentication', line: 'PasswordAuthentication yes' } + - name: Allow root password SSH login via high-precedence drop-in (debug builds only) + # Ubuntu's cloud image ships /etc/ssh/sshd_config.d/50-cloud-init.conf with + # `PasswordAuthentication no`. sshd honours the FIRST value seen for each + # keyword, and the main sshd_config Includes the drop-in dir near the top — + # so editing the main file is silently overridden by that drop-in and the + # debug image ends up key-only (unusable without the build-time key, which + # defeats the point of a debug image). Disabling cloud-init does NOT remove + # the drop-in. A `00-` drop-in sorts before `50-cloud-init.conf`, so its + # values are seen first and win — restoring password/console access. + ansible.builtin.copy: + dest: /etc/ssh/sshd_config.d/00-debug-access.conf + content: | + # Debug build only — restores root password SSH + console access so the + # image is usable without the build-time key (e.g. measurement capture). + # Named `00-` to win first-match precedence over 50-cloud-init.conf. + PermitRootLogin yes + PasswordAuthentication yes + KbdInteractiveAuthentication yes + owner: root + group: root + mode: '0644' when: debug_build | default(false) - name: Security Hardening @@ -256,6 +373,7 @@ become: true tags: - security + - build handlers: - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" @@ -264,11 +382,28 @@ ansible.builtin.include_role: name: security +- name: Install sr25519 signer (guest boot-time hotkey proof-of-possession) + hosts: vm + become: true + tags: + - sr25519 + - build + handlers: + - name: Global handlers + ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" + tasks: + # Before the luks play rebuilds the final initramfs — fetch_key copy_exec's /usr/bin/sr25519 + # into it, so the binary must exist first (and while the VM is still up). + - name: Install sr25519 signer + ansible.builtin.include_role: + name: sr25519 + - name: Extend RTMR3 with guest software stack hosts: vm become: true tags: - rtmr3-measure + - build handlers: - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" @@ -283,6 +418,7 @@ any_errors_fatal: false tags: - cleanup-orchestration + - build handlers: - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" @@ -297,6 +433,7 @@ any_errors_fatal: false tags: - cleanup-build-vm + - build handlers: - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" @@ -310,6 +447,7 @@ become: true tags: - remove-ssh + - build handlers: - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" @@ -324,6 +462,7 @@ become: true tags: - finalize-vm-image + - build handlers: - name: Global handlers ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" @@ -332,41 +471,123 @@ ansible.builtin.include_role: name: finalize-vm-image -- name: Compute expected RTMR3 from final image +# The whole measurement phase — gather-measurement-inputs + compute-measurements + +# image-manifest — is generated by the fork entirely OFFLINE (no TDX hardware) and cheap, so +# it always runs as part of a full build. There's no on/off variable and no per-profile +# selection: it generates every profile (pending ones self-skip). Every register (mrtd + rtmr0 +# + rtmr1/2 + rtmr3) is computed POST-luks in one `build` call below, since the CLI unlocks the +# encrypted root with LUKS_PASSPHRASE to read RTMR3's userspace files. To build the image +# WITHOUT measurements (e.g. non-Docker or fast local dev iteration), run `--tags build`, or +# `--skip-tags compute-measurements` — the whole chain shares the `compute-measurements` tag. +- name: Prepare boot image (encrypt + measured initramfs for prod / install debug initramfs) hosts: host become: true tags: - - compute-rtmr3 + - prepare-boot-image + - build + handlers: + - name: Global handlers + ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" tasks: - - name: Compute RTMR3 + - name: Prepare boot image ansible.builtin.include_role: - name: compute-rtmr3 + name: prepare-boot-image + apply: + tags: prepare-boot-image -- name: Encrypt disk +# ── Measurement GATHER (post-boot-image) ────────────────────────────────────── +# The initrd is now final (prepare-boot-image rebuilt it for prod AND debug). Stage this image's +# direct-boot vmlinuz/initrd/cmdline — RTMR1/2 measure these exact bytes. +# +# There is NO CCEL capture in this build flow: the tdx-measure fork self-generates the +# complete RTMR0 offline, so the build needs no TDX hardware. The `capture-ccel` role is +# retained on disk as a STANDALONE tool (for the occasional debug / new-CPU-gen cross- +# check against a real quote — include it from a one-off playbook when needed), but it is +# deliberately not part of the build: the fork is the source of truth, and the runtime +# attestation match is the ongoing guarantee. +- name: Gather measurement inputs from final image (post-luks) hosts: host become: true tags: - - luks - handlers: - - name: Global handlers - ansible.builtin.import_tasks: "{{ playbook_dir }}/../handlers/main.yml" + - gather-measurement-inputs + - compute-measurements tasks: - - name: Encrypt disk + - name: Stage direct-boot artifacts ansible.builtin.include_role: - name: luks - apply: - tags: luks - when: not (debug_build | default(false)) + name: stage-boot-artifacts -- name: Prime VM for stable TDX measurements +# ── Measurement COMPUTE (post-luks) ─────────────────────────────────────────── +# `chutes-cvm measurements generate` computes EVERY register in one call — mrtd + RTMR0 +# (per-topology, the fork) + RTMR1/RTMR2 (the staged direct-boot artifacts) + RTMR3 (mounting the +# root, unlocked with LUKS_PASSPHRASE) — writing the version's single measurements.yaml. +# tdx-measure builds the shared fork engine. All offline (no TDX/GPU). +- name: Compute measurements from gathered inputs (post-luks) hosts: host become: true tags: - - prime-vm + - compute-measurements tasks: - - name: Prime VM EFI boot variable state + - name: Provision the tdx-measure fork (RTMR0/1/2 engine) ansible.builtin.include_role: - name: prime-vm - apply: - tags: prime-vm + name: tdx-measure + + - name: Build the full measurements.yaml (mrtd + rtmr0 all topologies + rtmr1/2 + rtmr3) + ansible.builtin.command: + argv: + - chutes-cvm + - measurements + - generate + - --version + - "{{ vm_version }}" + - --image + - "{{ final_img_path }}" + - --output + - "{{ repo_root }}/measurements/{{ vm_version }}{{ '-debug' if debug_build | default(false) else '' }}/measurements.yaml" + - --tdx-measure-bin + - "{{ tdx_measure_bin | default('tdx-measure') }}" + # Source of the known host classes + their fingerprints (the build host must reach it). + - --api-base + - "{{ measurements_api_base | default('https://api.chutes.ai') }}" + # This build is the measurement generator, so it processes the full queue — including + # host classes awaiting generation (a newly submitted profile is 'pending' until its + # measurement exists here). Third-party verification runs omit this and see measured only. + - --include-pending + environment: + # RTMR3 mounts the root; LUKS_PASSPHRASE unlocks it when the image is encrypted (prod). + LUKS_PASSPHRASE: "{{ luks_passphrase | default('') }}" + # The tdx-measure fork shells out to `docker build` for offline ACPI generation; + # extend PATH so it is found even under snap or a minimal inherited PATH. + PATH: "{{ lookup('env', 'PATH') }}:/usr/local/bin:/usr/bin:/snap/bin" + register: _measurements_build + changed_when: _measurements_build.rc == 0 + + - name: Show measurements build output + ansible.builtin.debug: + var: _measurements_build.stderr_lines + +- name: Stage the published image-set manifest + hosts: host + become: true + tags: + - image-manifest + - compute-measurements + tasks: + # The publishable image is a SET: the finished qcow2 + its direct-boot artifacts + a + # manifest.json tying them together, all in the set's own /[-debug]/ + # directory (so prod and debug never share a manifest, and the directory can be copied + # straight into /var/lib/chutes/base-images// for testing). chutes_cvm.guest.image_set is the single manifest + # generator (also used by publish and the launcher), so the schema never drifts. + # Generated over the FINAL qcow2 so the recorded sha256 matches what miners + # download, and for whichever variant this build produced (debug or prod) — so the + # build (ansible) owns the published sha for both. Depends on the direct-boot sidecars + # from stage-boot-artifacts, so it carries the `compute-measurements` tag too — a + # `--skip-tags compute-measurements` skips the manifest with them. + - name: Generate manifest.json for the finished image set + ansible.builtin.command: + chdir: "{{ repo_root }}/host-tools/scripts" + argv: >- + {{ ['chutes-cvm', 'image', 'manifest', final_img_path, + '--version', vm_version] + + (['--debug'] if (debug_build | default(false)) else []) }} + changed_when: true diff --git a/ansible/guest/playbooks/group_vars/all.yml b/ansible/guest/playbooks/group_vars/all.yml index 922e6e72..dbda0148 100644 --- a/ansible/guest/playbooks/group_vars/all.yml +++ b/ansible/guest/playbooks/group_vars/all.yml @@ -47,7 +47,7 @@ ubuntu_minor: "04" # baseline. Pinning it to an exact version makes the guest kernel reproducible and # bumps deliberate. Trade-off: opts out of automatic HWE kernel security updates. # MUST stay >= 6.16 (RTMR3 depends on the tsm-mr sysfs interface merged in 6.16). -guest_hwe_kernel_version: "6.17.0-35.35~24.04.1" +guest_hwe_kernel_version: "7.0.0-28.28~24.04.1" # CUDA version - leave as-is unless skipping CUDA entirely (skip_cuda: true) for providers with pre-installed drivers cuda_version: "13-2" # NVIDIA driver branch diff --git a/ansible/guest/playbooks/group_vars/host.yml b/ansible/guest/playbooks/group_vars/host.yml index c98345ba..98094128 100644 --- a/ansible/guest/playbooks/group_vars/host.yml +++ b/ansible/guest/playbooks/group_vars/host.yml @@ -4,10 +4,24 @@ img_dir: "{{ repo_root }}/guest-tools/image" # VM image release version from repo root (distinct from Python package version) vm_version: "{{ lookup('file', playbook_dir + '/../VERSION') | trim }}" # build_env is set in inventory or role defaults (not here — playbook group_vars would override inventory) -final_img_path: "{{ img_dir }}/{{ build_env }}/{{ vm_version }}{{ '-debug' if debug_build | default(false) else '' }}.qcow2" +# The publishable output is an image SET, so it gets its OWN directory per version+variant: +# /[-debug]/ holds the qcow2, its .vmlinuz/.initrd/.cmdline sidecars and +# manifest.json. That directory is exactly what an image-set consumer expects (the launcher +# globs a single *.qcow2 and reads manifest.json beside it), so it can be copied straight +# into /var/lib/chutes/base-images// — and a debug build never clobbers the prod +# set's manifest. Intermediate checkpoint layers below stay flat in / (not a set). +image_set_name: "{{ vm_version }}{{ '-debug' if debug_build | default(false) else '' }}" +image_set_dir: "{{ img_dir }}/{{ build_env }}/{{ image_set_name }}" +final_img_path: "{{ image_set_dir }}/{{ image_set_name }}.qcow2" base_img_path: "{{ img_dir }}/tdx-guest-ubuntu-{{ ubuntu_version }}.qcow2" prepared_img_path: "{{ img_dir }}/{{ build_env }}/{{ vm_version }}-prepared.qcow2" gpu_img_path: "{{ img_dir }}/{{ build_env }}/{{ vm_version }}-gpu.qcow2" libvirt_image_dir: "/var/lib/libvirt/images" build_img_path: "{{ libvirt_image_dir }}/tdx-guest.qcow2" -vm_name: "tdx-build" \ No newline at end of file +vm_name: "tdx-build" +# Measurements (RTMR0/1/2/3) are generated by the tdx-measure fork entirely OFFLINE — no +# TDX hardware, cheap — so they always run as part of a full build; there is no on/off +# variable and no per-profile selection (every profile is generated; pending ones self- +# skip). To build the image WITHOUT them, run `--tags build` or `--skip-tags +# compute-measurements` (the whole measurement + manifest chain shares that tag). The +# `capture-ccel` role is retained standalone for ad-hoc debugging only, not in the build. \ No newline at end of file diff --git a/ansible/guest/playbooks/tee-gpu-vm.yml b/ansible/guest/playbooks/tee-gpu-vm.yml index f7c92acf..22ae88e5 100644 --- a/ansible/guest/playbooks/tee-gpu-vm.yml +++ b/ansible/guest/playbooks/tee-gpu-vm.yml @@ -205,19 +205,25 @@ tags: - compute-rtmr3 tasks: - - name: Compute RTMR3 - ansible.builtin.include_role: - name: compute-rtmr3 - -- name: Prime VM for stable TDX measurements - hosts: host - become: true - tags: - - prime-vm - tasks: - - name: Prime VM EFI boot variable state - ansible.builtin.include_role: - name: prime-vm - apply: - tags: prime-vm + # RTMR3 measures the partner's SSH key (+ the other tdx-measure.conf files) from the root. + # This build has no per-topology aggregation, so it generates only that one register. + - name: Compute RTMR3 (bare hex on stdout) + ansible.builtin.command: + argv: + - chutes-cvm + - measurements + - generate + - --register + - rtmr3 + - --image + - "{{ final_img_path }}" + environment: + # LUKS_PASSPHRASE unlocks the root when the image is already encrypted. + LUKS_PASSPHRASE: "{{ luks_passphrase | default('') }}" + register: _rtmr3_compute + changed_when: false + + - name: Show RTMR3 + ansible.builtin.debug: + msg: "RTMR3={{ _rtmr3_compute.stdout | trim }}" diff --git a/ansible/guest/roles/admission-controller/defaults/main.yml b/ansible/guest/roles/admission-controller/defaults/main.yml index 1517142e..79fe1b41 100644 --- a/ansible/guest/roles/admission-controller/defaults/main.yml +++ b/ansible/guest/roles/admission-controller/defaults/main.yml @@ -8,6 +8,10 @@ opa_version: "1.15.2" # which is stable across v2 and v3; staying on v2 avoids any v3 bundle-format changes. cosign_version: "2.6.3" opa_log_level: "info" +# Per-decision logging (full AdmissionReview input → journal). OFF in prod: it is +# high-volume noise that evicts the boot/attestation history from the journal window, +# and it puts tenant pod specs into a log the miner can read via the status API. +# Debug builds can opt in with -e opa_decision_logs=true. opa_decision_logs: false # Admission controller settings @@ -20,9 +24,6 @@ cache_enabled: true cache_ttl: 300 debug_mode: false metrics_enabled: true -# Chutes config -validator: 5Dt7HZ7Zpw4DppPxFM7Ke3Cm7sDAWhsZXmM5ZAmE7dSVJbcQ - # Registry allowlist allowed_registries: - docker.io @@ -32,5 +33,4 @@ allowed_registries: - rancher - nvcr.io - parachutes - - bitnami - - "{{ validator }}.localregistry.chutes.ai:30500" + - "registry.chutes.ai" diff --git a/ansible/guest/roles/admission-controller/files/admission-controller.service b/ansible/guest/roles/admission-controller/files/admission-controller.service index 8558d373..aad8ef1a 100644 --- a/ansible/guest/roles/admission-controller/files/admission-controller.service +++ b/ansible/guest/roles/admission-controller/files/admission-controller.service @@ -16,6 +16,9 @@ WorkingDirectory=/opt/sek8s # Use virtual environment Environment="PATH=/opt/sek8s/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" Environment="VIRTUAL_ENV=/opt/sek8s/venv" +# Keep the RTMR3-measured /opt/sek8s/src tree free of runtime-written .pyc (see the note in +# system-manager.service) so the measurement stays stable across reboots. +Environment="PYTHONDONTWRITEBYTECODE=1" EnvironmentFile=/etc/admission-controller/admission.env # Run the admission controller diff --git a/ansible/guest/roles/admission-controller/files/opa-config.yaml b/ansible/guest/roles/admission-controller/files/opa-config.yaml deleted file mode 100644 index d6bdcb9f..00000000 --- a/ansible/guest/roles/admission-controller/files/opa-config.yaml +++ /dev/null @@ -1,9 +0,0 @@ -# OPA Configuration -decision_logs: - console: true - reporting: - min_delay_seconds: 5 - max_delay_seconds: 10 - -status: - console: true diff --git a/ansible/guest/roles/admission-controller/files/policies/pods.rego b/ansible/guest/roles/admission-controller/files/policies/pods.rego index d7761870..ea622db7 100644 --- a/ansible/guest/roles/admission-controller/files/policies/pods.rego +++ b/ansible/guest/roles/admission-controller/files/policies/pods.rego @@ -129,6 +129,8 @@ dangerous_capabilities := { "SYS_RAWIO", "SYS_PTRACE", "SYS_BOOT", + "MAC_ADMIN", + "MAC_OVERRIDE", } has_dangerous_capability(container) if { diff --git a/ansible/guest/roles/admission-controller/tasks/configure-cosign.yml b/ansible/guest/roles/admission-controller/tasks/configure-cosign.yml index 52419af8..cdf2fcbb 100644 --- a/ansible/guest/roles/admission-controller/tasks/configure-cosign.yml +++ b/ansible/guest/roles/admission-controller/tasks/configure-cosign.yml @@ -25,58 +25,8 @@ group: admission mode: '0750' - - name: Setup cosign key - ansible.builtin.copy: - src: "{{ cosign_public_key_path }}" - dest: /etc/admission-controller/cosign/cosign.pub - owner: root - group: admission - mode: '0640' - notify: restart admission-controller - - - name: Add proxy hostname to /etc/hosts - lineinfile: - path: /etc/hosts - line: "127.0.0.1 {{ validator }}.localregistry.chutes.ai" - regexp: "^127\\.0\\.0\\.1\\s+{{ (validator + '.localregistry.chutes.ai') | regex_escape() }}$" - state: present - backup: yes - - - name: Ensure /etc/docker directory exists - file: - path: /etc/docker - state: directory - mode: '0755' - - - name: Read existing Docker daemon config - slurp: - path: /etc/docker/daemon.json - register: docker_daemon_config - ignore_errors: yes - - - name: Parse existing Docker daemon config - set_fact: - docker_config: "{{ docker_daemon_config.content | b64decode | from_json }}" - when: docker_daemon_config is succeeded - - - name: Set default Docker config if file doesn't exist - set_fact: - docker_config: {} - when: docker_daemon_config is failed - - - name: Update insecure-registries in Docker config - set_fact: - docker_config: "{{ docker_config | combine({'insecure-registries': insecure_registries_list}) }}" - vars: - insecure_registries_list: - - "{{ validator }}.localregistry.chutes.ai:{{ registry_port | default('30500') }}" - - "localhost:{{ registry_port | default('30500') }}" - - "127.0.0.1:{{ registry_port | default('30500') }}" - - - name: Write updated Docker daemon config - copy: - content: "{{ docker_config | to_nice_json }}" - dest: /etc/docker/daemon.json - mode: '0644' - backup: yes - notify: restart docker \ No newline at end of file + # registry.chutes.ai is a real, publicly-resolvable host reached over + # genuine mTLS — no /etc/hosts loopback alias and no insecure-registries + # entry are needed. The former localregistry.chutes.ai:30500 local-proxy + # plumbing (127.0.0.1 hosts alias + insecure-registries in daemon.json) has + # been removed now that pulls use per-VM mTLS to the upstream registry. \ No newline at end of file diff --git a/ansible/guest/roles/admission-controller/tasks/configure-opa-service.yml b/ansible/guest/roles/admission-controller/tasks/configure-opa-service.yml index b4b1930b..289d2cc5 100644 --- a/ansible/guest/roles/admission-controller/tasks/configure-opa-service.yml +++ b/ansible/guest/roles/admission-controller/tasks/configure-opa-service.yml @@ -2,8 +2,8 @@ - name: Configure OPA service block: - name: Copy OPA systemd service file - ansible.builtin.copy: - src: opa.service + ansible.builtin.template: + src: opa.service.j2 dest: /etc/systemd/system/opa.service owner: root group: root diff --git a/ansible/guest/roles/admission-controller/tasks/install-opa.yml b/ansible/guest/roles/admission-controller/tasks/install-opa.yml index 945cc19d..f25a6978 100644 --- a/ansible/guest/roles/admission-controller/tasks/install-opa.yml +++ b/ansible/guest/roles/admission-controller/tasks/install-opa.yml @@ -49,8 +49,8 @@ - { path: /var/log/opa, owner: opa, group: opa } - name: Create OPA configuration - ansible.builtin.copy: - src: opa-config.yaml + ansible.builtin.template: + src: opa-config.yaml.j2 dest: /etc/opa/opa.yaml owner: root group: root diff --git a/ansible/guest/roles/admission-controller/templates/admission-controller.env.j2 b/ansible/guest/roles/admission-controller/templates/admission-controller.env.j2 index 1fd7903b..dd17b911 100644 --- a/ansible/guest/roles/admission-controller/templates/admission-controller.env.j2 +++ b/ansible/guest/roles/admission-controller/templates/admission-controller.env.j2 @@ -21,6 +21,10 @@ ALLOWED_REGISTRIES='{{ allowed_registries | default(['docker.io', 'gcr.io', 'qua CACHE_ENABLED={{ cache_enabled | default(true) | lower }} CACHE_TTL={{ cache_ttl | default(300) }} +# Cosign key paths +CHUTES_PUBLIC_KEY_PATH=/run/chutes/signing-keys/cosign/chutes.pub +DOCKERHUB_PUBLIC_KEY_PATH=/run/chutes/signing-keys/cosign/dockerhub.pub + # Cosign validator (optional; CosignConfig defaults apply if unset — see sek8s/config.py) # COSIGN_SUCCESS_CACHE_TTL=3600 # COSIGN_FAILURE_CACHE_TTL=600 diff --git a/ansible/guest/roles/admission-controller/templates/cosign-registries.json.j2 b/ansible/guest/roles/admission-controller/templates/cosign-registries.json.j2 index ff79f486..44a486a7 100644 --- a/ansible/guest/roles/admission-controller/templates/cosign-registries.json.j2 +++ b/ansible/guest/roles/admission-controller/templates/cosign-registries.json.j2 @@ -9,23 +9,8 @@ "organization": "parachutes", "require_signature": true, "verification_method": "key", - "public_key": "/etc/admission-controller/cosign/cosign.pub", + "public_key": "/run/chutes/signing-keys/cosign/dockerhub.pub", "rekor_url": "https://rekor.sigstore.dev" - }, - "bitnami": { - "organization": "bitnami", - "require_signature": true, - "verification_method": "keyless", - "keyless_identity_regex": "^https://github.com/bitnami/.*", - "keyless_issuer": "https://token.actions.githubusercontent.com", - "rekor_url": "https://rekor.sigstore.dev", - "repositories": { - "kubectl": { - "repository": "kubectl", - "require_signature": false, - "verification_method": "disabled" - } - } } } }, @@ -60,19 +45,17 @@ "verification_method": "disabled" }, { - "registry": "{{ validator }}.localregistry.chutes.ai:{{ registry_port | default('30500') }}", + "registry": "registry.chutes.ai", "require_signature": true, "verification_method": "key", - "allow_http": true, - "allow_insecure": true, - "public_key": "/etc/admission-controller/cosign/cosign.pub", + "public_key": "/run/chutes/signing-keys/cosign/chutes.pub", "rekor_url": "https://rekor.sigstore.dev" }, { "registry": "*", "require_signature": true, "verification_method": "key", - "public_key": "/etc/admission-controller/cosign/cosign.pub", + "public_key": "/run/chutes/signing-keys/cosign/chutes.pub", "rekor_url": "https://rekor.sigstore.dev" } ], diff --git a/ansible/guest/roles/admission-controller/templates/opa-config-data.json.j2 b/ansible/guest/roles/admission-controller/templates/opa-config-data.json.j2 index 7ee9af4d..e0873993 100644 --- a/ansible/guest/roles/admission-controller/templates/opa-config-data.json.j2 +++ b/ansible/guest/roles/admission-controller/templates/opa-config-data.json.j2 @@ -1,5 +1,5 @@ { "config": { - "validator_registry": "{{ validator | lower }}.localregistry.chutes.ai:{{ registry_port | default('30500') }}" + "validator_registry": "registry.chutes.ai" } } diff --git a/ansible/guest/roles/admission-controller/templates/opa-config.yaml.j2 b/ansible/guest/roles/admission-controller/templates/opa-config.yaml.j2 new file mode 100644 index 00000000..86de4747 --- /dev/null +++ b/ansible/guest/roles/admission-controller/templates/opa-config.yaml.j2 @@ -0,0 +1,18 @@ +# OPA Configuration +# Managed by Ansible (see ansible/guest/roles/admission-controller). +{% if opa_decision_logs | default(false) %} +# Decision logs are OFF by default: `console: true` writes the full AdmissionReview +# input (complete pod specs — images, env, mounts) to the journal for every admitted +# object, which both floods the journal window that boot/attestation triage needs and +# puts tenant workload detail into a log the miner can read over the status API +# (/status/services/opa/logs). Enable only for debug builds: -e opa_decision_logs=true. +decision_logs: + console: true + reporting: + min_delay_seconds: 5 + max_delay_seconds: 10 + +{% endif %} +# Plugin/policy-load status only — low volume, no request data. +status: + console: true diff --git a/ansible/guest/roles/admission-controller/files/opa.service b/ansible/guest/roles/admission-controller/templates/opa.service.j2 similarity index 95% rename from ansible/guest/roles/admission-controller/files/opa.service rename to ansible/guest/roles/admission-controller/templates/opa.service.j2 index 82902634..033d4526 100644 --- a/ansible/guest/roles/admission-controller/files/opa.service +++ b/ansible/guest/roles/admission-controller/templates/opa.service.j2 @@ -10,7 +10,7 @@ Group=opa ExecStart=/usr/local/bin/opa run --server \ --config-file=/etc/opa/opa.yaml \ --addr=localhost:8181 \ - --log-level=info \ + --log-level={{ opa_log_level }} \ /etc/opa/policies Restart=always RestartSec=5 diff --git a/ansible/guest/roles/apparmor-hardening/defaults/main.yml b/ansible/guest/roles/apparmor-hardening/defaults/main.yml new file mode 100644 index 00000000..91084e2b --- /dev/null +++ b/ansible/guest/roles/apparmor-hardening/defaults/main.yml @@ -0,0 +1,2 @@ +--- +debug_build: false diff --git a/ansible/guest/roles/apparmor-hardening/files/profiles/sek8s.attestation-proxy b/ansible/guest/roles/apparmor-hardening/files/profiles/sek8s.attestation-proxy new file mode 100644 index 00000000..ed5baf9f --- /dev/null +++ b/ansible/guest/roles/apparmor-hardening/files/profiles/sek8s.attestation-proxy @@ -0,0 +1,100 @@ +# vim: ft=apparmor +# sek8s.attestation-proxy — AppArmor profile for the attestation-proxy container. +# Applied via Kubernetes annotation (not auto-attached). +# +# Posture: default-deny, explicit allowlist only. +# /run/chutes/ is never granted — the proxy reads its TLS certs only via the +# volume-mounted path /etc/ssl/host-certs/. No deny rules are needed because +# what is not allowed is denied by omission. + +abi , + +include + +# attach_disconnected: the attestation socket is a hostPath BIND MOUNT, so its inode is +# "disconnected" from this profile's mount-namespace root. Without this flag AppArmor fails the +# path lookup on connect() ("Failed name lookup - disconnected path") and returns EACCES before any +# rule is evaluated — even in complain mode. The flag reattaches such paths to / so the socket rule +# below can match (this is why Docker's default profile carries it). +profile sek8s.attestation-proxy flags=(enforce, attach_disconnected) { + include + include + include + + # Python interpreter + app code. The proxy runs its OWN lean image, NOT the /opt/sek8s guest + # layout: the interpreter is /usr/local/bin/python3.12 and the code + venv live under + # /app/src/attestation-proxy. (Confirmed against the live container's AppArmor audit; the old + # /opt/sek8s paths never matched, so in enforce the proxy couldn't read its own code.) mr because + # C-extension .so files are mmap'd. + /usr/local/bin/python3.12 mrix, + /usr/local/bin/python3{,.*} mrix, + # The image bundles both the proxy and the shared sek8s-common package under /app/src. + /app/src/ r, + /app/src/** mr, + + # Proxy server TLS certs + /etc/ssl/host-certs/server.crt r, + /etc/ssl/host-certs/server.key r, + + # System CA bundle for outbound TLS verification + /etc/ssl/certs/ r, + /etc/ssl/certs/** r, + + # Service config (read-only) + /etc/attestation-service/ r, + /etc/attestation-service/** r, + + # Attestation unix socket (host-side attestation service). + # It is bind-mounted from the host, so with attach_disconnected (above) AppArmor reattaches it + # under its SOURCE path — /run/attestation-service/... — which is the name that shows in the + # audit log, not the pod mount point (/var/run/attestation -> /run/attestation). Grant the source + # path for the connect (rw = write is what connect() to a pathname socket needs); the mount-point + # paths are kept as a belt-and-suspenders in case a kernel resolves it differently. + /run/attestation-service/ r, + /run/attestation-service/** rw, + /run/attestation/ r, + /run/attestation/** rw, + + # Bittensor data dir (emptyDir mount in k8s pod) + /home/chutes/.bittensor/ rw, + /home/chutes/.bittensor/** rwlk, + + # Temp dir (emptyDir mount in k8s pod) + /tmp/ r, + /tmp/** rwlk, + + # System libraries + /usr/lib/** rm, + /usr/local/lib/** rm, + /etc/ld.so.cache r, + /etc/ld.so.conf r, + /etc/ld.so.conf.d/ r, + /etc/ld.so.conf.d/** r, + + # Minimal /proc access for the process itself + @{PROC}/@{pid}/fd/ r, + @{PROC}/@{pid}/maps r, + @{PROC}/@{pid}/status r, + @{PROC}/sys/kernel/ngroups_max r, + @{PROC}/sys/kernel/overflowgid r, + @{PROC}/sys/kernel/overflowuid r, + # cgroup CPU-limit detection at startup (thread-pool sizing). + @{PROC}/1/cgroup r, + /sys/fs/cgroup/ r, + /sys/fs/cgroup/** r, + /dev/null rw, + /dev/urandom r, + + # Logging + /dev/log w, + /run/systemd/journal/socket w, + /run/systemd/journal/dev-log w, + + # Network for proxy operations (TLS + attestation socket) + network inet stream, + network inet dgram, + network inet6 stream, + network inet6 dgram, + network unix stream, + network unix dgram, +} diff --git a/ansible/guest/roles/apparmor-hardening/files/profiles/sek8s.chute-log-shipper b/ansible/guest/roles/apparmor-hardening/files/profiles/sek8s.chute-log-shipper new file mode 100644 index 00000000..6eb90dd1 --- /dev/null +++ b/ansible/guest/roles/apparmor-hardening/files/profiles/sek8s.chute-log-shipper @@ -0,0 +1,95 @@ +# vim: ft=apparmor +# sek8s.chute-log-shipper — AppArmor profile for the chute-log-shipper service. +# Applied via systemd AppArmorProfile= directive (not auto-attached). +# Confines the service to exactly: chute pod logs, the CRI socket + k3s/crictl +# wrapper, the registry-tls leaf, the cursor dir, and outbound network. + +abi , + +include + +# attach_disconnected: the service runs in a systemd sandbox mount namespace, which leaves /dev +# nodes (e.g. /dev/null) "disconnected" from this profile's root — without the flag AppArmor fails +# the path lookup (error=-13) even in complain. (Confirmed against the live audit.) +profile sek8s.chute-log-shipper flags=(enforce, attach_disconnected) { + include + include + include + + # Python interpreter and venv (the sek8s package, installed under /opt/sek8s). mr on the venv + # because C-extension .so files are mmap'd. /opt/sek8s/src is READ-ONLY: it is RTMR3-measured, and + # the service runs with PYTHONDONTWRITEBYTECODE=1 (see the unit file) so no .pyc are written into + # the measured tree at runtime — a runtime write would drift RTMR3 on the next boot. + /opt/sek8s/ r, + /opt/sek8s/venv/bin/python3{,.*} mrix, + /opt/sek8s/venv/** mr, + /opt/sek8s/src/** r, + + # Environment file. + /etc/chute-log-shipper/** r, + + # The unit drop-in grants CAP_DAC_READ_SEARCH (kubelet pod log dirs are root-owned 0750); + # AppArmor must permit its use separately or log capture fails with EACCES. + capability dac_read_search, + + # Chute pod logs — read-only, scoped to the chutes namespace pod dirs only. + /var/log/pods/ r, + /var/log/pods/chutes_*/ r, + /var/log/pods/chutes_*/** r, + + # Registry mTLS leaf — the egress client identity (read-only). + /run/chutes/registry-tls/ r, + /run/chutes/registry-tls/client.crt r, + /run/chutes/registry-tls/client.key r, + + # Cursor persistence — read/write. + /var/lib/chute-log-shipper/ rw, + /var/lib/chute-log-shipper/** rwk, + + # CRI socket for `k3s crictl` (via the restricted wrapper). + /run/k3s/containerd/containerd.sock rw, + + # Restricted crictl wrapper + the k3s binary it execs. /usr/local/bin/k3s is a symlink into the + # versioned data dir, and AppArmor mediates the RESOLVED path — so the real binary must be granted + # too, or the exec is denied in enforce. The k3s multi-call binary also shells out to the iptables + # backend on invocation (even for `crictl`), so those are granted to keep it functional; see the + # note below — a log reader touching iptables is worth revisiting. + /usr/local/bin/crictl-pods-helper rix, + /usr/local/bin/k3s mrix, + /var/lib/rancher/k3s/data/*/bin/k3s mrix, + /etc/rancher/k3s/config.yaml r, + /etc/machine-id r, + /bin/bash rix, + + # crictl reads its config here to learn the CRI (containerd) socket endpoint; without it + # `crictl pods` dies at config load before it can query anything. + /var/lib/rancher/k3s/agent/etc/ r, + /var/lib/rancher/k3s/agent/etc/crictl.yaml r, + + # System libraries. + /usr/lib/** rm, + /usr/local/lib/** rm, + /etc/ld.so.cache r, + /etc/ld.so.conf r, + /etc/ld.so.conf.d/** r, + + # Proc, sys, dev. + @{PROC}/** r, + @{sys}/** r, + /dev/null rw, + /dev/tty rw, + /dev/urandom r, + owner @{PROC}/@{pid}/fd/ r, + + # Network egress to the validator (mTLS) + DNS. + network inet stream, + network inet dgram, + network inet6 stream, + network inet6 dgram, + network unix stream, + network unix dgram, + + # Subprocesses (the crictl wrapper / k3s) inherit this profile. + signal send peer=sek8s.chute-log-shipper, + signal receive peer=sek8s.chute-log-shipper, +} diff --git a/ansible/guest/roles/apparmor-hardening/files/profiles/sek8s.deny-sensitive-default b/ansible/guest/roles/apparmor-hardening/files/profiles/sek8s.deny-sensitive-default new file mode 100644 index 00000000..d5cd1420 --- /dev/null +++ b/ansible/guest/roles/apparmor-hardening/files/profiles/sek8s.deny-sensitive-default @@ -0,0 +1,45 @@ +# vim: ft=apparmor +# sek8s.deny-sensitive-default — the blanket profile for common shells, interpreters, and +# data-transfer tools. This is what an ESCAPED OR STRAY shell gets: a chute pod that breaks out of +# its container, an exploited service that shells out, an injected command. It must therefore be +# the most restrictive shell profile in the system. +# +# It denies BOTH protected assets: +# * the model cache — private model weights, the highest-value asset on the box +# * /run/chutes — miner seed, hotkey, mTLS keys, k3s secretbox config +# and additionally hides /run/k3s-init, where the k3s-post-start wrapper stages per-script +# secrets (see sek8s.k3s-init.* profiles). +# +# Services that legitimately need one of those assets do NOT get an exception here — AppArmor +# gives `deny` precedence over any allow, so an exception is impossible to express in this file. +# They get their own profile composed from sek8s-shell-base plus whichever denies still apply, +# applied via systemd AppArmorProfile= or aa-exec, which overrides this auto-attachment. +# +# python3 is intentionally NOT in the attachment list: confining it would require explicit +# overrides for every Python-based systemd service. Python launched from a confined shell inherits +# this profile via ix. + +abi , + +include + +@{confined_bins} = /usr/bin/{bash,dash,cat,cp,tar,rsync,scp,curl,wget,perl,dd,socat,nc,ncat} /bin/{sh,dash} + +profile sek8s.deny-sensitive-default @{confined_bins} flags=(enforce) { + include + include + include + + @{confined_bins} mrix, + + # A stray shell must not be able to reboot or power off the box. Measured init scripts DO need + # this (their fail-closed handler powers the VM off on a security-critical failure), so the deny + # lives here rather than in sek8s-shell-base. + deny capability sys_boot, + + # The per-script staging area is invisible here. Only the sek8s.k3s-init.