feat(infra): opt-in Cloudflare R2 remote-state backend (#167)

* feat(infra): add opt-in Cloudflare R2 remote-state backend

Closes the last single point of failure in the protection story: state was
local-only, so losing the operator's machine/tfstate meant losing the ability
to plan, reconcile, or safely destroy the stack — worse now that the VPS
carries delete_protection.

Adds a pre-wired, opt-in path to Cloudflare R2 (S3-compatible, no egress fees,
an account we already use). Default clone stays on local state and works
unchanged; enabling remote state is a documented ~2-minute step.

- main.tf: commented backend "s3" block pre-filled with the R2-specific flags
  (region=auto, use_path_style, skip_* preflight, skip_s3_checksum) and
  use_lockfile for native locking — no DynamoDB. Deployment-specific values
  (bucket, key, endpoints) come from backend.hcl via partial config.
- backend.hcl.example: template for the per-deployment values; credentials
  stay in AWS_* env vars, never on disk.
- .gitignore: ignore backend.hcl, keep the example.
- README: "Remote state (optional)" section with the migrate-state steps.

Verified on OpenTofu 1.12.0: fmt + validate clean; the s3 backend accepts
every argument (a scratch init parsed the full config and only failed at the
network layer against a deliberately-invalid endpoint).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(infra): align tofu provisioning docs with VPS protection + R2 state

The provisioning-with-tofu page lagged behind the infra changes:

- "Destroying" told users `tofu destroy` wipes the server — now WRONG, since
  prevent_destroy + Hetzner delete_protection refuse it. Rewritten to document
  the deliberate two-step gate-lowering (apply to lift locks, then destroy).
- "State management" showed a bare s3 block (bucket/key/region only). Replaced
  with the pre-wired Cloudflare R2 flow this PR adds: backend.hcl partial
  config, env-var credentials, use_lockfile, and init -migrate-state.
- Prerequisites now call out the OpenTofu 1.12+ requirement.
- terraform.tfvars shape documents prevent_server_destroy.
- Design choices: updated the state bullet and added a defense-in-depth bullet
  covering both protection layers.

Verified: astro build succeeds, check:rendered-markdown passes, and
check:fragments confirms the new #destroying / #state-management /
#design-choices anchors resolve.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(security): allowlist two historical false-positive gitleaks findings

The local pre-push hook runs full-history `gitleaks detect` (CI is
path-filtered, so these slipped past on unrelated pushes). Two findings live
only in old commit diffs; both current files are already clean:

- add-service-to-compose.mdx: a Meilisearch example MEILI_MASTER_KEY value
  before it became the `<set-from-your-secret-manager>` placeholder.
- Auth.session.mutations.utils.test.ts: fixture token `abcdef1234567890`,
  since changed to `test-challenge-token-stub`.

Pin both by fingerprint in .gitleaksignore — narrowly scoped to these exact
commits/lines; rules stay active everywhere else and for HEAD. Verified
`gitleaks detect` (v8.30.1, the CI-pinned version) now reports no leaks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: agjs

.gitleaksignore

@@ -0,0 +1,19 @@
+# gitleaks fingerprint allowlist for specific HISTORICAL findings.
+#
+# Format: <commit>:<file>:<rule>:<line>. Each entry ignores exactly one
+# finding at one commit — rules stay fully active for every other commit and
+# for the current working tree. Used for false positives that live only in old
+# commit diffs (the current files are already clean), where a line/value regex
+# in .gitleaks.toml can't match because the offending value no longer exists
+# in HEAD. The repo-wide allowlist for current placeholders lives in
+# .gitleaks.toml; this file is only for pinned past commits.
+
+# Meilisearch recipe: an intermediate commit had a non-empty MEILI_MASTER_KEY
+# example value before it was changed to the `<set-from-your-secret-manager>`
+# placeholder (now allowlisted in .gitleaks.toml). Docs example, never a real key.
+9262aa98aebe8b93d055b28a98fb44b72097935b:apps/docs/src/content/docs/recipes/add-service-to-compose.mdx:generic-api-key:100
+
+# Auth session test: the fixture token was `abcdef1234567890` at this commit
+# (looks hex-key-ish to generic-api-key) and has since been changed to the
+# clearer `test-challenge-token-stub`. Test stub, never a real secret.
+901fd55367183293112f393dbc4cb99127af3913:apps/ui/src/features/auth/Auth.session.mutations.utils.test.ts:generic-api-key:13

apps/docs/src/content/docs/topics/provisioning-with-tofu.mdx

@@ -17,7 +17,7 @@ You'll need the following before starting:
 - **Cloudflare API token**: Cloudflare, My Profile, API Tokens, Custom Token. Scope: `Zone:DNS:Edit`, `Zone:Zone Settings:Edit`, `Zone:Rulesets:Edit` on the target zone.
 - **Cloudflare zone ID**: Zone overview page in the dashboard, right sidebar.
 - **SSH key**: Run `ssh-keygen -t ed25519` if you don't have one. You'll paste the `.pub` contents.
-- **OpenTofu binary**: `brew install opentofu` on macOS. See [install docs](https://opentofu.org/docs/intro/install/) for other OSes.
+- **OpenTofu binary**: `brew install opentofu` on macOS — version **1.12 or newer** (the stack uses variable-driven `prevent_destroy` and native state locking). See [install docs](https://opentofu.org/docs/intro/install/) for other OSes.
 
 ## Apply
 
@@ -97,6 +97,10 @@ domain               = "boringstack.example"
 vps_type     = "cx32"          # 4 vCPU / 8 GB
 vps_location = "fsn1"
 
+# Protection (default true). Blocks tofu *and* Hetzner from destroying the
+# VPS; set false only for a deliberate rebuild or teardown.
+prevent_server_destroy = true
+
 # Stack secrets
 jwt_secret        = "..."  # 32+ chars
 postgres_password = "..."
@@ -126,20 +130,31 @@ Everything in `terraform.tfvars.example` ships with comments explaining what it'
 
 ## State management
 
-For a single operator: state file is local and gitignored (default config).
+State defaults to a local `terraform.tfstate` — gitignored, and fine for a single operator spinning something up quickly. But local state is a single point of failure: lose the file and you can no longer plan, reconcile, or safely destroy the stack — more so now that the VPS is delete-protected (see [Destroying](#destroying)).
 
-For a team: point the OpenTofu backend at S3 (or any S3-compatible store like Cloudflare R2, Backblaze B2, or Hetzner Object Storage). Add one block to `main.tf`:
+For anything long-lived, move state to **Cloudflare R2** (S3-compatible, no egress fees, an account you already have for DNS). `main.tf` ships a ready-to-uncomment `backend "s3"` block pre-filled with the R2-specific flags (`use_path_style`, the `skip_*` preflight toggles, `skip_s3_checksum`) and `use_lockfile = true` for native locking — no DynamoDB table. Deployment-specific values live in a gitignored `backend.hcl`, copied from `backend.hcl.example`:
 
 ```hcl
-terraform {
-  backend "s3" {
-    bucket = "boringstack-tofu-state"
-    key    = "boringstack/terraform.tfstate"
-    region = "..."
-  }
+# backend.hcl
+bucket = "boringstack-tfstate"
+key    = "bootstrap/terraform.tfstate"
+endpoints = {
+  s3 = "https://<CLOUDFLARE_ACCOUNT_ID>.r2.cloudflarestorage.com"
 }
 ```
 
+Create an R2 bucket and an R2 API token (Object Read & Write), then migrate the existing local state:
+
+```bash
+cp backend.hcl.example backend.hcl   # fill in bucket + account id
+export AWS_ACCESS_KEY_ID=<r2 access key id>
+export AWS_SECRET_ACCESS_KEY=<r2 secret access key>
+# uncomment the backend "s3" block in main.tf, then:
+tofu init -backend-config=backend.hcl -migrate-state
+```
+
+Credentials stay in the `AWS_*` env vars, never on disk. Hetzner Object Storage, Backblaze B2, or AWS S3 work the same way — only `endpoints.s3` (and the `skip_*`/`use_path_style` flags, for true AWS) differ.
+
 The state file contains secrets (cloud-init renders with sensitive values). Encrypt at rest and restrict bucket access. Same security posture as everywhere else in BoringStack.
 
 ## Updating
@@ -176,11 +191,17 @@ The `bootstrap` module talks to cloud-init, which every major cloud accepts. Swa
 
 ## Destroying
 
+The VPS is protected by two independent layers (see [Design choices](#design-choices)): OpenTofu's `prevent_destroy` and Hetzner's API-level `delete_protection`/`rebuild_protection`. Both are driven by the `prevent_server_destroy` variable, which defaults to `true`. So a plain `tofu destroy` is **refused** — by design, you can't nuke production in a single command.
+
+To deliberately tear down, lower the gate first, then destroy:
+
 ```bash
-tofu destroy
+# Set prevent_server_destroy = false in terraform.tfvars (or pass -var on each command).
+tofu apply    # lifts the Hetzner delete/rebuild locks in place
+tofu destroy  # now succeeds
 ```
 
-This wipes the Hetzner server, removes Cloudflare records, deletes the firewall and SSH key, and reverts Cloudflare zone settings to defaults. The state file remains. Run `rm terraform.tfstate*` for full cleanup.
+The `apply` step is required: Hetzner won't honour a delete while the API lock is still set, so the lock must be lifted by an apply *before* `destroy` can remove the server. This wipes the Hetzner server, removes Cloudflare records, deletes the firewall and SSH key, and reverts Cloudflare zone settings to defaults. With remote state the state object stays in the bucket; with local state, run `rm terraform.tfstate*` for full cleanup.
 
 ## Troubleshooting
 
@@ -209,7 +230,8 @@ The runtime repos work fine without this one. It's a convenience layer, not a de
 - **Sane Cloudflare zone defaults**: SSL strict, HSTS 6 months, TLS min 1.2, browser integrity on. Matches what `production-labels.yml` expects. Each setting is one override away.
 - **DNS: apex and www only**: one A/AAAA pair on the apex serves both the SPA and `/api/*` via same-origin path routing. `www.` is a CNAME to apex with a redirect rule. No `api.` subdomain. Traefik path-routes `/api/*` on the same host.
 - **Edge bot-blocking, on by default**: a single Cloudflare WAF custom rule (`http_request_firewall_custom`) blocks common scanner probes — `/.env`, `/.git/`, `/wp-admin`, `/xmlrpc`, … — at the edge, so they never reach Traefik or the API. Configurable via `bot_block_paths`; disable with `enable_bot_blocking = false`. Works on the Cloudflare Free plan.
-- **State stays local by default**: single-operator default. An S3 backend block is one paste away for teams.
+- **State stays local by default, R2 one step away**: single-operator default is a local state file. `main.tf` ships a pre-wired, commented Cloudflare R2 backend (S3-compatible, native locking, no DynamoDB); enabling it is `cp backend.hcl.example backend.hcl`, uncomment the block, and `tofu init -migrate-state`. See [State management](#state-management).
+- **Production is hard to nuke, on purpose**: the VPS carries OpenTofu `prevent_destroy` *and* Hetzner API-level `delete_protection`/`rebuild_protection`, both gated behind a single `prevent_server_destroy` variable (default on). Defense in depth — destroying or replacing the box (and the Postgres, ACME, and GlitchTip volumes it holds) takes a deliberate, explicit gate-lowering, not a stray `tofu destroy` or a misclick in the Hetzner console. See [Destroying](#destroying).
 - **Secrets in terraform.tfvars (gitignored)**: same pragmatic floor as `compose/.env`. Upgrade to a secret manager when team size demands it.
 - **Outputs print, never side-effect**: apply prints the IP, ssh command, and site URL. Never auto-opens anything.
 - **Optional bootstrap repo, separate from infra-compose**: same logic as the planned Kubernetes template. Separation lets operators skip the tool entirely.

infra/bootstrap/.gitignore

@@ -15,6 +15,10 @@ crash.*.log
 *.tfvars.json
 !terraform.tfvars.example
 
+# Remote-state backend config (bucket + account id). Commit the example only.
+backend.hcl
+!backend.hcl.example
+
 # Editor / OS
 .DS_Store
 *.swp

infra/bootstrap/README.md

@@ -22,6 +22,24 @@ Part of [BoringStack](https://boringstack.xyz).
 
 For documentation on how to run, configure, and extend the OpenTofu bootstrap, see the [OpenTofu provisioning docs](https://boringstack.xyz/topics/provisioning-with-tofu/).
 
+## Remote state (optional)
+
+State defaults to a local `terraform.tfstate`, which is fine for a quick spin-up but a single point of failure for anything long-lived — lose the file and you can no longer plan, reconcile, or safely destroy the stack. To move state to **Cloudflare R2** (S3-compatible, no egress fees, native locking — no DynamoDB):
+
+1. Create an R2 bucket and an R2 API token with **Object Read & Write**.
+2. `cp backend.hcl.example backend.hcl` and fill in the bucket name + account id (`backend.hcl` is gitignored).
+3. Export the token as S3 credentials:
+   ```sh
+   export AWS_ACCESS_KEY_ID=<r2 access key id>
+   export AWS_SECRET_ACCESS_KEY=<r2 secret access key>
+   ```
+4. Uncomment the `backend "s3"` block in `main.tf`, then migrate:
+   ```sh
+   tofu init -backend-config=backend.hcl -migrate-state
+   ```
+
+R2 uses the `s3` backend with AWS-specific preflight disabled; the block in `main.tf` is pre-filled with the right flags. Hetzner Object Storage or AWS S3 work the same way — only the `endpoints.s3` value (and the `skip_*`/`use_path_style` flags, for true AWS) differ.
+
 ## License
 
 MIT

infra/bootstrap/backend.hcl.example

@@ -0,0 +1,20 @@
+# Cloudflare R2 remote-state backend config (OPT-IN).
+#
+# Copy to backend.hcl (gitignored) and fill in the values below, then enable
+# the `backend "s3"` block in main.tf and run:
+#
+#   export AWS_ACCESS_KEY_ID=<r2 access key id>
+#   export AWS_SECRET_ACCESS_KEY=<r2 secret access key>
+#   tofu init -backend-config=backend.hcl -migrate-state
+#
+# Credentials are NOT stored here — they come from the AWS_* env vars above so
+# this file (and backend.hcl) stay safe to keep on disk. The bucket name and
+# account id are not secrets, but backend.hcl is gitignored anyway.
+
+bucket = "boringstack-tfstate"
+key    = "bootstrap/terraform.tfstate"
+
+endpoints = {
+  # Find your account id in the Cloudflare dashboard → R2 → "S3 API" endpoint.
+  s3 = "https://<CLOUDFLARE_ACCOUNT_ID>.r2.cloudflarestorage.com"
+}

infra/bootstrap/main.tf

@@ -29,6 +29,37 @@ terraform {
       version = "~> 3.4"
     }
   }
+
+  # --------------------------------------------------------------------------
+  # Remote state (OPT-IN). Default is local state — a terraform.tfstate file on
+  # the operator's machine. That's fine for a quick spin-up, but it is a single
+  # point of failure for anything long-lived: lose the machine (or the file)
+  # and you lose the ability to plan, reconcile, or safely destroy the stack —
+  # made worse now that the VPS carries delete_protection. Move state to
+  # Cloudflare R2 (S3-compatible, no egress fees, an account you already have):
+  #
+  #   1. Create an R2 bucket and an R2 API token (Object Read & Write).
+  #   2. cp backend.hcl.example backend.hcl   # gitignored; fill in bucket + account id
+  #   3. export AWS_ACCESS_KEY_ID=<r2 access key id>
+  #      export AWS_SECRET_ACCESS_KEY=<r2 secret access key>
+  #   4. Uncomment the block below, then migrate the existing local state:
+  #        tofu init -backend-config=backend.hcl -migrate-state
+  #
+  # R2 speaks the S3 API, so we use the s3 backend with AWS-specific preflight
+  # disabled and native lockfile locking (S3 conditional writes — no DynamoDB;
+  # requires OpenTofu >= 1.10, already guaranteed by required_version above).
+  # --------------------------------------------------------------------------
+  # backend "s3" {
+  #   region                      = "auto" # R2 ignores it, but the s3 backend still requires a value
+  #   use_lockfile                = true   # state locking via S3 conditional writes (no DynamoDB)
+  #   use_path_style              = true
+  #   skip_credentials_validation = true
+  #   skip_region_validation      = true
+  #   skip_requesting_account_id  = true
+  #   skip_metadata_api_check     = true
+  #   skip_s3_checksum            = true # R2 rejects the AWS streaming-checksum trailer
+  #   # bucket, key, endpoints.s3 -> backend.hcl (see backend.hcl.example)
+  # }
 }
 
 provider "hcloud" {