Phase 5: self-hosted deployment (Helm, Terraform, SOPS, CD on tag) - #11
Merged
Conversation
Production now sets JasperFx.Profile.ResourceAutoCreate = AutoCreate.None inside the existing CritterStackDefaults call, next to GeneratedCodeMode. The wolverine schema comes from an explicit `resources setup` deploy step from here on, not from a race between replicas at startup (TECH-DEBT #4). Development is unaffected: FinmyApiFactory boots against a fresh Testcontainers Postgres and needs auto-create to keep working. docker-compose.yml adds a one-shot wolverine-setup service, built from the same final image with command ["resources", "setup"], running after migrate and before api. Without it, ASPNETCORE_ENVIRONMENT=Production in that file would now start api against a wolverine schema that does not exist, breaking the 'clone and docker compose up, no manual steps' guarantee. Verified end to end: docker compose down -v to wipe the volume, then up --build. wolverine-setup's logs show it creating the wolverine schema and its tables; api starts clean afterward and /health/live and /health/ready both return 200.
One chart, no sub-charts, templates named flat by component. Covers the Deployment/Service/Ingress/HPA/PDB the roadmap asks for, wired to the existing /health/live and /health/ready endpoints, plus Postgres, Redis and MinIO as StatefulSets/a Deployment on k3s's local-path storage class, and a nightly pg_dump CronJob for backup (ADR-0018). EF migrations and Wolverine's 'resources setup' run from one plain, non-hook Job rather than a Helm lifecycle hook: a pre-install hook blocks the whole release before Postgres exists in the same chart, and a post-install hook under --wait never fires because Helm is already blocked on the Deployment's rollout. Everything applies together instead, and the API's readiness probe (already checking GetPendingMigrationsAsync per ADR-0014) is what gates traffic on the Job finishing. The API's RollingUpdate strategy sets maxUnavailable: 0 so old replicas keep serving during that window. readOnlyRootFilesystem stays false: TypeLoadMode.Auto (ADR-0013) writes generated handler code to /app/Internal on first handler invocation, which a read-only rootfs would break at runtime, not startup. Verified with real helm and kubeconform binaries (installed via winget, none present in this environment beforehand): helm lint passes, helm template renders all 14 resources with dummy secrets, and every rendered manifest validates cleanly under kubeconform -strict.
Provisions one hcloud_server (default cx22, nbg1, Ubuntu 24.04), a firewall (22 and 6443 restricted to an admin CIDR, 80/443 open), and A/AAAA records via the Hetzner DNS provider. k3s and cert-manager install through the server's own cloud-init rather than a separate Ansible or Helm-from-a-workstation step: k3s's bundled Traefik, local-path and metrics-server are left at their defaults, and cert-manager plus a Let's Encrypt ClusterIssuer land as manifests k3s auto-applies on its own. Written and terraform validate-clean against real provider schemas (confirmed with the actual hcloud and hetznerdns providers via terraform init/validate, installed via winget for this session), but never applied: no server exists yet, so 'deployed and reachable over HTTPS' stays open in ROADMAP.md section 6. deploy/terraform/README.md says so plainly and documents the one manual step this configuration deliberately does not automate (copying the post-apply kubeconfig into the KUBE_CONFIG Actions secret). .gitignore gains .terraform/, *.tfstate*, *.tfvars (keeping *.tfvars.example) and *.agekey; the Terraform lock file is committed.
deploy/values-prod.sops.yaml holds the four values values.yaml cannot ship a real default for: Jwt__SigningKey, the Postgres password, and the MinIO root credentials. Keys stay readable in diffs; only the values are encrypted. .sops.yaml at the repo root points at one age recipient; the matching private key lives only in the SOPS_AGE_KEY GitHub Actions secret and the repo owner's password manager, never in this repository. Closes TECH-DEBT #15: the compose file's inline Jwt__SigningKey default stays exactly what it always was, a local-development convenience, no longer standing in for a real secret-management story anywhere. The path_regex uses a [/\] character class instead of a literal forward slash, because sops resolves the target path with the host OS's separator before matching, so a literal '/' silently fails to match on Windows (reproduced and fixed in this session; the same rule works unmodified in Linux CI). Verified end to end with real sops and age binaries (installed via winget, neither present in this environment beforehand): generated the actual age keypair used here, encrypted deploy/values-prod.sops.yaml against its public key, decrypted with the private key to confirm the round-trip, and fed the decrypted output into helm template to confirm it satisfies the chart's required-value checks.
New chart-and-infra job, parallel to build-and-test, no .NET involved. This is the only place any of deploy/ gets checked: no helm, kubectl, terraform, sops or age binary is assumed to exist on a dev machine. helm lint, then helm template with placeholder secret values piped through kubeconform -strict for schema validation against real Kubernetes API schemas (no live cluster needed for either check). terraform fmt -check and terraform init -backend=false && validate against the real provider schemas. A final grep confirms deploy/values-prod.sops.yaml is still actually encrypted, so a plaintext secret landing in the repo fails the build rather than merging quietly.
The Helm migrate Job needs an image for its EF migration initContainer, and release.yml only ever published target: final. Adds a second build-push-action step publishing target: migrator to ghcr.io/<repo>/migrator:<tag>. New deploy job, needs: publish, gated on the DEPLOY_ENABLED repository variable so tagging a release does not fail red while no cluster exists yet to deploy to (deploy/terraform has never been applied). Decrypts deploy/values-prod.sops.yaml with the SOPS_AGE_KEY secret, configures kubectl from the KUBE_CONFIG secret, and runs helm upgrade --install with --wait. environment: production scopes both secrets to this job.
Records the four decisions the roadmap's Phase 5 paragraph asked for: Postgres/Redis/MinIO in-cluster with a pg_dump CronJob (versus CloudNativePG or managed services), SOPS+age (versus Sealed Secrets or a cloud KMS), Actions-on-tag helm upgrade (versus Argo CD), and k3s's bundled Traefik (versus ingress-nginx). Also records the migrate Job ordering decision (plain Job, not a Helm hook, and why both hook lifecycles deadlock on a fresh cluster) and states plainly what none of this has been verified against: a live cluster.
…E.md Marks Phase 5 done in ROADMAP.md with what actually shipped and what did not (no server, so 'deployed and reachable over HTTPS' stays open in section 6). Drops TECH-DEBT #4 and #15, both closed this phase, and notes #17's drift widened by one more compose service. Adds deploy/ to README's project layout and ADR-0018 to its architecture-decisions list. CLAUDE.md's 'What exists'/'What does not exist yet' sections describe the deploy/ shape and its verification status.
deploy/README.md and deploy/terraform/README.md each document their own piece; neither sequences the full path from having a Hetzner account to the app answering HTTPS on a real domain. RUNBOOK.md is that sequence: provision, make the GHCR images pullable, extract a kubeconfig for Actions, confirm SOPS_AGE_KEY, point the Ingress at the real host, flip DEPLOY_ENABLED, tag, verify. Written against no live cluster, same caveat as the rest of deploy/ this phase; the closing section names the three failure modes ADR-0018 already flagged as unverified so a first-deploy troubleshooter starts there instead of assuming the chart is wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 5: self-hosted deployment
Closes the Phase 5 paragraph in
docs/ROADMAP.mdsection 5. See ADR-0018 for the decisions.What's here
deploy/helm/finmy): API Deployment/Service/Ingress/HPA/PDB wired to/health/liveand/health/ready, plus in-cluster Postgres/Redis/MinIO and a nightlypg_dumpbackup CronJob.resources setuprun from one plain, non-hook Job (see ADR-0018 for why a Helm hook deadlocks either way).deploy/terraform): Hetzner VPS, firewall, DNS, k3s + cert-manager via cloud-init.deploy/values-prod.sops.yaml), closing TECH-DEBT Bump the nuget-minor-and-patch group with 25 updates #15.Program.cssetsResourceAutoCreate = AutoCreate.Nonein production, closing TECH-DEBT build(deps): bump docker/login-action from 3 to 4 #4.docker-compose.ymlgains awolverine-setupservice so plaindocker compose upkeeps working.chart-and-infra) validates the chart and Terraform on every PR.release.ymlpublishes the migrator image and adds a tag-triggereddeployjob, inert behind aDEPLOY_ENABLEDrepo variable.Verified this session
dotnet build/dotnet test(112 tests) /scripts/coverage.ps1(64.26%/56.36%, above the 60/54 floor) all green.docker compose down -v && up --build:wolverine-setupcreates thewolverineschema itself,apistarts clean, both health endpoints return 200 — confirmsAutoCreate.Noneactually took effect.helm,terraform,sopsandagebinaries (none installed beforehand, added via winget for this session):helm lintclean,helm templaterenders all 14 resources,kubeconform -strictvalidates every one,terraform validatepasses against real provider schemas, and the full SOPS-decrypt-into-helm templatepipeline round-trips correctly.Not verified (no cluster exists)
Ingress, cert-manager issuance, HPA scaling, PDB behavior, the backup CronJob's restore path,
terraform apply, and thedeployjob end to end. All flagged in ADR-0018 anddeploy/terraform/README.md. "Deployed and reachable over HTTPS" stays unchecked inROADMAP.mdsection 6 by design — Terraform is written and validated, never applied.Also flagged, not fixed
TECH-DEBT #17 (
docker-compose.local.ymldrift) widens by one more service (wolverine-setup) added to the base compose file. No phase assigned to #17; left alone per the session brief.