Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 15 additions & 8 deletions docs/TENANTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,11 +64,15 @@ tenant ships a SOPS-encrypted Secret.

### App secrets (DB creds, API keys, … — only for tenants that need them)

A tenant gets a store + Vault role **only if it needs app secrets** — a static
site gets none. The **tenant owns its app secrets end-to-end**; the platform only
provisions the store + isolation and never seeds a tenant's app values. A value is
only ever introduced via a committed resource (a generator, or the tenant's own
push), **never written to OpenBao out of band**.
A tenant gets a store + Vault role **only if it needs app secrets** — a purely
static site with no integrations gets none. The **tenant owns its app secrets
end-to-end**; the platform only provisions the store + isolation and never seeds
a tenant's app values. A *generated* value is only ever introduced via a
committed resource (a generator, or the tenant's own push), never written to
OpenBao out of band; the one exception is an **externally-issued credential**
(a third-party API key, e.g. ascoachingogvaner's simply.com DNS credentials at
`apps/ascoachingogvaner/simply`), which the maintainer seeds into the tenant's
path once — there is nowhere to generate it from.

- **Tenant (`deploy/`)** — own your secret end-to-end. Your `edit` RoleBinding
aggregates `external-secrets-tenant-edit`, so you may create `Password`
Expand Down Expand Up @@ -124,9 +128,10 @@ artifacts produced by that trusted workflow are ever reconciled onto the cluster

## 5. Register the tenant on the platform

Add `k8s/bases/apps/<tenant>/` — copy `ascoachingogvaner/` (a static tenant with
no app secrets) or `wedding-app/` (a tenant with app secrets + a namespaced
SecretStore) and rename — with:
Add `k8s/bases/apps/<tenant>/` — copy `wedding-app/` (a tenant with app secrets
+ a namespaced SecretStore) or `ascoachingogvaner/` (a static tenant that also
runs a **tenant-owned external-dns** for its custom domain, with the extra
`external-dns-*` grants below) and rename — with:

| File | Purpose |
|---|---|
Expand All @@ -137,6 +142,8 @@ SecretStore) and rename — with:
| `networkpolicy.yaml` | Cilium policy: ingress from the Gateway on the app port; egress DNS (+ CNPG/metrics if needed) |
| `ghcr-auth-externalsecret.yaml` | OpenBao-backed `ExternalSecret` (shared `openbao` ClusterSecretStore, key `infrastructure/ghcr/auth`) producing the `ghcr-auth` pull secret |
| `secretstore.yaml` | *Only if the tenant needs app secrets* — namespaced `SecretStore` (`kind: SecretStore`, name `openbao`) authenticating via the tenant's Vault role (mirror `wedding-app/`) |
| `external-dns-rbac.yaml` | *Only if the tenant runs its own external-dns for a tenant-owned domain* — binds the tenant's `external-dns` SA to the `tenant-external-dns(-global)` ClusterRoles (HTTPRoutes in its namespace, the shared Gateway in kube-system, namespaces) — mirror `ascoachingogvaner/` |
| `external-dns-networkpolicy.yaml` | *Same condition* — egress for the external-dns pods: kube-apiserver + the DNS provider's API, FQDN-pinned |
| `sync.yaml` | `OCIRepository` (semver `>=1.0.0`, cosign `verify`) + `Kustomization` (prune, `serviceAccountName: <tenant>`) |

In `sync.yaml`, update the `name`/`namespace`/`url`
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Egress for the tenant-owned external-dns pods (deployed from the tenant's
# own deploy/ artifact). The namespace-wide allow-ascoachingogvaner policy
# only opens DNS, so the controller's two upstreams are allowed here, scoped
# to its pods: the kube-apiserver (HTTPRoute/Gateway watches) and the
# simply.com API the webhook sidecar manages DNS records through — FQDN-pinned
# rather than world:443 so a compromised pod cannot reach arbitrary external
# services (same rationale as the platform external-dns policy).
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: allow-ascoachingogvaner-external-dns
namespace: ascoachingogvaner
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: external-dns
egress:
- toEntities:
- kube-apiserver
- toFQDNs:
- matchName: "api.simply.com"
toPorts:
- ports:
- port: "443"
protocol: TCP
# DNS resolution via Cilium's L7 DNS proxy, restricted to the names this
# pod actually needs to resolve (closes off DNS-tunneling exfil).
- toEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: kube-system
k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: UDP
- port: "53"
protocol: TCP
rules:
dns:
- matchName: "api.simply.com"
- matchPattern: "*.cluster.local"
52 changes: 52 additions & 0 deletions k8s/bases/apps/ascoachingogvaner/external-dns-rbac.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# RBAC for the tenant-owned external-dns instance ascoachingogvaner ships in
# its own deploy/ artifact (manages its simply.com zone from its HTTPRoute
# hostnames). The tenant creates the dedicated `external-dns` ServiceAccount
# itself; only these grants are platform territory — a tenant's `edit` role
# cannot read the shared Gateway in kube-system, list namespaces, or create
# RBAC. Capabilities live in
# k8s/bases/infrastructure/cluster-roles/tenant-external-dns.yaml.
---
# HTTPRoute reads, limited to the tenant's own namespace.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: ascoachingogvaner-external-dns
namespace: ascoachingogvaner
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tenant-external-dns
subjects:
- kind: ServiceAccount
name: external-dns
namespace: ascoachingogvaner
---
# Gateway reads, limited to kube-system (where the shared `platform` Gateway
# lives) — external-dns resolves route targets from the Gateway's LB address.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: ascoachingogvaner-external-dns
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tenant-external-dns
subjects:
- kind: ServiceAccount
name: external-dns
namespace: ascoachingogvaner
---
# Namespace reads (cluster-scoped informer required by the gateway source).
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ascoachingogvaner-external-dns
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tenant-external-dns-global
subjects:
- kind: ServiceAccount
name: external-dns
namespace: ascoachingogvaner
3 changes: 3 additions & 0 deletions k8s/bases/apps/ascoachingogvaner/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,12 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- external-dns-networkpolicy.yaml
- external-dns-rbac.yaml
- ghcr-auth-externalsecret.yaml
- namespace.yaml
- rolebinding.yaml
- secretstore.yaml
- serviceaccount.yaml
- sync.yaml
- networkpolicy.yaml
32 changes: 32 additions & 0 deletions k8s/bases/apps/ascoachingogvaner/secretstore.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Tenant-scoped, namespaced SecretStore for ascoachingogvaner — provisioned by
# the platform so the tenant only has to add an ExternalSecret in its own
# deploy/ artifact to read secrets under its Vault path (today: the simply.com
# API credentials its tenant-owned external-dns consumes).
#
# Authenticates as the tenant's own `ascoachingogvaner` ServiceAccount via the
# dedicated `ascoachingogvaner` OpenBao Kubernetes auth role
# (k8s/bases/infrastructure/vault-config/job.yaml), which carries only the
# path-scoped `app-ascoachingogvaner` policy — read + seed-write limited to
# secret/{data,metadata}/apps/ascoachingogvaner/*. The tenant can therefore never reach
# infra or another tenant's secrets, unlike the shared cluster-scoped
# `openbao` ClusterSecretStore (which the `restrict-tenant-secret-stores`
# Kyverno policy blocks tenant ExternalSecrets from using).
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: openbao
namespace: ascoachingogvaner
labels:
app.kubernetes.io/managed-by: ksail
spec:
provider:
vault:
server: "http://openbao.openbao.svc.cluster.local:8200"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "ascoachingogvaner"
serviceAccountRef:
name: "ascoachingogvaner"
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,4 @@ resources:
- cnpg-tenant-edit.yaml
- external-secrets-tenant-edit.yaml
- gateway-tenant-edit.yaml
- tenant-external-dns.yaml
43 changes: 43 additions & 0 deletions k8s/bases/infrastructure/cluster-roles/tenant-external-dns.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Read capabilities for a TENANT-OWNED external-dns instance (a tenant runs
# its own external-dns Deployment in its namespace to manage a tenant-owned
# DNS zone — e.g. ascoachingogvaner's simply.com domain). The gateway-httproute
# source needs three reads a tenant's `edit` RoleBinding cannot provide:
# HTTPRoutes (its own namespace), the shared Gateway (kube-system, to resolve
# the LB address routes attach to), and Namespaces (cluster-wide informer for
# listener allowedRoutes selectors).
#
# Two ClusterRoles so the grant surface stays minimal:
# - `tenant-external-dns` carries the namespaced rules and is bound via
# RoleBindings (tenant namespace → httproutes; kube-system → gateways),
# so neither resource is readable cluster-wide.
# - `tenant-external-dns-global` carries only the namespace read and is
# bound via a ClusterRoleBinding (namespaces are cluster-scoped).
# Per-tenant bindings live in the tenant's registration dir
# (k8s/bases/apps/<tenant>/external-dns-rbac.yaml).
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tenant-external-dns
rules:
- apiGroups: ["gateway.networking.k8s.io"]
resources:
- gateways
- httproutes
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tenant-external-dns-global
rules:
- apiGroups: [""]
resources:
- namespaces
verbs:
- get
- list
- watch
42 changes: 41 additions & 1 deletion k8s/bases/infrastructure/vault-config/job.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -401,6 +401,34 @@ spec:
}
POLICY

# Read-only view of ascoachingogvaner's app path for the shared
# ClusterSecretStore bundle: the cert-manager simply.com DNS01
# solver consumes the tenant-owned simply.com API credentials
# (apps/ascoachingogvaner/simply) to issue the tenant's
# certificate on the shared Gateway.
bao policy write app-ascoachingogvaner-readonly - <<'POLICY'
path "secret/data/apps/ascoachingogvaner/*" {
capabilities = ["read"]
}
POLICY

# Tenant-scoped policy for ascoachingogvaner's OWN namespaced
# SecretStore (k8s/bases/apps/ascoachingogvaner/secretstore.yaml)
# and the ExternalSecrets the tenant ships in its deploy/ artifact
# (today: the simply.com credentials for its tenant-owned
# external-dns). Read + seed-write, confined to
# secret/{data,metadata}/apps/ascoachingogvaner/* — mirrors
# app-wedding-app. Bound via the dedicated `ascoachingogvaner`
# Kubernetes auth role below.
bao policy write app-ascoachingogvaner - <<'POLICY'
path "secret/data/apps/ascoachingogvaner/*" {
capabilities = ["create", "update", "read"]
}
path "secret/metadata/apps/ascoachingogvaner/*" {
capabilities = ["create", "update", "read"]
}
POLICY

bao policy write infra-oidc-readonly - <<'POLICY'
path "secret/data/infrastructure/oidc/*" {
capabilities = ["read"]
Expand Down Expand Up @@ -509,7 +537,7 @@ spec:
bao write auth/kubernetes/role/external-secrets \
bound_service_account_names=external-secrets \
bound_service_account_namespaces=external-secrets \
policies=infra-backup-readonly,infra-dns-readonly,infra-monitoring-readonly,infra-hcloud-readonly,infra-tls-readonly,infra-ghcr-readonly,app-fleetdm-readonly,app-wedding-readonly,app-umami-readonly,infra-oidc-readonly,vault-seed-write \
policies=infra-backup-readonly,infra-dns-readonly,infra-monitoring-readonly,infra-hcloud-readonly,infra-tls-readonly,infra-ghcr-readonly,app-fleetdm-readonly,app-wedding-readonly,app-umami-readonly,app-ascoachingogvaner-readonly,infra-oidc-readonly,vault-seed-write \
ttl=1h

# Dedicated role for the fleetdm VaultDynamicSecret generator.
Expand Down Expand Up @@ -546,6 +574,18 @@ spec:
policies=app-wedding-app \
ttl=1h

# Dedicated tenant role for ascoachingogvaner — same shape as
# wedding-app above. Authenticates the tenant's own
# `ascoachingogvaner` ServiceAccount (used by its namespaced
# SecretStore + deploy/ ExternalSecret) and carries ONLY the
# path-scoped `app-ascoachingogvaner` policy, confining the
# tenant to secret/{data,metadata}/apps/ascoachingogvaner/*.
bao write auth/kubernetes/role/ascoachingogvaner \
bound_service_account_names=ascoachingogvaner \
bound_service_account_namespaces=ascoachingogvaner \
policies=app-ascoachingogvaner \
ttl=1h

# Snapshot CronJob needs health check read
bao write auth/kubernetes/role/vault-snapshot \
bound_service_account_names=vault-snapshot \
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,10 @@ metadata:
namespace: ascoachingogvaner
spec:
patches:
- target:
kind: HTTPRoute
name: ascoachingogvaner
patch: |
- op: replace
path: /spec/hostnames
value:
- ascoachingogvaner.platform.devantler.tech
# NOTE: the hostname-replace patch that used to live here is gone — the
# tenant repo now owns its public hostnames (ascoachingogvaner.dk + the
# ascoachingogvaner.platform.devantler.tech subdomain) in its own
# HTTPRoute, since the domain is tenant-specific, not platform config.
# Surface this tenant on the homepage dashboard. The HTTPRoute ships
# from an external OCI artifact (ghcr.io/devantler-tech/ascoachingogvaner),
# so the gethomepage.dev/* annotations have to be injected here rather
Expand All @@ -33,5 +29,5 @@ spec:
gethomepage.dev/description: Personal/business static site.
gethomepage.dev/group: Customer Sites
gethomepage.dev/icon: mdi-account-tie
gethomepage.dev/href: https://ascoachingogvaner.platform.devantler.tech
gethomepage.dev/href: https://ascoachingogvaner.dk
gethomepage.dev/pod-selector: app.kubernetes.io/name=ascoachingogvaner
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Serve the tenant-owned ascoachingogvaner.dk domain on the shared platform
# Gateway. SNI picks the most specific listener, so these terminate TLS with
# the domain's own certificate while every other hostname keeps hitting the
# hostname-less `https` listener with the ${domain} wildcard cert. JSON patch
# (not strategic merge) because Gateway is a CRD — a strategic-merge patch
# would replace the whole listeners list instead of appending.
- op: add
path: /spec/listeners/-
value:
name: https-ascoachingogvaner-dk
hostname: ascoachingogvaner.dk
port: 443
protocol: HTTPS
tls:
mode: Terminate
certificateRefs:
- name: ascoachingogvaner-dk-tls
allowedRoutes:
namespaces:
from: All
- op: add
path: /spec/listeners/-
value:
name: https-www-ascoachingogvaner-dk
hostname: www.ascoachingogvaner.dk
port: 443
protocol: HTTPS
tls:
mode: Terminate
certificateRefs:
- name: ascoachingogvaner-dk-tls
allowedRoutes:
namespaces:
from: All
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: ascoachingogvaner-dk
namespace: kube-system
spec:
secretName: ascoachingogvaner-dk-tls
dnsNames:
- ascoachingogvaner.dk
- www.ascoachingogvaner.dk
issuerRef:
group: cert-manager.io
kind: ClusterIssuer
# Hardcoded (not ${issuer_name}): this zone is DNS-hosted at simply.com,
# so only the letsencrypt-prod issuer (with its simply.com DNS01 solver —
# see ../cluster-issuers/) can validate it; the Cloudflare Origin CA
# alternative doesn't apply to non-Cloudflare zones.
name: letsencrypt-prod
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Prod-only certificates for tenant-owned custom domains terminated on the
# shared platform Gateway. The wildcard ${domain} certificate stays in
# bases/infrastructure/certificates/; these zones exist only on Hetzner.
resources:
- ascoachingogvaner-dk-certificate.yaml
Loading
Loading