From 758636be7dc4854d3f9448a058430df3d92c7e95 Mon Sep 17 00:00:00 2001 From: Robert Moucha Date: Fri, 14 Aug 2026 09:46:41 +0200 Subject: [PATCH 1/2] feat(chart): configurable pod and container securityContext Adds two global values, `podSecurityContext` and `containerSecurityContext`, applied to every pod and every container the chart itself renders, including initContainers and the init/cleanup Jobs. Each is merged with a matching per-component override that wins on a per-key basis: podSecurityContext <- .securityContext containerSecurityContext <- .containerSecurityContext Before this change the chart exposed a pod-level securityContext for only zookeeper, bookkeeper, broker and oxia.server, and had no container-level securityContext support at all, so allowPrivilegeEscalation, capabilities, seccompProfile and readOnlyRootFilesystem were not settable at any value. Clusters running a restrictive pod security policy (Pod Security Admission, OPA Gatekeeper, Kyverno) had no supported way to make the chart conform. The merge uses mergeOverwrite rather than merge, because merge treats zero values in its destination as absent and would silently discard a per-component `fsGroup: 0` or `allowPrivilegeEscalation: false`. Containers supplied through `.initContainers`, `oxia.coordinator.extraContainers` and `dekaf.deployment.extraContainers` are raw passthrough and are not merged with these settings; the README says so. Both globals default to empty, so rendered output is unchanged apart from two things: - The zookeeper and broker sts-cleanup upgrade-hook Jobs gain pod template labels via pulsar.template.labels. They were the only pod templates in the chart without them, so .Values.labels never reached their pods. - Three Jobs (bookkeeper cluster-initialize and the two sts-cleanup hooks) now render the `fsGroup: 0` of the component they belong to. Those pods mount only ConfigMaps, Secrets and the service account token, so this has no effect on volume ownership. It is not entirely inert everywhere: fsGroup is an admission input for OpenShift SCCs. The autorecovery StatefulSet gains liveness, readiness and startup probes, all disabled by default, so this adds a knob rather than behaviour. It had none and no values knob to add them. The daemon does not start BookKeeper's HTTP service, so the probes target the Prometheus stats endpoint on autorecovery.ports.http -- the same endpoint the PodMonitor already scrapes. They default to off because that endpoint comes from the image's BookKeeper stats provider rather than from anything the chart configures: a cluster overriding statsProviderClass or prometheusStatsHttpPort through autorecovery.configData would otherwise get a restart loop from liveness, or a permanently unready pod stalling the rollout from readiness. Every probe value is read with `dig`, because `autorecovery.probe` is a new key: under `helm upgrade --reuse-values` the new chart's defaults are not coalesced in, so the map is absent and a direct dereference aborts the render for every existing release. `fsGroup` is applied as a supplementary group, so `runAsGroup` does not need to match it. The docs say so rather than the opposite. Adds two worked examples. examples/values-psa-restricted.yaml targets the Kubernetes `restricted` Pod Security Standard and deliberately leaves the per-component `fsGroup: 0` defaults alone, since `restricted` places no constraint on fsGroup, fsGroupChangePolicy or supplementalGroups. examples/values-restricted-group-ranges.yaml covers the narrower case of a policy that constrains group IDs to a numeric range -- Gatekeeper's PodSecurityPolicy-derived K8sPSPAllowedUsers, an equivalent Kyverno policy, or an OpenShift SCC -- and overrides fsGroup on the four components that ship it, with a warning about the recursive volume ownership change. Both examples set `runAsUser` explicitly rather than relying on the images' own USER, because not every image the chart runs has one: the zookeeper and broker sts-cleanup pre-upgrade hooks use alpine/k8s, which has no USER directive and runs as UID 0. With runAsNonRoot and no runAsUser the kubelet refuses those pods ("container has runAsNonRoot and image will run as root"), and because they are pre-upgrade hooks that blocks `helm upgrade` outright. kubectl in that image was verified to work as UID 10000. Both examples also disable the bundled monitoring stack, because these values do not reach subchart templates: prometheus-node-exporter requires hostNetwork, hostPID and hostPath, Grafana chowns its data directory as root, and some of the stack's workloads set no seccompProfile. Left enabled its pods are rejected on admission while Pulsar itself comes up fine. With it disabled, the whole rendered release is admitted by a namespace enforcing `restricted` with no violations. Validated with helm lint and kubeconform (k8s 1.25/1.31/1.36) against the default values, every .ci/clusters/* config, and .ci/templates-all-values.yaml. Also verified on a 4-node kind cluster (k8s 1.36.1) and on EKS 1.34 with the AWS EBS CSI driver, upgrading in place from published 4.7.0. Moving fsGroup from 0 to 10000 triggers the recursive relabel and preserves data: a 5000-message unacked backlog was consumed intact afterwards, with identical storageSize and no pod restarts. --- README.md | 99 +++++++++++++ charts/pulsar/templates/_helpers.tpl | 47 ++++++ .../templates/autorecovery-statefulset.yaml | 51 +++++++ .../bookkeeper-cluster-initialize.yaml | 5 + .../templates/bookkeeper-statefulset.yaml | 8 +- .../templates/broker-statefulset-upgrade.yaml | 6 + .../pulsar/templates/broker-statefulset.yaml | 10 +- charts/pulsar/templates/dekaf-deployment.yaml | 2 + .../function-worker-statefulset.yaml | 4 + charts/pulsar/templates/jwt-secret-init.yaml | 3 + .../oxia-coordinator-deployment.yaml | 2 + .../templates/oxia-server-statefulset.yaml | 6 +- .../pulsar/templates/proxy-statefulset.yaml | 6 + .../templates/pulsar-cluster-initialize.yaml | 7 + .../pulsar-manager-cluster-initialize.yaml | 4 + .../templates/pulsar-manager-statefulset.yaml | 2 + .../templates/standalone-deployment.yaml | 2 + .../pulsar/templates/toolset-statefulset.yaml | 3 + .../zookeeper-statefulset-upgrade.yaml | 6 + .../templates/zookeeper-statefulset.yaml | 7 +- charts/pulsar/values.yaml | 136 +++++++++++++++++- examples/README.md | 2 + examples/values-psa-restricted.yaml | 58 ++++++++ examples/values-restricted-group-ranges.yaml | 107 ++++++++++++++ 24 files changed, 566 insertions(+), 17 deletions(-) create mode 100644 examples/values-psa-restricted.yaml create mode 100644 examples/values-restricted-group-ranges.yaml diff --git a/README.md b/README.md index f12e77a1..c7a6fa34 100644 --- a/README.md +++ b/README.md @@ -285,6 +285,105 @@ multiple times (later files win), or use the [`merge-values.sh`](examples/merge- helper to merge several into a single file. See [`examples/README.md`](examples/README.md) for the full list and usage details. +## Pod and container security contexts + +Two global values apply a security context to every pod and every container the chart +itself renders, including its initContainers and the init/cleanup Jobs. Containers you +supply through `.initContainers`, `oxia.coordinator.extraContainers` or +`dekaf.deployment.extraContainers` are passed through verbatim and are not merged with +these settings -- set a `securityContext` on those yourself: + +```yaml +podSecurityContext: + runAsNonRoot: true + runAsUser: 10000 + runAsGroup: 10000 + fsGroup: 10000 + fsGroupChangePolicy: OnRootMismatch + supplementalGroups: [10000] +containerSecurityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + seccompProfile: + type: RuntimeDefault +``` + +Both are empty by default, so the chart's rendered output is unchanged unless you set +them. Use them when a cluster policy engine (Pod Security Admission, OPA Gatekeeper, +Kyverno) requires settings the chart does not set on its own. + +The `containerSecurityContext` block above is what the Kubernetes `restricted` Pod +Security Standard actually enforces, together with `runAsNonRoot`. See +[`examples/values-psa-restricted.yaml`](examples/values-psa-restricted.yaml). + +Each global value is merged with the matching per-component override, and the +per-component value wins on a per-key basis: + +| Global | Per-component | +|---|---| +| `podSecurityContext` | `.securityContext` | +| `containerSecurityContext` | `.containerSecurityContext` | + +The per-component keys exist for `zookeeper`, `bookkeeper`, `broker`, `autorecovery`, +`proxy`, `toolset`, `function_worker`, `pulsar_manager`, `standalone`, `oxia.server`, +`oxia.coordinator`, `pulsar_metadata`, `dekaf.deployment` and +`auth.authentication.jwt.generateSecrets`. Component-owned Jobs follow the component +they belong to: the bookkeeper cluster-initialize Job uses `bookkeeper.*`, the +zookeeper and broker `sts-cleanup` upgrade hooks use `zookeeper.*` and `broker.*`, and +the `pulsar-cluster-initialize` Job uses `pulsar_metadata.*`. + +### Overriding fsGroup, if a policy constrains group IDs + +`zookeeper`, `bookkeeper`, `broker` and `oxia.server` ship `securityContext.fsGroup: 0` +so that mounted volumes are group-owned by GID 0, the group the `pulsar` user (UID 10000) +belongs to. That is deliberate, and it is what lets the images run under an arbitrary +assigned UID — the model OpenShift uses. **`fsGroup: 0` is not a privilege**: group 0 +inside a container is an ordinary group, and root power comes from UID 0 and capabilities, +neither of which this grants. + +Most policies do not require you to change it. In particular the Kubernetes `restricted` +Pod Security Standard places **no constraint on `fsGroup`, `fsGroupChangePolicy` or +`supplementalGroups`**, and the PSS policy sets that Gatekeeper and Kyverno ship mirror +PSA, so `fsGroup: 0` is admitted there as-is. + +Override it only when a policy actually constrains group IDs to a numeric range — for +example Gatekeeper's PodSecurityPolicy-derived `K8sPSPAllowedUsers` with +`fsGroup: {rule: MustRunAs, ranges: [{min: 1, ...}]}`, or an OpenShift SCC. Because a +per-component value takes precedence over the global one, setting a global `fsGroup` +alone does **not** change these four components — override their own `securityContext` +as well: + +```yaml +podSecurityContext: + runAsNonRoot: true + runAsUser: 10000 + runAsGroup: 10000 + fsGroup: 10000 + fsGroupChangePolicy: OnRootMismatch + supplementalGroups: [10000] +bookkeeper: + securityContext: + fsGroup: 10000 + fsGroupChangePolicy: OnRootMismatch +``` + +`fsGroup` is applied as a *supplementary* group, so `runAsGroup` does not need to match +it -- the process keeps its own primary GID and still gets access to the volume. + +On a cluster with existing volumes, changing `fsGroup` triggers a recursive ownership +change on first mount; `fsGroupChangePolicy: OnRootMismatch` keeps that to the first pass, +but it can still take a long time on large bookie ledger volumes. Test on a copy before +changing a production cluster. + +### readOnlyRootFilesystem + +`containerSecurityContext.readOnlyRootFilesystem: true` is accepted but does not work +out of the box: the Pulsar images rewrite their configuration under `/pulsar/conf` on +startup (`bin/apply-config-from-env.py`) and write logs under `/pulsar/logs`. Mount +writable volumes over those paths using `.extraVolumes` and +`.extraVolumeMounts` before enabling it. + ## Disabling victoria-metrics-k8s-stack components In order to disable the victoria-metrics-k8s-stack, you can add the following to your `values.yaml`. diff --git a/charts/pulsar/templates/_helpers.tpl b/charts/pulsar/templates/_helpers.tpl index d8bc866d..99cdde03 100644 --- a/charts/pulsar/templates/_helpers.tpl +++ b/charts/pulsar/templates/_helpers.tpl @@ -110,6 +110,53 @@ app: {{ template "pulsar.name" . }} release: {{ .Release.Name }} {{- end }} +{{/* +Render a pod-level securityContext. + +Merges the chart-wide `.Values.podSecurityContext` defaults with a per-component +`.securityContext` override, where the per-component value wins. Renders +nothing when both are empty, so charts that set neither are unaffected. + +Usage: + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.broker.securityContext "root" . "indent" 6) }} +*/}} +{{- define "pulsar.podSecurityContext" -}} +{{- include "pulsar.mergedSecurityContext" (dict "global" .root.Values.podSecurityContext "component" .securityContext "indent" .indent) -}} +{{- end -}} + +{{/* +Render a container-level securityContext. + +Merges the chart-wide `.Values.containerSecurityContext` defaults with a per-component +`.containerSecurityContext` override, where the per-component value wins. +Applies to both containers and initContainers. Renders nothing when both are empty. + +Usage: + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.broker.containerSecurityContext "root" . "indent" 8) }} +*/}} +{{- define "pulsar.containerSecurityContext" -}} +{{- include "pulsar.mergedSecurityContext" (dict "global" .root.Values.containerSecurityContext "component" .securityContext "indent" .indent) -}} +{{- end -}} + +{{/* +Merge a global and a per-component securityContext and render the resulting block, +indented by `indent` spaces. Nested maps are deep-merged; the per-component value +takes precedence. Renders nothing when the merged result is empty. + +`mergeOverwrite` is used rather than `merge` because `merge` treats zero values +(`0`, `false`, `""`) in its destination as absent, which would silently discard a +per-component `fsGroup: 0` or `allowPrivilegeEscalation: false` override. +Both operands are deep-copied so that `.Values` is never mutated. +*/}} +{{- define "pulsar.mergedSecurityContext" -}} +{{- $global := deepCopy (.global | default dict) -}} +{{- $component := deepCopy (.component | default dict) -}} +{{- $merged := mergeOverwrite $global $component -}} +{{- if $merged -}} +{{- printf "securityContext:\n%s" (toYaml $merged | indent 2) | nindent (int .indent) -}} +{{- end -}} +{{- end -}} + {{/* Create ImagePullSecrets */}} diff --git a/charts/pulsar/templates/autorecovery-statefulset.yaml b/charts/pulsar/templates/autorecovery-statefulset.yaml index d8432245..c32116e1 100644 --- a/charts/pulsar/templates/autorecovery-statefulset.yaml +++ b/charts/pulsar/templates/autorecovery-statefulset.yaml @@ -115,11 +115,13 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.autorecovery.gracePeriod }} serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.autorecovery.component }}" + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.autorecovery.securityContext "root" . "indent" 6) }} initContainers: {{- if .Values.tls.autorecovery.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.autorecovery "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.autorecovery "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.autorecovery.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["sh", "-c"] args: @@ -137,6 +139,7 @@ spec: - name: pulsar-bookkeeper-verify-clusterid image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.autorecovery "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.autorecovery "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.autorecovery.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.autorecovery.waitBookkeeperTimeout }}", "sh", "-c"] args: @@ -158,6 +161,54 @@ spec: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.autorecovery.component }}" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.autorecovery "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.autorecovery "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.autorecovery.containerSecurityContext "root" . "indent" 8) }} + {{- /* + The autorecovery daemon does not start BookKeeper's HTTP service, so there is no + /api/v1 endpoint to probe as there is on a bookie. It does start the Prometheus + stats endpoint on autorecovery.ports.http, which is the same endpoint the + PodMonitor scrapes, so /metrics is what these probes target. + + Every lookup below goes through `dig`, because `autorecovery.probe` is a new key: + under `helm upgrade --reuse-values` the new chart's defaults are not coalesced in, + so the map is absent and a direct dereference aborts the whole render. + + All three are off by default, so this adds a knob rather than behaviour. /metrics + is served by the image's BookKeeper stats provider rather than by anything the + chart configures, so a cluster overriding statsProviderClass or + prometheusStatsHttpPort through autorecovery.configData would break liveness + (restart loop) and readiness (permanently unready, stalling the rollout) alike. + */}} + {{- $probe := .Values.autorecovery.probe | default dict }} + {{- if dig "liveness" "enabled" false $probe }} + livenessProbe: + httpGet: + path: /metrics + port: {{ .Values.autorecovery.ports.http }} + initialDelaySeconds: {{ dig "liveness" "initialDelaySeconds" 10 $probe }} + periodSeconds: {{ dig "liveness" "periodSeconds" 30 $probe }} + timeoutSeconds: {{ dig "liveness" "timeoutSeconds" 5 $probe }} + failureThreshold: {{ dig "liveness" "failureThreshold" 60 $probe }} + {{- end }} + {{- if dig "readiness" "enabled" false $probe }} + readinessProbe: + httpGet: + path: /metrics + port: {{ .Values.autorecovery.ports.http }} + initialDelaySeconds: {{ dig "readiness" "initialDelaySeconds" 10 $probe }} + periodSeconds: {{ dig "readiness" "periodSeconds" 30 $probe }} + timeoutSeconds: {{ dig "readiness" "timeoutSeconds" 5 $probe }} + failureThreshold: {{ dig "readiness" "failureThreshold" 60 $probe }} + {{- end }} + {{- if dig "startup" "enabled" false $probe }} + startupProbe: + httpGet: + path: /metrics + port: {{ .Values.autorecovery.ports.http }} + initialDelaySeconds: {{ dig "startup" "initialDelaySeconds" 10 $probe }} + periodSeconds: {{ dig "startup" "periodSeconds" 30 $probe }} + timeoutSeconds: {{ dig "startup" "timeoutSeconds" 5 $probe }} + failureThreshold: {{ dig "startup" "failureThreshold" 60 $probe }} + {{- end }} {{- if .Values.autorecovery.resources }} resources: {{ toYaml .Values.autorecovery.resources | indent 10 }} diff --git a/charts/pulsar/templates/bookkeeper-cluster-initialize.yaml b/charts/pulsar/templates/bookkeeper-cluster-initialize.yaml index 34137ebb..07f9c511 100755 --- a/charts/pulsar/templates/bookkeeper-cluster-initialize.yaml +++ b/charts/pulsar/templates/bookkeeper-cluster-initialize.yaml @@ -44,6 +44,7 @@ spec: spec: {{- include "pulsar.imagePullSecrets" . | nindent 6 }} serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}" + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.bookkeeper.securityContext "root" . "indent" 6) }} nodeSelector: {{- if .Values.pulsar_metadata.nodeSelector }} {{ toYaml .Values.pulsar_metadata.nodeSelector | indent 8 }} @@ -57,6 +58,7 @@ spec: - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.bookie "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.bookie "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["sh", "-c"] args: @@ -69,6 +71,7 @@ spec: - name: wait-zookeeper-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.bookie "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.bookie "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.bookkeeper.metadata.waitZookeeperTimeout }}", "sh", "-c"] args: @@ -88,6 +91,7 @@ spec: - name: wait-oxia-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.bookie "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.bookie "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.bookkeeper.metadata.waitOxiaTimeout }}", "sh", "-c"] args: @@ -100,6 +104,7 @@ spec: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}-init" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.bookie "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.bookie "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "root" . "indent" 8) }} {{- if .Values.bookkeeper.metadata.resources }} resources: {{ toYaml .Values.bookkeeper.metadata.resources | indent 10 }} diff --git a/charts/pulsar/templates/bookkeeper-statefulset.yaml b/charts/pulsar/templates/bookkeeper-statefulset.yaml index 4b76472a..b06eb4d2 100644 --- a/charts/pulsar/templates/bookkeeper-statefulset.yaml +++ b/charts/pulsar/templates/bookkeeper-statefulset.yaml @@ -112,10 +112,7 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.bookkeeper.gracePeriod }} serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}" - {{- if .Values.bookkeeper.securityContext }} - securityContext: -{{ toYaml .Values.bookkeeper.securityContext | indent 8 }} - {{- end }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.bookkeeper.securityContext "root" . "indent" 6) }} {{- if and .Values.bookkeeper.waitMetadataTimeout (gt (.Values.bookkeeper.waitMetadataTimeout | int) 0) }} initContainers: {{- if .Values.tls.bookie.cacerts.enabled }} @@ -123,6 +120,7 @@ spec: image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.bookie "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.bookie "root" .) }}" resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "root" . "indent" 8) }} command: ["sh", "-c"] args: - | @@ -136,6 +134,7 @@ spec: image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.bookie "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.bookie "root" .) }}" resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "root" . "indent" 8) }} command: ["timeout", "{{ .Values.bookkeeper.waitMetadataTimeout }}", "sh", "-c"] args: # only reformat bookie if bookkeeper is running without persistence @@ -154,6 +153,7 @@ spec: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.bookie "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.bookie "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "root" . "indent" 8) }} {{- if .Values.bookkeeper.probe.liveness.enabled }} livenessProbe: httpGet: diff --git a/charts/pulsar/templates/broker-statefulset-upgrade.yaml b/charts/pulsar/templates/broker-statefulset-upgrade.yaml index 5552ccd4..eb5a7506 100644 --- a/charts/pulsar/templates/broker-statefulset-upgrade.yaml +++ b/charts/pulsar/templates/broker-statefulset-upgrade.yaml @@ -87,12 +87,18 @@ metadata: spec: backoffLimit: 1 template: + metadata: + labels: + {{- include "pulsar.template.labels" . | nindent 8 }} + component: {{ .Values.broker.component }}-sts-cleanup spec: serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.broker.component }}-sts-cleanup" restartPolicy: Never + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.broker.securityContext "root" . "indent" 6) }} containers: - name: sts-cleanup image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.kubectl "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.broker.containerSecurityContext "root" . "indent" 10) }} command: - sh - -c diff --git a/charts/pulsar/templates/broker-statefulset.yaml b/charts/pulsar/templates/broker-statefulset.yaml index dfb95094..87b2eb86 100644 --- a/charts/pulsar/templates/broker-statefulset.yaml +++ b/charts/pulsar/templates/broker-statefulset.yaml @@ -75,10 +75,7 @@ spec: {{- end }} spec: serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.broker.component }}-acct" - {{- if .Values.broker.securityContext }} - securityContext: -{{ toYaml .Values.broker.securityContext | indent 8 }} - {{- end }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.broker.securityContext "root" . "indent" 6) }} {{- if .Values.broker.nodeSelector }} nodeSelector: {{ toYaml .Values.broker.nodeSelector | indent 8 }} @@ -141,6 +138,7 @@ spec: - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.broker "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.broker "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.broker.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["sh", "-c"] args: @@ -155,6 +153,7 @@ spec: - name: wait-zookeeper-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.broker "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.broker "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.broker.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.broker.waitZookeeperTimeout }}", "sh", "-c"] args: @@ -176,6 +175,7 @@ spec: - name: wait-oxia-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.broker "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.broker "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.broker.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.broker.waitOxiaTimeout }}", "sh", "-c"] args: @@ -190,6 +190,7 @@ spec: - name: wait-bookkeeper-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.broker "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.broker "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.broker.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.broker.waitBookkeeperTimeout }}", "sh", "-c"] args: @@ -222,6 +223,7 @@ spec: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.broker.component }}" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.broker "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.broker "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.broker.containerSecurityContext "root" . "indent" 8) }} {{- if .Values.broker.probe.liveness.enabled }} livenessProbe: httpGet: diff --git a/charts/pulsar/templates/dekaf-deployment.yaml b/charts/pulsar/templates/dekaf-deployment.yaml index 79871da0..ca50d329 100644 --- a/charts/pulsar/templates/dekaf-deployment.yaml +++ b/charts/pulsar/templates/dekaf-deployment.yaml @@ -57,10 +57,12 @@ spec: priorityClassName: {{ .Values.dekaf.deployment.priorityClassName }} {{- end }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" ((.Values.dekaf).deployment).securityContext "root" . "indent" 6) }} containers: - name: dekaf image: "{{ .Values.images.dekaf.repository }}:{{ .Values.images.dekaf.tag }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.dekaf "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" ((.Values.dekaf).deployment).containerSecurityContext "root" . "indent" 10) }} env: - name: DEKAF_PULSAR_WEB_URL value: "http://{{ template "pulsar.fullname" . }}-{{ .Values.broker.component }}:{{ .Values.broker.ports.http }}" diff --git a/charts/pulsar/templates/function-worker-statefulset.yaml b/charts/pulsar/templates/function-worker-statefulset.yaml index 242c2b80..b1cdcb93 100644 --- a/charts/pulsar/templates/function-worker-statefulset.yaml +++ b/charts/pulsar/templates/function-worker-statefulset.yaml @@ -55,6 +55,7 @@ spec: {{- end }} spec: serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.function_worker.component }}-acct" + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.function_worker.securityContext "root" . "indent" 6) }} {{- if .Values.function_worker.nodeSelector }} nodeSelector: {{ toYaml .Values.function_worker.nodeSelector | indent 8 }} @@ -117,6 +118,7 @@ spec: - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.function_worker "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.function_worker "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.function_worker.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["sh", "-c"] args: @@ -130,6 +132,7 @@ spec: - name: wait-broker-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.function_worker "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.function_worker "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.function_worker.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.function_worker.waitBrokerTimeout }}", "sh", "-c"] args: @@ -146,6 +149,7 @@ spec: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.function_worker.component }}" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.function_worker "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.function_worker "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.function_worker.containerSecurityContext "root" . "indent" 8) }} {{- if .Values.function_worker.probe.liveness.enabled }} livenessProbe: tcpSocket: diff --git a/charts/pulsar/templates/jwt-secret-init.yaml b/charts/pulsar/templates/jwt-secret-init.yaml index c4bd8b5b..9bc6fee4 100644 --- a/charts/pulsar/templates/jwt-secret-init.yaml +++ b/charts/pulsar/templates/jwt-secret-init.yaml @@ -98,6 +98,7 @@ spec: {{- include "pulsar.imagePullSecrets" . | nindent 6 }} serviceAccountName: "{{ template "pulsar.fullname" . }}-jwt-secret-init" restartPolicy: Never + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.auth.authentication.jwt.generateSecrets.securityContext "root" . "indent" 6) }} {{- if .Values.auth.authentication.jwt.generateSecrets.nodeSelector }} nodeSelector: {{ toYaml .Values.auth.authentication.jwt.generateSecrets.nodeSelector | indent 8 }} @@ -155,6 +156,7 @@ spec: - name: generate-jwt-artifacts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.broker "root" .) }}" imagePullPolicy: {{ include "pulsar.imagePullPolicy" (dict "image" .Values.images.broker "root" .) }} + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.auth.authentication.jwt.generateSecrets.containerSecurityContext "root" . "indent" 10) }} command: - /bin/bash - -c @@ -195,6 +197,7 @@ spec: - name: create-secrets image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.kubectl "root" .) }}" imagePullPolicy: {{ include "pulsar.imagePullPolicy" (dict "image" .Values.images.kubectl "root" .) }} + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.auth.authentication.jwt.generateSecrets.containerSecurityContext "root" . "indent" 10) }} env: - name: NAMESPACE valueFrom: diff --git a/charts/pulsar/templates/oxia-coordinator-deployment.yaml b/charts/pulsar/templates/oxia-coordinator-deployment.yaml index e26d017b..fef77cf1 100644 --- a/charts/pulsar/templates/oxia-coordinator-deployment.yaml +++ b/charts/pulsar/templates/oxia-coordinator-deployment.yaml @@ -61,6 +61,7 @@ spec: priorityClassName: {{ .Values.oxia.coordinator.priorityClassName }} {{- end }} serviceAccountName: {{ template "pulsar.fullname" . }}-{{ .Values.oxia.component }}-coordinator + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.oxia.coordinator.securityContext "root" . "indent" 6) }} containers: - command: {{- if .Values.oxia.coordinator.entrypoint }} @@ -70,6 +71,7 @@ spec: {{- end }} image: "{{ .Values.images.oxia.repository }}:{{ .Values.images.oxia.tag }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.oxia "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.oxia.coordinator.containerSecurityContext "root" . "indent" 10) }} name: coordinator ports: {{- range $key, $value := .Values.oxia.coordinator.ports }} diff --git a/charts/pulsar/templates/oxia-server-statefulset.yaml b/charts/pulsar/templates/oxia-server-statefulset.yaml index ff606953..9a103347 100644 --- a/charts/pulsar/templates/oxia-server-statefulset.yaml +++ b/charts/pulsar/templates/oxia-server-statefulset.yaml @@ -106,10 +106,7 @@ spec: {{ end }} {{- end }} serviceAccountName: {{ template "pulsar.fullname" . }}-{{ .Values.oxia.component }} - {{- if .Values.oxia.server.securityContext }} - securityContext: -{{ toYaml .Values.oxia.server.securityContext | indent 8 }} - {{- end }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.oxia.server.securityContext "root" . "indent" 6) }} containers: - command: - "oxia" @@ -126,6 +123,7 @@ spec: {{- end}} image: "{{ .Values.images.oxia.repository }}:{{ .Values.images.oxia.tag }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.oxia "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.oxia.server.containerSecurityContext "root" . "indent" 10) }} name: server ports: {{- range $key, $value := .Values.oxia.server.ports }} diff --git a/charts/pulsar/templates/proxy-statefulset.yaml b/charts/pulsar/templates/proxy-statefulset.yaml index 3460f882..2f9c611f 100644 --- a/charts/pulsar/templates/proxy-statefulset.yaml +++ b/charts/pulsar/templates/proxy-statefulset.yaml @@ -113,11 +113,13 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.proxy.gracePeriod }} serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.proxy.component }}" + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.proxy.securityContext "root" . "indent" 6) }} initContainers: {{- if .Values.tls.proxy.cacerts.enabled }} - name: combine-certs image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.proxy "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.proxy "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.proxy.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["sh", "-c"] args: @@ -132,6 +134,7 @@ spec: - name: wait-zookeeper-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.proxy "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.proxy "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.proxy.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.proxy.waitZookeeperTimeout }}", "sh", "-c"] args: @@ -155,6 +158,7 @@ spec: - name: wait-oxia-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.proxy "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.proxy "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.proxy.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.proxy.waitOxiaTimeout }}", "sh", "-c"] args: @@ -171,6 +175,7 @@ spec: - name: wait-broker-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.proxy "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.proxy "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.proxy.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.proxy.waitBrokerTimeout }}", "sh", "-c"] args: @@ -190,6 +195,7 @@ spec: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.proxy.component }}" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.proxy "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.proxy "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.proxy.containerSecurityContext "root" . "indent" 8) }} {{- if .Values.proxy.probe.liveness.enabled }} livenessProbe: httpGet: diff --git a/charts/pulsar/templates/pulsar-cluster-initialize.yaml b/charts/pulsar/templates/pulsar-cluster-initialize.yaml index 873035cf..91a25db7 100755 --- a/charts/pulsar/templates/pulsar-cluster-initialize.yaml +++ b/charts/pulsar/templates/pulsar-cluster-initialize.yaml @@ -48,11 +48,13 @@ spec: nodeSelector: {{ toYaml .Values.pulsar_metadata.nodeSelector | indent 8 }} {{- end }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.pulsar_metadata.securityContext "root" . "indent" 6) }} initContainers: {{- if .Values.tls.toolset.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.pulsar_metadata.image "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.pulsar_metadata.image "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["sh", "-c"] args: @@ -66,6 +68,7 @@ spec: - name: wait-zk-cs-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.pulsar_metadata.image "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.pulsar_metadata.image "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.pulsar_metadata.waitZookeeperTimeout }}", "sh", "-c"] args: @@ -81,6 +84,7 @@ spec: - name: wait-zk-metastore-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.pulsar_metadata.image "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.pulsar_metadata.image "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.pulsar_metadata.waitZookeeperTimeout }}", "sh", "-c"] args: @@ -100,6 +104,7 @@ spec: - name: wait-oxia-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.pulsar_metadata.image "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.pulsar_metadata.image "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.pulsar_metadata.waitOxiaTimeout }}", "sh", "-c"] args: @@ -114,6 +119,7 @@ spec: - name: pulsar-bookkeeper-verify-clusterid image: {{ template "pulsar.imageFullName" (dict "image" .Values.pulsar_metadata.image "root" .) }} imagePullPolicy: {{ template "pulsar.imagePullPolicy" (dict "image" .Values.pulsar_metadata.image "root" .) }} + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.pulsar_metadata.waitBookkeeperTimeout }}", "sh", "-c"] args: @@ -136,6 +142,7 @@ spec: - name: {{ template "pulsar.fullname" . }}-{{ .Values.pulsar_metadata.component }} image: {{ template "pulsar.imageFullName" (dict "image" .Values.pulsar_metadata.image "root" .) }} imagePullPolicy: {{ template "pulsar.imagePullPolicy" (dict "image" .Values.pulsar_metadata.image "root" .) }} + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "root" . "indent" 8) }} {{- if .Values.pulsar_metadata.resources }} resources: {{ toYaml .Values.pulsar_metadata.resources | indent 10 }} diff --git a/charts/pulsar/templates/pulsar-manager-cluster-initialize.yaml b/charts/pulsar/templates/pulsar-manager-cluster-initialize.yaml index c7eb4f68..e89ed266 100755 --- a/charts/pulsar/templates/pulsar-manager-cluster-initialize.yaml +++ b/charts/pulsar/templates/pulsar-manager-cluster-initialize.yaml @@ -51,10 +51,12 @@ spec: {{ toYaml .Values.pulsar_metadata.tolerations | indent 8 }} {{- end }} restartPolicy: OnFailure + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.pulsar_manager.securityContext "root" . "indent" 6) }} initContainers: - name: wait-pulsar-manager-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.pulsar_metadata.image "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.pulsar_metadata.image "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.pulsar_manager.containerSecurityContext "root" . "indent" 10) }} resources: {{ toYaml .Values.initContainer.resources | nindent 12 }} command: [ "sh", "-c" ] args: @@ -69,6 +71,7 @@ spec: - name: wait-broker-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.proxy "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.pulsar_metadata.image "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.pulsar_manager.containerSecurityContext "root" . "indent" 10) }} resources: {{ toYaml .Values.initContainer.resources | nindent 12 }} command: [ "sh", "-c" ] args: @@ -85,6 +88,7 @@ spec: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.pulsar_manager.component }}-init" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.pulsar_metadata.image "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.pulsar_metadata.image "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.pulsar_manager.containerSecurityContext "root" . "indent" 10) }} {{- if .Values.pulsar_metadata.resources }} resources: {{ toYaml .Values.pulsar_metadata.resources | nindent 12 }} {{- end }} diff --git a/charts/pulsar/templates/pulsar-manager-statefulset.yaml b/charts/pulsar/templates/pulsar-manager-statefulset.yaml index 5dfd06e8..398da8a6 100755 --- a/charts/pulsar/templates/pulsar-manager-statefulset.yaml +++ b/charts/pulsar/templates/pulsar-manager-statefulset.yaml @@ -63,6 +63,7 @@ spec: {{- toYaml .Values.pulsar_manager.topologySpreadConstraints | nindent 8 }} {{- end }} terminationGracePeriodSeconds: {{ .Values.pulsar_manager.gracePeriod }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.pulsar_manager.securityContext "root" . "indent" 6) }} {{- if .Values.pulsar_manager.initContainers }} initContainers: {{- toYaml .Values.pulsar_manager.initContainers | nindent 6 }} @@ -71,6 +72,7 @@ spec: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.pulsar_manager.component }}" image: "{{ .Values.images.pulsar_manager.repository }}:{{ .Values.images.pulsar_manager.tag }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.pulsar_manager "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.pulsar_manager.containerSecurityContext "root" . "indent" 10) }} {{- if .Values.pulsar_manager.resources }} resources: {{ toYaml .Values.pulsar_manager.resources | indent 12 }} diff --git a/charts/pulsar/templates/standalone-deployment.yaml b/charts/pulsar/templates/standalone-deployment.yaml index 804c1602..9ae9dcca 100644 --- a/charts/pulsar/templates/standalone-deployment.yaml +++ b/charts/pulsar/templates/standalone-deployment.yaml @@ -102,6 +102,7 @@ spec: {{ toYaml .Values.standalone.tolerations | indent 8 }} {{- end }} terminationGracePeriodSeconds: {{ .Values.standalone.gracePeriod }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.standalone.securityContext "root" . "indent" 6) }} {{- if .Values.standalone.initContainers }} initContainers: {{- toYaml .Values.standalone.initContainers | nindent 6 }} @@ -110,6 +111,7 @@ spec: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.standalone.component }}" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.standalone "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.standalone "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.standalone.containerSecurityContext "root" . "indent" 8) }} {{- if .Values.standalone.probe.liveness.enabled }} livenessProbe: tcpSocket: diff --git a/charts/pulsar/templates/toolset-statefulset.yaml b/charts/pulsar/templates/toolset-statefulset.yaml index 8480eff7..fee0404d 100644 --- a/charts/pulsar/templates/toolset-statefulset.yaml +++ b/charts/pulsar/templates/toolset-statefulset.yaml @@ -67,11 +67,13 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.toolset.gracePeriod }} serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.toolset.component }}" + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.toolset.securityContext "root" . "indent" 6) }} initContainers: {{- if .Values.tls.toolset.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.toolset "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.toolset "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.toolset.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["sh", "-c"] args: @@ -92,6 +94,7 @@ spec: image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.broker "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.broker "root" .) }}" {{- end }} + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.toolset.containerSecurityContext "root" . "indent" 8) }} {{- if .Values.toolset.resources }} resources: {{ toYaml .Values.toolset.resources | indent 10 }} diff --git a/charts/pulsar/templates/zookeeper-statefulset-upgrade.yaml b/charts/pulsar/templates/zookeeper-statefulset-upgrade.yaml index fa6ed661..7d1e1fe0 100644 --- a/charts/pulsar/templates/zookeeper-statefulset-upgrade.yaml +++ b/charts/pulsar/templates/zookeeper-statefulset-upgrade.yaml @@ -87,12 +87,18 @@ metadata: spec: backoffLimit: 1 template: + metadata: + labels: + {{- include "pulsar.template.labels" . | nindent 8 }} + component: {{ .Values.zookeeper.component }}-sts-cleanup spec: serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.zookeeper.component }}-sts-cleanup" restartPolicy: Never + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.zookeeper.securityContext "root" . "indent" 6) }} containers: - name: sts-cleanup image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.kubectl "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.zookeeper.containerSecurityContext "root" . "indent" 10) }} command: - sh - -c diff --git a/charts/pulsar/templates/zookeeper-statefulset.yaml b/charts/pulsar/templates/zookeeper-statefulset.yaml index 1d2c7acc..ceaae0ad 100755 --- a/charts/pulsar/templates/zookeeper-statefulset.yaml +++ b/charts/pulsar/templates/zookeeper-statefulset.yaml @@ -113,15 +113,13 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.zookeeper.gracePeriod }} serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.zookeeper.component }}" - {{- if .Values.zookeeper.securityContext }} - securityContext: -{{ toYaml .Values.zookeeper.securityContext | indent 8 }} - {{- end }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.zookeeper.securityContext "root" . "indent" 6) }} initContainers: {{- if .Values.tls.zookeeper.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.zookeeper "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.zookeeper "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.zookeeper.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["sh", "-c"] args: @@ -137,6 +135,7 @@ spec: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.zookeeper.component }}" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.zookeeper "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.zookeeper "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.zookeeper.containerSecurityContext "root" . "indent" 8) }} {{- if .Values.zookeeper.resources }} resources: {{ toYaml .Values.zookeeper.resources | indent 10 }} diff --git a/charts/pulsar/values.yaml b/charts/pulsar/values.yaml index 772f2826..e9b29ebe 100755 --- a/charts/pulsar/values.yaml +++ b/charts/pulsar/values.yaml @@ -116,6 +116,50 @@ affinity: # preferredDuringSchedulingIgnoredDuringExecution - scheduler will try to enforce but not guranentee type: requiredDuringSchedulingIgnoredDuringExecution +## SecurityContext +## +## Global security contexts applied to every pod and every container this chart +## renders, including initContainers and the init/cleanup Jobs. Both are empty by +## default, so the chart's rendered output is unchanged unless you set them. +## +## Each is merged with the matching per-component override, and the per-component +## value wins on a per-key basis: +## +## podSecurityContext <- merged with .securityContext +## containerSecurityContext <- merged with .containerSecurityContext +## +## Use these when a cluster policy engine (Pod Security Admission, OPA Gatekeeper, +## Kyverno) requires settings the chart does not set by default. For example: +## +## podSecurityContext: +## runAsNonRoot: true +## runAsUser: 10000 +## runAsGroup: 10000 +## fsGroup: 10000 +## fsGroupChangePolicy: OnRootMismatch +## supplementalGroups: [10000] +## containerSecurityContext: +## allowPrivilegeEscalation: false +## capabilities: +## drop: ["ALL"] +## seccompProfile: +## type: RuntimeDefault +## +## Note that zookeeper, bookkeeper, broker and oxia.server ship +## `securityContext.fsGroup: 0` by default, and a per-component value takes +## precedence over the global one. To move those four components off GID 0 you must +## also override their own `securityContext` (see the comment on +## `zookeeper.securityContext`), and be aware that changing `fsGroup` on a cluster +## with existing volumes triggers a recursive ownership change on first mount. +## +## `readOnlyRootFilesystem: true` is not supported out of the box: the Pulsar +## images rewrite their configuration under /pulsar/conf on startup and write logs +## under /pulsar/logs. Mount writable volumes over those paths with the +## `.extraVolumes` / `.extraVolumeMounts` settings before +## enabling it. +podSecurityContext: {} +containerSecurityContext: {} + ## Components ## ## Control what components of Apache Pulsar to deploy for the cluster @@ -422,6 +466,9 @@ auth: enabled: false # nodeSelector: # cloud.google.com/gke-nodepool: default-pool + # Merged over the global `podSecurityContext` / `containerSecurityContext`. + securityContext: {} + containerSecurityContext: {} tolerations: [] affinity: anti_affinity: false @@ -610,9 +657,15 @@ zookeeper: extraVolumes: [] extraVolumeMounts: [] # Ensures 2.10.0 non-root docker image works correctly. + # Merged over the global `podSecurityContext`; these keys win over it. Override + # `fsGroup` here if a cluster policy constrains group IDs. `fsGroup` is added as a + # supplementary group, so `runAsGroup` does not need to match it. securityContext: fsGroup: 0 fsGroupChangePolicy: "OnRootMismatch" + # Container-level securityContext, merged over the global `containerSecurityContext`. + # Applied to the zookeeper container and its initContainers. + containerSecurityContext: {} volumes: useSeparateDiskForTxlog: false # use a persistent volume or emptyDir @@ -729,6 +782,9 @@ oxia: annotations: {} service_account: annotations: {} + # Merged over the global `podSecurityContext` / `containerSecurityContext`. + securityContext: {} + containerSecurityContext: {} tolerations: [] # nodeSelector: # cloud.google.com/gke-nodepool: default-pool @@ -785,9 +841,14 @@ oxia: annotations: {} service_account: annotations: {} + # Merged over the global `podSecurityContext`; these keys win over it. Override + # `fsGroup` here if a cluster policy constrains group IDs. `fsGroup` is added as a + # supplementary group, so `runAsGroup` does not need to match it. securityContext: fsGroup: 0 fsGroupChangePolicy: "OnRootMismatch" + # Merged over the global `containerSecurityContext`. + containerSecurityContext: {} affinity: anti_affinity: true anti_affinity_topology_key: kubernetes.io/hostname @@ -899,9 +960,16 @@ bookkeeper: extraVolumes: [] extraVolumeMounts: [] # Ensures 2.10.0 non-root docker image works correctly. + # Merged over the global `podSecurityContext`; these keys win over it. Override + # `fsGroup` here if a cluster policy constrains group IDs. `fsGroup` is added as a + # supplementary group, so `runAsGroup` does not need to match it. + # This context is also applied to the bookkeeper cluster-initialize Job. securityContext: fsGroup: 0 fsGroupChangePolicy: "OnRootMismatch" + # Container-level securityContext, merged over the global `containerSecurityContext`. + # Applied to the bookie container, its initContainers and the cluster-initialize Job. + containerSecurityContext: {} volumes: # use a persistent volume or emptyDir persistence: true @@ -1076,6 +1144,44 @@ autorecovery: restartPodsOnConfigMapChange: false ports: http: 8000 + # Merged over the global `podSecurityContext` / `containerSecurityContext`. + # Unlike zookeeper/bookkeeper/broker this component mounts no data volume, so it + # carries no `fsGroup: 0` default. + securityContext: {} + containerSecurityContext: {} + # The autorecovery daemon does not start BookKeeper's HTTP service, so the probes + # target the Prometheus stats endpoint on `ports.http` -- the same endpoint the + # PodMonitor scrapes -- rather than a /api/v1 health endpoint. + # + # Both are off by default on purpose, so enabling them is opt-in and this chart's + # rendered output is unchanged for existing deployments. + # + # /metrics is served by the image's BookKeeper stats provider, not by anything this + # chart configures. A cluster that overrides `statsProviderClass` or + # `prometheusStatsHttpPort` through `autorecovery.configData` -- settings that were + # previously inert here -- would break both probes: liveness turns a working daemon + # into a restart loop, and readiness leaves the pod permanently unready, which stalls + # the StatefulSet rollout. Enable them once you have confirmed that `ports.http` + # serves /metrics in your deployment. + probe: + liveness: + enabled: false + failureThreshold: 60 + initialDelaySeconds: 10 + periodSeconds: 30 + timeoutSeconds: 5 + readiness: + enabled: false + failureThreshold: 60 + initialDelaySeconds: 10 + periodSeconds: 30 + timeoutSeconds: 5 + startup: + enabled: false + failureThreshold: 30 + initialDelaySeconds: 60 + periodSeconds: 30 + timeoutSeconds: 5 # nodeSelector: # cloud.google.com/gke-nodepool: default-pool # priorityClassName: high-priority-nonpreempting @@ -1148,6 +1254,10 @@ pulsar_metadata: waitBookkeeperTimeout: 120 ## Timeout for running metadata initialization initTimeout: 60 + ## Security contexts for the `pulsar-cluster-initialize` job, merged over the + ## global `podSecurityContext` / `containerSecurityContext`. + securityContext: {} + containerSecurityContext: {} ## Allow read-only operations on the metadata store when the metadata store is not available. ## This is useful when you want to continue serving requests even if the metadata store is not fully available with quorum. @@ -1242,6 +1352,9 @@ standalone: https: 8443 pulsar: 6650 pulsarssl: 6651 + # Merged over the global `podSecurityContext` / `containerSecurityContext`. + securityContext: {} + containerSecurityContext: {} # nodeSelector: # cloud.google.com/gke-nodepool: default-pool # priorityClassName: high-priority-nonpreempting @@ -1310,10 +1423,15 @@ broker: # GID 0 (the group the pulsar user, uid 10000, belongs to) and made group-writable, so the broker can # read/write it. Required when packageManagement.fileSystemStorage is enabled (the broker mounts a # PersistentVolumeClaim as the FileSystemPackagesStorage directory); harmless otherwise. Mirrors the - # bookkeeper/zookeeper securityContext. + # bookkeeper/zookeeper securityContext. Merged over the global `podSecurityContext`; + # these keys win over it. This context is also applied to the broker sts-cleanup + # pre-upgrade hook Job. securityContext: fsGroup: 0 fsGroupChangePolicy: "OnRootMismatch" + # Container-level securityContext, merged over the global `containerSecurityContext`. + # Applied to the broker container, its initContainers and the sts-cleanup hook Job. + containerSecurityContext: {} autoscaling: enabled: false minReplicas: 1 @@ -1650,6 +1768,9 @@ function_worker: ports: http: 6750 https: 6751 + # Merged over the global `podSecurityContext` / `containerSecurityContext`. + securityContext: {} + containerSecurityContext: {} # podManagementPolicy for the StatefulSet podManagementPolicy: Parallel # This is how Victoria Metrics or Prometheus discovers this component @@ -1753,6 +1874,9 @@ proxy: # so the metrics are correctly rendered in grafana dashboard component: proxy replicaCount: 3 + # Merged over the global `podSecurityContext` / `containerSecurityContext`. + securityContext: {} + containerSecurityContext: {} autoscaling: enabled: false minReplicas: 1 @@ -1970,6 +2094,9 @@ toolset: useProxy: true replicaCount: 1 initContainers: [] + # Merged over the global `podSecurityContext` / `containerSecurityContext`. + securityContext: {} + containerSecurityContext: {} # True includes annotation for statefulset that contains hash of corresponding configmap, which will cause pods to restart on configmap change restartPodsOnConfigMapChange: false # nodeSelector: @@ -2257,6 +2384,10 @@ pulsar_manager: component: pulsar-manager replicaCount: 1 initContainers: [] + # Merged over the global `podSecurityContext` / `containerSecurityContext`. Also + # applied to the pulsar-manager cluster-initialize Job. + securityContext: {} + containerSecurityContext: {} # True includes annotation for statefulset that contains hash of corresponding configmap, which will cause pods to restart on configmap change restartPodsOnConfigMapChange: false # nodeSelector: @@ -2362,6 +2493,9 @@ dekaf: annotations: {} podAnnotations: {} nodeSelector: {} + # Merged over the global `podSecurityContext` / `containerSecurityContext`. + securityContext: {} + containerSecurityContext: {} tolerations: [] # priorityClassName: high-priority-nonpreempting extraVolumes: [] diff --git a/examples/README.md b/examples/README.md index 59a8fa0d..8d35ba84 100644 --- a/examples/README.md +++ b/examples/README.md @@ -198,6 +198,8 @@ topics). Function instances run with the Kubernetes runtime (one pod per instanc | [`values-tls-ca.yaml`](values-tls-ca.yaml) | Enable TLS (including the bookie) using a CA issuer that references an existing `ca-key-pair` secret, with custom DNS SANs for the proxy. Requires cert-manager. | | [`values-jwt-symmetric.yaml`](values-jwt-symmetric.yaml) | Enable JWT authentication and authorization using a symmetric (shared secret) key (`usingSecretKey: true`). The signing key and per-superuser tokens are generated [in-chart](../README.md#in-chart-jwt-secret-generation) (`generateSecrets.enabled: true`), so a fully authenticated cluster deploys with a single `helm install`. | | [`values-jwt-asymmetric.yaml`](values-jwt-asymmetric.yaml) | Enable JWT authentication and authorization using an asymmetric (RSA private/public) key pair (`usingSecretKey: false`). The signing key pair and per-superuser tokens are generated [in-chart](../README.md#in-chart-jwt-secret-generation) (`generateSecrets.enabled: true`), so a fully authenticated cluster deploys with a single `helm install`. | +| [`values-psa-restricted.yaml`](values-psa-restricted.yaml) | Run under the Kubernetes `restricted` Pod Security Standard: `runAsNonRoot` plus no privilege escalation, all capabilities dropped and `RuntimeDefault` seccomp. Leaves the chart's per-component `fsGroup: 0` defaults alone, since `restricted` places no constraint on `fsGroup`. See [Pod and container security contexts](../README.md#pod-and-container-security-contexts). | +| [`values-restricted-group-ranges.yaml`](values-restricted-group-ranges.yaml) | For the narrower case of a policy that constrains group IDs to a numeric range (Gatekeeper's `K8sPSPAllowedUsers`, an equivalent Kyverno policy, or an OpenShift SCC) and so rejects `fsGroup: 0`. Overrides `fsGroup` on the four components that ship it. Not required by PSA `restricted` — changing `fsGroup` on existing volumes triggers a recursive ownership change, so read the warning in the file first. | ### Monitoring diff --git a/examples/values-psa-restricted.yaml b/examples/values-psa-restricted.yaml new file mode 100644 index 00000000..ff40c64d --- /dev/null +++ b/examples/values-psa-restricted.yaml @@ -0,0 +1,58 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +# Run the cluster under the Kubernetes `restricted` Pod Security Standard, i.e. in a +# namespace labelled: +# +# pod-security.kubernetes.io/enforce: restricted +# +# These are the controls `restricted` actually enforces. It places no constraint on +# `fsGroup`, `fsGroupChangePolicy` or `supplementalGroups`, so the chart's per-component +# `fsGroup: 0` defaults are left exactly as they are -- they are what makes the Pulsar +# images work under an arbitrary assigned UID, and `fsGroup: 0` is admitted by +# `restricted` unchanged. +# +# If your cluster additionally runs a policy that constrains group IDs to a numeric +# range, see values-restricted-group-ranges.yaml. + +# The bundled monitoring stack is not restricted-compliant, and the securityContext +# settings below do not reach subchart templates: node-exporter needs hostNetwork, +# hostPID and hostPath, Grafana chowns as root, and some workloads set no seccompProfile. +# Left enabled its pods are rejected on admission while Pulsar itself comes up fine, so +# deploy monitoring separately into a namespace with a policy that suits it. +victoria-metrics-k8s-stack: + enabled: false + +# Applied to every pod the chart renders, including the init/cleanup Jobs. +podSecurityContext: + runAsNonRoot: true + # Set explicitly because the sts-cleanup pre-upgrade hooks run alpine/k8s, which has + # no USER directive: with runAsNonRoot and no runAsUser the kubelet refuses them + # ("image will run as root") and `helm upgrade` is blocked. 10000 is the uid of the + # `pulsar` user in the apachepulsar images, and kubectl runs fine under it too. + runAsUser: 10000 + +# Applied to every container and initContainer. This matches the container-level controls +# of both PSS `restricted` and OpenShift's `restricted-v2` SCC. +containerSecurityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + seccompProfile: + type: RuntimeDefault diff --git a/examples/values-restricted-group-ranges.yaml b/examples/values-restricted-group-ranges.yaml new file mode 100644 index 00000000..6ddc5698 --- /dev/null +++ b/examples/values-restricted-group-ranges.yaml @@ -0,0 +1,107 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +# Run the cluster under a policy that constrains *group IDs* to a numeric range, and so +# rejects the chart's `fsGroup: 0` defaults. Examples of such a policy: +# +# - Gatekeeper's PodSecurityPolicy-derived K8sPSPAllowedUsers constraint template with +# fsGroup: {rule: MustRunAs, ranges: [{min: 1, max: 65535}]} +# supplementalGroups: {rule: MustRunAs, ranges: [{min: 1, max: 65535}]} +# runAsGroup: {rule: MustRunAs, ranges: [{min: 1, max: 65535}]} +# - a Kyverno policy doing the same +# - an OpenShift SCC whose fsGroup rule is MustRunAs against a preallocated range +# +# This is not what the Kubernetes `restricted` Pod Security Standard requires: PSA +# `restricted` places no constraint on fsGroup, fsGroupChangePolicy or supplementalGroups +# and admits `fsGroup: 0` unchanged. If `restricted` is all you need, prefer +# values-psa-restricted.yaml, which leaves the fsGroup defaults alone. +# +# Changing fsGroup on a cluster that already has provisioned volumes triggers a recursive +# ownership change on first mount. `fsGroupChangePolicy: OnRootMismatch` limits that to the +# first pass, but it can still take a long time on large bookie ledger volumes, so test on +# a copy first. +# +# Not an OpenShift example: under the stock `restricted-v2` SCC an explicit fsGroup of +# 10000 is rejected just as 0 is, since MustRunAs validates against the namespace's +# preallocated range. There you want to omit the group settings and let admission inject +# them, which means clearing the chart defaults (`zookeeper: {securityContext: null}` and +# so on) rather than overriding them. + +# The bundled monitoring stack is disabled: the settings below do not reach subchart +# templates, so its pods are rejected by a group-range policy while Pulsar comes up fine. +# Deploy monitoring separately into a namespace with a policy that suits it. +victoria-metrics-k8s-stack: + enabled: false + +# Applied to every pod the chart renders, including the init/cleanup Jobs. +podSecurityContext: + runAsNonRoot: true + # 10000 is the uid of the `pulsar` user in the apachepulsar images. + runAsUser: 10000 + runAsGroup: 10000 + fsGroup: 10000 + fsGroupChangePolicy: OnRootMismatch + supplementalGroups: [10000] + +# Applied to every container and initContainer the chart renders. +containerSecurityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + seccompProfile: + type: RuntimeDefault + +# If your policy also requires a liveness and readiness probe on every long-running +# workload, note that the autorecovery StatefulSet has none by default. You can enable +# them, but only if you have not overridden `statsProviderClass` or +# `prometheusStatsHttpPort` in `autorecovery.configData` -- both probes target the +# /metrics endpoint served by the image's BookKeeper stats provider, so changing either +# setting breaks them. +# +# autorecovery: +# probe: +# liveness: +# enabled: true +# readiness: +# enabled: true + +# zookeeper, bookkeeper, broker and oxia.server ship `securityContext.fsGroup: 0`, and a +# per-component value takes precedence over the global one, so each has to be overridden +# explicitly for a group-range policy to admit them. +# +# `fsGroup` is added as a supplementary group, so `runAsGroup` above does not need to +# match these values. +zookeeper: + securityContext: + fsGroup: 10000 + fsGroupChangePolicy: OnRootMismatch +bookkeeper: + securityContext: + fsGroup: 10000 + fsGroupChangePolicy: OnRootMismatch +broker: + securityContext: + fsGroup: 10000 + fsGroupChangePolicy: OnRootMismatch +# Only used when the metadata store is Oxia (components.oxia: true). +oxia: + server: + securityContext: + fsGroup: 10000 + fsGroupChangePolicy: OnRootMismatch From d4126752e0779523ed483af225ddb71ba3d0db21 Mon Sep 17 00:00:00 2001 From: Robert Moucha Date: Wed, 19 Aug 2026 12:13:09 +0200 Subject: [PATCH 2/2] feat(chart): support readOnlyRootFilesystem Setting `containerSecurityContext.readOnlyRootFilesystem: true` previously rendered but did not run: the Pulsar images rewrite their configuration under /pulsar/conf on startup (bin/apply-config-from-env.py), write logs under /pulsar/logs, and the JVM and functions worker use /tmp. The chart now mounts an emptyDir over each path in the new `emptyDirVolumes` list on every container and initContainer of the component, and prepends a copy-pulsar-conf initContainer that seeds the entries marked `seedFromImage` from the image, because an emptyDir starts empty and apply-config-from-env.py edits files that must already exist. The default list describes the Pulsar images: emptyDirVolumes: - path: /pulsar/conf seedFromImage: true - path: /pulsar/logs sizeLimit: 1Gi - path: /tmp sizeLimit: 1Gi A list rather than a boolean, so that an existing deployment is not broken by a mountPath the chart cannot know about: an entry whose path the component already mounts through extraVolumeMounts is dropped, since a duplicate mountPath is rejected outright by the API server. `sizeLimit` is set because an uncapped emptyDir can fill a node's ephemeral storage and get pods evicted. This could not be done from values alone. Values-supplied `.initContainers` are appended after the built-in ones, so a seeding container cannot run first, and only autorecovery exposes initContainersExtraVolumeMounts, so the built-in init containers elsewhere could not be given the conf volume. broker/wait-bookkeeper-ready in particular runs apply-config-from-env.py and would fail. The volumes are driven by the effective container securityContext, using the same global/per-component merge as `pulsar.containerSecurityContext`, so one global setting covers the whole release and a component that overrides readOnlyRootFilesystem back to false also loses the volumes. `.emptyDirVolumes` replaces the list it would otherwise inherit; `[]` opts a component out entirely, for users who would rather declare the mounts themselves through extraVolumes/extraVolumeMounts. Resolution is `--reuse-values` safe. The helper carries a copy of the values.yaml default and distinguishes an absent key from an explicit `[]` with `hasKey`, because `helm upgrade --reuse-values` does not coalesce in a new chart's defaults: the key is simply absent there, and treating that as an empty list would render a read-only root filesystem with none of the volumes that make it work. Components that do not run a Pulsar image must not receive that Pulsar-shaped list -- the seed step would try to copy a /pulsar/conf that does not exist in their images -- so they carry their own default at the call site rather than in values.yaml, which keeps them correct when the key is absent. Measured by running each image read-only: * oxia server and coordinator get /tmp. Oxia keeps its own state under the data directory -- WAL, Pebble SSTables and snapshots all live there, and an incoming snapshot is written straight into the db path rather than staged elsewhere -- and /tmp stayed empty through 260MB of write load per server, compaction, and a full snapshot install into a wiped follower, measured with a writable /tmp mounted so that any use of it would have been visible. Neither Pebble nor hashicorp/raft references os.TempDir outside their tests. It is provided regardless: the cost is one emptyDir, and a dependency or a later oxia version falling back to os.TempDir() would otherwise take the metadata store down under load rather than at startup. * dekaf needs /tmp; without it the JVM fails with java.nio.file.FileSystemException: /tmp/dekaf...: Read-only file system. * the sts-cleanup pre-upgrade hooks run kubectl from alpine/k8s, which needs nothing writable, and are left outside the mechanism because the values key they would read is shared with a Pulsar-image workload. The pulsar_manager StatefulSet cannot support it. Beyond /run/postgresql and /tmp the apachepulsar/pulsar-manager image needs /var/log/nginx, /var/lib/nginx and /run/nginx.pid, and writes a supervisord socket into /pulsar-manager, its own install directory, which an emptyDir cannot cover without hiding the application. Enabling both now fails the render with a message naming the reason and the opt-out, rather than producing a pod that never becomes ready. Note that opting it out also opts out its cluster-initialize Job, which shares the pulsar_manager securityContext even though it runs a Pulsar image and would otherwise work. A path is rejected at render time rather than at apply time when it is relative, carries an unknown key (`seedFromimage` would otherwise silently leave the conf directory unseeded), collides with another entry, reduces to the same volume name as another entry (/pulsar/logs and /pulsar-logs both give pulsar-logs), or is long enough to exceed the 63-character limit on a Kubernetes name. The bookkeeper StatefulSet emitted `initContainers:` only when waitMetadataTimeout was greater than zero. It is now emitted whenever anything needs it -- waitMetadataTimeout, this feature, cacerts, or user-supplied bookkeeper.initContainers -- with verify-clusterid still gated on the timeout. Besides being required here, this fixes a latent bug that predates this change: with `bookkeeper.waitMetadataTimeout: 0` and `bookkeeper.initContainers` set, the user's init containers were emitted without their parent key and the template failed to parse. Since readOnlyRootFilesystem is unset by default, rendered output is unchanged. Verified against the previous commit for the default values, all 19 .ci/clusters/* configs, .ci/values-common.yaml and .ci/templates-all-values.yaml, comparing only the parent chart's documents (the bundled victoria-metrics-k8s-stack regenerates its self-signed webhook certificates on every helm invocation). With it enabled, 51 clean kubeconform runs across k8s 1.25/1.31/1.36 over 17 value sets covering TLS, JWT, standalone, oxia, dekaf, all components, and three --reuse-values shapes. Verified at runtime on EKS 1.34 by installing without it and then running `helm upgrade --reuse-values --set containerSecurityContext.readOnlyRootFilesystem=true`: every StatefulSet gains the seed container and the volumes, copy-pulsar-conf completes first, the seeded conf is rewritten by apply-config-from-env.py, and 200 messages produced before the upgrade are consumed after it with no read-only filesystem errors in any container. oxia and dekaf were confirmed read-only at runtime with the volumes above and nothing else. Logs on an emptyDir do not survive pod replacement, which is noted in values.yaml and the README. --- README.md | 84 +++++++- charts/pulsar/templates/_helpers.tpl | 189 ++++++++++++++++++ .../templates/autorecovery-statefulset.yaml | 16 ++ .../bookkeeper-cluster-initialize.yaml | 21 ++ .../templates/bookkeeper-statefulset.yaml | 29 ++- .../pulsar/templates/broker-statefulset.yaml | 23 +++ charts/pulsar/templates/dekaf-deployment.yaml | 7 + .../function-worker-statefulset.yaml | 21 +- charts/pulsar/templates/jwt-secret-init.yaml | 18 ++ .../oxia-coordinator-deployment.yaml | 19 +- .../templates/oxia-server-statefulset.yaml | 8 + .../pulsar/templates/proxy-statefulset.yaml | 31 ++- .../templates/pulsar-cluster-initialize.yaml | 27 +++ .../pulsar-manager-cluster-initialize.yaml | 26 +++ .../templates/pulsar-manager-statefulset.yaml | 4 + .../templates/standalone-deployment.yaml | 14 +- .../pulsar/templates/toolset-statefulset.yaml | 17 ++ .../templates/zookeeper-statefulset.yaml | 13 ++ charts/pulsar/values.yaml | 34 +++- examples/README.md | 2 +- examples/values-psa-restricted.yaml | 6 + examples/values-restricted-group-ranges.yaml | 1 + 22 files changed, 588 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index c7a6fa34..4d7428d5 100644 --- a/README.md +++ b/README.md @@ -378,11 +378,85 @@ changing a production cluster. ### readOnlyRootFilesystem -`containerSecurityContext.readOnlyRootFilesystem: true` is accepted but does not work -out of the box: the Pulsar images rewrite their configuration under `/pulsar/conf` on -startup (`bin/apply-config-from-env.py`) and write logs under `/pulsar/logs`. Mount -writable volumes over those paths using `.extraVolumes` and -`.extraVolumeMounts` before enabling it. +The Pulsar images do not run on a read-only root filesystem unaided. They rewrite their +configuration under `/pulsar/conf` on startup (`bin/apply-config-from-env.py`), write logs +under `/pulsar/logs`, and the JVM and the functions worker use `/tmp`. + +Setting `containerSecurityContext.readOnlyRootFilesystem: true` is nevertheless enough. +For each entry in `emptyDirVolumes` the chart then mounts an `emptyDir` at that path on +every container and initContainer of the component, and for entries marked +`seedFromImage: true` it prepends a `copy-pulsar-conf` initContainer that copies the +image's contents into the volume first — an `emptyDir` starts empty, and +`apply-config-from-env.py` edits files that must already exist. + +The chart-wide default describes the Pulsar images: + +```yaml +emptyDirVolumes: + - path: /pulsar/conf + seedFromImage: true + - path: /pulsar/logs + sizeLimit: 1Gi + - path: /tmp + sizeLimit: 1Gi +``` + +`sizeLimit` matters: without it an `emptyDir` is unbounded and a busy `/pulsar/logs` can +fill a node's ephemeral storage and get pods evicted. + +This is driven by the effective (merged) container securityContext, so a single global +`readOnlyRootFilesystem: true` covers the release, and a component that overrides it back +to `false` also loses the volumes. + +#### Per-component paths + +Every component takes its own `.emptyDirVolumes`, which **replaces** the list +it would otherwise inherit rather than adding to it — Helm merges maps per key but +replaces lists wholesale. Use `[]` for "this component needs none". + +The three paths above describe the Pulsar images, so they are not applied to components +that run something else. Those carry their own default and never inherit the chart-wide +list: + +| Component | Default | Why | +|---|---|---| +| `oxia.server`, `oxia.coordinator` | `/tmp` | the `oxia` binary keeps its own state under its data directory, but `/tmp` is provided so that a Go dependency or a future version falling back to `os.TempDir()` cannot take the metadata store down | +| `dekaf.deployment` | `/tmp` | the JVM writes scratch files there | +| everything else | the three above | Pulsar images | + +Setting `.emptyDirVolumes` still overrides either default. + +To hand a path back to yourself, drop its entry and declare it through +`.extraVolumes` / `.extraVolumeMounts`. Declaring just the mount is +enough: an entry whose path the component already mounts via `extraVolumeMounts` is +skipped, because a duplicate `mountPath` is rejected by the API server. Bear in mind that +whatever you mount over `/pulsar/conf` has to be populated, since nothing seeds it for you. + +Rejected at render time rather than at apply time: a relative path, an unknown entry key, +two paths that reduce to the same volume name (`/pulsar/logs` and `/pulsar-logs` both give +`pulsar-logs`), and a path long enough to exceed the 63-character limit on a name. + +#### pulsar_manager is not supported + +`pulsar_manager` cannot run with a read-only root filesystem. Besides `/run`, +`/var/log/nginx` and `/var/lib/nginx`, the `apachepulsar/pulsar-manager` image writes a +supervisord socket into `/pulsar-manager` — its own install directory, which an `emptyDir` +cannot cover without hiding the application. Enabling both fails the render with an +explanatory message; set + +```yaml +pulsar_manager: + containerSecurityContext: + readOnlyRootFilesystem: false +``` + +to exclude that component. Note this also excludes its cluster-initialize Job, which +shares the `pulsar_manager` securityContext even though it runs a Pulsar image and would +otherwise support a read-only root filesystem. + +Note that logs written to an `emptyDir` do not survive pod replacement. If you rely on +reading `/pulsar/logs` from inside a pod, ship them off-node or keep +`readOnlyRootFilesystem` disabled. ## Disabling victoria-metrics-k8s-stack components diff --git a/charts/pulsar/templates/_helpers.tpl b/charts/pulsar/templates/_helpers.tpl index 99cdde03..63913fb4 100644 --- a/charts/pulsar/templates/_helpers.tpl +++ b/charts/pulsar/templates/_helpers.tpl @@ -157,6 +157,195 @@ Both operands are deep-copied so that `.Values` is never mutated. {{- end -}} {{- end -}} +{{/* +Resolve the effective emptyDirVolumes list for a component, returned as JSON (a template +can only return a string; callers pipe it through `fromJsonArray`). + +Precedence, highest first: + 1. `.emptyDirVolumes` + 2. the caller's `default`, passed only by components that do not run the Pulsar image, + so that the chart-wide list -- which describes the Pulsar layout -- cannot apply + 3. `.Values.emptyDirVolumes` + 4. the chart default below, a copy of the values.yaml default. `helm upgrade + --reuse-values` does not coalesce in new chart defaults, so the key is absent there; + treating that as `[]` would give a read-only root filesystem and none of the volumes + that make it work. Keep the two copies in sync. +A list REPLACES the one above it rather than extending it, because Helm merges maps per +key but replaces lists whole. + +Entries whose path the component already mounts via `extraVolumeMounts` are dropped: a +duplicate mountPath is rejected by the API server, so an existing setup that provides one +of these paths itself keeps working. + +Usage: + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.broker "root" .)) }} +*/}} +{{- define "pulsar.emptyDirVolumes.resolve" -}} +{{- $list := list (dict "path" "/pulsar/conf" "seedFromImage" true) (dict "path" "/pulsar/logs" "sizeLimit" "1Gi") (dict "path" "/tmp" "sizeLimit" "1Gi") -}} +{{- if hasKey .root.Values "emptyDirVolumes" -}} +{{- $list = .root.Values.emptyDirVolumes | default list -}} +{{- end -}} +{{- if hasKey . "default" -}} +{{- $list = .default | default list -}} +{{- end -}} +{{- $component := .component | default dict -}} +{{- if hasKey $component "emptyDirVolumes" -}} +{{- $list = (index $component "emptyDirVolumes") | default list -}} +{{- end -}} +{{- include "pulsar.emptyDirVolumes.validate" $list -}} +{{- $mounted := list -}} +{{- range (index $component "extraVolumeMounts") | default list -}} +{{- $mounted = append $mounted .mountPath -}} +{{- end -}} +{{- $effective := list -}} +{{- range $list -}} +{{- if not (has .path $mounted) -}} +{{- $effective = append $effective . -}} +{{- end -}} +{{- end -}} +{{- $effective | toJson -}} +{{- end -}} + +{{/* +Whether to render the emptyDir volumes for a component: true when the effective container +securityContext sets `readOnlyRootFilesystem` and the resolved list is non-empty. The +merge matches `pulsar.containerSecurityContext`, so a component that opts out of a +read-only root filesystem also opts out of these volumes. + +Usage (as a guard): + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.broker.containerSecurityContext "volumes" $edv "root" .) }} +*/}} +{{- define "pulsar.emptyDirVolumes.enabled" -}} +{{- $global := deepCopy (.root.Values.containerSecurityContext | default dict) -}} +{{- $override := deepCopy (.securityContext | default dict) -}} +{{- if and (mergeOverwrite $global $override).readOnlyRootFilesystem .volumes -}} +true +{{- end -}} +{{- end -}} + +{{/* +Turn a path into a DNS-1123 volume name: /pulsar/conf -> pulsar-conf, /tmp -> tmp. +*/}} +{{- define "pulsar.emptyDirVolumes.name" -}} +{{- $n := . | trim | trimAll "/" | replace "/" "-" | replace "_" "-" | replace "." "-" | lower -}} +{{- if not (regexMatch "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$" $n) -}} +{{- fail (printf "emptyDirVolumes: path %q does not yield a valid volume name (got %q)" . $n) -}} +{{- end -}} +{{- if gt (len $n) 63 -}} +{{- fail (printf "emptyDirVolumes: path %q yields the volume name %q, which exceeds the 63 character limit for a Kubernetes name" . $n) -}} +{{- end -}} +{{- $n -}} +{{- end -}} + +{{/* +Validate an emptyDirVolumes list. Every entry needs an absolute `path` and no key beyond +path/seedFromImage/sizeLimit -- a typo such as `seedFromimage` would otherwise leave the +conf directory unseeded and only fail at runtime. No two entries may collide on path or on +derived name (/pulsar/logs and /pulsar-logs both give pulsar-logs). +*/}} +{{- define "pulsar.emptyDirVolumes.validate" -}} +{{- if not (kindIs "slice" .) -}} +{{- fail (printf "emptyDirVolumes must be a list, got %v (%s). Note that `--set emptyDirVolumes=[]` assigns the string \"[]\"; use a values file to set an empty list." . (kindOf .)) -}} +{{- end -}} +{{- $names := dict -}} +{{- $paths := dict -}} +{{- range . -}} +{{- if not (kindIs "map" .) -}} +{{- fail (printf "emptyDirVolumes: every entry must be a mapping with a `path` key, got %v (%s)." . (kindOf .)) -}} +{{- end -}} +{{- range $k, $_ := . -}} +{{- if not (has $k (list "path" "seedFromImage" "sizeLimit")) -}} +{{- fail (printf "emptyDirVolumes: unknown key %q; supported keys are path, seedFromImage and sizeLimit" $k) -}} +{{- end -}} +{{- end -}} +{{- if not .path -}} +{{- fail "emptyDirVolumes: every entry requires a `path`" -}} +{{- end -}} +{{- if not (hasPrefix "/" .path) -}} +{{- fail (printf "emptyDirVolumes: path %q must be absolute" .path) -}} +{{- end -}} +{{- if hasKey $paths .path -}} +{{- fail (printf "emptyDirVolumes: path %q is listed twice" .path) -}} +{{- end -}} +{{- $_ := set $paths .path true -}} +{{- $n := include "pulsar.emptyDirVolumes.name" .path -}} +{{- if hasKey $names $n -}} +{{- fail (printf "emptyDirVolumes: paths %q and %q both map to volume name %q; rename one" (index $names $n) .path $n) -}} +{{- end -}} +{{- $_ := set $names $n .path -}} +{{- end -}} +{{- end -}} + +{{/* +The emptyDir volumes for a resolved list. + +Usage: + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} +*/}} +{{- define "pulsar.emptyDirVolumes.volumes" -}} +{{- range . }} +- name: {{ include "pulsar.emptyDirVolumes.name" .path }} + emptyDir: + {{- if .sizeLimit }} + sizeLimit: {{ .sizeLimit }} + {{- else }} + {} + {{- end }} +{{- end -}} +{{- end -}} + +{{/* +The matching mounts, applied to every container and initContainer of the component. + +Usage: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} +*/}} +{{- define "pulsar.emptyDirVolumes.mounts" -}} +{{- range . }} +- name: {{ include "pulsar.emptyDirVolumes.name" .path }} + mountPath: {{ .path }} +{{- end -}} +{{- end -}} + +{{/* +An initContainer that seeds the entries marked `seedFromImage: true`, because an emptyDir +starts empty and the Pulsar images rewrite files under /pulsar/conf that must exist. + +Must render as the *first* initContainer, ahead of any built-in one that reads the seeded +path. Each seeded volume is mounted at /mnt/, not at its real path, so it does not +shadow the copy in the image. `cp -r` rather than `cp -a`: preserving ownership fails as a +non-root user. Renders nothing when no entry is seeded. + +Usage: + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.broker "securityContext" .Values.broker.containerSecurityContext "root" .) | nindent 6 }} +*/}} +{{- define "pulsar.emptyDirVolumes.seedContainer" -}} +{{- $seed := list -}} +{{- range .volumes -}} +{{- if .seedFromImage -}} +{{- $seed = append $seed . -}} +{{- end -}} +{{- end -}} +{{- if $seed -}} +- name: copy-pulsar-conf + image: "{{ template "pulsar.imageFullName" (dict "image" .image "root" .root) }}" + imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .image "root" .root) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .securityContext "root" .root "indent" 2) }} + resources: {{ toYaml .root.Values.initContainer.resources | nindent 4 }} + command: ["sh", "-c"] + args: + - | + {{- range $seed }} + cp -r {{ .path }}/. /mnt/{{ include "pulsar.emptyDirVolumes.name" .path }}/ + {{- end }} + volumeMounts: + {{- range $seed }} + - name: {{ include "pulsar.emptyDirVolumes.name" .path }} + mountPath: /mnt/{{ include "pulsar.emptyDirVolumes.name" .path }} + {{- end }} +{{- end -}} +{{- end -}} + {{/* Create ImagePullSecrets */}} diff --git a/charts/pulsar/templates/autorecovery-statefulset.yaml b/charts/pulsar/templates/autorecovery-statefulset.yaml index c32116e1..94b25880 100644 --- a/charts/pulsar/templates/autorecovery-statefulset.yaml +++ b/charts/pulsar/templates/autorecovery-statefulset.yaml @@ -115,8 +115,12 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.autorecovery.gracePeriod }} serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.autorecovery.component }}" + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.autorecovery "root" .)) }} {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.autorecovery.securityContext "root" . "indent" 6) }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.autorecovery.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.autorecovery "securityContext" .Values.autorecovery.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} {{- if .Values.tls.autorecovery.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.autorecovery "root" .) }}" @@ -132,6 +136,9 @@ spec: {{ toYaml .Values.autorecovery.initContainersExtraVolumeMounts | indent 8 }} {{- end }} {{- include "pulsar.autorecovery.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.autorecovery.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.autorecovery.waitBookkeeperTimeout (gt (.Values.autorecovery.waitBookkeeperTimeout | int) 0) }} # This initContainer will wait for bookkeeper initnewcluster to complete @@ -153,6 +160,9 @@ spec: {{ toYaml .Values.autorecovery.initContainersExtraVolumeMounts | indent 8 }} {{- end }} {{- include "pulsar.autorecovery.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.autorecovery.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if .Values.autorecovery.initContainers }} {{- toYaml .Values.autorecovery.initContainers | nindent 6 }} @@ -235,7 +245,13 @@ spec: {{ toYaml .Values.autorecovery.extraVolumeMounts | indent 8 }} {{- end }} {{- include "pulsar.autorecovery.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.autorecovery.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} volumes: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.autorecovery.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} {{- include "pulsar.autorecovery.certs.volumes" . | nindent 6 }} {{- if .Values.autorecovery.extraVolumes }} {{ toYaml .Values.autorecovery.extraVolumes | indent 6 }} diff --git a/charts/pulsar/templates/bookkeeper-cluster-initialize.yaml b/charts/pulsar/templates/bookkeeper-cluster-initialize.yaml index 07f9c511..76fcbdd2 100755 --- a/charts/pulsar/templates/bookkeeper-cluster-initialize.yaml +++ b/charts/pulsar/templates/bookkeeper-cluster-initialize.yaml @@ -44,6 +44,7 @@ spec: spec: {{- include "pulsar.imagePullSecrets" . | nindent 6 }} serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}" + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.bookkeeper "root" .)) }} {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.bookkeeper.securityContext "root" . "indent" 6) }} nodeSelector: {{- if .Values.pulsar_metadata.nodeSelector }} @@ -54,6 +55,9 @@ spec: {{ toYaml .Values.pulsar_metadata.tolerations | indent 8 }} {{- end }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.bookie "securityContext" .Values.bookkeeper.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} {{- if .Values.tls.bookie.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.bookie "root" .) }}" @@ -66,6 +70,9 @@ spec: bin/certs-combine-pem.sh /pulsar/certs/cacerts/ca-combined.pem {{ template "pulsar.certs.cacerts" (dict "certs" .Values.tls.bookie.cacerts.certs) }} volumeMounts: {{- include "pulsar.toolset.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.components.zookeeper .Values.bookkeeper.metadata.waitZookeeperTimeout (gt (.Values.bookkeeper.metadata.waitZookeeperTimeout | int) 0) }} - name: wait-zookeeper-ready @@ -86,6 +93,10 @@ spec: sleep 3; done; {{- end}} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end}} {{- if and .Values.components.oxia .Values.bookkeeper.metadata.waitOxiaTimeout (gt (.Values.bookkeeper.metadata.waitOxiaTimeout | int) 0) }} - name: wait-oxia-ready @@ -99,6 +110,10 @@ spec: until nslookup {{ template "pulsar.oxia.server.service" . }}; do sleep 3; done; + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} containers: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}-init" @@ -131,8 +146,14 @@ spec: name: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}" volumeMounts: {{- include "pulsar.toolset.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} volumes: {{- include "pulsar.toolset.certs.volumes" . | nindent 6 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} restartPolicy: OnFailure {{- end }} {{- end }} diff --git a/charts/pulsar/templates/bookkeeper-statefulset.yaml b/charts/pulsar/templates/bookkeeper-statefulset.yaml index b06eb4d2..3ff36992 100644 --- a/charts/pulsar/templates/bookkeeper-statefulset.yaml +++ b/charts/pulsar/templates/bookkeeper-statefulset.yaml @@ -112,9 +112,22 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.bookkeeper.gracePeriod }} serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}" + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.bookkeeper "root" .)) }} {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.bookkeeper.securityContext "root" . "indent" 6) }} - {{- if and .Values.bookkeeper.waitMetadataTimeout (gt (.Values.bookkeeper.waitMetadataTimeout | int) 0) }} + {{- /* + `initContainers:` is emitted whenever anything needs it, not only when + waitMetadataTimeout is set. The conf-seeding container has to run regardless, or a + read-only root filesystem gets an empty emptyDir over /pulsar/conf and the bookie + cannot start. This also keeps user-supplied bookkeeper.initContainers from being + emitted without their parent key when waitMetadataTimeout is 0. + */}} + {{- $bkWaitMetadata := and .Values.bookkeeper.waitMetadataTimeout (gt (.Values.bookkeeper.waitMetadataTimeout | int) 0) }} + {{- $bkRootfs := include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- if or $bkWaitMetadata $bkRootfs .Values.tls.bookie.cacerts.enabled .Values.bookkeeper.initContainers }} initContainers: + {{- if $bkRootfs }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.bookie "securityContext" .Values.bookkeeper.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} {{- if .Values.tls.bookie.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.bookie "root" .) }}" @@ -127,7 +140,11 @@ spec: bin/certs-combine-pem.sh /pulsar/certs/cacerts/ca-combined.pem {{ template "pulsar.certs.cacerts" (dict "certs" .Values.tls.bookie.cacerts.certs) }} volumeMounts: {{- include "pulsar.bookkeeper.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} + {{- if $bkWaitMetadata }} # This initContainer will wait for bookkeeper initnewcluster to complete # before deploying the bookies - name: pulsar-bookkeeper-verify-clusterid @@ -145,10 +162,14 @@ spec: name: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}" volumeMounts: {{- include "pulsar.bookkeeper.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if .Values.bookkeeper.initContainers }} {{- toYaml .Values.bookkeeper.initContainers | nindent 6 }} {{- end }} + {{- end }} containers: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.bookie "root" .) }}" @@ -278,7 +299,13 @@ spec: {{ toYaml .Values.bookkeeper.extraVolumeMounts | indent 8 }} {{- end }} {{- include "pulsar.bookkeeper.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} volumes: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} {{- if not (and (and .Values.persistence .Values.volumes.persistence) .Values.bookkeeper.volumes.persistence) }} - name: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}-{{ .Values.bookkeeper.volumes.journal.name }}" emptyDir: {} diff --git a/charts/pulsar/templates/broker-statefulset.yaml b/charts/pulsar/templates/broker-statefulset.yaml index 87b2eb86..d28b4a16 100644 --- a/charts/pulsar/templates/broker-statefulset.yaml +++ b/charts/pulsar/templates/broker-statefulset.yaml @@ -75,6 +75,7 @@ spec: {{- end }} spec: serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.broker.component }}-acct" + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.broker "root" .)) }} {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.broker.securityContext "root" . "indent" 6) }} {{- if .Values.broker.nodeSelector }} nodeSelector: @@ -134,6 +135,9 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.broker.gracePeriod }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.broker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.broker "securityContext" .Values.broker.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} {{- if .Values.tls.broker.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.broker "root" .) }}" @@ -146,6 +150,9 @@ spec: bin/certs-combine-pem.sh /pulsar/certs/cacerts/ca-combined.pem {{ template "pulsar.certs.cacerts" (dict "certs" .Values.tls.broker.cacerts.certs) }} volumeMounts: {{- include "pulsar.broker.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.broker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.components.zookeeper .Values.broker.waitZookeeperTimeout (gt (.Values.broker.waitZookeeperTimeout | int) 0) }} # This init container will wait for zookeeper to be ready before @@ -170,6 +177,9 @@ spec: done; volumeMounts: {{- include "pulsar.broker.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.broker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.components.oxia .Values.broker.waitOxiaTimeout (gt (.Values.broker.waitOxiaTimeout | int) 0) }} - name: wait-oxia-ready @@ -183,6 +193,10 @@ spec: until nslookup {{ template "pulsar.oxia.server.service" . }}; do sleep 3; done; + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.broker.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.broker.waitBookkeeperTimeout (gt (.Values.broker.waitBookkeeperTimeout | int) 0) }} # This init container will wait for bookkeeper to be ready before @@ -215,6 +229,9 @@ spec: name: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}" volumeMounts: {{- include "pulsar.broker.certs.volumeMounts" . | nindent 10 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.broker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 10 }} + {{- end }} {{- end }} {{- if .Values.broker.initContainers }} {{- toYaml .Values.broker.initContainers | nindent 6 }} @@ -326,6 +343,9 @@ spec: mountPath: {{ .Values.broker.packageManagement.fileSystemStorage.storagePath }} {{- end }} {{- include "pulsar.broker.certs.volumeMounts" . | nindent 10 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.broker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 10 }} + {{- end }} env: {{- if and (and .Values.broker.storageOffload (eq .Values.broker.storageOffload.driver "aws-s3")) .Values.broker.storageOffload.secret }} - name: AWS_ACCESS_KEY_ID @@ -355,6 +375,9 @@ spec: {{- toYaml .Values.broker.extraEnvs | nindent 10 }} {{- end }} volumes: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.broker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} {{- if .Values.broker.extraVolumes }} {{ toYaml .Values.broker.extraVolumes | indent 6 }} {{- end }} diff --git a/charts/pulsar/templates/dekaf-deployment.yaml b/charts/pulsar/templates/dekaf-deployment.yaml index ca50d329..7fbe584f 100644 --- a/charts/pulsar/templates/dekaf-deployment.yaml +++ b/charts/pulsar/templates/dekaf-deployment.yaml @@ -57,6 +57,7 @@ spec: priorityClassName: {{ .Values.dekaf.deployment.priorityClassName }} {{- end }} + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" (((.Values.dekaf).deployment) | default dict) "default" (list (dict "path" "/tmp" "sizeLimit" "1Gi")) "root" .)) }} {{- include "pulsar.podSecurityContext" (dict "securityContext" ((.Values.dekaf).deployment).securityContext "root" . "indent" 6) }} containers: - name: dekaf @@ -94,6 +95,9 @@ spec: {{- if ((.Values.dekaf).persistence).enabled }} volumeMounts: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" ((.Values.dekaf).deployment).containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 12 }} + {{- end }} - name: {{ template "pulsar.fullname" . }}-{{ template "dekaf.component" . }}-library mountPath: /dekaf/data/library {{- end }} @@ -107,6 +111,9 @@ spec: {{- end }} volumes: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" ((.Values.dekaf).deployment).containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} {{- if ((.Values.dekaf).persistence).enabled }} - name: {{ template "pulsar.fullname" . }}-{{ template "dekaf.component" . }}-library persistentVolumeClaim: diff --git a/charts/pulsar/templates/function-worker-statefulset.yaml b/charts/pulsar/templates/function-worker-statefulset.yaml index b1cdcb93..61411112 100644 --- a/charts/pulsar/templates/function-worker-statefulset.yaml +++ b/charts/pulsar/templates/function-worker-statefulset.yaml @@ -55,6 +55,7 @@ spec: {{- end }} spec: serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.function_worker.component }}-acct" + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.function_worker "root" .)) }} {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.function_worker.securityContext "root" . "indent" 6) }} {{- if .Values.function_worker.nodeSelector }} nodeSelector: @@ -114,6 +115,9 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.function_worker.gracePeriod }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.function_worker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.function_worker "securityContext" .Values.function_worker.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} {{- if .Values.tls.function_worker.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.function_worker "root" .) }}" @@ -126,6 +130,9 @@ spec: bin/certs-combine-pem.sh /pulsar/certs/cacerts/ca-combined.pem {{ template "pulsar.certs.cacerts" (dict "certs" .Values.tls.function_worker.cacerts.certs) }} volumeMounts: {{- include "pulsar.function_worker.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.function_worker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.components.broker .Values.function_worker.waitBrokerTimeout (gt (.Values.function_worker.waitBrokerTimeout | int) 0) }} # Wait for the broker to be resolvable before starting the worker (the worker is broker-attached). @@ -141,6 +148,10 @@ spec: echo "waiting for broker {{ template "pulsar.fullname" . }}-{{ .Values.broker.component }} ... check in 3 seconds ..."; sleep 3; done; + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.function_worker.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if .Values.function_worker.initContainers }} {{- toYaml .Values.function_worker.initContainers | nindent 6 }} @@ -206,8 +217,11 @@ spec: - configMapRef: name: "{{ template "pulsar.fullname" . }}-{{ .Values.function_worker.component }}" {{- $jwtAuth := and .Values.auth.authentication.enabled .Values.auth.authentication.jwt.enabled }} - {{- if or $jwtAuth .Values.function_worker.extraVolumeMounts .Values.tls.enabled }} + {{- if or $jwtAuth .Values.function_worker.extraVolumeMounts .Values.tls.enabled (include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.function_worker.containerSecurityContext "volumes" $edv "root" .)) }} volumeMounts: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.function_worker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 10 }} + {{- end }} {{- if $jwtAuth }} - mountPath: "/pulsar/keys" name: token-keys @@ -225,8 +239,11 @@ spec: env: {{- toYaml .Values.function_worker.extraEnvs | nindent 10 }} {{- end }} - {{- if or $jwtAuth .Values.function_worker.extraVolumes .Values.tls.enabled }} + {{- if or $jwtAuth .Values.function_worker.extraVolumes .Values.tls.enabled (include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.function_worker.containerSecurityContext "volumes" $edv "root" .)) }} volumes: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.function_worker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} {{- if .Values.function_worker.extraVolumes }} {{ toYaml .Values.function_worker.extraVolumes | indent 6 }} {{- end }} diff --git a/charts/pulsar/templates/jwt-secret-init.yaml b/charts/pulsar/templates/jwt-secret-init.yaml index 9bc6fee4..619a5c48 100644 --- a/charts/pulsar/templates/jwt-secret-init.yaml +++ b/charts/pulsar/templates/jwt-secret-init.yaml @@ -98,6 +98,7 @@ spec: {{- include "pulsar.imagePullSecrets" . | nindent 6 }} serviceAccountName: "{{ template "pulsar.fullname" . }}-jwt-secret-init" restartPolicy: Never + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.auth.authentication.jwt.generateSecrets "root" .)) }} {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.auth.authentication.jwt.generateSecrets.securityContext "root" . "indent" 6) }} {{- if .Values.auth.authentication.jwt.generateSecrets.nodeSelector }} nodeSelector: @@ -152,7 +153,13 @@ spec: - name: jwt-secrets emptyDir: sizeLimit: 1Mi + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.auth.authentication.jwt.generateSecrets.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 8 }} + {{- end }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.auth.authentication.jwt.generateSecrets.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.broker "securityContext" .Values.auth.authentication.jwt.generateSecrets.containerSecurityContext "root" .) | nindent 8 }} + {{- end }} - name: generate-jwt-artifacts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.broker "root" .) }}" imagePullPolicy: {{ include "pulsar.imagePullPolicy" (dict "image" .Values.images.broker "root" .) }} @@ -193,6 +200,9 @@ spec: volumeMounts: - mountPath: /jwt-secrets name: jwt-secrets + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.auth.authentication.jwt.generateSecrets.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 12 }} + {{- end }} containers: - name: create-secrets image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.kubectl "root" .) }}" @@ -252,5 +262,13 @@ spec: volumeMounts: - mountPath: /jwt-secrets name: jwt-secrets + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.auth.authentication.jwt.generateSecrets.containerSecurityContext "volumes" $edv "root" .) }} + {{- /* + This container runs the kubectl image rather than a Pulsar image. The mounts + are harmless there -- it has no /pulsar tree for them to shadow -- and it does + need a writable /tmp, so it takes the same list as the rest of the Job. + */}} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 12 }} + {{- end }} {{- end }} {{- end }} diff --git a/charts/pulsar/templates/oxia-coordinator-deployment.yaml b/charts/pulsar/templates/oxia-coordinator-deployment.yaml index fef77cf1..a239a9db 100644 --- a/charts/pulsar/templates/oxia-coordinator-deployment.yaml +++ b/charts/pulsar/templates/oxia-coordinator-deployment.yaml @@ -61,6 +61,7 @@ spec: priorityClassName: {{ .Values.oxia.coordinator.priorityClassName }} {{- end }} serviceAccountName: {{ template "pulsar.fullname" . }}-{{ .Values.oxia.component }}-coordinator + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.oxia.coordinator "default" (list (dict "path" "/tmp" "sizeLimit" "1Gi")) "root" .)) }} {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.oxia.coordinator.securityContext "root" . "indent" 6) }} containers: - command: @@ -82,9 +83,14 @@ spec: limits: cpu: {{ .Values.oxia.coordinator.cpuLimit }} memory: {{ .Values.oxia.coordinator.memoryLimit }} - {{- if .Values.oxia.coordinator.extraVolumeMounts }} + {{- if or .Values.oxia.coordinator.extraVolumeMounts (include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.oxia.coordinator.containerSecurityContext "volumes" $edv "root" .)) }} volumeMounts: - {{- toYaml .Values.oxia.coordinator.extraVolumeMounts | nindent 12 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.oxia.coordinator.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 12 }} + {{- end }} + {{- if .Values.oxia.coordinator.extraVolumeMounts }} + {{- toYaml .Values.oxia.coordinator.extraVolumeMounts | nindent 12 }} + {{- end }} {{- end }} livenessProbe: {{- include "oxia-cluster.probe" .Values.oxia.coordinator.ports.internal | nindent 12 }} @@ -93,8 +99,13 @@ spec: {{- if .Values.oxia.coordinator.extraContainers }} {{- toYaml .Values.oxia.coordinator.extraContainers | nindent 8 }} {{- end }} - {{- if .Values.oxia.coordinator.extraVolumes }} + {{- if or .Values.oxia.coordinator.extraVolumes (include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.oxia.coordinator.containerSecurityContext "volumes" $edv "root" .)) }} volumes: -{{- toYaml .Values.oxia.coordinator.extraVolumes | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.oxia.coordinator.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 8 }} + {{- end }} + {{- if .Values.oxia.coordinator.extraVolumes }} + {{- toYaml .Values.oxia.coordinator.extraVolumes | nindent 8 }} + {{- end }} {{- end }} {{- end }} diff --git a/charts/pulsar/templates/oxia-server-statefulset.yaml b/charts/pulsar/templates/oxia-server-statefulset.yaml index 9a103347..4e652213 100644 --- a/charts/pulsar/templates/oxia-server-statefulset.yaml +++ b/charts/pulsar/templates/oxia-server-statefulset.yaml @@ -106,7 +106,12 @@ spec: {{ end }} {{- end }} serviceAccountName: {{ template "pulsar.fullname" . }}-{{ .Values.oxia.component }} + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.oxia.server "default" (list (dict "path" "/tmp" "sizeLimit" "1Gi")) "root" .)) }} {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.oxia.server.securityContext "root" . "indent" 6) }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.oxia.server.containerSecurityContext "volumes" $edv "root" .) }} + volumes: + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} containers: - command: - "oxia" @@ -135,6 +140,9 @@ spec: cpu: {{ .Values.oxia.server.cpuLimit }} memory: {{ .Values.oxia.server.memoryLimit }} volumeMounts: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.oxia.server.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 12 }} + {{- end }} - name: {{ template "pulsar.fullname" . }}-{{ .Values.oxia.component }}-data mountPath: /data livenessProbe: diff --git a/charts/pulsar/templates/proxy-statefulset.yaml b/charts/pulsar/templates/proxy-statefulset.yaml index 2f9c611f..478f7f2a 100644 --- a/charts/pulsar/templates/proxy-statefulset.yaml +++ b/charts/pulsar/templates/proxy-statefulset.yaml @@ -113,8 +113,12 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.proxy.gracePeriod }} serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.proxy.component }}" + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.proxy "root" .)) }} {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.proxy.securityContext "root" . "indent" 6) }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.proxy.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.proxy "securityContext" .Values.proxy.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} {{- if .Values.tls.proxy.cacerts.enabled }} - name: combine-certs image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.proxy "root" .) }}" @@ -127,6 +131,9 @@ spec: bin/certs-combine-pem.sh /pulsar/certs/cacerts/ca-combined.pem {{ template "pulsar.certs.cacerts" (dict "certs" .Values.tls.proxy.cacerts.certs) }} volumeMounts: {{- include "pulsar.proxy.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.proxy.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.components.zookeeper (not .Values.standalone.enabled) .Values.proxy.waitZookeeperTimeout (gt (.Values.proxy.waitZookeeperTimeout | int) 0) }} # This init container will wait for zookeeper to be ready before @@ -153,6 +160,10 @@ spec: echo "pulsar cluster {{ template "pulsar.cluster.name" . }} isn't initialized yet ... check in 3 seconds ..." && sleep 3; done; {{- end}} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.proxy.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end}} {{- if and .Values.components.oxia (not .Values.standalone.enabled) .Values.proxy.waitOxiaTimeout (gt (.Values.proxy.waitOxiaTimeout | int) 0) }} - name: wait-oxia-ready @@ -166,7 +177,11 @@ spec: until nslookup {{ template "pulsar.oxia.server.service" . }}; do sleep 3; done; - {{- end }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.proxy.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} + {{- end }} {{- if and .Values.proxy.waitBrokerTimeout (gt (.Values.proxy.waitBrokerTimeout | int) 0) }} {{- $backendService := ternary (include "pulsar.standalone.service" .) (printf "%s-%s" (include "pulsar.fullname" .) .Values.broker.component) .Values.standalone.enabled }} # This init container waits for the broker backend (either the broker @@ -187,6 +202,10 @@ spec: sleep 10; brokerServiceNumber="$(nslookup -timeout=10 {{ $backendService }} | grep Name | wc -l)"; done; + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.proxy.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end}} {{- if .Values.proxy.initContainers }} {{- toYaml .Values.proxy.initContainers | nindent 6 }} @@ -265,8 +284,11 @@ spec: envFrom: - configMapRef: name: "{{ template "pulsar.fullname" . }}-{{ .Values.proxy.component }}" - {{- if or .Values.proxy.extraVolumeMounts .Values.auth.authentication.enabled (and .Values.tls.enabled (or .Values.tls.proxy.enabled .Values.tls.broker.enabled)) }} + {{- if or .Values.proxy.extraVolumeMounts .Values.auth.authentication.enabled (and .Values.tls.enabled (or .Values.tls.proxy.enabled .Values.tls.broker.enabled)) (include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.proxy.containerSecurityContext "volumes" $edv "root" .)) }} volumeMounts: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.proxy.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 10 }} + {{- end }} {{- if .Values.auth.authentication.enabled }} {{- if .Values.auth.authentication.jwt.enabled }} - mountPath: "/pulsar/keys" @@ -283,8 +305,11 @@ spec: {{- end }} {{- end}} {{- include "pulsar.imagePullSecrets" . | nindent 6}} - {{- if or .Values.proxy.extraVolumes .Values.auth.authentication.enabled (and .Values.tls.enabled .Values.tls.proxy.enabled) }} + {{- if or .Values.proxy.extraVolumes .Values.auth.authentication.enabled (and .Values.tls.enabled .Values.tls.proxy.enabled) (include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.proxy.containerSecurityContext "volumes" $edv "root" .)) }} volumes: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.proxy.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 8 }} + {{- end }} {{- if .Values.proxy.extraVolumes }} {{ toYaml .Values.proxy.extraVolumes | indent 8 }} {{- end }} diff --git a/charts/pulsar/templates/pulsar-cluster-initialize.yaml b/charts/pulsar/templates/pulsar-cluster-initialize.yaml index 91a25db7..cdd93cbd 100755 --- a/charts/pulsar/templates/pulsar-cluster-initialize.yaml +++ b/charts/pulsar/templates/pulsar-cluster-initialize.yaml @@ -48,8 +48,12 @@ spec: nodeSelector: {{ toYaml .Values.pulsar_metadata.nodeSelector | indent 8 }} {{- end }} + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.pulsar_metadata "root" .)) }} {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.pulsar_metadata.securityContext "root" . "indent" 6) }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.pulsar_metadata.image "securityContext" .Values.pulsar_metadata.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} {{- if .Values.tls.toolset.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.pulsar_metadata.image "root" .) }}" @@ -62,6 +66,9 @@ spec: bin/certs-combine-pem.sh /pulsar/certs/cacerts/ca-combined.pem {{ template "pulsar.certs.cacerts" (dict "certs" .Values.tls.toolset.cacerts.certs) }} volumeMounts: {{- include "pulsar.toolset.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.components.zookeeper .Values.pulsar_metadata.waitZookeeperTimeout (gt (.Values.pulsar_metadata.waitZookeeperTimeout | int) 0) }} {{- if .Values.pulsar_metadata.configurationStore }} @@ -80,6 +87,9 @@ spec: done; volumeMounts: {{- include "pulsar.toolset.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} - name: wait-zk-metastore-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.pulsar_metadata.image "root" .) }}" @@ -99,6 +109,10 @@ spec: sleep 3; done; {{- end }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.components.oxia .Values.pulsar_metadata.waitOxiaTimeout (gt (.Values.pulsar_metadata.waitOxiaTimeout | int) 0) }} - name: wait-oxia-ready @@ -112,6 +126,10 @@ spec: until nslookup {{ template "pulsar.oxia.server.service" . }}; do sleep 3; done; + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.pulsar_metadata.waitBookkeeperTimeout (gt (.Values.pulsar_metadata.waitBookkeeperTimeout | int) 0) }} # This initContainer will wait for bookkeeper initnewcluster to complete @@ -137,6 +155,9 @@ spec: name: {{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }} volumeMounts: {{- include "pulsar.toolset.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} containers: - name: {{ template "pulsar.fullname" . }}-{{ .Values.pulsar_metadata.component }} @@ -186,8 +207,14 @@ spec: {{- end }} volumeMounts: {{- include "pulsar.toolset.certs.volumeMounts" . | nindent 10 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 10 }} + {{- end }} volumes: {{- include "pulsar.toolset.certs.volumes" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 8 }} + {{- end }} restartPolicy: OnFailure {{- if .Values.pulsar_metadata.nodeSelector }} nodeSelector: diff --git a/charts/pulsar/templates/pulsar-manager-cluster-initialize.yaml b/charts/pulsar/templates/pulsar-manager-cluster-initialize.yaml index e89ed266..cf78f3e7 100755 --- a/charts/pulsar/templates/pulsar-manager-cluster-initialize.yaml +++ b/charts/pulsar/templates/pulsar-manager-cluster-initialize.yaml @@ -51,13 +51,27 @@ spec: {{ toYaml .Values.pulsar_metadata.tolerations | indent 8 }} {{- end }} restartPolicy: OnFailure + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.pulsar_manager "root" .)) }} {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.pulsar_manager.securityContext "root" . "indent" 6) }} initContainers: + {{- /* + This Job runs a Pulsar image and its main container does `cd /tmp` and + `curl -D headers.txt`, so it needs the writable paths like every other + Pulsar-image workload. The pulsar_manager StatefulSet itself does not -- that + runs the apachepulsar/pulsar-manager image and has no /pulsar tree. + */}} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_manager.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.pulsar_metadata.image "securityContext" .Values.pulsar_manager.containerSecurityContext "root" .) | nindent 8 }} + {{- end }} - name: wait-pulsar-manager-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.pulsar_metadata.image "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.pulsar_metadata.image "root" .) }}" {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.pulsar_manager.containerSecurityContext "root" . "indent" 10) }} resources: {{ toYaml .Values.initContainer.resources | nindent 12 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_manager.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 10 }} + {{- end }} command: [ "sh", "-c" ] args: - | @@ -73,6 +87,10 @@ spec: imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.pulsar_metadata.image "root" .) }}" {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.pulsar_manager.containerSecurityContext "root" . "indent" 10) }} resources: {{ toYaml .Values.initContainer.resources | nindent 12 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_manager.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 10 }} + {{- end }} command: [ "sh", "-c" ] args: - | @@ -92,6 +110,10 @@ spec: {{- if .Values.pulsar_metadata.resources }} resources: {{ toYaml .Values.pulsar_metadata.resources | nindent 12 }} {{- end }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_manager.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 10 }} + {{- end }} command: [ "sh", "-c" ] args: - | @@ -192,5 +214,9 @@ spec: name: "{{ template "pulsar.fullname" . }}-{{ .Values.pulsar_manager.component }}-secret" {{- end }} key: UI_PASSWORD + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_manager.containerSecurityContext "volumes" $edv "root" .) }} + volumes: + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} {{- end }} {{- end }} diff --git a/charts/pulsar/templates/pulsar-manager-statefulset.yaml b/charts/pulsar/templates/pulsar-manager-statefulset.yaml index 398da8a6..e066f349 100755 --- a/charts/pulsar/templates/pulsar-manager-statefulset.yaml +++ b/charts/pulsar/templates/pulsar-manager-statefulset.yaml @@ -18,6 +18,10 @@ # {{- if .Values.components.pulsar_manager }} +{{- $pmSC := mergeOverwrite (deepCopy (.Values.containerSecurityContext | default dict)) (deepCopy (.Values.pulsar_manager.containerSecurityContext | default dict)) }} +{{- if $pmSC.readOnlyRootFilesystem }} +{{- fail "ERROR: pulsar_manager does not support readOnlyRootFilesystem. The apachepulsar/pulsar-manager image runs nginx, PostgreSQL and supervisord, and writes a supervisor socket inside its own install directory (/pulsar-manager), which cannot be covered by a mounted volume without shadowing the application. Set pulsar_manager.containerSecurityContext.readOnlyRootFilesystem=false to exclude this component, or disable components.pulsar_manager. Every other component the chart renders does support it." }} +{{- end }} apiVersion: apps/v1 kind: StatefulSet metadata: diff --git a/charts/pulsar/templates/standalone-deployment.yaml b/charts/pulsar/templates/standalone-deployment.yaml index 9ae9dcca..fae655c6 100644 --- a/charts/pulsar/templates/standalone-deployment.yaml +++ b/charts/pulsar/templates/standalone-deployment.yaml @@ -102,11 +102,17 @@ spec: {{ toYaml .Values.standalone.tolerations | indent 8 }} {{- end }} terminationGracePeriodSeconds: {{ .Values.standalone.gracePeriod }} + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.standalone "root" .)) }} {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.standalone.securityContext "root" . "indent" 6) }} - {{- if .Values.standalone.initContainers }} + {{- if or .Values.standalone.initContainers (include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.standalone.containerSecurityContext "volumes" $edv "root" .)) }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.standalone.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.standalone "securityContext" .Values.standalone.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} + {{- if .Values.standalone.initContainers }} {{- toYaml .Values.standalone.initContainers | nindent 6 }} {{- end }} + {{- end }} containers: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.standalone.component }}" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.standalone "root" .) }}" @@ -196,10 +202,16 @@ spec: {{ toYaml .Values.standalone.extraVolumeMounts | indent 10 }} {{- end }} {{- include "pulsar.standalone.certs.volumeMounts" . | nindent 10 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.standalone.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 10 }} + {{- end }} {{- if .Values.standalone.extraEnvs }} {{- toYaml .Values.standalone.extraEnvs | nindent 10 }} {{- end }} volumes: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.standalone.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} - name: "{{ template "pulsar.fullname" . }}-{{ .Values.standalone.component }}-{{ .Values.standalone.volumes.data.name }}" persistentVolumeClaim: claimName: "{{ template "pulsar.fullname" . }}-{{ .Values.standalone.component }}-{{ .Values.standalone.volumes.data.name }}" diff --git a/charts/pulsar/templates/toolset-statefulset.yaml b/charts/pulsar/templates/toolset-statefulset.yaml index fee0404d..2f9e7562 100644 --- a/charts/pulsar/templates/toolset-statefulset.yaml +++ b/charts/pulsar/templates/toolset-statefulset.yaml @@ -67,8 +67,16 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.toolset.gracePeriod }} serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.toolset.component }}" + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.toolset "root" .)) }} {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.toolset.securityContext "root" . "indent" 6) }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.toolset.containerSecurityContext "volumes" $edv "root" .) }} + {{- $toolsetImage := .Values.images.broker }} + {{- if (and .Values.images.toolset .Values.images.toolset.repository) }} + {{- $toolsetImage = .Values.images.toolset }} + {{- end }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" $toolsetImage "securityContext" .Values.toolset.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} {{- if .Values.tls.toolset.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.toolset "root" .) }}" @@ -81,6 +89,9 @@ spec: bin/certs-combine-pem.sh /pulsar/certs/cacerts/ca-combined.pem {{ template "pulsar.certs.cacerts" (dict "certs" .Values.tls.toolset.cacerts.certs) }} volumeMounts: {{- include "pulsar.toolset.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.toolset.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if .Values.toolset.initContainers }} {{- toYaml .Values.toolset.initContainers | nindent 6 }} @@ -129,7 +140,13 @@ spec: {{ toYaml .Values.toolset.extraVolumeMounts | indent 8 }} {{- end }} {{- include "pulsar.toolset.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.toolset.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} volumes: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.toolset.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} {{- if .Values.auth.authentication.enabled }} {{- if .Values.auth.authentication.jwt.enabled }} - name: client-token diff --git a/charts/pulsar/templates/zookeeper-statefulset.yaml b/charts/pulsar/templates/zookeeper-statefulset.yaml index ceaae0ad..89570def 100755 --- a/charts/pulsar/templates/zookeeper-statefulset.yaml +++ b/charts/pulsar/templates/zookeeper-statefulset.yaml @@ -113,8 +113,12 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.zookeeper.gracePeriod }} serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.zookeeper.component }}" + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.zookeeper "root" .)) }} {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.zookeeper.securityContext "root" . "indent" 6) }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.zookeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.zookeeper "securityContext" .Values.zookeeper.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} {{- if .Values.tls.zookeeper.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.zookeeper "root" .) }}" @@ -127,6 +131,9 @@ spec: bin/certs-combine-pem.sh /pulsar/certs/cacerts/ca-combined.pem {{ template "pulsar.certs.cacerts" (dict "certs" .Values.tls.zookeeper.cacerts.certs) }} volumeMounts: {{- include "pulsar.zookeeper.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.zookeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if .Values.zookeeper.initContainers }} {{- toYaml .Values.zookeeper.initContainers | nindent 6 }} @@ -244,10 +251,16 @@ spec: mountPath: /pulsar/data-log {{- end }} {{- include "pulsar.zookeeper.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.zookeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- if .Values.zookeeper.extraVolumeMounts }} {{ toYaml .Values.zookeeper.extraVolumeMounts | indent 8 }} {{- end }} volumes: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.zookeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} {{- if not (and (and .Values.volumes.persistence .Values.volumes.persistence) .Values.zookeeper.volumes.persistence) }} - name: "{{ template "pulsar.fullname" . }}-{{ .Values.zookeeper.component }}-{{ .Values.zookeeper.volumes.data.name }}" emptyDir: {} diff --git a/charts/pulsar/values.yaml b/charts/pulsar/values.yaml index e9b29ebe..40f9fa1a 100755 --- a/charts/pulsar/values.yaml +++ b/charts/pulsar/values.yaml @@ -152,14 +152,38 @@ affinity: ## `zookeeper.securityContext`), and be aware that changing `fsGroup` on a cluster ## with existing volumes triggers a recursive ownership change on first mount. ## -## `readOnlyRootFilesystem: true` is not supported out of the box: the Pulsar -## images rewrite their configuration under /pulsar/conf on startup and write logs -## under /pulsar/logs. Mount writable volumes over those paths with the -## `.extraVolumes` / `.extraVolumeMounts` settings before -## enabling it. +## Setting `readOnlyRootFilesystem: true` is supported: the chart then mounts the +## writable volumes the Pulsar images need. See `emptyDirVolumes` below. podSecurityContext: {} containerSecurityContext: {} +## Writable volumes mounted when a component sets `readOnlyRootFilesystem: true`; the +## Pulsar images rewrite /pulsar/conf on startup, log to /pulsar/logs, and use /tmp. +## Ignored while `readOnlyRootFilesystem` is unset, so the default output is unchanged. +## +## Per entry: `path` (required, absolute; the volume name is derived from it, so two +## paths that sanitise alike are rejected), `seedFromImage` (copy the image's contents +## there first, via a `copy-pulsar-conf` initContainer, since an emptyDir starts empty), +## and `sizeLimit` (an uncapped emptyDir can fill the node and get the pod evicted). +## +## `.emptyDirVolumes` REPLACES this list rather than extending it; use `[]` +## for "this component needs none" -- but a component with `readOnlyRootFilesystem: true` +## and an empty list has no writable path at all, so supply them yourself through +## `.extraVolumes` / `.extraVolumeMounts`. A path the component +## already mounts through `extraVolumeMounts` is skipped, since duplicate mountPaths are +## rejected. +## See the README for the components that need something other than this default. +## +## Keep in sync with the fallback in the `pulsar.emptyDirVolumes.resolve` helper, which +## applies when the key is absent (`helm upgrade --reuse-values`). +emptyDirVolumes: + - path: /pulsar/conf + seedFromImage: true + - path: /pulsar/logs + sizeLimit: 1Gi + - path: /tmp + sizeLimit: 1Gi + ## Components ## ## Control what components of Apache Pulsar to deploy for the cluster diff --git a/examples/README.md b/examples/README.md index 8d35ba84..39dd1f88 100644 --- a/examples/README.md +++ b/examples/README.md @@ -198,7 +198,7 @@ topics). Function instances run with the Kubernetes runtime (one pod per instanc | [`values-tls-ca.yaml`](values-tls-ca.yaml) | Enable TLS (including the bookie) using a CA issuer that references an existing `ca-key-pair` secret, with custom DNS SANs for the proxy. Requires cert-manager. | | [`values-jwt-symmetric.yaml`](values-jwt-symmetric.yaml) | Enable JWT authentication and authorization using a symmetric (shared secret) key (`usingSecretKey: true`). The signing key and per-superuser tokens are generated [in-chart](../README.md#in-chart-jwt-secret-generation) (`generateSecrets.enabled: true`), so a fully authenticated cluster deploys with a single `helm install`. | | [`values-jwt-asymmetric.yaml`](values-jwt-asymmetric.yaml) | Enable JWT authentication and authorization using an asymmetric (RSA private/public) key pair (`usingSecretKey: false`). The signing key pair and per-superuser tokens are generated [in-chart](../README.md#in-chart-jwt-secret-generation) (`generateSecrets.enabled: true`), so a fully authenticated cluster deploys with a single `helm install`. | -| [`values-psa-restricted.yaml`](values-psa-restricted.yaml) | Run under the Kubernetes `restricted` Pod Security Standard: `runAsNonRoot` plus no privilege escalation, all capabilities dropped and `RuntimeDefault` seccomp. Leaves the chart's per-component `fsGroup: 0` defaults alone, since `restricted` places no constraint on `fsGroup`. See [Pod and container security contexts](../README.md#pod-and-container-security-contexts). | +| [`values-psa-restricted.yaml`](values-psa-restricted.yaml) | Run under the Kubernetes `restricted` Pod Security Standard: `runAsNonRoot` plus no privilege escalation, all capabilities dropped, `RuntimeDefault` seccomp and a read-only root filesystem. Leaves the chart's per-component `fsGroup: 0` defaults alone, since `restricted` places no constraint on `fsGroup`. See [Pod and container security contexts](../README.md#pod-and-container-security-contexts). | | [`values-restricted-group-ranges.yaml`](values-restricted-group-ranges.yaml) | For the narrower case of a policy that constrains group IDs to a numeric range (Gatekeeper's `K8sPSPAllowedUsers`, an equivalent Kyverno policy, or an OpenShift SCC) and so rejects `fsGroup: 0`. Overrides `fsGroup` on the four components that ship it. Not required by PSA `restricted` — changing `fsGroup` on existing volumes triggers a recursive ownership change, so read the warning in the file first. | ### Monitoring diff --git a/examples/values-psa-restricted.yaml b/examples/values-psa-restricted.yaml index ff40c64d..41a68b52 100644 --- a/examples/values-psa-restricted.yaml +++ b/examples/values-psa-restricted.yaml @@ -50,8 +50,14 @@ podSecurityContext: # Applied to every container and initContainer. This matches the container-level controls # of both PSS `restricted` and OpenShift's `restricted-v2` SCC. +# +# readOnlyRootFilesystem is in neither standard but is supported here; see the README for +# what it mounts, and note that logs on an emptyDir do not survive pod replacement. +# `pulsar_manager` is the one component that cannot run read-only, and enabling both fails +# the render with an explanatory message. containerSecurityContext: allowPrivilegeEscalation: false + readOnlyRootFilesystem: true capabilities: drop: ["ALL"] seccompProfile: diff --git a/examples/values-restricted-group-ranges.yaml b/examples/values-restricted-group-ranges.yaml index 6ddc5698..82b9d614 100644 --- a/examples/values-restricted-group-ranges.yaml +++ b/examples/values-restricted-group-ranges.yaml @@ -62,6 +62,7 @@ podSecurityContext: # Applied to every container and initContainer the chart renders. containerSecurityContext: allowPrivilegeEscalation: false + readOnlyRootFilesystem: true capabilities: drop: ["ALL"] seccompProfile: