diff --git a/README.md b/README.md index f8760978..c40c35b7 100644 --- a/README.md +++ b/README.md @@ -302,6 +302,161 @@ multiple times (later files win), or use the [`merge-values.sh`](examples/merge- helper to merge several into a single file. See [`examples/README.md`](examples/README.md) for the full list and usage details. +## Pod and container security contexts + +Two global values apply a security context to every pod and every container the chart +itself renders, including its initContainers and the init/cleanup Jobs. Containers you +supply through `.initContainers`, `oxia.coordinator.extraContainers` or +`dekaf.deployment.extraContainers` are passed through verbatim and are not merged with +these settings -- set a `securityContext` on those yourself: + +```yaml +podSecurityContext: + runAsNonRoot: true + runAsUser: 10000 + runAsGroup: 10000 + fsGroup: 10000 + fsGroupChangePolicy: OnRootMismatch + supplementalGroups: [10000] +containerSecurityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + seccompProfile: + type: RuntimeDefault +``` + +Both are empty by default, so the chart's rendered output is unchanged unless you set +them. Use them when a cluster policy engine (Pod Security Admission, OPA Gatekeeper, +Kyverno) requires settings the chart does not set on its own. + +The `containerSecurityContext` block above is what the Kubernetes `restricted` Pod +Security Standard actually enforces, together with `runAsNonRoot`. See +[`examples/values-psa-restricted.yaml`](examples/values-psa-restricted.yaml). + +Each global value is merged with the matching per-component override, and the +per-component value wins on a per-key basis: + +| Global | Per-component | +|---|---| +| `podSecurityContext` | `.securityContext` | +| `containerSecurityContext` | `.containerSecurityContext` | + +The per-component keys exist for `zookeeper`, `bookkeeper`, `broker`, `autorecovery`, +`proxy`, `toolset`, `function_worker`, `standalone`, `oxia.server`, +`oxia.coordinator`, `pulsar_metadata`, `dekaf.deployment` and +`auth.authentication.jwt.generateSecrets`. Component-owned Jobs follow the component +they belong to: the bookkeeper cluster-initialize Job uses `bookkeeper.*`, the +zookeeper and broker `sts-cleanup` upgrade hooks use `zookeeper.*` and `broker.*`, and +the `pulsar-cluster-initialize` Job uses `pulsar_metadata.*`. + +### Overriding fsGroup, if a policy constrains group IDs + +`zookeeper`, `bookkeeper`, `broker` and `oxia.server` ship `securityContext.fsGroup: 0` +so that mounted volumes are group-owned by GID 0, the group the `pulsar` user (UID 10000) +belongs to. That is deliberate, and it is what lets the images run under an arbitrary +assigned UID — the model OpenShift uses. **`fsGroup: 0` is not a privilege**: group 0 +inside a container is an ordinary group, and root power comes from UID 0 and capabilities, +neither of which this grants. + +Most policies do not require you to change it. In particular the Kubernetes `restricted` +Pod Security Standard places **no constraint on `fsGroup`, `fsGroupChangePolicy` or +`supplementalGroups`**, and the PSS policy sets that Gatekeeper and Kyverno ship mirror +PSA, so `fsGroup: 0` is admitted there as-is. + +Override it only when a policy actually constrains group IDs to a numeric range — for +example Gatekeeper's PodSecurityPolicy-derived `K8sPSPAllowedUsers` with +`fsGroup: {rule: MustRunAs, ranges: [{min: 1, ...}]}`, or an OpenShift SCC. Because a +per-component value takes precedence over the global one, setting a global `fsGroup` +alone does **not** change these four components — override their own `securityContext` +as well: + +```yaml +podSecurityContext: + runAsNonRoot: true + runAsUser: 10000 + runAsGroup: 10000 + fsGroup: 10000 + fsGroupChangePolicy: OnRootMismatch + supplementalGroups: [10000] +bookkeeper: + securityContext: + fsGroup: 10000 + fsGroupChangePolicy: OnRootMismatch +``` + +`fsGroup` is applied as a *supplementary* group, so `runAsGroup` does not need to match +it -- the process keeps its own primary GID and still gets access to the volume. + +On a cluster with existing volumes, changing `fsGroup` triggers a recursive ownership +change on first mount; `fsGroupChangePolicy: OnRootMismatch` keeps that to the first pass, +but it can still take a long time on large bookie ledger volumes. Test on a copy before +changing a production cluster. + +### readOnlyRootFilesystem + +The Pulsar images do not run on a read-only root filesystem unaided. They rewrite their +configuration under `/pulsar/conf` on startup (`bin/apply-config-from-env.py`), write logs +under `/pulsar/logs`, and the JVM and the functions worker use `/tmp`. + +Setting `containerSecurityContext.readOnlyRootFilesystem: true` is nevertheless enough. +For each entry in `emptyDirVolumes` the chart then mounts an `emptyDir` at that path on +every container and initContainer of the component, and for entries marked +`seedFromImage: true` it prepends a `copy-pulsar-conf` initContainer that copies the +image's contents into the volume first — an `emptyDir` starts empty, and +`apply-config-from-env.py` edits files that must already exist. + +The chart-wide default describes the Pulsar images: + +```yaml +emptyDirVolumes: + - path: /pulsar/conf + seedFromImage: true + - path: /pulsar/logs + sizeLimit: 1Gi + - path: /tmp + sizeLimit: 1Gi +``` + +`sizeLimit` matters: without it an `emptyDir` is unbounded and a busy `/pulsar/logs` can +fill a node's ephemeral storage and get pods evicted. + +This is driven by the effective (merged) container securityContext, so a single global +`readOnlyRootFilesystem: true` covers the release, and a component that overrides it back +to `false` also loses the volumes. + +#### Per-component paths + +Every component takes its own `.emptyDirVolumes`, which **replaces** the list +it would otherwise inherit rather than adding to it — Helm merges maps per key but +replaces lists wholesale. Use `[]` for "this component needs none". + +The three paths above describe the Pulsar images, so they are not applied to components +that run something else. Those carry their own default and never inherit the chart-wide +list: + +| Component | Default | Why | +|---|---|---| +| `oxia.server`, `oxia.coordinator` | `/tmp` | the `oxia` binary keeps its own state under its data directory, but `/tmp` is provided so that a Go dependency or a future version falling back to `os.TempDir()` cannot take the metadata store down | +| `dekaf.deployment` | `/tmp` | the JVM writes scratch files there | +| everything else | the three above | Pulsar images | + +Setting `.emptyDirVolumes` still overrides either default. + +To hand a path back to yourself, drop its entry and declare it through +`.extraVolumes` / `.extraVolumeMounts`. Declaring just the mount is +enough: an entry whose path the component already mounts via `extraVolumeMounts` is +skipped, because a duplicate `mountPath` is rejected by the API server. Bear in mind that +whatever you mount over `/pulsar/conf` has to be populated, since nothing seeds it for you. + +Rejected at render time rather than at apply time: a relative path, an unknown entry key, +two paths that reduce to the same volume name (`/pulsar/logs` and `/pulsar-logs` both give +`pulsar-logs`), and a path long enough to exceed the 63-character limit on a name. + +Note that logs written to an `emptyDir` do not survive pod replacement. If you rely on +reading `/pulsar/logs` from inside a pod, ship them off-node or keep +`readOnlyRootFilesystem` disabled. + ## Disabling victoria-metrics-k8s-stack components In order to disable the victoria-metrics-k8s-stack, you can add the following to your `values.yaml`. diff --git a/charts/pulsar/templates/_helpers.tpl b/charts/pulsar/templates/_helpers.tpl index d8bc866d..63913fb4 100644 --- a/charts/pulsar/templates/_helpers.tpl +++ b/charts/pulsar/templates/_helpers.tpl @@ -110,6 +110,242 @@ app: {{ template "pulsar.name" . }} release: {{ .Release.Name }} {{- end }} +{{/* +Render a pod-level securityContext. + +Merges the chart-wide `.Values.podSecurityContext` defaults with a per-component +`.securityContext` override, where the per-component value wins. Renders +nothing when both are empty, so charts that set neither are unaffected. + +Usage: + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.broker.securityContext "root" . "indent" 6) }} +*/}} +{{- define "pulsar.podSecurityContext" -}} +{{- include "pulsar.mergedSecurityContext" (dict "global" .root.Values.podSecurityContext "component" .securityContext "indent" .indent) -}} +{{- end -}} + +{{/* +Render a container-level securityContext. + +Merges the chart-wide `.Values.containerSecurityContext` defaults with a per-component +`.containerSecurityContext` override, where the per-component value wins. +Applies to both containers and initContainers. Renders nothing when both are empty. + +Usage: + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.broker.containerSecurityContext "root" . "indent" 8) }} +*/}} +{{- define "pulsar.containerSecurityContext" -}} +{{- include "pulsar.mergedSecurityContext" (dict "global" .root.Values.containerSecurityContext "component" .securityContext "indent" .indent) -}} +{{- end -}} + +{{/* +Merge a global and a per-component securityContext and render the resulting block, +indented by `indent` spaces. Nested maps are deep-merged; the per-component value +takes precedence. Renders nothing when the merged result is empty. + +`mergeOverwrite` is used rather than `merge` because `merge` treats zero values +(`0`, `false`, `""`) in its destination as absent, which would silently discard a +per-component `fsGroup: 0` or `allowPrivilegeEscalation: false` override. +Both operands are deep-copied so that `.Values` is never mutated. +*/}} +{{- define "pulsar.mergedSecurityContext" -}} +{{- $global := deepCopy (.global | default dict) -}} +{{- $component := deepCopy (.component | default dict) -}} +{{- $merged := mergeOverwrite $global $component -}} +{{- if $merged -}} +{{- printf "securityContext:\n%s" (toYaml $merged | indent 2) | nindent (int .indent) -}} +{{- end -}} +{{- end -}} + +{{/* +Resolve the effective emptyDirVolumes list for a component, returned as JSON (a template +can only return a string; callers pipe it through `fromJsonArray`). + +Precedence, highest first: + 1. `.emptyDirVolumes` + 2. the caller's `default`, passed only by components that do not run the Pulsar image, + so that the chart-wide list -- which describes the Pulsar layout -- cannot apply + 3. `.Values.emptyDirVolumes` + 4. the chart default below, a copy of the values.yaml default. `helm upgrade + --reuse-values` does not coalesce in new chart defaults, so the key is absent there; + treating that as `[]` would give a read-only root filesystem and none of the volumes + that make it work. Keep the two copies in sync. +A list REPLACES the one above it rather than extending it, because Helm merges maps per +key but replaces lists whole. + +Entries whose path the component already mounts via `extraVolumeMounts` are dropped: a +duplicate mountPath is rejected by the API server, so an existing setup that provides one +of these paths itself keeps working. + +Usage: + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.broker "root" .)) }} +*/}} +{{- define "pulsar.emptyDirVolumes.resolve" -}} +{{- $list := list (dict "path" "/pulsar/conf" "seedFromImage" true) (dict "path" "/pulsar/logs" "sizeLimit" "1Gi") (dict "path" "/tmp" "sizeLimit" "1Gi") -}} +{{- if hasKey .root.Values "emptyDirVolumes" -}} +{{- $list = .root.Values.emptyDirVolumes | default list -}} +{{- end -}} +{{- if hasKey . "default" -}} +{{- $list = .default | default list -}} +{{- end -}} +{{- $component := .component | default dict -}} +{{- if hasKey $component "emptyDirVolumes" -}} +{{- $list = (index $component "emptyDirVolumes") | default list -}} +{{- end -}} +{{- include "pulsar.emptyDirVolumes.validate" $list -}} +{{- $mounted := list -}} +{{- range (index $component "extraVolumeMounts") | default list -}} +{{- $mounted = append $mounted .mountPath -}} +{{- end -}} +{{- $effective := list -}} +{{- range $list -}} +{{- if not (has .path $mounted) -}} +{{- $effective = append $effective . -}} +{{- end -}} +{{- end -}} +{{- $effective | toJson -}} +{{- end -}} + +{{/* +Whether to render the emptyDir volumes for a component: true when the effective container +securityContext sets `readOnlyRootFilesystem` and the resolved list is non-empty. The +merge matches `pulsar.containerSecurityContext`, so a component that opts out of a +read-only root filesystem also opts out of these volumes. + +Usage (as a guard): + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.broker.containerSecurityContext "volumes" $edv "root" .) }} +*/}} +{{- define "pulsar.emptyDirVolumes.enabled" -}} +{{- $global := deepCopy (.root.Values.containerSecurityContext | default dict) -}} +{{- $override := deepCopy (.securityContext | default dict) -}} +{{- if and (mergeOverwrite $global $override).readOnlyRootFilesystem .volumes -}} +true +{{- end -}} +{{- end -}} + +{{/* +Turn a path into a DNS-1123 volume name: /pulsar/conf -> pulsar-conf, /tmp -> tmp. +*/}} +{{- define "pulsar.emptyDirVolumes.name" -}} +{{- $n := . | trim | trimAll "/" | replace "/" "-" | replace "_" "-" | replace "." "-" | lower -}} +{{- if not (regexMatch "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$" $n) -}} +{{- fail (printf "emptyDirVolumes: path %q does not yield a valid volume name (got %q)" . $n) -}} +{{- end -}} +{{- if gt (len $n) 63 -}} +{{- fail (printf "emptyDirVolumes: path %q yields the volume name %q, which exceeds the 63 character limit for a Kubernetes name" . $n) -}} +{{- end -}} +{{- $n -}} +{{- end -}} + +{{/* +Validate an emptyDirVolumes list. Every entry needs an absolute `path` and no key beyond +path/seedFromImage/sizeLimit -- a typo such as `seedFromimage` would otherwise leave the +conf directory unseeded and only fail at runtime. No two entries may collide on path or on +derived name (/pulsar/logs and /pulsar-logs both give pulsar-logs). +*/}} +{{- define "pulsar.emptyDirVolumes.validate" -}} +{{- if not (kindIs "slice" .) -}} +{{- fail (printf "emptyDirVolumes must be a list, got %v (%s). Note that `--set emptyDirVolumes=[]` assigns the string \"[]\"; use a values file to set an empty list." . (kindOf .)) -}} +{{- end -}} +{{- $names := dict -}} +{{- $paths := dict -}} +{{- range . -}} +{{- if not (kindIs "map" .) -}} +{{- fail (printf "emptyDirVolumes: every entry must be a mapping with a `path` key, got %v (%s)." . (kindOf .)) -}} +{{- end -}} +{{- range $k, $_ := . -}} +{{- if not (has $k (list "path" "seedFromImage" "sizeLimit")) -}} +{{- fail (printf "emptyDirVolumes: unknown key %q; supported keys are path, seedFromImage and sizeLimit" $k) -}} +{{- end -}} +{{- end -}} +{{- if not .path -}} +{{- fail "emptyDirVolumes: every entry requires a `path`" -}} +{{- end -}} +{{- if not (hasPrefix "/" .path) -}} +{{- fail (printf "emptyDirVolumes: path %q must be absolute" .path) -}} +{{- end -}} +{{- if hasKey $paths .path -}} +{{- fail (printf "emptyDirVolumes: path %q is listed twice" .path) -}} +{{- end -}} +{{- $_ := set $paths .path true -}} +{{- $n := include "pulsar.emptyDirVolumes.name" .path -}} +{{- if hasKey $names $n -}} +{{- fail (printf "emptyDirVolumes: paths %q and %q both map to volume name %q; rename one" (index $names $n) .path $n) -}} +{{- end -}} +{{- $_ := set $names $n .path -}} +{{- end -}} +{{- end -}} + +{{/* +The emptyDir volumes for a resolved list. + +Usage: + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} +*/}} +{{- define "pulsar.emptyDirVolumes.volumes" -}} +{{- range . }} +- name: {{ include "pulsar.emptyDirVolumes.name" .path }} + emptyDir: + {{- if .sizeLimit }} + sizeLimit: {{ .sizeLimit }} + {{- else }} + {} + {{- end }} +{{- end -}} +{{- end -}} + +{{/* +The matching mounts, applied to every container and initContainer of the component. + +Usage: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} +*/}} +{{- define "pulsar.emptyDirVolumes.mounts" -}} +{{- range . }} +- name: {{ include "pulsar.emptyDirVolumes.name" .path }} + mountPath: {{ .path }} +{{- end -}} +{{- end -}} + +{{/* +An initContainer that seeds the entries marked `seedFromImage: true`, because an emptyDir +starts empty and the Pulsar images rewrite files under /pulsar/conf that must exist. + +Must render as the *first* initContainer, ahead of any built-in one that reads the seeded +path. Each seeded volume is mounted at /mnt/, not at its real path, so it does not +shadow the copy in the image. `cp -r` rather than `cp -a`: preserving ownership fails as a +non-root user. Renders nothing when no entry is seeded. + +Usage: + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.broker "securityContext" .Values.broker.containerSecurityContext "root" .) | nindent 6 }} +*/}} +{{- define "pulsar.emptyDirVolumes.seedContainer" -}} +{{- $seed := list -}} +{{- range .volumes -}} +{{- if .seedFromImage -}} +{{- $seed = append $seed . -}} +{{- end -}} +{{- end -}} +{{- if $seed -}} +- name: copy-pulsar-conf + image: "{{ template "pulsar.imageFullName" (dict "image" .image "root" .root) }}" + imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .image "root" .root) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .securityContext "root" .root "indent" 2) }} + resources: {{ toYaml .root.Values.initContainer.resources | nindent 4 }} + command: ["sh", "-c"] + args: + - | + {{- range $seed }} + cp -r {{ .path }}/. /mnt/{{ include "pulsar.emptyDirVolumes.name" .path }}/ + {{- end }} + volumeMounts: + {{- range $seed }} + - name: {{ include "pulsar.emptyDirVolumes.name" .path }} + mountPath: /mnt/{{ include "pulsar.emptyDirVolumes.name" .path }} + {{- end }} +{{- end -}} +{{- end -}} + {{/* Create ImagePullSecrets */}} diff --git a/charts/pulsar/templates/autorecovery-statefulset.yaml b/charts/pulsar/templates/autorecovery-statefulset.yaml index 7c6b3ca8..5a1c6fde 100644 --- a/charts/pulsar/templates/autorecovery-statefulset.yaml +++ b/charts/pulsar/templates/autorecovery-statefulset.yaml @@ -119,11 +119,17 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.autorecovery.gracePeriod }} serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.autorecovery.component }}" + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.autorecovery "root" .)) }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.autorecovery.securityContext "root" . "indent" 6) }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.autorecovery.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.autorecovery "securityContext" .Values.autorecovery.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} {{- if .Values.tls.autorecovery.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.autorecovery "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.autorecovery "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.autorecovery.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["sh", "-c"] args: @@ -134,6 +140,9 @@ spec: {{ toYaml .Values.autorecovery.initContainersExtraVolumeMounts | indent 8 }} {{- end }} {{- include "pulsar.autorecovery.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.autorecovery.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.autorecovery.waitBookkeeperTimeout (gt (.Values.autorecovery.waitBookkeeperTimeout | int) 0) }} # This initContainer will wait for bookkeeper initnewcluster to complete @@ -141,6 +150,7 @@ spec: - name: pulsar-bookkeeper-verify-clusterid image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.autorecovery "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.autorecovery "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.autorecovery.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.autorecovery.waitBookkeeperTimeout }}", "sh", "-c"] args: @@ -154,6 +164,9 @@ spec: {{ toYaml .Values.autorecovery.initContainersExtraVolumeMounts | indent 8 }} {{- end }} {{- include "pulsar.autorecovery.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.autorecovery.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if .Values.autorecovery.initContainers }} {{- toYaml .Values.autorecovery.initContainers | nindent 6 }} @@ -162,6 +175,54 @@ spec: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.autorecovery.component }}" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.autorecovery "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.autorecovery "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.autorecovery.containerSecurityContext "root" . "indent" 8) }} + {{- /* + The autorecovery daemon does not start BookKeeper's HTTP service, so there is no + /api/v1 endpoint to probe as there is on a bookie. It does start the Prometheus + stats endpoint on autorecovery.ports.http, which is the same endpoint the + PodMonitor scrapes, so /metrics is what these probes target. + + Every lookup below goes through `dig`, because `autorecovery.probe` is a new key: + under `helm upgrade --reuse-values` the new chart's defaults are not coalesced in, + so the map is absent and a direct dereference aborts the whole render. + + All three are off by default, so this adds a knob rather than behaviour. /metrics + is served by the image's BookKeeper stats provider rather than by anything the + chart configures, so a cluster overriding statsProviderClass or + prometheusStatsHttpPort through autorecovery.configData would break liveness + (restart loop) and readiness (permanently unready, stalling the rollout) alike. + */}} + {{- $probe := .Values.autorecovery.probe | default dict }} + {{- if dig "liveness" "enabled" false $probe }} + livenessProbe: + httpGet: + path: /metrics + port: {{ .Values.autorecovery.ports.http }} + initialDelaySeconds: {{ dig "liveness" "initialDelaySeconds" 10 $probe }} + periodSeconds: {{ dig "liveness" "periodSeconds" 30 $probe }} + timeoutSeconds: {{ dig "liveness" "timeoutSeconds" 5 $probe }} + failureThreshold: {{ dig "liveness" "failureThreshold" 60 $probe }} + {{- end }} + {{- if dig "readiness" "enabled" false $probe }} + readinessProbe: + httpGet: + path: /metrics + port: {{ .Values.autorecovery.ports.http }} + initialDelaySeconds: {{ dig "readiness" "initialDelaySeconds" 10 $probe }} + periodSeconds: {{ dig "readiness" "periodSeconds" 30 $probe }} + timeoutSeconds: {{ dig "readiness" "timeoutSeconds" 5 $probe }} + failureThreshold: {{ dig "readiness" "failureThreshold" 60 $probe }} + {{- end }} + {{- if dig "startup" "enabled" false $probe }} + startupProbe: + httpGet: + path: /metrics + port: {{ .Values.autorecovery.ports.http }} + initialDelaySeconds: {{ dig "startup" "initialDelaySeconds" 10 $probe }} + periodSeconds: {{ dig "startup" "periodSeconds" 30 $probe }} + timeoutSeconds: {{ dig "startup" "timeoutSeconds" 5 $probe }} + failureThreshold: {{ dig "startup" "failureThreshold" 60 $probe }} + {{- end }} {{- if .Values.autorecovery.resources }} resources: {{ toYaml .Values.autorecovery.resources | indent 10 }} @@ -188,7 +249,13 @@ spec: {{ toYaml .Values.autorecovery.extraVolumeMounts | indent 8 }} {{- end }} {{- include "pulsar.autorecovery.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.autorecovery.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} volumes: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.autorecovery.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} {{- include "pulsar.autorecovery.certs.volumes" . | nindent 6 }} {{- if .Values.autorecovery.extraVolumes }} {{ toYaml .Values.autorecovery.extraVolumes | indent 6 }} diff --git a/charts/pulsar/templates/bookkeeper-cluster-initialize.yaml b/charts/pulsar/templates/bookkeeper-cluster-initialize.yaml index 81389b35..d4176bf6 100755 --- a/charts/pulsar/templates/bookkeeper-cluster-initialize.yaml +++ b/charts/pulsar/templates/bookkeeper-cluster-initialize.yaml @@ -48,6 +48,8 @@ spec: spec: {{- include "pulsar.imagePullSecrets" . | nindent 6 }} serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}" + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.bookkeeper "root" .)) }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.bookkeeper.securityContext "root" . "indent" 6) }} nodeSelector: {{- if .Values.pulsar_metadata.nodeSelector }} {{ toYaml .Values.pulsar_metadata.nodeSelector | indent 8 }} @@ -62,10 +64,14 @@ spec: {{- toYaml . | nindent 10 }} {{- end }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.bookie "securityContext" .Values.bookkeeper.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} {{- if .Values.tls.bookie.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.bookie "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.bookie "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["sh", "-c"] args: @@ -73,11 +79,15 @@ spec: bin/certs-combine-pem.sh /pulsar/certs/cacerts/ca-combined.pem {{ template "pulsar.certs.cacerts" (dict "certs" .Values.tls.bookie.cacerts.certs) }} volumeMounts: {{- include "pulsar.toolset.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.components.zookeeper .Values.bookkeeper.metadata.waitZookeeperTimeout (gt (.Values.bookkeeper.metadata.waitZookeeperTimeout | int) 0) }} - name: wait-zookeeper-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.bookie "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.bookie "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.bookkeeper.metadata.waitZookeeperTimeout }}", "sh", "-c"] args: @@ -92,11 +102,16 @@ spec: sleep 3; done; {{- end}} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end}} {{- if and .Values.components.oxia .Values.bookkeeper.metadata.waitOxiaTimeout (gt (.Values.bookkeeper.metadata.waitOxiaTimeout | int) 0) }} - name: wait-oxia-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.bookie "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.bookie "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.bookkeeper.metadata.waitOxiaTimeout }}", "sh", "-c"] args: @@ -104,11 +119,16 @@ spec: until nslookup {{ template "pulsar.oxia.server.service" . }}; do sleep 3; done; + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} containers: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}-init" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.bookie "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.bookie "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "root" . "indent" 8) }} {{- if .Values.bookkeeper.metadata.resources }} resources: {{ toYaml .Values.bookkeeper.metadata.resources | indent 10 }} @@ -135,8 +155,14 @@ spec: name: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}" volumeMounts: {{- include "pulsar.toolset.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} volumes: {{- include "pulsar.toolset.certs.volumes" . | nindent 6 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} restartPolicy: OnFailure {{- end }} {{- end }} diff --git a/charts/pulsar/templates/bookkeeper-statefulset.yaml b/charts/pulsar/templates/bookkeeper-statefulset.yaml index 71b92297..88ca6917 100644 --- a/charts/pulsar/templates/bookkeeper-statefulset.yaml +++ b/charts/pulsar/templates/bookkeeper-statefulset.yaml @@ -116,30 +116,46 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.bookkeeper.gracePeriod }} serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}" - {{- if .Values.bookkeeper.securityContext }} - securityContext: -{{ toYaml .Values.bookkeeper.securityContext | indent 8 }} - {{- end }} - {{- if and .Values.bookkeeper.waitMetadataTimeout (gt (.Values.bookkeeper.waitMetadataTimeout | int) 0) }} + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.bookkeeper "root" .)) }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.bookkeeper.securityContext "root" . "indent" 6) }} + {{- /* + `initContainers:` is emitted whenever anything needs it, not only when + waitMetadataTimeout is set. The conf-seeding container has to run regardless, or a + read-only root filesystem gets an empty emptyDir over /pulsar/conf and the bookie + cannot start. This also keeps user-supplied bookkeeper.initContainers from being + emitted without their parent key when waitMetadataTimeout is 0. + */}} + {{- $bkWaitMetadata := and .Values.bookkeeper.waitMetadataTimeout (gt (.Values.bookkeeper.waitMetadataTimeout | int) 0) }} + {{- $bkRootfs := include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- if or $bkWaitMetadata $bkRootfs .Values.tls.bookie.cacerts.enabled .Values.bookkeeper.initContainers }} initContainers: + {{- if $bkRootfs }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.bookie "securityContext" .Values.bookkeeper.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} {{- if .Values.tls.bookie.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.bookie "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.bookie "root" .) }}" resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "root" . "indent" 8) }} command: ["sh", "-c"] args: - | bin/certs-combine-pem.sh /pulsar/certs/cacerts/ca-combined.pem {{ template "pulsar.certs.cacerts" (dict "certs" .Values.tls.bookie.cacerts.certs) }} volumeMounts: {{- include "pulsar.bookkeeper.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} + {{- if $bkWaitMetadata }} # This initContainer will wait for bookkeeper initnewcluster to complete # before deploying the bookies - name: pulsar-bookkeeper-verify-clusterid image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.bookie "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.bookie "root" .) }}" resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "root" . "indent" 8) }} command: ["timeout", "{{ .Values.bookkeeper.waitMetadataTimeout }}", "sh", "-c"] args: # only reformat bookie if bookkeeper is running without persistence @@ -150,14 +166,19 @@ spec: name: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}" volumeMounts: {{- include "pulsar.bookkeeper.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if .Values.bookkeeper.initContainers }} {{- toYaml .Values.bookkeeper.initContainers | nindent 6 }} {{- end }} + {{- end }} containers: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.bookie "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.bookie "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "root" . "indent" 8) }} {{- if .Values.bookkeeper.probe.liveness.enabled }} livenessProbe: httpGet: @@ -282,7 +303,13 @@ spec: {{ toYaml .Values.bookkeeper.extraVolumeMounts | indent 8 }} {{- end }} {{- include "pulsar.bookkeeper.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} volumes: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.bookkeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} {{- if not (and (and .Values.persistence .Values.volumes.persistence) .Values.bookkeeper.volumes.persistence) }} - name: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}-{{ .Values.bookkeeper.volumes.journal.name }}" emptyDir: {} diff --git a/charts/pulsar/templates/broker-statefulset-upgrade.yaml b/charts/pulsar/templates/broker-statefulset-upgrade.yaml index f4774340..c3cdc894 100644 --- a/charts/pulsar/templates/broker-statefulset-upgrade.yaml +++ b/charts/pulsar/templates/broker-statefulset-upgrade.yaml @@ -87,14 +87,18 @@ metadata: spec: backoffLimit: 1 template: - {{- with .Values.job.podAnnotations }} metadata: + labels: + {{- include "pulsar.template.labels" . | nindent 8 }} + component: {{ .Values.broker.component }}-sts-cleanup + {{- with .Values.job.podAnnotations }} annotations: {{- toYaml . | nindent 8 }} - {{- end }} + {{- end }} spec: serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.broker.component }}-sts-cleanup" restartPolicy: Never + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.broker.securityContext "root" . "indent" 6) }} {{- include "pulsar.imagePullSecrets" . | nindent 6 }} {{- if .Values.broker.nodeSelector }} nodeSelector: @@ -124,6 +128,7 @@ spec: - name: sts-cleanup image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.kubectl "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.kubectl "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.broker.containerSecurityContext "root" . "indent" 10) }} command: - sh - -c diff --git a/charts/pulsar/templates/broker-statefulset.yaml b/charts/pulsar/templates/broker-statefulset.yaml index b2c49465..69402461 100644 --- a/charts/pulsar/templates/broker-statefulset.yaml +++ b/charts/pulsar/templates/broker-statefulset.yaml @@ -75,10 +75,8 @@ spec: {{- end }} spec: serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.broker.component }}-acct" - {{- if .Values.broker.securityContext }} - securityContext: -{{ toYaml .Values.broker.securityContext | indent 8 }} - {{- end }} + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.broker "root" .)) }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.broker.securityContext "root" . "indent" 6) }} {{- if .Values.broker.nodeSelector }} nodeSelector: {{ toYaml .Values.broker.nodeSelector | indent 8 }} @@ -141,10 +139,14 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.broker.gracePeriod }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.broker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.broker "securityContext" .Values.broker.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} {{- if .Values.tls.broker.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.broker "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.broker "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.broker.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["sh", "-c"] args: @@ -152,6 +154,9 @@ spec: bin/certs-combine-pem.sh /pulsar/certs/cacerts/ca-combined.pem {{ template "pulsar.certs.cacerts" (dict "certs" .Values.tls.broker.cacerts.certs) }} volumeMounts: {{- include "pulsar.broker.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.broker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.components.zookeeper .Values.broker.waitZookeeperTimeout (gt (.Values.broker.waitZookeeperTimeout | int) 0) }} # This init container will wait for zookeeper to be ready before @@ -159,6 +164,7 @@ spec: - name: wait-zookeeper-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.broker "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.broker "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.broker.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.broker.waitZookeeperTimeout }}", "sh", "-c"] args: @@ -175,11 +181,15 @@ spec: done; volumeMounts: {{- include "pulsar.broker.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.broker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.components.oxia .Values.broker.waitOxiaTimeout (gt (.Values.broker.waitOxiaTimeout | int) 0) }} - name: wait-oxia-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.broker "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.broker "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.broker.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.broker.waitOxiaTimeout }}", "sh", "-c"] args: @@ -187,6 +197,10 @@ spec: until nslookup {{ template "pulsar.oxia.server.service" . }}; do sleep 3; done; + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.broker.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.broker.waitBookkeeperTimeout (gt (.Values.broker.waitBookkeeperTimeout | int) 0) }} # This init container will wait for bookkeeper to be ready before @@ -194,6 +208,7 @@ spec: - name: wait-bookkeeper-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.broker "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.broker "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.broker.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.broker.waitBookkeeperTimeout }}", "sh", "-c"] args: @@ -218,6 +233,9 @@ spec: name: "{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}" volumeMounts: {{- include "pulsar.broker.certs.volumeMounts" . | nindent 10 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.broker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 10 }} + {{- end }} {{- end }} {{- if .Values.broker.initContainers }} {{- toYaml .Values.broker.initContainers | nindent 6 }} @@ -226,6 +244,7 @@ spec: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.broker.component }}" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.broker "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.broker "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.broker.containerSecurityContext "root" . "indent" 8) }} {{- if .Values.broker.probe.liveness.enabled }} livenessProbe: httpGet: @@ -328,6 +347,9 @@ spec: mountPath: {{ .Values.broker.packageManagement.fileSystemStorage.storagePath }} {{- end }} {{- include "pulsar.broker.certs.volumeMounts" . | nindent 10 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.broker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 10 }} + {{- end }} env: {{- if and (and .Values.broker.storageOffload (eq .Values.broker.storageOffload.driver "aws-s3")) .Values.broker.storageOffload.secret }} - name: AWS_ACCESS_KEY_ID @@ -357,6 +379,9 @@ spec: {{- toYaml .Values.broker.extraEnvs | nindent 10 }} {{- end }} volumes: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.broker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} {{- if .Values.broker.extraVolumes }} {{ toYaml .Values.broker.extraVolumes | indent 6 }} {{- end }} diff --git a/charts/pulsar/templates/dekaf-deployment.yaml b/charts/pulsar/templates/dekaf-deployment.yaml index db7d940e..5dddb245 100644 --- a/charts/pulsar/templates/dekaf-deployment.yaml +++ b/charts/pulsar/templates/dekaf-deployment.yaml @@ -57,6 +57,8 @@ spec: priorityClassName: {{ .Values.dekaf.deployment.priorityClassName }} {{- end }} + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" (((.Values.dekaf).deployment) | default dict) "default" (list (dict "path" "/tmp" "sizeLimit" "1Gi")) "root" .)) }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" ((.Values.dekaf).deployment).securityContext "root" . "indent" 6) }} {{- with (((.Values.dekaf).deployment).affinity).nodeAffinity }} affinity: nodeAffinity: @@ -67,6 +69,7 @@ spec: - name: dekaf image: "{{ .Values.images.dekaf.repository }}:{{ .Values.images.dekaf.tag }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.dekaf "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" ((.Values.dekaf).deployment).containerSecurityContext "root" . "indent" 10) }} env: - name: DEKAF_PULSAR_WEB_URL value: "http://{{ template "pulsar.fullname" . }}-{{ .Values.broker.component }}:{{ .Values.broker.ports.http }}" @@ -98,6 +101,9 @@ spec: {{- if ((.Values.dekaf).persistence).enabled }} volumeMounts: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" ((.Values.dekaf).deployment).containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 12 }} + {{- end }} - name: {{ template "pulsar.fullname" . }}-{{ template "dekaf.component" . }}-library mountPath: /dekaf/data/library {{- end }} @@ -111,6 +117,9 @@ spec: {{- end }} volumes: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" ((.Values.dekaf).deployment).containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} {{- if ((.Values.dekaf).persistence).enabled }} - name: {{ template "pulsar.fullname" . }}-{{ template "dekaf.component" . }}-library persistentVolumeClaim: diff --git a/charts/pulsar/templates/function-worker-statefulset.yaml b/charts/pulsar/templates/function-worker-statefulset.yaml index 51d57de2..0360a582 100644 --- a/charts/pulsar/templates/function-worker-statefulset.yaml +++ b/charts/pulsar/templates/function-worker-statefulset.yaml @@ -55,6 +55,8 @@ spec: {{- end }} spec: serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.function_worker.component }}-acct" + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.function_worker "root" .)) }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.function_worker.securityContext "root" . "indent" 6) }} {{- if .Values.function_worker.nodeSelector }} nodeSelector: {{ toYaml .Values.function_worker.nodeSelector | indent 8 }} @@ -117,10 +119,14 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.function_worker.gracePeriod }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.function_worker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.function_worker "securityContext" .Values.function_worker.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} {{- if .Values.tls.function_worker.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.function_worker "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.function_worker "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.function_worker.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["sh", "-c"] args: @@ -128,12 +134,16 @@ spec: bin/certs-combine-pem.sh /pulsar/certs/cacerts/ca-combined.pem {{ template "pulsar.certs.cacerts" (dict "certs" .Values.tls.function_worker.cacerts.certs) }} volumeMounts: {{- include "pulsar.function_worker.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.function_worker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.components.broker .Values.function_worker.waitBrokerTimeout (gt (.Values.function_worker.waitBrokerTimeout | int) 0) }} # Wait for the broker to be resolvable before starting the worker (the worker is broker-attached). - name: wait-broker-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.function_worker "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.function_worker "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.function_worker.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.function_worker.waitBrokerTimeout }}", "sh", "-c"] args: @@ -142,6 +152,10 @@ spec: echo "waiting for broker {{ template "pulsar.fullname" . }}-{{ .Values.broker.component }} ... check in 3 seconds ..."; sleep 3; done; + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.function_worker.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if .Values.function_worker.initContainers }} {{- toYaml .Values.function_worker.initContainers | nindent 6 }} @@ -150,6 +164,7 @@ spec: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.function_worker.component }}" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.function_worker "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.function_worker "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.function_worker.containerSecurityContext "root" . "indent" 8) }} {{- if .Values.function_worker.probe.liveness.enabled }} livenessProbe: tcpSocket: @@ -206,8 +221,11 @@ spec: - configMapRef: name: "{{ template "pulsar.fullname" . }}-{{ .Values.function_worker.component }}" {{- $jwtAuth := and .Values.auth.authentication.enabled .Values.auth.authentication.jwt.enabled }} - {{- if or $jwtAuth .Values.function_worker.extraVolumeMounts .Values.tls.enabled }} + {{- if or $jwtAuth .Values.function_worker.extraVolumeMounts .Values.tls.enabled (include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.function_worker.containerSecurityContext "volumes" $edv "root" .)) }} volumeMounts: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.function_worker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 10 }} + {{- end }} {{- if $jwtAuth }} - mountPath: "/pulsar/keys" name: token-keys @@ -225,8 +243,11 @@ spec: env: {{- toYaml .Values.function_worker.extraEnvs | nindent 10 }} {{- end }} - {{- if or $jwtAuth .Values.function_worker.extraVolumes .Values.tls.enabled }} + {{- if or $jwtAuth .Values.function_worker.extraVolumes .Values.tls.enabled (include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.function_worker.containerSecurityContext "volumes" $edv "root" .)) }} volumes: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.function_worker.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} {{- if .Values.function_worker.extraVolumes }} {{ toYaml .Values.function_worker.extraVolumes | indent 6 }} {{- end }} diff --git a/charts/pulsar/templates/jwt-secret-init.yaml b/charts/pulsar/templates/jwt-secret-init.yaml index 129ec5d1..1de1c999 100644 --- a/charts/pulsar/templates/jwt-secret-init.yaml +++ b/charts/pulsar/templates/jwt-secret-init.yaml @@ -102,6 +102,8 @@ spec: {{- include "pulsar.imagePullSecrets" . | nindent 6 }} serviceAccountName: "{{ template "pulsar.fullname" . }}-jwt-secret-init" restartPolicy: Never + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.auth.authentication.jwt.generateSecrets "root" .)) }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.auth.authentication.jwt.generateSecrets.securityContext "root" . "indent" 6) }} {{- if .Values.auth.authentication.jwt.generateSecrets.nodeSelector }} nodeSelector: {{ toYaml .Values.auth.authentication.jwt.generateSecrets.nodeSelector | indent 8 }} @@ -161,10 +163,17 @@ spec: - name: jwt-secrets emptyDir: sizeLimit: 1Mi + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.auth.authentication.jwt.generateSecrets.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 8 }} + {{- end }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.auth.authentication.jwt.generateSecrets.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.broker "securityContext" .Values.auth.authentication.jwt.generateSecrets.containerSecurityContext "root" .) | nindent 8 }} + {{- end }} - name: generate-jwt-artifacts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.broker "root" .) }}" imagePullPolicy: {{ include "pulsar.imagePullPolicy" (dict "image" .Values.images.broker "root" .) }} + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.auth.authentication.jwt.generateSecrets.containerSecurityContext "root" . "indent" 10) }} command: - /bin/bash - -c @@ -201,10 +210,14 @@ spec: volumeMounts: - mountPath: /jwt-secrets name: jwt-secrets + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.auth.authentication.jwt.generateSecrets.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 12 }} + {{- end }} containers: - name: create-secrets image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.kubectl "root" .) }}" imagePullPolicy: {{ include "pulsar.imagePullPolicy" (dict "image" .Values.images.kubectl "root" .) }} + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.auth.authentication.jwt.generateSecrets.containerSecurityContext "root" . "indent" 10) }} env: - name: NAMESPACE valueFrom: @@ -259,5 +272,13 @@ spec: volumeMounts: - mountPath: /jwt-secrets name: jwt-secrets + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.auth.authentication.jwt.generateSecrets.containerSecurityContext "volumes" $edv "root" .) }} + {{- /* + This container runs the kubectl image rather than a Pulsar image. The mounts + are harmless there -- it has no /pulsar tree for them to shadow -- and it does + need a writable /tmp, so it takes the same list as the rest of the Job. + */}} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 12 }} + {{- end }} {{- end }} {{- end }} diff --git a/charts/pulsar/templates/oxia-coordinator-deployment.yaml b/charts/pulsar/templates/oxia-coordinator-deployment.yaml index 6953000e..3edbc838 100644 --- a/charts/pulsar/templates/oxia-coordinator-deployment.yaml +++ b/charts/pulsar/templates/oxia-coordinator-deployment.yaml @@ -66,6 +66,8 @@ spec: {{- toYaml . | nindent 10 }} {{- end }} serviceAccountName: {{ template "pulsar.fullname" . }}-{{ .Values.oxia.component }}-coordinator + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.oxia.coordinator "default" (list (dict "path" "/tmp" "sizeLimit" "1Gi")) "root" .)) }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.oxia.coordinator.securityContext "root" . "indent" 6) }} containers: - command: {{- if .Values.oxia.coordinator.entrypoint }} @@ -75,6 +77,7 @@ spec: {{- end }} image: "{{ .Values.images.oxia.repository }}:{{ .Values.images.oxia.tag }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.oxia "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.oxia.coordinator.containerSecurityContext "root" . "indent" 10) }} name: coordinator ports: {{- range $key, $value := .Values.oxia.coordinator.ports }} @@ -85,9 +88,14 @@ spec: limits: cpu: {{ .Values.oxia.coordinator.cpuLimit }} memory: {{ .Values.oxia.coordinator.memoryLimit }} - {{- if .Values.oxia.coordinator.extraVolumeMounts }} + {{- if or .Values.oxia.coordinator.extraVolumeMounts (include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.oxia.coordinator.containerSecurityContext "volumes" $edv "root" .)) }} volumeMounts: - {{- toYaml .Values.oxia.coordinator.extraVolumeMounts | nindent 12 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.oxia.coordinator.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 12 }} + {{- end }} + {{- if .Values.oxia.coordinator.extraVolumeMounts }} + {{- toYaml .Values.oxia.coordinator.extraVolumeMounts | nindent 12 }} + {{- end }} {{- end }} livenessProbe: {{- include "oxia-cluster.probe" .Values.oxia.coordinator.ports.internal | nindent 12 }} @@ -96,8 +104,13 @@ spec: {{- if .Values.oxia.coordinator.extraContainers }} {{- toYaml .Values.oxia.coordinator.extraContainers | nindent 8 }} {{- end }} - {{- if .Values.oxia.coordinator.extraVolumes }} + {{- if or .Values.oxia.coordinator.extraVolumes (include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.oxia.coordinator.containerSecurityContext "volumes" $edv "root" .)) }} volumes: -{{- toYaml .Values.oxia.coordinator.extraVolumes | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.oxia.coordinator.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 8 }} + {{- end }} + {{- if .Values.oxia.coordinator.extraVolumes }} + {{- toYaml .Values.oxia.coordinator.extraVolumes | nindent 8 }} + {{- end }} {{- end }} {{- end }} diff --git a/charts/pulsar/templates/oxia-server-statefulset.yaml b/charts/pulsar/templates/oxia-server-statefulset.yaml index ccfd630c..2d579555 100644 --- a/charts/pulsar/templates/oxia-server-statefulset.yaml +++ b/charts/pulsar/templates/oxia-server-statefulset.yaml @@ -110,9 +110,11 @@ spec: {{ end }} {{- end }} serviceAccountName: {{ template "pulsar.fullname" . }}-{{ .Values.oxia.component }} - {{- if .Values.oxia.server.securityContext }} - securityContext: -{{ toYaml .Values.oxia.server.securityContext | indent 8 }} + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.oxia.server "default" (list (dict "path" "/tmp" "sizeLimit" "1Gi")) "root" .)) }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.oxia.server.securityContext "root" . "indent" 6) }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.oxia.server.containerSecurityContext "volumes" $edv "root" .) }} + volumes: + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} {{- end }} containers: - command: @@ -130,6 +132,7 @@ spec: {{- end}} image: "{{ .Values.images.oxia.repository }}:{{ .Values.images.oxia.tag }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.oxia "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.oxia.server.containerSecurityContext "root" . "indent" 10) }} name: server ports: {{- range $key, $value := .Values.oxia.server.ports }} @@ -141,6 +144,9 @@ spec: cpu: {{ .Values.oxia.server.cpuLimit }} memory: {{ .Values.oxia.server.memoryLimit }} volumeMounts: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.oxia.server.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 12 }} + {{- end }} - name: {{ template "pulsar.fullname" . }}-{{ .Values.oxia.component }}-data mountPath: /data livenessProbe: diff --git a/charts/pulsar/templates/proxy-statefulset.yaml b/charts/pulsar/templates/proxy-statefulset.yaml index a89dad12..798b62b5 100644 --- a/charts/pulsar/templates/proxy-statefulset.yaml +++ b/charts/pulsar/templates/proxy-statefulset.yaml @@ -117,11 +117,17 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.proxy.gracePeriod }} serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.proxy.component }}" + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.proxy "root" .)) }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.proxy.securityContext "root" . "indent" 6) }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.proxy.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.proxy "securityContext" .Values.proxy.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} {{- if .Values.tls.proxy.cacerts.enabled }} - name: combine-certs image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.proxy "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.proxy "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.proxy.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["sh", "-c"] args: @@ -129,6 +135,9 @@ spec: bin/certs-combine-pem.sh /pulsar/certs/cacerts/ca-combined.pem {{ template "pulsar.certs.cacerts" (dict "certs" .Values.tls.proxy.cacerts.certs) }} volumeMounts: {{- include "pulsar.proxy.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.proxy.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.components.zookeeper (not .Values.standalone.enabled) .Values.proxy.waitZookeeperTimeout (gt (.Values.proxy.waitZookeeperTimeout | int) 0) }} # This init container will wait for zookeeper to be ready before @@ -136,6 +145,7 @@ spec: - name: wait-zookeeper-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.proxy "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.proxy "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.proxy.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.proxy.waitZookeeperTimeout }}", "sh", "-c"] args: @@ -154,11 +164,16 @@ spec: echo "pulsar cluster {{ template "pulsar.cluster.name" . }} isn't initialized yet ... check in 3 seconds ..." && sleep 3; done; {{- end}} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.proxy.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end}} {{- if and .Values.components.oxia (not .Values.standalone.enabled) .Values.proxy.waitOxiaTimeout (gt (.Values.proxy.waitOxiaTimeout | int) 0) }} - name: wait-oxia-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.proxy "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.proxy "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.proxy.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.proxy.waitOxiaTimeout }}", "sh", "-c"] args: @@ -166,7 +181,11 @@ spec: until nslookup {{ template "pulsar.oxia.server.service" . }}; do sleep 3; done; - {{- end }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.proxy.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} + {{- end }} {{- if and .Values.proxy.waitBrokerTimeout (gt (.Values.proxy.waitBrokerTimeout | int) 0) }} {{- $backendService := ternary (include "pulsar.standalone.service" .) (printf "%s-%s" (include "pulsar.fullname" .) .Values.broker.component) .Values.standalone.enabled }} # This init container waits for the broker backend (either the broker @@ -175,6 +194,7 @@ spec: - name: wait-broker-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.proxy "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.proxy "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.proxy.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.proxy.waitBrokerTimeout }}", "sh", "-c"] args: @@ -186,6 +206,10 @@ spec: sleep 10; brokerServiceNumber="$(nslookup -timeout=10 {{ $backendService }} | grep Name | wc -l)"; done; + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.proxy.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end}} {{- if .Values.proxy.initContainers }} {{- toYaml .Values.proxy.initContainers | nindent 6 }} @@ -194,6 +218,7 @@ spec: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.proxy.component }}" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.proxy "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.proxy "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.proxy.containerSecurityContext "root" . "indent" 8) }} {{- if .Values.proxy.probe.liveness.enabled }} livenessProbe: httpGet: @@ -263,8 +288,11 @@ spec: envFrom: - configMapRef: name: "{{ template "pulsar.fullname" . }}-{{ .Values.proxy.component }}" - {{- if or .Values.proxy.extraVolumeMounts .Values.auth.authentication.enabled (and .Values.tls.enabled (or .Values.tls.proxy.enabled .Values.tls.broker.enabled)) }} + {{- if or .Values.proxy.extraVolumeMounts .Values.auth.authentication.enabled (and .Values.tls.enabled (or .Values.tls.proxy.enabled .Values.tls.broker.enabled)) (include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.proxy.containerSecurityContext "volumes" $edv "root" .)) }} volumeMounts: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.proxy.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 10 }} + {{- end }} {{- if .Values.auth.authentication.enabled }} {{- if .Values.auth.authentication.jwt.enabled }} - mountPath: "/pulsar/keys" @@ -281,8 +309,11 @@ spec: {{- end }} {{- end}} {{- include "pulsar.imagePullSecrets" . | nindent 6}} - {{- if or .Values.proxy.extraVolumes .Values.auth.authentication.enabled (and .Values.tls.enabled .Values.tls.proxy.enabled) }} + {{- if or .Values.proxy.extraVolumes .Values.auth.authentication.enabled (and .Values.tls.enabled .Values.tls.proxy.enabled) (include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.proxy.containerSecurityContext "volumes" $edv "root" .)) }} volumes: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.proxy.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 8 }} + {{- end }} {{- if .Values.proxy.extraVolumes }} {{ toYaml .Values.proxy.extraVolumes | indent 8 }} {{- end }} diff --git a/charts/pulsar/templates/pulsar-cluster-initialize.yaml b/charts/pulsar/templates/pulsar-cluster-initialize.yaml index b10c192f..d5c4fab3 100755 --- a/charts/pulsar/templates/pulsar-cluster-initialize.yaml +++ b/charts/pulsar/templates/pulsar-cluster-initialize.yaml @@ -52,16 +52,22 @@ spec: nodeSelector: {{ toYaml .Values.pulsar_metadata.nodeSelector | indent 8 }} {{- end }} + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.pulsar_metadata "root" .)) }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.pulsar_metadata.securityContext "root" . "indent" 6) }} {{- with (.Values.pulsar_metadata.affinity).nodeAffinity }} affinity: nodeAffinity: {{- toYaml . | nindent 10 }} {{- end }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.pulsar_metadata.image "securityContext" .Values.pulsar_metadata.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} {{- if .Values.tls.toolset.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.pulsar_metadata.image "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.pulsar_metadata.image "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["sh", "-c"] args: @@ -69,12 +75,16 @@ spec: bin/certs-combine-pem.sh /pulsar/certs/cacerts/ca-combined.pem {{ template "pulsar.certs.cacerts" (dict "certs" .Values.tls.toolset.cacerts.certs) }} volumeMounts: {{- include "pulsar.toolset.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.components.zookeeper .Values.pulsar_metadata.waitZookeeperTimeout (gt (.Values.pulsar_metadata.waitZookeeperTimeout | int) 0) }} {{- if .Values.pulsar_metadata.configurationStore }} - name: wait-zk-cs-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.pulsar_metadata.image "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.pulsar_metadata.image "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.pulsar_metadata.waitZookeeperTimeout }}", "sh", "-c"] args: @@ -86,10 +96,14 @@ spec: done; volumeMounts: {{- include "pulsar.toolset.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} - name: wait-zk-metastore-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.pulsar_metadata.image "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.pulsar_metadata.image "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.pulsar_metadata.waitZookeeperTimeout }}", "sh", "-c"] args: @@ -104,11 +118,16 @@ spec: sleep 3; done; {{- end }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.components.oxia .Values.pulsar_metadata.waitOxiaTimeout (gt (.Values.pulsar_metadata.waitOxiaTimeout | int) 0) }} - name: wait-oxia-ready image: "{{ template "pulsar.imageFullName" (dict "image" .Values.pulsar_metadata.image "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.pulsar_metadata.image "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.pulsar_metadata.waitOxiaTimeout }}", "sh", "-c"] args: @@ -116,6 +135,10 @@ spec: until nslookup {{ template "pulsar.oxia.server.service" . }}; do sleep 3; done; + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "volumes" $edv "root" .) }} + volumeMounts: + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if and .Values.pulsar_metadata.waitBookkeeperTimeout (gt (.Values.pulsar_metadata.waitBookkeeperTimeout | int) 0) }} # This initContainer will wait for bookkeeper initnewcluster to complete @@ -123,6 +146,7 @@ spec: - name: pulsar-bookkeeper-verify-clusterid image: {{ template "pulsar.imageFullName" (dict "image" .Values.pulsar_metadata.image "root" .) }} imagePullPolicy: {{ template "pulsar.imagePullPolicy" (dict "image" .Values.pulsar_metadata.image "root" .) }} + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["timeout", "{{ .Values.pulsar_metadata.waitBookkeeperTimeout }}", "sh", "-c"] args: @@ -140,11 +164,15 @@ spec: name: {{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }} volumeMounts: {{- include "pulsar.toolset.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} containers: - name: {{ template "pulsar.fullname" . }}-{{ .Values.pulsar_metadata.component }} image: {{ template "pulsar.imageFullName" (dict "image" .Values.pulsar_metadata.image "root" .) }} imagePullPolicy: {{ template "pulsar.imagePullPolicy" (dict "image" .Values.pulsar_metadata.image "root" .) }} + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "root" . "indent" 8) }} {{- if .Values.pulsar_metadata.resources }} resources: {{ toYaml .Values.pulsar_metadata.resources | indent 10 }} @@ -188,8 +216,14 @@ spec: {{- end }} volumeMounts: {{- include "pulsar.toolset.certs.volumeMounts" . | nindent 10 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 10 }} + {{- end }} volumes: {{- include "pulsar.toolset.certs.volumes" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.pulsar_metadata.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 8 }} + {{- end }} restartPolicy: OnFailure {{- if .Values.pulsar_metadata.nodeSelector }} nodeSelector: diff --git a/charts/pulsar/templates/standalone-deployment.yaml b/charts/pulsar/templates/standalone-deployment.yaml index 54d65021..1631774b 100644 --- a/charts/pulsar/templates/standalone-deployment.yaml +++ b/charts/pulsar/templates/standalone-deployment.yaml @@ -107,14 +107,22 @@ spec: {{- toYaml . | nindent 10 }} {{- end }} terminationGracePeriodSeconds: {{ .Values.standalone.gracePeriod }} - {{- if .Values.standalone.initContainers }} + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.standalone "root" .)) }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.standalone.securityContext "root" . "indent" 6) }} + {{- if or .Values.standalone.initContainers (include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.standalone.containerSecurityContext "volumes" $edv "root" .)) }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.standalone.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.standalone "securityContext" .Values.standalone.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} + {{- if .Values.standalone.initContainers }} {{- toYaml .Values.standalone.initContainers | nindent 6 }} {{- end }} + {{- end }} containers: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.standalone.component }}" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.standalone "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.standalone "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.standalone.containerSecurityContext "root" . "indent" 8) }} {{- if .Values.standalone.probe.liveness.enabled }} livenessProbe: tcpSocket: @@ -199,10 +207,16 @@ spec: {{ toYaml .Values.standalone.extraVolumeMounts | indent 10 }} {{- end }} {{- include "pulsar.standalone.certs.volumeMounts" . | nindent 10 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.standalone.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 10 }} + {{- end }} {{- if .Values.standalone.extraEnvs }} {{- toYaml .Values.standalone.extraEnvs | nindent 10 }} {{- end }} volumes: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.standalone.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} - name: "{{ template "pulsar.fullname" . }}-{{ .Values.standalone.component }}-{{ .Values.standalone.volumes.data.name }}" persistentVolumeClaim: claimName: "{{ template "pulsar.fullname" . }}-{{ .Values.standalone.component }}-{{ .Values.standalone.volumes.data.name }}" diff --git a/charts/pulsar/templates/toolset-statefulset.yaml b/charts/pulsar/templates/toolset-statefulset.yaml index 257c33b3..3ac927ab 100644 --- a/charts/pulsar/templates/toolset-statefulset.yaml +++ b/charts/pulsar/templates/toolset-statefulset.yaml @@ -72,11 +72,21 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.toolset.gracePeriod }} serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.toolset.component }}" + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.toolset "root" .)) }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.toolset.securityContext "root" . "indent" 6) }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.toolset.containerSecurityContext "volumes" $edv "root" .) }} + {{- $toolsetImage := .Values.images.broker }} + {{- if (and .Values.images.toolset .Values.images.toolset.repository) }} + {{- $toolsetImage = .Values.images.toolset }} + {{- end }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" $toolsetImage "securityContext" .Values.toolset.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} {{- if .Values.tls.toolset.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.toolset "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.toolset "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.toolset.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["sh", "-c"] args: @@ -84,6 +94,9 @@ spec: bin/certs-combine-pem.sh /pulsar/certs/cacerts/ca-combined.pem {{ template "pulsar.certs.cacerts" (dict "certs" .Values.tls.toolset.cacerts.certs) }} volumeMounts: {{- include "pulsar.toolset.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.toolset.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if .Values.toolset.initContainers }} {{- toYaml .Values.toolset.initContainers | nindent 6 }} @@ -97,6 +110,7 @@ spec: image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.broker "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.broker "root" .) }}" {{- end }} + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.toolset.containerSecurityContext "root" . "indent" 8) }} {{- if .Values.toolset.resources }} resources: {{ toYaml .Values.toolset.resources | indent 10 }} @@ -131,7 +145,13 @@ spec: {{ toYaml .Values.toolset.extraVolumeMounts | indent 8 }} {{- end }} {{- include "pulsar.toolset.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.toolset.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} volumes: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.toolset.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} {{- if .Values.auth.authentication.enabled }} {{- if .Values.auth.authentication.jwt.enabled }} - name: client-token diff --git a/charts/pulsar/templates/zookeeper-statefulset-upgrade.yaml b/charts/pulsar/templates/zookeeper-statefulset-upgrade.yaml index d0b24d7e..d74d8f5f 100644 --- a/charts/pulsar/templates/zookeeper-statefulset-upgrade.yaml +++ b/charts/pulsar/templates/zookeeper-statefulset-upgrade.yaml @@ -87,14 +87,18 @@ metadata: spec: backoffLimit: 1 template: - {{- with .Values.job.podAnnotations }} metadata: + labels: + {{- include "pulsar.template.labels" . | nindent 8 }} + component: {{ .Values.zookeeper.component }}-sts-cleanup + {{- with .Values.job.podAnnotations }} annotations: {{- toYaml . | nindent 8 }} - {{- end }} + {{- end }} spec: serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.zookeeper.component }}-sts-cleanup" restartPolicy: Never + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.zookeeper.securityContext "root" . "indent" 6) }} {{- include "pulsar.imagePullSecrets" . | nindent 6 }} {{- if .Values.zookeeper.nodeSelector }} nodeSelector: @@ -124,6 +128,7 @@ spec: - name: sts-cleanup image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.kubectl "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.kubectl "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.zookeeper.containerSecurityContext "root" . "indent" 10) }} command: - sh - -c diff --git a/charts/pulsar/templates/zookeeper-statefulset.yaml b/charts/pulsar/templates/zookeeper-statefulset.yaml index f2a282a6..cba1bf97 100755 --- a/charts/pulsar/templates/zookeeper-statefulset.yaml +++ b/charts/pulsar/templates/zookeeper-statefulset.yaml @@ -117,15 +117,17 @@ spec: {{- end }} terminationGracePeriodSeconds: {{ .Values.zookeeper.gracePeriod }} serviceAccountName: "{{ template "pulsar.fullname" . }}-{{ .Values.zookeeper.component }}" - {{- if .Values.zookeeper.securityContext }} - securityContext: -{{ toYaml .Values.zookeeper.securityContext | indent 8 }} - {{- end }} + {{- $edv := fromJsonArray (include "pulsar.emptyDirVolumes.resolve" (dict "component" .Values.zookeeper "root" .)) }} + {{- include "pulsar.podSecurityContext" (dict "securityContext" .Values.zookeeper.securityContext "root" . "indent" 6) }} initContainers: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.zookeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.seedContainer" (dict "volumes" $edv "image" .Values.images.zookeeper "securityContext" .Values.zookeeper.containerSecurityContext "root" .) | nindent 6 }} + {{- end }} {{- if .Values.tls.zookeeper.cacerts.enabled }} - name: cacerts image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.zookeeper "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.zookeeper "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.zookeeper.containerSecurityContext "root" . "indent" 8) }} resources: {{ toYaml .Values.initContainer.resources | nindent 10 }} command: ["sh", "-c"] args: @@ -133,6 +135,9 @@ spec: bin/certs-combine-pem.sh /pulsar/certs/cacerts/ca-combined.pem {{ template "pulsar.certs.cacerts" (dict "certs" .Values.tls.zookeeper.cacerts.certs) }} volumeMounts: {{- include "pulsar.zookeeper.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.zookeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- end }} {{- if .Values.zookeeper.initContainers }} {{- toYaml .Values.zookeeper.initContainers | nindent 6 }} @@ -141,6 +146,7 @@ spec: - name: "{{ template "pulsar.fullname" . }}-{{ .Values.zookeeper.component }}" image: "{{ template "pulsar.imageFullName" (dict "image" .Values.images.zookeeper "root" .) }}" imagePullPolicy: "{{ template "pulsar.imagePullPolicy" (dict "image" .Values.images.zookeeper "root" .) }}" + {{- include "pulsar.containerSecurityContext" (dict "securityContext" .Values.zookeeper.containerSecurityContext "root" . "indent" 8) }} {{- if .Values.zookeeper.resources }} resources: {{ toYaml .Values.zookeeper.resources | indent 10 }} @@ -249,10 +255,16 @@ spec: mountPath: /pulsar/data-log {{- end }} {{- include "pulsar.zookeeper.certs.volumeMounts" . | nindent 8 }} + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.zookeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.mounts" $edv | nindent 8 }} + {{- end }} {{- if .Values.zookeeper.extraVolumeMounts }} {{ toYaml .Values.zookeeper.extraVolumeMounts | indent 8 }} {{- end }} volumes: + {{- if include "pulsar.emptyDirVolumes.enabled" (dict "securityContext" .Values.zookeeper.containerSecurityContext "volumes" $edv "root" .) }} + {{- include "pulsar.emptyDirVolumes.volumes" $edv | nindent 6 }} + {{- end }} {{- if not (and (and .Values.volumes.persistence .Values.volumes.persistence) .Values.zookeeper.volumes.persistence) }} - name: "{{ template "pulsar.fullname" . }}-{{ .Values.zookeeper.component }}-{{ .Values.zookeeper.volumes.data.name }}" emptyDir: {} diff --git a/charts/pulsar/values.yaml b/charts/pulsar/values.yaml index b620903b..a4340e5c 100755 --- a/charts/pulsar/values.yaml +++ b/charts/pulsar/values.yaml @@ -116,6 +116,74 @@ affinity: # preferredDuringSchedulingIgnoredDuringExecution - scheduler will try to enforce but not guranentee type: requiredDuringSchedulingIgnoredDuringExecution +## SecurityContext +## +## Global security contexts applied to every pod and every container this chart +## renders, including initContainers and the init/cleanup Jobs. Both are empty by +## default, so the chart's rendered output is unchanged unless you set them. +## +## Each is merged with the matching per-component override, and the per-component +## value wins on a per-key basis: +## +## podSecurityContext <- merged with .securityContext +## containerSecurityContext <- merged with .containerSecurityContext +## +## Use these when a cluster policy engine (Pod Security Admission, OPA Gatekeeper, +## Kyverno) requires settings the chart does not set by default. For example: +## +## podSecurityContext: +## runAsNonRoot: true +## runAsUser: 10000 +## runAsGroup: 10000 +## fsGroup: 10000 +## fsGroupChangePolicy: OnRootMismatch +## supplementalGroups: [10000] +## containerSecurityContext: +## allowPrivilegeEscalation: false +## capabilities: +## drop: ["ALL"] +## seccompProfile: +## type: RuntimeDefault +## +## Note that zookeeper, bookkeeper, broker and oxia.server ship +## `securityContext.fsGroup: 0` by default, and a per-component value takes +## precedence over the global one. To move those four components off GID 0 you must +## also override their own `securityContext` (see the comment on +## `zookeeper.securityContext`), and be aware that changing `fsGroup` on a cluster +## with existing volumes triggers a recursive ownership change on first mount. +## +## Setting `readOnlyRootFilesystem: true` is supported: the chart then mounts the +## writable volumes the Pulsar images need. See `emptyDirVolumes` below. +podSecurityContext: {} +containerSecurityContext: {} + +## Writable volumes mounted when a component sets `readOnlyRootFilesystem: true`; the +## Pulsar images rewrite /pulsar/conf on startup, log to /pulsar/logs, and use /tmp. +## Ignored while `readOnlyRootFilesystem` is unset, so the default output is unchanged. +## +## Per entry: `path` (required, absolute; the volume name is derived from it, so two +## paths that sanitise alike are rejected), `seedFromImage` (copy the image's contents +## there first, via a `copy-pulsar-conf` initContainer, since an emptyDir starts empty), +## and `sizeLimit` (an uncapped emptyDir can fill the node and get the pod evicted). +## +## `.emptyDirVolumes` REPLACES this list rather than extending it; use `[]` +## for "this component needs none" -- but a component with `readOnlyRootFilesystem: true` +## and an empty list has no writable path at all, so supply them yourself through +## `.extraVolumes` / `.extraVolumeMounts`. A path the component +## already mounts through `extraVolumeMounts` is skipped, since duplicate mountPaths are +## rejected. +## See the README for the components that need something other than this default. +## +## Keep in sync with the fallback in the `pulsar.emptyDirVolumes.resolve` helper, which +## applies when the key is absent (`helm upgrade --reuse-values`). +emptyDirVolumes: + - path: /pulsar/conf + seedFromImage: true + - path: /pulsar/logs + sizeLimit: 1Gi + - path: /tmp + sizeLimit: 1Gi + ## Components ## ## Control what components of Apache Pulsar to deploy for the cluster @@ -442,6 +510,9 @@ auth: enabled: false # nodeSelector: # cloud.google.com/gke-nodepool: default-pool + # Merged over the global `podSecurityContext` / `containerSecurityContext`. + securityContext: {} + containerSecurityContext: {} tolerations: [] affinity: anti_affinity: false @@ -650,9 +721,15 @@ zookeeper: extraVolumes: [] extraVolumeMounts: [] # Ensures 2.10.0 non-root docker image works correctly. + # Merged over the global `podSecurityContext`; these keys win over it. Override + # `fsGroup` here if a cluster policy constrains group IDs. `fsGroup` is added as a + # supplementary group, so `runAsGroup` does not need to match it. securityContext: fsGroup: 0 fsGroupChangePolicy: "OnRootMismatch" + # Container-level securityContext, merged over the global `containerSecurityContext`. + # Applied to the zookeeper container and its initContainers. + containerSecurityContext: {} volumes: useSeparateDiskForTxlog: false # use a persistent volume or emptyDir @@ -766,6 +843,9 @@ oxia: annotations: {} service_account: annotations: {} + # Merged over the global `podSecurityContext` / `containerSecurityContext`. + securityContext: {} + containerSecurityContext: {} tolerations: [] # nodeSelector: # cloud.google.com/gke-nodepool: default-pool @@ -834,9 +914,14 @@ oxia: annotations: {} service_account: annotations: {} + # Merged over the global `podSecurityContext`; these keys win over it. Override + # `fsGroup` here if a cluster policy constrains group IDs. `fsGroup` is added as a + # supplementary group, so `runAsGroup` does not need to match it. securityContext: fsGroup: 0 fsGroupChangePolicy: "OnRootMismatch" + # Merged over the global `containerSecurityContext`. + containerSecurityContext: {} affinity: anti_affinity: true anti_affinity_topology_key: kubernetes.io/hostname @@ -970,9 +1055,16 @@ bookkeeper: extraVolumes: [] extraVolumeMounts: [] # Ensures 2.10.0 non-root docker image works correctly. + # Merged over the global `podSecurityContext`; these keys win over it. Override + # `fsGroup` here if a cluster policy constrains group IDs. `fsGroup` is added as a + # supplementary group, so `runAsGroup` does not need to match it. + # This context is also applied to the bookkeeper cluster-initialize Job. securityContext: fsGroup: 0 fsGroupChangePolicy: "OnRootMismatch" + # Container-level securityContext, merged over the global `containerSecurityContext`. + # Applied to the bookie container, its initContainers and the cluster-initialize Job. + containerSecurityContext: {} volumes: # use a persistent volume or emptyDir persistence: true @@ -1144,6 +1236,44 @@ autorecovery: restartPodsOnConfigMapChange: false ports: http: 8000 + # Merged over the global `podSecurityContext` / `containerSecurityContext`. + # Unlike zookeeper/bookkeeper/broker this component mounts no data volume, so it + # carries no `fsGroup: 0` default. + securityContext: {} + containerSecurityContext: {} + # The autorecovery daemon does not start BookKeeper's HTTP service, so the probes + # target the Prometheus stats endpoint on `ports.http` -- the same endpoint the + # PodMonitor scrapes -- rather than a /api/v1 health endpoint. + # + # Both are off by default on purpose, so enabling them is opt-in and this chart's + # rendered output is unchanged for existing deployments. + # + # /metrics is served by the image's BookKeeper stats provider, not by anything this + # chart configures. A cluster that overrides `statsProviderClass` or + # `prometheusStatsHttpPort` through `autorecovery.configData` -- settings that were + # previously inert here -- would break both probes: liveness turns a working daemon + # into a restart loop, and readiness leaves the pod permanently unready, which stalls + # the StatefulSet rollout. Enable them once you have confirmed that `ports.http` + # serves /metrics in your deployment. + probe: + liveness: + enabled: false + failureThreshold: 60 + initialDelaySeconds: 10 + periodSeconds: 30 + timeoutSeconds: 5 + readiness: + enabled: false + failureThreshold: 60 + initialDelaySeconds: 10 + periodSeconds: 30 + timeoutSeconds: 5 + startup: + enabled: false + failureThreshold: 30 + initialDelaySeconds: 60 + periodSeconds: 30 + timeoutSeconds: 5 # nodeSelector: # cloud.google.com/gke-nodepool: default-pool # priorityClassName: high-priority-nonpreempting @@ -1227,6 +1357,10 @@ pulsar_metadata: waitBookkeeperTimeout: 120 ## Timeout for running metadata initialization initTimeout: 60 + ## Security contexts for the `pulsar-cluster-initialize` job, merged over the + ## global `podSecurityContext` / `containerSecurityContext`. + securityContext: {} + containerSecurityContext: {} ## Allow read-only operations on the metadata store when the metadata store is not available. ## This is useful when you want to continue serving requests even if the metadata store is not fully available with quorum. @@ -1333,6 +1467,9 @@ standalone: https: 8443 pulsar: 6650 pulsarssl: 6651 + # Merged over the global `podSecurityContext` / `containerSecurityContext`. + securityContext: {} + containerSecurityContext: {} # nodeSelector: # cloud.google.com/gke-nodepool: default-pool # priorityClassName: high-priority-nonpreempting @@ -1411,10 +1548,15 @@ broker: # GID 0 (the group the pulsar user, uid 10000, belongs to) and made group-writable, so the broker can # read/write it. Required when packageManagement.fileSystemStorage is enabled (the broker mounts a # PersistentVolumeClaim as the FileSystemPackagesStorage directory); harmless otherwise. Mirrors the - # bookkeeper/zookeeper securityContext. + # bookkeeper/zookeeper securityContext. Merged over the global `podSecurityContext`; + # these keys win over it. This context is also applied to the broker sts-cleanup + # pre-upgrade hook Job. securityContext: fsGroup: 0 fsGroupChangePolicy: "OnRootMismatch" + # Container-level securityContext, merged over the global `containerSecurityContext`. + # Applied to the broker container, its initContainers and the sts-cleanup hook Job. + containerSecurityContext: {} autoscaling: enabled: false minReplicas: 1 @@ -1759,6 +1901,9 @@ function_worker: ports: http: 6750 https: 6751 + # Merged over the global `podSecurityContext` / `containerSecurityContext`. + securityContext: {} + containerSecurityContext: {} # podManagementPolicy for the StatefulSet podManagementPolicy: Parallel # This is how Victoria Metrics or Prometheus discovers this component @@ -1873,6 +2018,9 @@ proxy: # so the metrics are correctly rendered in grafana dashboard component: proxy replicaCount: 3 + # Merged over the global `podSecurityContext` / `containerSecurityContext`. + securityContext: {} + containerSecurityContext: {} autoscaling: enabled: false minReplicas: 1 @@ -2098,6 +2246,9 @@ toolset: useProxy: true replicaCount: 1 initContainers: [] + # Merged over the global `podSecurityContext` / `containerSecurityContext`. + securityContext: {} + containerSecurityContext: {} # True includes annotation for statefulset that contains hash of corresponding configmap, which will cause pods to restart on configmap change restartPodsOnConfigMapChange: false # nodeSelector: @@ -2396,6 +2547,9 @@ dekaf: annotations: {} podAnnotations: {} nodeSelector: {} + # Merged over the global `podSecurityContext` / `containerSecurityContext`. + securityContext: {} + containerSecurityContext: {} tolerations: [] # priorityClassName: high-priority-nonpreempting affinity: diff --git a/examples/README.md b/examples/README.md index 9440e06f..81b7b517 100644 --- a/examples/README.md +++ b/examples/README.md @@ -198,6 +198,8 @@ topics). Function instances run with the Kubernetes runtime (one pod per instanc | [`values-tls-ca.yaml`](values-tls-ca.yaml) | Enable TLS (including the bookie) using a CA issuer that references an existing `ca-key-pair` secret, with custom DNS SANs for the proxy. Requires cert-manager. | | [`values-jwt-symmetric.yaml`](values-jwt-symmetric.yaml) | Enable JWT authentication and authorization using a symmetric (shared secret) key (`usingSecretKey: true`). The signing key and per-superuser tokens are generated [in-chart](../README.md#in-chart-jwt-secret-generation) (`generateSecrets.enabled: true`), so a fully authenticated cluster deploys with a single `helm install`. | | [`values-jwt-asymmetric.yaml`](values-jwt-asymmetric.yaml) | Enable JWT authentication and authorization using an asymmetric (RSA private/public) key pair (`usingSecretKey: false`). The signing key pair and per-superuser tokens are generated [in-chart](../README.md#in-chart-jwt-secret-generation) (`generateSecrets.enabled: true`), so a fully authenticated cluster deploys with a single `helm install`. | +| [`values-psa-restricted.yaml`](values-psa-restricted.yaml) | Run under the Kubernetes `restricted` Pod Security Standard: `runAsNonRoot` plus no privilege escalation, all capabilities dropped, `RuntimeDefault` seccomp and a read-only root filesystem. Leaves the chart's per-component `fsGroup: 0` defaults alone, since `restricted` places no constraint on `fsGroup`. See [Pod and container security contexts](../README.md#pod-and-container-security-contexts). | +| [`values-restricted-group-ranges.yaml`](values-restricted-group-ranges.yaml) | For the narrower case of a policy that constrains group IDs to a numeric range (Gatekeeper's `K8sPSPAllowedUsers`, an equivalent Kyverno policy, or an OpenShift SCC) and so rejects `fsGroup: 0`. Overrides `fsGroup` on the four components that ship it. Not required by PSA `restricted` — changing `fsGroup` on existing volumes triggers a recursive ownership change, so read the warning in the file first. | ### Monitoring diff --git a/examples/values-psa-restricted.yaml b/examples/values-psa-restricted.yaml new file mode 100644 index 00000000..98a22fc7 --- /dev/null +++ b/examples/values-psa-restricted.yaml @@ -0,0 +1,62 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +# Run the cluster under the Kubernetes `restricted` Pod Security Standard, i.e. in a +# namespace labelled: +# +# pod-security.kubernetes.io/enforce: restricted +# +# These are the controls `restricted` actually enforces. It places no constraint on +# `fsGroup`, `fsGroupChangePolicy` or `supplementalGroups`, so the chart's per-component +# `fsGroup: 0` defaults are left exactly as they are -- they are what makes the Pulsar +# images work under an arbitrary assigned UID, and `fsGroup: 0` is admitted by +# `restricted` unchanged. +# +# If your cluster additionally runs a policy that constrains group IDs to a numeric +# range, see values-restricted-group-ranges.yaml. + +# The bundled monitoring stack is not restricted-compliant, and the securityContext +# settings below do not reach subchart templates: node-exporter needs hostNetwork, +# hostPID and hostPath, Grafana chowns as root, and some workloads set no seccompProfile. +# Left enabled its pods are rejected on admission while Pulsar itself comes up fine, so +# deploy monitoring separately into a namespace with a policy that suits it. +victoria-metrics-k8s-stack: + enabled: false + +# Applied to every pod the chart renders, including the init/cleanup Jobs. +podSecurityContext: + runAsNonRoot: true + # Set explicitly because the sts-cleanup pre-upgrade hooks run alpine/k8s, which has + # no USER directive: with runAsNonRoot and no runAsUser the kubelet refuses them + # ("image will run as root") and `helm upgrade` is blocked. 10000 is the uid of the + # `pulsar` user in the apachepulsar images, and kubectl runs fine under it too. + runAsUser: 10000 + +# Applied to every container and initContainer. This matches the container-level controls +# of both PSS `restricted` and OpenShift's `restricted-v2` SCC. +# +# readOnlyRootFilesystem is in neither standard but is supported here; see the README for +# what it mounts, and note that logs on an emptyDir do not survive pod replacement. +containerSecurityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + seccompProfile: + type: RuntimeDefault diff --git a/examples/values-restricted-group-ranges.yaml b/examples/values-restricted-group-ranges.yaml new file mode 100644 index 00000000..82b9d614 --- /dev/null +++ b/examples/values-restricted-group-ranges.yaml @@ -0,0 +1,108 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +# Run the cluster under a policy that constrains *group IDs* to a numeric range, and so +# rejects the chart's `fsGroup: 0` defaults. Examples of such a policy: +# +# - Gatekeeper's PodSecurityPolicy-derived K8sPSPAllowedUsers constraint template with +# fsGroup: {rule: MustRunAs, ranges: [{min: 1, max: 65535}]} +# supplementalGroups: {rule: MustRunAs, ranges: [{min: 1, max: 65535}]} +# runAsGroup: {rule: MustRunAs, ranges: [{min: 1, max: 65535}]} +# - a Kyverno policy doing the same +# - an OpenShift SCC whose fsGroup rule is MustRunAs against a preallocated range +# +# This is not what the Kubernetes `restricted` Pod Security Standard requires: PSA +# `restricted` places no constraint on fsGroup, fsGroupChangePolicy or supplementalGroups +# and admits `fsGroup: 0` unchanged. If `restricted` is all you need, prefer +# values-psa-restricted.yaml, which leaves the fsGroup defaults alone. +# +# Changing fsGroup on a cluster that already has provisioned volumes triggers a recursive +# ownership change on first mount. `fsGroupChangePolicy: OnRootMismatch` limits that to the +# first pass, but it can still take a long time on large bookie ledger volumes, so test on +# a copy first. +# +# Not an OpenShift example: under the stock `restricted-v2` SCC an explicit fsGroup of +# 10000 is rejected just as 0 is, since MustRunAs validates against the namespace's +# preallocated range. There you want to omit the group settings and let admission inject +# them, which means clearing the chart defaults (`zookeeper: {securityContext: null}` and +# so on) rather than overriding them. + +# The bundled monitoring stack is disabled: the settings below do not reach subchart +# templates, so its pods are rejected by a group-range policy while Pulsar comes up fine. +# Deploy monitoring separately into a namespace with a policy that suits it. +victoria-metrics-k8s-stack: + enabled: false + +# Applied to every pod the chart renders, including the init/cleanup Jobs. +podSecurityContext: + runAsNonRoot: true + # 10000 is the uid of the `pulsar` user in the apachepulsar images. + runAsUser: 10000 + runAsGroup: 10000 + fsGroup: 10000 + fsGroupChangePolicy: OnRootMismatch + supplementalGroups: [10000] + +# Applied to every container and initContainer the chart renders. +containerSecurityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + seccompProfile: + type: RuntimeDefault + +# If your policy also requires a liveness and readiness probe on every long-running +# workload, note that the autorecovery StatefulSet has none by default. You can enable +# them, but only if you have not overridden `statsProviderClass` or +# `prometheusStatsHttpPort` in `autorecovery.configData` -- both probes target the +# /metrics endpoint served by the image's BookKeeper stats provider, so changing either +# setting breaks them. +# +# autorecovery: +# probe: +# liveness: +# enabled: true +# readiness: +# enabled: true + +# zookeeper, bookkeeper, broker and oxia.server ship `securityContext.fsGroup: 0`, and a +# per-component value takes precedence over the global one, so each has to be overridden +# explicitly for a group-range policy to admit them. +# +# `fsGroup` is added as a supplementary group, so `runAsGroup` above does not need to +# match these values. +zookeeper: + securityContext: + fsGroup: 10000 + fsGroupChangePolicy: OnRootMismatch +bookkeeper: + securityContext: + fsGroup: 10000 + fsGroupChangePolicy: OnRootMismatch +broker: + securityContext: + fsGroup: 10000 + fsGroupChangePolicy: OnRootMismatch +# Only used when the metadata store is Oxia (components.oxia: true). +oxia: + server: + securityContext: + fsGroup: 10000 + fsGroupChangePolicy: OnRootMismatch