diff --git a/OPENSHELL_VERSION b/OPENSHELL_VERSION index df9bd5bdd..cce8f6ef5 100644 --- a/OPENSHELL_VERSION +++ b/OPENSHELL_VERSION @@ -1,7 +1,7 @@ # OpenShell upstream coordinates. Gateway, supervisor, sandbox runtime, and CLI image tags # and the chart ref use OPENSHELL_TAG. The console image is pinned by digest # (it uses its own tagging scheme). Update this file when bumping versions. -OPENSHELL_TAG=v0.1.5-rhaiv.3 +OPENSHELL_TAG=v0.1.6-rhaiv.0 OPENSHELL_CHART_REPO=https://github.com/opendatahub-io/openshell.git OPENSHELL_GATEWAY_IMAGE=quay.io/opendatahub/odh-openshell-gateway OPENSHELL_SUPERVISOR_IMAGE=quay.io/opendatahub/odh-openshell-supervisor diff --git a/charts/openshell/README.md b/charts/openshell/README.md index a4dea45a6..7f0b7c6fb 100644 --- a/charts/openshell/README.md +++ b/charts/openshell/README.md @@ -207,9 +207,81 @@ See [`values.yaml`](values.yaml) for source defaults. Selected overlays: - [`ci/values-keycloak.yaml`](ci/values-keycloak.yaml) - Keycloak OIDC integration - [`ci/values-high-availability.yaml`](ci/values-high-availability.yaml) - CI overlay for multi-replica external PostgreSQL testing - [`ci/values-autoscaling.yaml`](ci/values-autoscaling.yaml) - CI overlay for rendering the optional gateway HorizontalPodAutoscaler +- [`ci/values-supervisor-autoscaling.yaml`](ci/values-supervisor-autoscaling.yaml) - CI overlay for rendering the optional shared supervisor memory HorizontalPodAutoscaler - [`ci/values-spire.yaml`](ci/values-spire.yaml) - SPIFFE/SPIRE provider token grants - [`ci/values-spire-stack.yaml`](ci/values-spire-stack.yaml) - SPIRE hardened chart values for local development +### Additional destination CAs + +Create an operator-owned ConfigMap with a `ca.crt` key when sandboxes must reach +TLS destinations signed by a private CA: + +```shell +kubectl create configmap private-destination-ca -n openshell \ + --from-file=ca.crt=/path/to/ca.crt +``` + +Then set the source ConfigMap name: + +```yaml +supervisor: + network: + additionalCaConfigMapName: private-destination-ca +``` + +The chart mounts the source into the gateway and renders +`[openshell.supervisor.network].additional_ca_bundle`. The gateway validates +and normalizes the one certificate-only PEM bundle at startup. The Kubernetes +driver creates a separate managed ConfigMap in each target sandbox namespace +and mounts it only in the dedicated supervisor Pod at +`/etc/openshell/tls/additional/ca-bundle.pem`. That supervisor receives the +path through `--additional-ca-bundle` and folds the bundle into system roots. +The configured roots augment default destination trust. They do not configure +the corporate proxy, OIDC, or gateway callback mTLS trust. + +Changing only the source ConfigMap does not roll or reload the gateway. Restart +the gateway, then recreate or restart affected sandboxes to use the new roots. +Removing `additionalCaConfigMapName` removes the global TOML setting; newly +created or recreated sandbox pods then have no additional-CA volume or mount. +Existing running sandbox supervisors retain their startup trust until they are +restarted or recreated. + +The driver creates one immutable, content-addressed ConfigMap generation in +each target namespace. Names begin with +`openshell-additional-ca-` and include DNS-safe hashes of the effective +gateway ID and normalized bundle generation. Existing objects are accepted only +when their ownership labels, generation annotation, immutability, and exact +`ca.crt` data match; the driver never patches or updates them. + +Kubernetes cannot restrict `create` by `resourceNames`, and `resourceNames` +cannot express the digest-bearing name prefix needed for `get`. Enabling this +feature therefore gives the gateway/driver service account namespace-wide +ConfigMap `get` and `create` permissions in shared mode. In managed/operator +modes, the chart's ClusterRoleBinding makes those permissions cluster-wide for +the bound service account. The chart does not grant ConfigMap `list`, `watch`, +`patch`, `update`, or `delete`. Use a dedicated gateway service account and +namespace-scoped bindings when this boundary matters. If you render the TOML +setting outside this Helm value, grant `get` and `create` in every target +sandbox namespace. + +Disabling the setting intentionally does not delete driver-managed generations: +the gateway has no ConfigMap list/delete permissions, and an older object may +still be referenced by a running or stopped sandbox. An operator with separate +list/delete authority can inventory the gateway's generations by label: + +```shell +kubectl -n get configmap \ + -l 'openshell.ai/managed-by=openshell,openshell.ai/gateway-id=' +``` + +Before deleting a named generation, verify that no Sandbox resource or pod +volume in that namespace references it. Delete only the confirmed unreferenced +name; removing a referenced generation can make a later pod start or reschedule +fail closed. + +A configured CA can authenticate every policy-permitted TLS endpoint with a +matching certificate chain and hostname. Scope private CAs accordingly. + ### Database backend By default, OpenShell uses SQLite and runs the gateway as a StatefulSet so the @@ -377,6 +449,19 @@ Disabling autoscaling renders `spec.replicas` from `replicaCount` again, which defaults to 1, so set `replicaCount` to the replica count you want in that upgrade. +Set `supervisor.autoscaling.enabled=true` with the shared supervisor topology +to autoscale its Deployment independently of the gateway. It uses average +memory per Pod (`supervisor.autoscaling.targetMemoryAverageValue`, default +`512Mi`) through the resource metrics API, usually provided by Metrics Server. +Defaults keep two to four members and remove at most one member every two +minutes after a five-minute stabilization window. The HPA owns replicas, so +`supervisor.replicas` is ignored. Enabling it on an existing Deployment has +the same temporary reset-to-one risk described above; pick up new defaults +with `--reset-then-reuse-values`. When disabling it, set `supervisor.replicas` +explicitly. Existing sandboxes stay on their owners when scaling up, and +memory-based scaling does not guarantee spare capacity for handoff. See the +[shared supervisor autoscaling guide](https://docs.nvidia.com/openshell/latest/kubernetes/high-availability#autoscale-shared-supervisors-by-memory). + ## Secret bootstrap By default, a pre-install/pre-upgrade hook Job runs `openshell-gateway generate-certs` @@ -586,6 +671,7 @@ discovery endpoint or its TLS CA. | server.sandboxJwt.ttlSecs | int | `3600` | Token TTL in seconds. Defaults to 3600 (1h). | | server.sandboxNamespace | string | `""` | Namespace where sandbox pods are created. Defaults to the Helm release namespace (.Release.Namespace) when left empty. | | server.sandboxUid | string | `""` | UID for sandbox pods (`sandbox_uid`). Empty (default) = use the OpenShift SCC namespace annotation if present, otherwise the driver default. Must be an integer between 1 and 4294967294. | +| server.serviceHostnameFormat | string | `""` | Hostname format for exposed sandbox services. Empty or "readable" keeps readable `--[--]` hostnames; "hashed" mints opaque one-label hostnames discriminated by the gateway installation name (server.name, defaulting to the release name). The name must be unique among all gateways sharing the service base domain, and renaming the installation changes every minted hostname. | | server.telemetryEnabled | bool | `true` | Enable anonymous OpenShell telemetry from the gateway and the sandbox supervisors it launches. | | server.tls.certSecretName | string | `"openshell-server-tls"` | K8s secret (type kubernetes.io/tls) with tls.crt and tls.key for the server. | | server.tls.clientCaSecretName | string | `"openshell-server-client-ca"` | K8s secret with ca.crt for client certificate verification (mTLS). Only used when enableMtls is true. Set to "" to disable client certificate verification for HTTPS-only mode. | @@ -600,16 +686,26 @@ discovery endpoint or its TLS CA. | serviceAccount.annotations | object | `{}` | Annotations to add to the generated service account. | | serviceAccount.create | bool | `true` | Create a service account for the gateway. | | serviceAccount.name | string | `""` | Existing service account name to use when serviceAccount.create is false. | +| supervisor.affinity | object | `{}` | Affinity rules for the shared supervisor pod, rendered only when gatewayConfig selects supervisor_topology = "gateway". Top-level affinity applies only to the gateway pod. | +| supervisor.autoscaling.behavior | object | `{"scaleDown":{"policies":[{"periodSeconds":120,"type":"Pods","value":1}],"stabilizationWindowSeconds":300}}` | HPA scaling behavior. Remove at most one member every two minutes, after a five-minute stabilization window, to allow ownership handoff. | +| supervisor.autoscaling.enabled | bool | `false` | Enable memory-based autoscaling of the shared supervisor Deployment. Requires supervisor_topology = "gateway" and the Kubernetes resource metrics API (usually Metrics Server). When enabled, the HPA owns the replica count and supervisor.replicas is ignored. | +| supervisor.autoscaling.maxReplicas | int | `4` | Maximum supervisor replicas. | +| supervisor.autoscaling.minReplicas | int | `2` | Minimum supervisor replicas. Keep at least 2 for failover capacity. | +| supervisor.autoscaling.targetMemoryAverageValue | string | `"512Mi"` | Target average memory working set per supervisor Pod, as a positive Kubernetes quantity. This is an absolute value, not a percentage of requests. Scaling up adds capacity for new sandboxes; it does not move existing sandboxes away from a busy member. | | supervisor.image.digest | string | `""` | Supervisor image digest. When set, this takes precedence over tag. | | supervisor.image.pullPolicy | string | `nil` | Supervisor image pull policy. Empty uses global.image.pullPolicy. Prefer always, if_not_present, or never; the chart also accepts legacy Kubernetes spellings Always, IfNotPresent, and Never. | | supervisor.image.registry | string | `""` | Supervisor image registry. Empty uses global.image.registry. | | supervisor.image.repository | string | `"openshell/supervisor"` | Supervisor image repository. | | supervisor.image.tag | string | `""` | Supervisor image tag. Defaults to the chart appVersion when empty. | +| supervisor.network.additionalCaConfigMapName | string | `""` | Name of an operator-owned ConfigMap with a `ca.crt` key containing additional destination CA certificates for sandbox supervisors. Empty disables additional destination trust. | +| supervisor.podDisruptionBudget.enabled | bool | `true` | Render a PodDisruptionBudget with maxUnavailable 1 for the shared supervisor, used only when gatewayConfig selects supervisor_topology = "gateway". Disable it only when maintenance must evict more than one supervisor Pod at a time, or evict a member while another is unavailable. | | supervisor.podSecurityContext.fsGroup | int | `65534` | fsGroup assigned to the shared supervisor pod. | | supervisor.podSecurityContext.runAsGroup | int | `65534` | Primary GID assigned to the shared supervisor pod. | | supervisor.podSecurityContext.runAsNonRoot | bool | `true` | Require the shared supervisor to run as a non-root user. | | supervisor.podSecurityContext.runAsUser | int | `65534` | UID assigned to the shared supervisor pod. | | supervisor.podSecurityContext.seccompProfile.type | string | `"RuntimeDefault"` | Seccomp profile applied to the shared supervisor pod. | +| supervisor.recoveryTimeoutSeconds | int | `60` | Seconds a sandbox workload stays frozen after losing its pool member's supervision before the sandbox runtime terminates it, rendered as [openshell.gateway.supervisor_pool] recovery_timeout_secs. Must be an integer from 30 to 300. Null leaves the gateway's pool default of 60 seconds. | +| supervisor.replicas | int | `1` | Shared supervisor pool members, used only when gatewayConfig selects supervisor_topology = "gateway". The gateway always coordinates the shared supervisor as a high-availability pool: when a member fails or leaves, another member takes over its sandboxes without restarting their workloads. 1 has no failover; use 2 or more to survive losing a member. Only the supervisor Deployment's replica count depends on this value, so changing it neither restarts the gateway nor replaces running members. Ignored when supervisor.autoscaling.enabled is true. | | supervisor.sandboxRuntime.boundaryPort | int | `5500` | Workload boundary TLS listener port. | | tolerations | list | `[]` | Tolerations for the gateway pod. | | upstreamProxy | object | `{"authAllowInsecure":false,"authSecret":{"key":"","name":""},"caBundle":{"configMapName":"","key":"ca.crt"},"connectByHostname":false,"noProxy":"","url":""}` | Operator-owned corporate forward proxy for policy-approved TLS egress from Kubernetes sandboxes. The workload cannot select or override it. | diff --git a/charts/openshell/README.md.gotmpl b/charts/openshell/README.md.gotmpl index 46b3aa59b..479ead340 100644 --- a/charts/openshell/README.md.gotmpl +++ b/charts/openshell/README.md.gotmpl @@ -208,9 +208,81 @@ See [`values.yaml`](values.yaml) for source defaults. Selected overlays: - [`ci/values-keycloak.yaml`](ci/values-keycloak.yaml) - Keycloak OIDC integration - [`ci/values-high-availability.yaml`](ci/values-high-availability.yaml) - CI overlay for multi-replica external PostgreSQL testing - [`ci/values-autoscaling.yaml`](ci/values-autoscaling.yaml) - CI overlay for rendering the optional gateway HorizontalPodAutoscaler +- [`ci/values-supervisor-autoscaling.yaml`](ci/values-supervisor-autoscaling.yaml) - CI overlay for rendering the optional shared supervisor memory HorizontalPodAutoscaler - [`ci/values-spire.yaml`](ci/values-spire.yaml) - SPIFFE/SPIRE provider token grants - [`ci/values-spire-stack.yaml`](ci/values-spire-stack.yaml) - SPIRE hardened chart values for local development +### Additional destination CAs + +Create an operator-owned ConfigMap with a `ca.crt` key when sandboxes must reach +TLS destinations signed by a private CA: + +```shell +kubectl create configmap private-destination-ca -n openshell \ + --from-file=ca.crt=/path/to/ca.crt +``` + +Then set the source ConfigMap name: + +```yaml +supervisor: + network: + additionalCaConfigMapName: private-destination-ca +``` + +The chart mounts the source into the gateway and renders +`[openshell.supervisor.network].additional_ca_bundle`. The gateway validates +and normalizes the one certificate-only PEM bundle at startup. The Kubernetes +driver creates a separate managed ConfigMap in each target sandbox namespace +and mounts it only in the dedicated supervisor Pod at +`/etc/openshell/tls/additional/ca-bundle.pem`. That supervisor receives the +path through `--additional-ca-bundle` and folds the bundle into system roots. +The configured roots augment default destination trust. They do not configure +the corporate proxy, OIDC, or gateway callback mTLS trust. + +Changing only the source ConfigMap does not roll or reload the gateway. Restart +the gateway, then recreate or restart affected sandboxes to use the new roots. +Removing `additionalCaConfigMapName` removes the global TOML setting; newly +created or recreated sandbox pods then have no additional-CA volume or mount. +Existing running sandbox supervisors retain their startup trust until they are +restarted or recreated. + +The driver creates one immutable, content-addressed ConfigMap generation in +each target namespace. Names begin with +`openshell-additional-ca-` and include DNS-safe hashes of the effective +gateway ID and normalized bundle generation. Existing objects are accepted only +when their ownership labels, generation annotation, immutability, and exact +`ca.crt` data match; the driver never patches or updates them. + +Kubernetes cannot restrict `create` by `resourceNames`, and `resourceNames` +cannot express the digest-bearing name prefix needed for `get`. Enabling this +feature therefore gives the gateway/driver service account namespace-wide +ConfigMap `get` and `create` permissions in shared mode. In managed/operator +modes, the chart's ClusterRoleBinding makes those permissions cluster-wide for +the bound service account. The chart does not grant ConfigMap `list`, `watch`, +`patch`, `update`, or `delete`. Use a dedicated gateway service account and +namespace-scoped bindings when this boundary matters. If you render the TOML +setting outside this Helm value, grant `get` and `create` in every target +sandbox namespace. + +Disabling the setting intentionally does not delete driver-managed generations: +the gateway has no ConfigMap list/delete permissions, and an older object may +still be referenced by a running or stopped sandbox. An operator with separate +list/delete authority can inventory the gateway's generations by label: + +```shell +kubectl -n get configmap \ + -l 'openshell.ai/managed-by=openshell,openshell.ai/gateway-id=' +``` + +Before deleting a named generation, verify that no Sandbox resource or pod +volume in that namespace references it. Delete only the confirmed unreferenced +name; removing a referenced generation can make a later pod start or reschedule +fail closed. + +A configured CA can authenticate every policy-permitted TLS endpoint with a +matching certificate chain and hostname. Scope private CAs accordingly. + ### Database backend By default, OpenShell uses SQLite and runs the gateway as a StatefulSet so the @@ -378,6 +450,19 @@ Disabling autoscaling renders `spec.replicas` from `replicaCount` again, which defaults to 1, so set `replicaCount` to the replica count you want in that upgrade. +Set `supervisor.autoscaling.enabled=true` with the shared supervisor topology +to autoscale its Deployment independently of the gateway. It uses average +memory per Pod (`supervisor.autoscaling.targetMemoryAverageValue`, default +`512Mi`) through the resource metrics API, usually provided by Metrics Server. +Defaults keep two to four members and remove at most one member every two +minutes after a five-minute stabilization window. The HPA owns replicas, so +`supervisor.replicas` is ignored. Enabling it on an existing Deployment has +the same temporary reset-to-one risk described above; pick up new defaults +with `--reset-then-reuse-values`. When disabling it, set `supervisor.replicas` +explicitly. Existing sandboxes stay on their owners when scaling up, and +memory-based scaling does not guarantee spare capacity for handoff. See the +[shared supervisor autoscaling guide](https://docs.nvidia.com/openshell/latest/kubernetes/high-availability#autoscale-shared-supervisors-by-memory). + ## Secret bootstrap By default, a pre-install/pre-upgrade hook Job runs `openshell-gateway generate-certs` diff --git a/charts/openshell/ci/values-supervisor-autoscaling.yaml b/charts/openshell/ci/values-supervisor-autoscaling.yaml new file mode 100644 index 000000000..84e2fd0f0 --- /dev/null +++ b/charts/openshell/ci/values-supervisor-autoscaling.yaml @@ -0,0 +1,8 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway +supervisor: + autoscaling: + enabled: true diff --git a/charts/openshell/ci/values-supervisor-ha-scaling.yaml b/charts/openshell/ci/values-supervisor-ha-scaling.yaml new file mode 100644 index 000000000..7a68522f5 --- /dev/null +++ b/charts/openshell/ci/values-supervisor-ha-scaling.yaml @@ -0,0 +1,14 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# E2E overlay: supervise new sandbox generations with a shared supervisor pool +# that starts with one member, so the test can scale it to two members and +# back with `helm upgrade --set supervisor.replicas=N` without changing the +# gateway configuration or the supervisor Pod template. +gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + +supervisor: + replicas: 1 + recoveryTimeoutSeconds: 60 diff --git a/charts/openshell/ci/values-supervisor-ha.yaml b/charts/openshell/ci/values-supervisor-ha.yaml new file mode 100644 index 000000000..4a1dda47a --- /dev/null +++ b/charts/openshell/ci/values-supervisor-ha.yaml @@ -0,0 +1,14 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# E2E overlay: supervise new sandbox generations with a high-availability pool +# of two shared supervisors, so a failed or leaving member's sandboxes move to +# the other member. Layer it on a workspace-mode overlay to cover managed or +# operator namespaces. +gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + +supervisor: + replicas: 2 + recoveryTimeoutSeconds: 60 diff --git a/charts/openshell/ci/values-supervisor.yaml b/charts/openshell/ci/values-supervisor.yaml index 0361016d1..3e530a941 100644 --- a/charts/openshell/ci/values-supervisor.yaml +++ b/charts/openshell/ci/values-supervisor.yaml @@ -1,8 +1,8 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 # -# E2E overlay: supervise new sandbox generations with one supervisor process -# shared by the gateway. Layer it on a workspace-mode overlay +# E2E overlay: supervise new sandbox generations with a one-member shared +# supervisor pool. Layer it on a workspace-mode overlay # to cover managed or operator namespaces. gatewayConfig: openshell.drivers.kubernetes: diff --git a/charts/openshell/ci/values-tls-port-forward.yaml b/charts/openshell/ci/values-tls-port-forward.yaml new file mode 100644 index 000000000..a1f27d578 --- /dev/null +++ b/charts/openshell/ci/values-tls-port-forward.yaml @@ -0,0 +1,10 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# E2E overlay: keep gateway TLS on over the Skaffold values, for topologies +# that require it, such as the shared supervisor. The harness reaches the +# gateway through its port-forward with the chart-generated client +# certificate (OPENSHELL_E2E_KUBE_TLS=1); the server certificate names +# 127.0.0.1. +server: + disableTls: false diff --git a/charts/openshell/skaffold.yaml b/charts/openshell/skaffold.yaml index 4f30cb549..67ec8d51e 100644 --- a/charts/openshell/skaffold.yaml +++ b/charts/openshell/skaffold.yaml @@ -169,6 +169,18 @@ profiles: - op: add path: /deploy/helm/releases/1/valuesFiles/- value: ci/values-gateway.yaml + # Shared supervisor pool: two supervisor replicas that take over each other's + # sandboxes. The topology requires gateway TLS, so this profile turns it back + # on over ci/values-skaffold.yaml. + - name: shared-supervisor-high-availability + patches: + - op: add + path: /deploy/helm/releases/0/valuesFiles/- + value: ci/values-supervisor-ha.yaml + - op: add + path: /deploy/helm/releases/0/setValues + value: + server.disableTls: "false" - name: credential-driver-kubernetes-secrets patches: - op: add diff --git a/charts/openshell/templates/_gateway-workload.tpl b/charts/openshell/templates/_gateway-workload.tpl index 860be9421..a727b0af8 100644 --- a/charts/openshell/templates/_gateway-workload.tpl +++ b/charts/openshell/templates/_gateway-workload.tpl @@ -17,12 +17,14 @@ Gateway pod template shared by the StatefulSet and Deployment workload shapes. {{- $legacyCredentialDrivers := .Values.server.credentialDrivers | default dict -}} {{- $legacyVaultResources := get $legacyCredentialDrivers "vault" | default dict -}} {{- $vaultCaConfigMapName := get $vaultResources "caConfigMapName" | default (get $legacyVaultResources "caConfigMapName") -}} +{{- $supervisorNetwork := .Values.supervisor.network | default dict -}} metadata: annotations: # Roll the gateway workload when the rendered gateway TOML changes - the # gateway only reads /etc/openshell/gateway.toml at startup, so without # this annotation a `helm upgrade` that only mutates the ConfigMap would - # leave pods running with stale config. + # leave pods running with stale config. The operator-owned additional-CA + # ConfigMap is not checksummed; restart the gateway after changing its data. checksum/gateway-config: {{ include (print $.Template.BasePath "/gateway-config.yaml") . | sha256sum }} {{- with .Values.podAnnotations }} {{- toYaml . | nindent 4 }} @@ -134,6 +136,11 @@ spec: - name: SSL_CERT_FILE value: /etc/openshell-tls/oidc-ca/ca.crt {{- end }} + # The gateway stages normalized, content-addressed supervisor trust + # artifacts under its XDG state directory. Keep that state on a + # dedicated writable volume for both StatefulSet and Deployment modes. + - name: XDG_STATE_HOME + value: /var/lib/openshell-state - name: OPENSHELL_TELEMETRY_ENABLED value: {{ .Values.server.telemetryEnabled | quote }} {{- if $spiffeSocketPath }} @@ -161,6 +168,12 @@ spec: - name: gateway-peer-token mountPath: /var/run/secrets/openshell-peer readOnly: true + {{- if get $supervisorNetwork "additionalCaConfigMapName" }} + - name: additional-ca-source + mountPath: /etc/openshell-tls/additional-ca-source/ca.crt + subPath: ca.crt + readOnly: true + {{- end }} {{- if not .Values.server.disableTls }} - name: tls-cert mountPath: /etc/openshell-tls/server @@ -202,6 +215,8 @@ spec: {{- with .Values.server.extraVolumeMounts }} {{- toYaml . | nindent 8 }} {{- end }} + - name: gateway-state + mountPath: /var/lib/openshell-state ports: - name: grpc containerPort: {{ .Values.service.port }} @@ -255,6 +270,14 @@ spec: path: token audience: openshell-gateway-peer expirationSeconds: 3600 + {{- if get $supervisorNetwork "additionalCaConfigMapName" }} + - name: additional-ca-source + configMap: + name: {{ get $supervisorNetwork "additionalCaConfigMapName" | quote }} + items: + - key: ca.crt + path: ca.crt + {{- end }} {{- if not .Values.server.disableTls }} - name: tls-cert secret: @@ -310,6 +333,8 @@ spec: {{- with .Values.server.extraVolumes }} {{- toYaml . | nindent 4 }} {{- end }} + - name: gateway-state + emptyDir: {} {{- with .Values.nodeSelector }} nodeSelector: {{- toYaml . | nindent 4 }} diff --git a/charts/openshell/templates/_helpers.tpl b/charts/openshell/templates/_helpers.tpl index e42750801..5ee4e3310 100644 --- a/charts/openshell/templates/_helpers.tpl +++ b/charts/openshell/templates/_helpers.tpl @@ -362,6 +362,16 @@ raw gateway configuration so openshell.effectiveKubernetesConfig can use it. {{- if eq (get $kubernetes "supervisor_topology" | default "") "gateway" -}}true{{- end -}} {{- end }} +{{/* +Whether the shared supervisor PodDisruptionBudget renders. A missing +podDisruptionBudget map or enabled value means enabled on older releases. +*/}} +{{- define "openshell.supervisorPodDisruptionBudgetEnabled" -}} +{{- $pdb := get (.Values.supervisor | default dict) "podDisruptionBudget" | default dict -}} +{{- $enabled := ternary (get $pdb "enabled") true (kindIs "map" $pdb) -}} +{{- if not (and (kindIs "bool" $enabled) (not $enabled)) -}}true{{- end -}} +{{- end }} + {{/* Name of the shared supervisor Deployment and Service (and, with `-token`, its token Secret): the fullname with a fixed suffix, within the 63-character Service name limit. @@ -387,6 +397,79 @@ app.kubernetes.io/instance: {{ .Release.Name }} app.kubernetes.io/component: supervisor {{- end }} +{{/* +A chart value as a whole number, or nothing when it is not one. Helm parses +YAML numbers as float64, so whole floats count; booleans, strings, and +fractions do not. +*/}} +{{- define "openshell.wholeNumber" -}} +{{- if has (kindOf .) (list "int" "int8" "int16" "int32" "int64" "uint" "uint8" "uint16" "uint32" "uint64" "float32" "float64") -}} +{{- $whole := int64 . -}} +{{- if eq (float64 .) (float64 $whole) -}}{{- $whole -}}{{- end -}} +{{- end -}} +{{- end }} + +{{/* +Shared supervisor replica count. A missing value (for example after `helm +upgrade --reuse-values` from a release that predates it) means 1. +*/}} +{{- define "openshell.supervisorReplicas" -}} +{{- $supervisor := .Values.supervisor | default dict -}} +{{- $value := get $supervisor "replicas" -}} +{{- if or (not (hasKey $supervisor "replicas")) (kindIs "invalid" $value) -}} +1 +{{- else -}} +{{- $replicas := include "openshell.wholeNumber" $value -}} +{{- if or (eq $replicas "") (lt (int64 $replicas) 1) -}} +{{- fail "supervisor.replicas must be an integer of at least 1" -}} +{{- end -}} +{{- $replicas -}} +{{- end -}} +{{- end }} + +{{/* +Seconds a sandbox runtime keeps its workload frozen after losing a pool +member's supervision, or nothing when unset so the gateway applies its pool +default. +*/}} +{{- define "openshell.supervisorRecoveryTimeoutSeconds" -}} +{{- $supervisor := .Values.supervisor | default dict -}} +{{- $value := get $supervisor "recoveryTimeoutSeconds" -}} +{{- if and (hasKey $supervisor "recoveryTimeoutSeconds") (not (kindIs "invalid" $value)) (ne (toString $value) "") -}} +{{- $timeout := include "openshell.wholeNumber" $value -}} +{{- if or (eq $timeout "") (lt (int64 $timeout) 30) (gt (int64 $timeout) 300) -}} +{{- fail "supervisor.recoveryTimeoutSeconds must be an integer between 30 and 300, or null for the default" -}} +{{- end -}} +{{- $timeout -}} +{{- end -}} +{{- end }} + +{{/* +The gateway's [openshell.gateway.supervisor_pool] tuning table as YAML, or +nothing. The shared supervisor always runs as a high-availability pool that +the gateway coordinates; the chart owns the table only to set the recovery +deadline (supervisor.recoveryTimeoutSeconds; none leaves the gateway default). +It never depends on supervisor.replicas, so scaling never changes the gateway +configuration. Renders nothing without the gateway topology. Validates the +supervisor values whenever the topology is selected, and refuses a +hand-written table in either TOML form. +*/}} +{{- define "openshell.supervisorPoolConfig" -}} +{{- $gatewayConfig := .Values.gatewayConfig | default dict -}} +{{- if or (hasKey $gatewayConfig "openshell.gateway.supervisor_pool") (hasKey (get $gatewayConfig "openshell.gateway" | default dict) "supervisor_pool") -}} +{{- fail "supervisor_pool is rendered by the chart; set supervisor.replicas and supervisor.recoveryTimeoutSeconds" -}} +{{- end -}} +{{- if include "openshell.supervisorEnabled" . -}} +{{- if ne (include "openshell.supervisorAutoscalingEnabled" .) "true" -}} +{{- $_ := include "openshell.supervisorReplicas" . -}} +{{- end -}} +{{- $recoveryTimeout := include "openshell.supervisorRecoveryTimeoutSeconds" . -}} +{{- if $recoveryTimeout -}} +{{- toYaml (dict "recovery_timeout_secs" (int64 $recoveryTimeout)) -}} +{{- end -}} +{{- end -}} +{{- end }} + {{/* Namespace selector for the sandbox namespaces of this gateway's workspace mode: the shared namespace, the managed namespaces it labels, or the @@ -654,6 +737,15 @@ Validate chart values that Helm would otherwise accept silently. {{- if hasKey (get (.Values.gatewayConfig | default dict) "openshell.drivers.kubernetes" | default dict) "shared_supervisor" -}} {{- fail "shared_supervisor is rendered by the chart; set only supervisor_topology" -}} {{- end -}} +{{- include "openshell.validateSupervisorAutoscaling" . -}} +{{- $pdb := get (.Values.supervisor | default dict) "podDisruptionBudget" | default dict -}} +{{- if not (kindIs "map" $pdb) -}} +{{- fail "supervisor.podDisruptionBudget must be a map" -}} +{{- end -}} +{{- if and (hasKey $pdb "enabled") (not (kindIs "invalid" (get $pdb "enabled"))) (not (kindIs "bool" (get $pdb "enabled"))) -}} +{{- fail "supervisor.podDisruptionBudget.enabled must be a boolean" -}} +{{- end -}} +{{- $_ := include "openshell.supervisorPoolConfig" . -}} {{- $workloadKind := include "openshell.workloadKind" . -}} {{- $workload := .Values.workload | default dict -}} {{- $maxReplicas := int (include "openshell.maxReplicas" .) -}} diff --git a/charts/openshell/templates/_supervisor-autoscaling.tpl b/charts/openshell/templates/_supervisor-autoscaling.tpl new file mode 100644 index 000000000..3d25ebcf0 --- /dev/null +++ b/charts/openshell/templates/_supervisor-autoscaling.tpl @@ -0,0 +1,35 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{/* A missing supervisor autoscaling map means disabled on older releases. */}} +{{- define "openshell.supervisorAutoscalingEnabled" -}} +{{- $a := get (.Values.supervisor | default dict) "autoscaling" | default dict -}} +{{- if and (kindIs "map" $a) (get $a "enabled") -}}true{{- end -}} +{{- end }} + +{{- define "openshell.validateSupervisorAutoscaling" -}} +{{- $a := get (.Values.supervisor | default dict) "autoscaling" | default dict -}} +{{- if not (kindIs "map" $a) -}} +{{- fail "supervisor.autoscaling must be a map" -}} +{{- end -}} +{{- if and (hasKey $a "enabled") (not (kindIs "invalid" (get $a "enabled"))) (not (kindIs "bool" (get $a "enabled"))) -}} +{{- fail "supervisor.autoscaling.enabled must be a boolean" -}} +{{- end -}} +{{- if eq (include "openshell.supervisorAutoscalingEnabled" .) "true" -}} +{{- if not (include "openshell.supervisorEnabled" .) -}} +{{- fail "supervisor.autoscaling.enabled requires supervisor_topology = gateway" -}} +{{- end -}} +{{- $min := include "openshell.wholeNumber" (get $a "minReplicas") -}} +{{- $max := include "openshell.wholeNumber" (get $a "maxReplicas") -}} +{{- if or (eq $min "") (lt (int64 $min) 1) -}} +{{- fail "supervisor.autoscaling.minReplicas must be an integer of at least 1; when upgrading older values, use --reset-then-reuse-values or set all supervisor.autoscaling values explicitly" -}} +{{- end -}} +{{- if or (eq $max "") (lt (int64 $max) (int64 $min)) -}} +{{- fail "supervisor.autoscaling.maxReplicas must be an integer greater than or equal to minReplicas" -}} +{{- end -}} +{{- $memory := toString (get $a "targetMemoryAverageValue") -}} +{{- if or (not (regexMatch "^[+]?([0-9]+([.][0-9]*)?|[.][0-9]+)([KMGTPE]i|[numkMGTPE]|[eE][+-]?[0-9]+)?$" $memory)) (not (regexMatch "[1-9]" (regexFind "^[+]?[0-9.]*" $memory))) -}} +{{- fail "supervisor.autoscaling.targetMemoryAverageValue must be a positive Kubernetes memory quantity, such as 512Mi" -}} +{{- end -}} +{{- end -}} +{{- end }} diff --git a/charts/openshell/templates/_toml.tpl b/charts/openshell/templates/_toml.tpl index 121c93159..cf8bf6120 100644 --- a/charts/openshell/templates/_toml.tpl +++ b/charts/openshell/templates/_toml.tpl @@ -110,6 +110,11 @@ Top-level lists represent TOML arrays of tables and preserve their YAML order. * {{- range $legacyKey, $runtimeKey := dict "logLevel" "log_level" "enableLoopbackServiceHttp" "enable_loopback_service_http" "enableWebsocketTunnel" "enable_websocket_tunnel" "policyValidationFailureMode" "policy_validation_failure_mode" -}} {{- if not (hasKey $gateway $runtimeKey) -}}{{- $_ := set $gateway $runtimeKey (get $legacyServer $legacyKey) -}}{{- end -}} {{- end -}} +{{/* An empty hostname format means "unset"; only translate it when set. The + discriminator is the gateway installation name, rendered above. */}} +{{- if and (get $legacyServer "serviceHostnameFormat") (not (hasKey $gateway "service_hostname_format")) -}} +{{- $_ := set $gateway "service_hostname_format" (get $legacyServer "serviceHostnameFormat") -}} +{{- end -}} {{- if not (hasKey $gateway "compute_driver") -}}{{- $_ := set $gateway "compute_driver" "kubernetes" -}}{{- end -}} {{- $serverDnsNames := .Values.pkiInitJob.serverDnsNames | default list -}} {{- if .Values.certManager.enabled -}}{{- $serverDnsNames = .Values.certManager.serverDnsNames | default list -}}{{- end -}} @@ -304,6 +309,22 @@ owner: server.*. Override any gatewayConfig copies before serializing TOML. */}} {{- end -}} {{- $_ := set $config "openshell.gateway.tls" $gatewayTls -}} {{- end -}} +{{/* The chart owns the shared supervisor pool; see +openshell.supervisorPoolConfig. */}} +{{- $supervisorPool := include "openshell.supervisorPoolConfig" . | fromYaml -}} +{{- if $supervisorPool -}} +{{- $_ := set $config "openshell.gateway.supervisor_pool" $supervisorPool -}} +{{- end -}} +{{/* Additional destination CA trust is chart plumbing: the runtime table is +owned by the chart, not gatewayConfig. Derive it only from the chart-owned +operator ConfigMap reference. */}} +{{- $supervisorNetwork := .Values.supervisor.network | default dict -}} +{{- if get $supervisorNetwork "additionalCaConfigMapName" -}} +{{- if hasKey $config "openshell.supervisor.network" -}} +{{- fail "gatewayConfig must not define openshell.supervisor.network; the chart derives it from supervisor.network.additionalCaConfigMapName" -}} +{{- end -}} +{{- $_ := set $config "openshell.supervisor.network" (dict "additional_ca_bundle" "/etc/openshell-tls/additional-ca-source/ca.crt") -}} +{{- end -}} {{/* A nested YAML map and its dotted top-level table would serialize to conflicting TOML definitions. Fail before rendering an invalid ConfigMap. */}} {{- range $tableName := keys $config | sortAlpha -}} diff --git a/charts/openshell/templates/clusterrole.yaml b/charts/openshell/templates/clusterrole.yaml index a8f2e6f57..8af91e190 100644 --- a/charts/openshell/templates/clusterrole.yaml +++ b/charts/openshell/templates/clusterrole.yaml @@ -4,6 +4,7 @@ {{ $kubernetesConfig := include "openshell.effectiveKubernetesConfig" . | fromYaml -}} {{- $workspaceMode := get $kubernetesConfig "workspace_mode" | default "shared" }} +{{- $supervisorNetwork := .Values.supervisor.network | default dict }} {{- $allowDriverConfig := get $kubernetesConfig "allow_driver_config" | default false -}} {{- $managedSshIngressEnabled := .Values.networkPolicy.enabled -}} {{- $gatewayConfig := .Values.gatewayConfig | default dict -}} @@ -98,6 +99,23 @@ rules: - patch - watch {{- end }} + {{- if and (ne $workspaceMode "shared") (get $supervisorNetwork "additionalCaConfigMapName") }} + # The Kubernetes driver creates and verifies immutable, content-addressed + # destination-CA ConfigMaps. Kubernetes RBAC supports exact resourceNames + # only, not the digest-bearing name prefix, and CREATE cannot be restricted + # by resourceNames. This ClusterRole is bound cluster-wide in managed and + # operator modes, so keep the required capability to get/create only: the + # driver never lists, watches, patches, updates, or deletes these objects. + # Use a dedicated gateway service account and namespace-scoped bindings when + # this read boundary matters. + - apiGroups: + - "" + resources: + - configmaps + verbs: + - get + - create + {{- end }} {{- if ne $workspaceMode "shared" }} - apiGroups: [""] resources: ["services"] diff --git a/charts/openshell/templates/role.yaml b/charts/openshell/templates/role.yaml index 4097976ce..85d2e954d 100644 --- a/charts/openshell/templates/role.yaml +++ b/charts/openshell/templates/role.yaml @@ -1,7 +1,9 @@ {{- $kubernetesConfig := include "openshell.effectiveKubernetesConfig" . | fromYaml -}} {{- $workspaceMode := get $kubernetesConfig "workspace_mode" | default "shared" -}} {{- $allowDriverConfig := get $kubernetesConfig "allow_driver_config" | default false -}} -{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) (include "openshell.rbacCreate" .) }} +{{- $supervisorNetwork := .Values.supervisor.network | default dict -}} +{{- $additionalCaEnabled := ne (get $supervisorNetwork "additionalCaConfigMapName" | default "") "" -}} +{{- if and (eq $workspaceMode "shared") (or (include "openshell.workspaceResourcesEnabled" .) $additionalCaEnabled) (include "openshell.rbacCreate" .) }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 apiVersion: rbac.authorization.k8s.io/v1 @@ -12,6 +14,7 @@ metadata: labels: {{- include "openshell.labels" . | nindent 4 }} rules: + {{- if include "openshell.workspaceResourcesEnabled" . }} {{- if $allowDriverConfig }} # Metadata-only admission of caller-selected PVCs requires get permission. - apiGroups: [""] @@ -67,4 +70,13 @@ rules: - apiGroups: ["networking.k8s.io"] resources: ["networkpolicies"] verbs: ["create", "get"] + {{- end }} + {{- if $additionalCaEnabled }} + # Immutable, content-addressed destination-CA generations are verified by + # exact name. The driver intentionally has no list/watch/patch/update/delete + # capability for ConfigMaps, so retained generations cannot be mutated. + - apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "create"] + {{- end }} {{- end }} diff --git a/charts/openshell/templates/rolebinding.yaml b/charts/openshell/templates/rolebinding.yaml index a81d8c56c..ab487477c 100644 --- a/charts/openshell/templates/rolebinding.yaml +++ b/charts/openshell/templates/rolebinding.yaml @@ -1,6 +1,8 @@ {{- $kubernetesConfig := include "openshell.effectiveKubernetesConfig" . | fromYaml -}} {{- $workspaceMode := get $kubernetesConfig "workspace_mode" | default "shared" -}} -{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) (include "openshell.rbacCreate" .) }} +{{- $supervisorNetwork := .Values.supervisor.network | default dict -}} +{{- $additionalCaEnabled := ne (get $supervisorNetwork "additionalCaConfigMapName" | default "") "" -}} +{{- if and (eq $workspaceMode "shared") (include "openshell.rbacCreate" .) (or (include "openshell.workspaceResourcesEnabled" .) $additionalCaEnabled) }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 apiVersion: rbac.authorization.k8s.io/v1 diff --git a/charts/openshell/templates/supervisor-hpa.yaml b/charts/openshell/templates/supervisor-hpa.yaml new file mode 100644 index 000000000..fb6b48757 --- /dev/null +++ b/charts/openshell/templates/supervisor-hpa.yaml @@ -0,0 +1,31 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +{{- include "openshell.validateValues" . }} +{{- if eq (include "openshell.supervisorAutoscalingEnabled" .) "true" }} +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: {{ include "openshell.supervisorName" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: supervisor +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: {{ include "openshell.supervisorName" . }} + minReplicas: {{ .Values.supervisor.autoscaling.minReplicas }} + maxReplicas: {{ .Values.supervisor.autoscaling.maxReplicas }} + metrics: + - type: Resource + resource: + name: memory + target: + type: AverageValue + averageValue: {{ .Values.supervisor.autoscaling.targetMemoryAverageValue | quote }} + {{- with .Values.supervisor.autoscaling.behavior }} + behavior: + {{- toYaml . | nindent 4 }} + {{- end }} +{{- end }} diff --git a/charts/openshell/templates/supervisor-networkpolicy.yaml b/charts/openshell/templates/supervisor-networkpolicy.yaml new file mode 100644 index 000000000..02c2a6132 --- /dev/null +++ b/charts/openshell/templates/supervisor-networkpolicy.yaml @@ -0,0 +1,41 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +{{- include "openshell.validateValues" . }} +{{- if and (include "openshell.supervisorEnabled" .) .Values.networkPolicy.enabled }} +{{- $kubernetes := include "openshell.effectiveKubernetesConfig" . | fromYaml }} +{{- $shared := get $kubernetes "shared_supervisor" }} +# Only sandbox workloads in this gateway's sandbox namespaces reach the routed +# runtime listener. Each sandbox's own egress allowance limits its workload to +# that port, and the routing preface plus the generation's pinned TLS identity +# decide which instance it reaches. The supervisor dials the gateway itself, +# so no ingress from the gateway is needed. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ include "openshell.supervisorName" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: supervisor +spec: + podSelector: + matchLabels: + {{- include "openshell.supervisorSelectorLabels" . | nindent 6 }} + policyTypes: + - Ingress + - Egress + ingress: + - from: + - namespaceSelector: + {{- include "openshell.supervisorSandboxNamespaces" (dict "kubernetes" $kubernetes) | nindent 12 }} + podSelector: + matchLabels: + openshell.ai/boundary-role: workload + ports: + - protocol: TCP + port: {{ get $shared "runtime_port" }} + # Instances carry policy-approved sandbox egress, as dedicated supervisors + # do, and the host reaches the gateway. + egress: + - {} +{{- end }} diff --git a/charts/openshell/templates/supervisor-pdb.yaml b/charts/openshell/templates/supervisor-pdb.yaml new file mode 100644 index 000000000..ab1f8bc37 --- /dev/null +++ b/charts/openshell/templates/supervisor-pdb.yaml @@ -0,0 +1,20 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +{{- include "openshell.validateValues" . }} +{{- if and (include "openshell.supervisorEnabled" .) (eq (include "openshell.supervisorPodDisruptionBudgetEnabled" .) "true") }} +# Voluntary disruptions, such as node drains, evict one member at a time, so +# the remaining members can take over its instances. +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: {{ include "openshell.supervisorName" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: supervisor +spec: + maxUnavailable: 1 + selector: + matchLabels: + {{- include "openshell.supervisorSelectorLabels" . | nindent 6 }} +{{- end }} diff --git a/charts/openshell/templates/supervisor-service.yaml b/charts/openshell/templates/supervisor-service.yaml new file mode 100644 index 000000000..e91832afd --- /dev/null +++ b/charts/openshell/templates/supervisor-service.yaml @@ -0,0 +1,23 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +{{- include "openshell.validateValues" . }} +{{- if include "openshell.supervisorEnabled" . }} +{{- $kubernetes := include "openshell.effectiveKubernetesConfig" . | fromYaml }} +{{- $shared := get $kubernetes "shared_supervisor" }} +apiVersion: v1 +kind: Service +metadata: + name: {{ include "openshell.supervisorName" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: supervisor +spec: + selector: + {{- include "openshell.supervisorSelectorLabels" . | nindent 4 }} + ports: + - name: runtime + port: {{ get $shared "runtime_port" }} + targetPort: runtime + protocol: TCP +{{- end }} diff --git a/charts/openshell/templates/supervisor-token-secret.yaml b/charts/openshell/templates/supervisor-token-secret.yaml new file mode 100644 index 000000000..d49d1897d --- /dev/null +++ b/charts/openshell/templates/supervisor-token-secret.yaml @@ -0,0 +1,18 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +{{- include "openshell.validateValues" . }} +{{- if include "openshell.supervisorEnabled" . }} +# Delivery of the gateway-minted supervisor token. The chart creates the +# Secret empty; the gateway replaces its `token` on every renewal, and the +# supervisor re-reads the mounted file on each connection attempt. Helm +# upgrades leave the live token in place. +apiVersion: v1 +kind: Secret +metadata: + name: {{ include "openshell.supervisorName" . }}-token + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + app.kubernetes.io/component: supervisor +type: Opaque +{{- end }} diff --git a/charts/openshell/templates/supervisor.yaml b/charts/openshell/templates/supervisor.yaml index d6027d244..205803481 100644 --- a/charts/openshell/templates/supervisor.yaml +++ b/charts/openshell/templates/supervisor.yaml @@ -12,12 +12,12 @@ {{- $endpoint := get $kubernetes "grpc_endpoint" }} {{- $endpointUrl := urlParse $endpoint }} {{- $gatewayTls := and (not .Values.server.disableTls) .Values.server.tls.clientTlsSecretName }} -# One supervisor process hosting a sandbox instance per sandbox generation. -# The process dials the gateway over TLS with the supervisor token the gateway -# mints for it and receives its assignments from it; nothing but sandbox -# runtimes and kubelet probes connect to it. The Pod holds no Kubernetes -# identity of its own: each instance acts only with the credentials the -# gateway issued for its generation. +# One supervisor process per replica hosts a sandbox instance for each sandbox +# generation placed on it. The process dials the gateway over TLS with the +# supervisor token the gateway mints for it and receives its assignments from +# it; nothing but sandbox runtimes and kubelet probes connect to it. The Pod +# holds no Kubernetes identity of its own: each instance acts only with the +# credentials the gateway issued for its generation. apiVersion: apps/v1 kind: Deployment metadata: @@ -27,10 +27,21 @@ metadata: {{- include "openshell.labels" . | nindent 4 }} app.kubernetes.io/component: supervisor spec: - # Instances live in this process; a second replica could not serve them. - replicas: 1 + # The replicas form a high-availability pool. Each instance lives in the + # process hosting it; the pool rolls one member at a time, starting a new + # member before an old one leaves, and the gateway hands a failed or leaving + # member's instances to the remaining members. One replica has no failover + # but scales like any other count: only the replica count depends on the + # pool's size, so scaling changes nothing else here or in the gateway + # configuration. + {{- if ne (include "openshell.supervisorAutoscalingEnabled" .) "true" }} + replicas: {{ include "openshell.supervisorReplicas" . }} + {{- end }} strategy: - type: Recreate + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 0 selector: matchLabels: {{- include "openshell.supervisorSelectorLabels" . | nindent 6 }} @@ -41,7 +52,9 @@ spec: spec: automountServiceAccountToken: false enableServiceLinks: false - terminationGracePeriodSeconds: 30 + # A stopping member announces that it is leaving, and the gateway moves + # its instances to the remaining members before it exits. + terminationGracePeriodSeconds: 60 {{- with ($imagePullSecrets | uniq) }} imagePullSecrets: {{- range . }} @@ -52,6 +65,10 @@ spec: securityContext: {{- toYaml . | nindent 8 }} {{- end }} + {{- with .Values.supervisor.affinity }} + affinity: + {{- toYaml . | nindent 8 }} + {{- end }} containers: - name: supervisor image: {{ get $kubernetes "supervisor_image" | quote }} @@ -81,6 +98,17 @@ spec: - name: OPENSHELL_GATEWAY_TLS_SERVER_NAME value: {{ get $endpointUrl "hostname" | quote }} {{- end }} + # Pool membership: the Pod name is this member's stable ID, and + # other members redirect the runtimes of its instances to its Pod + # IP. + - name: OPENSHELL_SUPERVISOR_POOL_MEMBER_ID + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: OPENSHELL_SUPERVISOR_POOL_ADVERTISE_IP + valueFrom: + fieldRef: + fieldPath: status.podIP ports: - name: runtime containerPort: {{ get $shared "runtime_port" }} @@ -137,72 +165,4 @@ spec: - key: ca.crt path: ca.crt {{- end }} ---- -apiVersion: v1 -kind: Service -metadata: - name: {{ $name }} - namespace: {{ .Release.Namespace }} - labels: - {{- include "openshell.labels" . | nindent 4 }} - app.kubernetes.io/component: supervisor -spec: - selector: - {{- include "openshell.supervisorSelectorLabels" . | nindent 4 }} - ports: - - name: runtime - port: {{ get $shared "runtime_port" }} - targetPort: runtime - protocol: TCP -{{- if .Values.networkPolicy.enabled }} ---- -# Only sandbox workloads in this gateway's sandbox namespaces reach the routed -# runtime listener. Each sandbox's own egress allowance limits its workload to -# that port, and the routing preface plus the generation's pinned TLS identity -# decide which instance it reaches. The supervisor dials the gateway itself, -# so no ingress from the gateway is needed. -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: - name: {{ $name }} - namespace: {{ .Release.Namespace }} - labels: - {{- include "openshell.labels" . | nindent 4 }} - app.kubernetes.io/component: supervisor -spec: - podSelector: - matchLabels: - {{- include "openshell.supervisorSelectorLabels" . | nindent 6 }} - policyTypes: - - Ingress - - Egress - ingress: - - from: - - namespaceSelector: - {{- include "openshell.supervisorSandboxNamespaces" (dict "kubernetes" $kubernetes) | nindent 12 }} - podSelector: - matchLabels: - openshell.ai/boundary-role: workload - ports: - - protocol: TCP - port: {{ get $shared "runtime_port" }} - # Instances carry policy-approved sandbox egress, as dedicated supervisors - # do, and the host reaches the gateway. - egress: - - {} -{{- end }} ---- -# Delivery of the gateway-minted supervisor token. The chart creates the -# Secret empty; the gateway replaces its `token` on every renewal, and the -# supervisor re-reads the mounted file on each connection attempt. Helm -# upgrades leave the live token in place. -apiVersion: v1 -kind: Secret -metadata: - name: {{ $name }}-token - namespace: {{ .Release.Namespace }} - labels: - {{- include "openshell.labels" . | nindent 4 }} - app.kubernetes.io/component: supervisor -type: Opaque {{- end }} diff --git a/charts/openshell/tests/gateway_config_test.yaml b/charts/openshell/tests/gateway_config_test.yaml index a28e2db37..16639d892 100644 --- a/charts/openshell/tests/gateway_config_test.yaml +++ b/charts/openshell/tests/gateway_config_test.yaml @@ -29,6 +29,50 @@ tests: path: data["gateway.toml"] pattern: '(?m)^enable_websocket_tunnel\s*=\s*true$' + - it: omits service_hostname_format unless a format is configured + template: templates/gateway-config.yaml + asserts: + - notMatchRegex: + path: data["gateway.toml"] + pattern: 'service_hostname_format\s*=' + + - it: renders service_hostname_format from server.serviceHostnameFormat + template: templates/gateway-config.yaml + set: + server.serviceHostnameFormat: hashed + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: '(?m)^service_hostname_format\s*=\s*"hashed"$' + - matchRegex: + path: data["gateway.toml"] + pattern: '(?m)^name\s*=\s*"openshell"$' + + - it: renders service_hostname_format with the configured installation name + template: templates/gateway-config.yaml + set: + server.serviceHostnameFormat: hashed + server.name: gateway-west + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: '(?m)^service_hostname_format\s*=\s*"hashed"$' + - matchRegex: + path: data["gateway.toml"] + pattern: '(?m)^name\s*=\s*"gateway-west"$' + + - it: preserves an explicit gatewayConfig service_hostname_format over the value + template: templates/gateway-config.yaml + set: + server.serviceHostnameFormat: hashed + gatewayConfig: + openshell.gateway: + service_hostname_format: readable + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: '(?m)^service_hostname_format\s*=\s*"readable"$' + - it: defaults to label admission with caller driver config disabled template: templates/gateway-config.yaml asserts: @@ -1250,6 +1294,25 @@ tests: - failedTemplate: errorPattern: "replicaCount > 1 requires server.externalDbSecret" + - it: renders writable ephemeral XDG state for gateway trust artifacts + template: templates/statefulset.yaml + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: XDG_STATE_HOME + value: /var/lib/openshell-state + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: gateway-state + mountPath: /var/lib/openshell-state + - contains: + path: spec.template.spec.volumes + content: + name: gateway-state + emptyDir: {} + - it: renders a Deployment for external database-backed gateway workloads template: templates/deployment.yaml set: @@ -1287,6 +1350,16 @@ tests: secretKeyRef: name: my-pg-secret key: uri + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: gateway-state + mountPath: /var/lib/openshell-state + - contains: + path: spec.template.spec.volumes + content: + name: gateway-state + emptyDir: {} - it: fails when a Deployment uses the default SQLite database template: templates/statefulset.yaml diff --git a/charts/openshell/tests/network_additional_ca_test.yaml b/charts/openshell/tests/network_additional_ca_test.yaml new file mode 100644 index 000000000..3b3208483 --- /dev/null +++ b/charts/openshell/tests/network_additional_ca_test.yaml @@ -0,0 +1,223 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: network additional CA delivery +release: + name: openshell + namespace: my-namespace +templates: + - templates/gateway-config.yaml + - templates/statefulset.yaml + - templates/role.yaml + - templates/rolebinding.yaml + - templates/clusterrole.yaml + +tests: + - it: omits the global additional CA setting by default + template: templates/gateway-config.yaml + asserts: + - notMatchRegex: + path: data["gateway.toml"] + pattern: '\[openshell\.supervisor\.network\]' + + - it: treats a missing supervisor network map as disabled + template: templates/gateway-config.yaml + set: + supervisor.network: null + asserts: + - notMatchRegex: + path: data["gateway.toml"] + pattern: '\[openshell\.supervisor\.network\]' + + - it: omits the gateway source mount by default + template: templates/statefulset.yaml + asserts: + - notContains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: additional-ca-source + any: true + - notContains: + path: spec.template.spec.volumes + content: + name: additional-ca-source + any: true + + - it: omits shared namespace ConfigMap permissions by default + template: templates/role.yaml + asserts: + - notContains: + path: rules + content: + apiGroups: [""] + resources: ["configmaps"] + any: true + + - it: omits cluster-wide ConfigMap permissions by default + template: templates/clusterrole.yaml + asserts: + - notContains: + path: rules + content: + apiGroups: [""] + resources: ["configmaps"] + any: true + + - it: renders the global setting outside the Kubernetes driver table + template: templates/gateway-config.yaml + set: + supervisor.network.additionalCaConfigMapName: private-destination-ca + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: '(?ms)\[openshell\.supervisor\.network\]\s*additional_ca_bundle\s*=\s*"/etc/openshell-tls/additional-ca-source/ca\.crt"' + - notMatchRegex: + path: data["gateway.toml"] + pattern: '(?ms)\[openshell\.drivers\.kubernetes\][^\[]*additional_ca_bundle' + + - it: mounts the operator source ConfigMap into the gateway + template: templates/statefulset.yaml + set: + supervisor.network.additionalCaConfigMapName: private-destination-ca + asserts: + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: additional-ca-source + mountPath: /etc/openshell-tls/additional-ca-source/ca.crt + subPath: ca.crt + readOnly: true + - contains: + path: spec.template.spec.volumes + content: + name: additional-ca-source + configMap: + name: private-destination-ca + items: + - key: ca.crt + path: ca.crt + - exists: + path: spec.template.metadata.annotations["checksum/gateway-config"] + + - it: grants only namespace-wide ConfigMap get and create in shared mode + template: templates/role.yaml + set: + supervisor.network.additionalCaConfigMapName: private-destination-ca + workspaceResources.enabled: false + asserts: + - contains: + path: rules + content: + apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "create"] + - notContains: + path: rules + content: + apiGroups: [""] + resources: ["configmaps"] + resourceNames: ["openshell-additional-ca-openshell"] + any: true + - notContains: + path: rules + content: + apiGroups: ["agents.x-k8s.io"] + any: true + + - it: binds shared namespace ConfigMap permissions when workspace resources are external + template: templates/rolebinding.yaml + set: + supervisor.network.additionalCaConfigMapName: private-destination-ca + workspaceResources.enabled: false + asserts: + - hasDocuments: + count: 1 + + - it: grants only ConfigMap get/create when shared workspace resources are external + template: templates/role.yaml + set: + supervisor.network.additionalCaConfigMapName: private-destination-ca + workspaceResources.enabled: false + asserts: + - contains: + path: rules + content: + apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "create"] + - notContains: + path: rules + content: + apiGroups: [""] + resources: ["pods"] + any: true + - notContains: + path: rules + content: + apiGroups: [""] + resources: ["secrets"] + any: true + - notContains: + path: rules + content: + apiGroups: ["networking.k8s.io"] + resources: ["networkpolicies"] + any: true + + - it: grants only cluster-wide ConfigMap get and create in managed mode + template: templates/clusterrole.yaml + set: + supervisor.network.additionalCaConfigMapName: private-destination-ca + server.drivers.kubernetes.workspaceMode: managed + asserts: + - contains: + path: rules + content: + apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "create"] + - notContains: + path: rules + content: + apiGroups: [""] + resources: ["configmaps"] + resourceNames: ["openshell-additional-ca-openshell"] + any: true + + - it: omits a shared Role in managed workspace mode + template: templates/role.yaml + set: + supervisor.network.additionalCaConfigMapName: private-destination-ca + server.drivers.kubernetes.workspaceMode: managed + asserts: + - hasDocuments: + count: 0 + + - it: grants only cluster-wide ConfigMap get and create in operator mode + template: templates/clusterrole.yaml + set: + supervisor.network.additionalCaConfigMapName: private-destination-ca + server.drivers.kubernetes.workspaceMode: operator + asserts: + - contains: + path: rules + content: + apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "create"] + - notContains: + path: rules + content: + apiGroups: [""] + resources: ["configmaps"] + resourceNames: ["openshell-additional-ca-openshell"] + any: true + + - it: omits a shared Role in operator workspace mode + template: templates/role.yaml + set: + supervisor.network.additionalCaConfigMapName: private-destination-ca + server.drivers.kubernetes.workspaceMode: operator + asserts: + - hasDocuments: + count: 0 diff --git a/charts/openshell/tests/supervisor_affinity_test.yaml b/charts/openshell/tests/supervisor_affinity_test.yaml new file mode 100644 index 000000000..6b426a2dc --- /dev/null +++ b/charts/openshell/tests/supervisor_affinity_test.yaml @@ -0,0 +1,86 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +suite: supervisor affinity +templates: + - templates/supervisor.yaml + - templates/statefulset.yaml + - templates/gateway-config.yaml +release: + name: openshell + namespace: gateway +set: + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway +tests: + - it: renders supervisor.affinity unchanged on the supervisor Pod + template: templates/supervisor.yaml + documentIndex: 0 + set: + supervisor: + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchLabels: + app.kubernetes.io/name: openshell-supervisor + app.kubernetes.io/instance: openshell + app.kubernetes.io/component: supervisor + topologyKey: kubernetes.io/hostname + asserts: + - equal: + path: spec.template.spec.affinity + value: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchLabels: + app.kubernetes.io/name: openshell-supervisor + app.kubernetes.io/instance: openshell + app.kubernetes.io/component: supervisor + topologyKey: kubernetes.io/hostname + - it: keeps gateway and supervisor affinity separate + set: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: example.com/pool + operator: In + values: [gateways] + supervisor: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: example.com/pool + operator: In + values: [supervisors] + asserts: + - equal: + path: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].values + value: [gateways] + template: templates/statefulset.yaml + - equal: + path: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].values + value: [supervisors] + template: templates/supervisor.yaml + documentIndex: 0 + - it: renders no gateway affinity when only supervisor.affinity is set + template: templates/statefulset.yaml + set: + supervisor: + affinity: + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchLabels: + app.kubernetes.io/component: supervisor + topologyKey: kubernetes.io/hostname + asserts: + - notExists: + path: spec.template.spec.affinity diff --git a/charts/openshell/tests/supervisor_autoscaling_test.yaml b/charts/openshell/tests/supervisor_autoscaling_test.yaml new file mode 100644 index 000000000..36523206b --- /dev/null +++ b/charts/openshell/tests/supervisor_autoscaling_test.yaml @@ -0,0 +1,214 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +suite: shared supervisor memory autoscaling +templates: + - templates/supervisor-hpa.yaml + - templates/supervisor.yaml + - templates/supervisor-pdb.yaml + - templates/hpa.yaml +release: + name: openshell + namespace: gateway +set: + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + supervisor.autoscaling.enabled: true +tests: + - it: renders the memory HPA with gradual downscaling + template: templates/supervisor-hpa.yaml + asserts: + - isKind: + of: HorizontalPodAutoscaler + - isAPIVersion: + of: autoscaling/v2 + - equal: + path: metadata.namespace + value: gateway + - equal: + path: metadata.name + value: openshell-supervisor + - equal: + path: spec.scaleTargetRef + value: + apiVersion: apps/v1 + kind: Deployment + name: openshell-supervisor + - equal: + path: spec.minReplicas + value: 2 + - equal: + path: spec.maxReplicas + value: 4 + - equal: + path: spec.metrics + value: + - type: Resource + resource: + name: memory + target: + type: AverageValue + averageValue: 512Mi + - equal: + path: spec.behavior.scaleDown + value: + stabilizationWindowSeconds: 300 + policies: + - type: Pods + value: 1 + periodSeconds: 120 + - it: omits fixed replicas when the HPA owns them + template: templates/supervisor.yaml + set: + supervisor.replicas: 0 + documentIndex: 0 + asserts: + - notExists: + path: spec.replicas + - it: preserves fixed replicas when autoscaling is disabled + template: templates/supervisor.yaml + set: + supervisor.autoscaling.enabled: false + supervisor.replicas: 3 + documentIndex: 0 + asserts: + - equal: + path: spec.replicas + value: 3 + - it: keeps the PodDisruptionBudget alongside the HPA by default + template: templates/supervisor-pdb.yaml + asserts: + - hasDocuments: + count: 1 + - equal: + path: spec.maxUnavailable + value: 1 + - it: drops only the PodDisruptionBudget when disabled under the HPA + set: + supervisor.podDisruptionBudget.enabled: false + asserts: + - hasDocuments: + count: 0 + template: templates/supervisor-pdb.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-hpa.yaml + - hasDocuments: + count: 1 + template: templates/supervisor.yaml + - notExists: + path: spec.replicas + template: templates/supervisor.yaml + - it: adds no supervisor HPA when disabled + template: templates/supervisor-hpa.yaml + set: + supervisor.autoscaling.enabled: false + asserts: + - hasDocuments: + count: 0 + - it: treats missing autoscaling values as disabled + template: templates/supervisor-hpa.yaml + set: + supervisor.autoscaling: null + asserts: + - hasDocuments: + count: 0 + - it: leaves gateway autoscaling disabled + template: templates/hpa.yaml + asserts: + - hasDocuments: + count: 0 + - it: lets gateway and supervisor HPAs coexist + template: templates/hpa.yaml + values: + - ../ci/values-autoscaling.yaml + asserts: + - equal: + path: spec.scaleTargetRef.name + value: openshell + - it: uses the same overridden name for the HPA and its target + template: templates/supervisor-hpa.yaml + set: + fullnameOverride: my-gateway + supervisor.autoscaling.minReplicas: 3 + supervisor.autoscaling.maxReplicas: 6 + supervisor.autoscaling.targetMemoryAverageValue: 768Mi + supervisor.autoscaling.behavior.scaleUp.stabilizationWindowSeconds: 30 + asserts: + - equal: + path: metadata.name + value: my-gateway-supervisor + - equal: + path: spec.scaleTargetRef.name + value: my-gateway-supervisor + - equal: + path: spec.minReplicas + value: 3 + - equal: + path: spec.maxReplicas + value: 6 + - equal: + path: spec.metrics[0].resource.target.averageValue + value: 768Mi + - equal: + path: spec.behavior.scaleUp.stabilizationWindowSeconds + value: 30 + - equal: + path: spec.behavior.scaleDown.stabilizationWindowSeconds + value: 300 + - it: requires the shared supervisor topology + template: templates/supervisor.yaml + set: + gatewayConfig: null + asserts: + - failedTemplate: + errorMessage: supervisor.autoscaling.enabled requires supervisor_topology = gateway + - it: rejects non-boolean enabling values + template: templates/supervisor.yaml + set: + supervisor.autoscaling.enabled: "false" + asserts: + - failedTemplate: + errorMessage: supervisor.autoscaling.enabled must be a boolean + - it: rejects zero minimum replicas + template: templates/supervisor.yaml + set: + supervisor.autoscaling.minReplicas: 0 + asserts: + - failedTemplate: + errorMessage: supervisor.autoscaling.minReplicas must be an integer of at least 1; when upgrading older values, use --reset-then-reuse-values or set all supervisor.autoscaling values explicitly + - it: rejects a fractional maximum + template: templates/supervisor.yaml + set: + supervisor.autoscaling.maxReplicas: 4.5 + asserts: + - failedTemplate: + errorMessage: supervisor.autoscaling.maxReplicas must be an integer greater than or equal to minReplicas + - it: rejects a maximum below the minimum + template: templates/supervisor.yaml + set: + supervisor.autoscaling.maxReplicas: 1 + asserts: + - failedTemplate: + errorMessage: supervisor.autoscaling.maxReplicas must be an integer greater than or equal to minReplicas + - it: rejects a zero memory target + template: templates/supervisor.yaml + set: + supervisor.autoscaling.targetMemoryAverageValue: 0Mi + asserts: + - failedTemplate: + errorMessage: supervisor.autoscaling.targetMemoryAverageValue must be a positive Kubernetes memory quantity, such as 512Mi + - it: rejects malformed memory quantities + template: templates/supervisor.yaml + set: + supervisor.autoscaling.targetMemoryAverageValue: 512megabytes + asserts: + - failedTemplate: + errorMessage: supervisor.autoscaling.targetMemoryAverageValue must be a positive Kubernetes memory quantity, such as 512Mi + - it: rejects missing memory targets in older reused values + template: templates/supervisor.yaml + set: + supervisor.autoscaling.targetMemoryAverageValue: null + asserts: + - failedTemplate: + errorMessage: supervisor.autoscaling.targetMemoryAverageValue must be a positive Kubernetes memory quantity, such as 512Mi diff --git a/charts/openshell/tests/supervisor_test.yaml b/charts/openshell/tests/supervisor_test.yaml index 0686a800e..49a9916cb 100644 --- a/charts/openshell/tests/supervisor_test.yaml +++ b/charts/openshell/tests/supervisor_test.yaml @@ -3,6 +3,10 @@ suite: supervisor deployment templates: - templates/supervisor.yaml + - templates/supervisor-service.yaml + - templates/supervisor-networkpolicy.yaml + - templates/supervisor-token-secret.yaml + - templates/supervisor-pdb.yaml - templates/gateway-config.yaml release: name: openshell @@ -13,7 +17,20 @@ tests: asserts: - hasDocuments: count: 0 - - it: deploys one instance host without Kubernetes identity in the gateway namespace + template: templates/supervisor.yaml + - hasDocuments: + count: 0 + template: templates/supervisor-service.yaml + - hasDocuments: + count: 0 + template: templates/supervisor-networkpolicy.yaml + - hasDocuments: + count: 0 + template: templates/supervisor-token-secret.yaml + - hasDocuments: + count: 0 + template: templates/supervisor-pdb.yaml + - it: deploys the supervisor pool without Kubernetes identity in the gateway namespace template: templates/supervisor.yaml set: gatewayConfig: @@ -22,7 +39,20 @@ tests: supervisor_image: example.com/supervisor:shared asserts: - hasDocuments: - count: 4 + count: 1 + template: templates/supervisor.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-service.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-networkpolicy.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-token-secret.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-pdb.yaml - equal: path: kind value: Deployment @@ -87,7 +117,8 @@ tests: port: 5500 targetPort: runtime protocol: TCP - documentIndex: 1 + template: templates/supervisor-service.yaml + documentIndex: 0 - equal: path: spec.template.spec.automountServiceAccountToken value: false @@ -121,21 +152,26 @@ tests: - equal: path: kind value: Secret - documentIndex: 3 + template: templates/supervisor-token-secret.yaml + documentIndex: 0 - equal: path: metadata.name value: openshell-supervisor-token - documentIndex: 3 + template: templates/supervisor-token-secret.yaml + documentIndex: 0 - equal: path: type value: Opaque - documentIndex: 3 + template: templates/supervisor-token-secret.yaml + documentIndex: 0 - notExists: path: data - documentIndex: 3 + template: templates/supervisor-token-secret.yaml + documentIndex: 0 - notExists: path: stringData - documentIndex: 3 + template: templates/supervisor-token-secret.yaml + documentIndex: 0 - contains: path: spec.template.spec.volumes content: @@ -149,22 +185,26 @@ tests: - equal: path: kind value: Service - documentIndex: 1 + template: templates/supervisor-service.yaml + documentIndex: 0 - equal: path: spec.selector value: app.kubernetes.io/name: openshell-supervisor app.kubernetes.io/instance: openshell app.kubernetes.io/component: supervisor - documentIndex: 1 + template: templates/supervisor-service.yaml + documentIndex: 0 - equal: path: kind value: NetworkPolicy - documentIndex: 2 + template: templates/supervisor-networkpolicy.yaml + documentIndex: 0 - lengthEqual: path: spec.ingress count: 1 - documentIndex: 2 + template: templates/supervisor-networkpolicy.yaml + documentIndex: 0 - equal: path: spec.ingress[0] value: @@ -178,7 +218,8 @@ tests: ports: - protocol: TCP port: 5500 - documentIndex: 2 + template: templates/supervisor-networkpolicy.yaml + documentIndex: 0 - it: admits runtimes only from this gateway's managed namespaces template: templates/supervisor.yaml set: @@ -194,7 +235,8 @@ tests: matchLabels: openshell.ai/managed-by: openshell openshell.ai/gateway-id: gw-a - documentIndex: 2 + template: templates/supervisor-networkpolicy.yaml + documentIndex: 0 - it: admits runtimes only from operator-labeled namespaces template: templates/supervisor.yaml set: @@ -209,7 +251,8 @@ tests: value: matchLabels: example.com/openshell-workspace: "true" - documentIndex: 2 + template: templates/supervisor-networkpolicy.yaml + documentIndex: 0 - it: rejects operator namespace selectors it cannot scope template: templates/supervisor.yaml set: @@ -232,11 +275,13 @@ tests: - matchRegex: path: metadata.name pattern: ^openshell-gateway-release-with-a-very-very-[0-9a-f]{8}-supervisor$ - documentIndex: 1 + template: templates/supervisor-service.yaml + documentIndex: 0 - notEqual: path: metadata.name value: openshell-gateway-release-with-a-very-very-long-name-0057 - documentIndex: 1 + template: templates/supervisor-service.yaml + documentIndex: 0 - it: requires gateway TLS for the supervisor token template: templates/supervisor.yaml set: @@ -261,6 +306,470 @@ tests: - matchRegex: path: data["gateway.toml"] pattern: 'shared_supervisor = \{ runtime_port = 5500, service_name = "openshell-supervisor", service_namespace = "gateway", token_secret_name = "openshell-supervisor-token" \}' + - it: runs one replica as a high-availability pool by default + template: templates/supervisor.yaml + set: + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + asserts: + - hasDocuments: + count: 1 + template: templates/supervisor.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-service.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-networkpolicy.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-token-secret.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-pdb.yaml + - equal: + path: spec.replicas + value: 1 + documentIndex: 0 + - equal: + path: spec.strategy + value: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 0 + documentIndex: 0 + - equal: + path: spec.template.spec.terminationGracePeriodSeconds + value: 60 + documentIndex: 0 + - notExists: + path: spec.template.spec.affinity + documentIndex: 0 + - contains: + path: spec.template.spec.containers[0].env + content: + name: OPENSHELL_SUPERVISOR_POOL_MEMBER_ID + valueFrom: + fieldRef: + fieldPath: metadata.name + documentIndex: 0 + - contains: + path: spec.template.spec.containers[0].env + content: + name: OPENSHELL_SUPERVISOR_POOL_ADVERTISE_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + documentIndex: 0 + - equal: + path: spec.maxUnavailable + value: 1 + template: templates/supervisor-pdb.yaml + documentSelector: + path: kind + value: PodDisruptionBudget + - it: runs two or more replicas as a high-availability pool + template: templates/supervisor.yaml + set: + supervisor: + replicas: 3 + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + asserts: + - hasDocuments: + count: 1 + template: templates/supervisor.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-service.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-networkpolicy.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-token-secret.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-pdb.yaml + - equal: + path: spec.replicas + value: 3 + documentIndex: 0 + - equal: + path: spec.strategy + value: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 0 + documentIndex: 0 + - equal: + path: spec.template.spec.terminationGracePeriodSeconds + value: 60 + documentIndex: 0 + - contains: + path: spec.template.spec.containers[0].env + content: + name: OPENSHELL_SUPERVISOR_POOL_MEMBER_ID + valueFrom: + fieldRef: + fieldPath: metadata.name + documentIndex: 0 + - contains: + path: spec.template.spec.containers[0].env + content: + name: OPENSHELL_SUPERVISOR_POOL_ADVERTISE_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + documentIndex: 0 + - equal: + path: spec.selector + value: + app.kubernetes.io/name: openshell-supervisor + app.kubernetes.io/instance: openshell + app.kubernetes.io/component: supervisor + template: templates/supervisor-service.yaml + documentIndex: 0 + - equal: + path: spec.ingress[0].ports + value: + - protocol: TCP + port: 5500 + template: templates/supervisor-networkpolicy.yaml + documentIndex: 0 + - equal: + path: apiVersion + value: policy/v1 + template: templates/supervisor-pdb.yaml + documentSelector: + path: kind + value: PodDisruptionBudget + - equal: + path: metadata.name + value: openshell-supervisor + template: templates/supervisor-pdb.yaml + documentSelector: + path: kind + value: PodDisruptionBudget + - equal: + path: spec + value: + maxUnavailable: 1 + selector: + matchLabels: + app.kubernetes.io/name: openshell-supervisor + app.kubernetes.io/instance: openshell + app.kubernetes.io/component: supervisor + template: templates/supervisor-pdb.yaml + documentSelector: + path: kind + value: PodDisruptionBudget + - it: protects the pool from voluntary disruption without a NetworkPolicy + template: templates/supervisor.yaml + set: + networkPolicy: + enabled: false + supervisor: + replicas: 2 + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + asserts: + - hasDocuments: + count: 1 + template: templates/supervisor.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-service.yaml + - hasDocuments: + count: 0 + template: templates/supervisor-networkpolicy.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-token-secret.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-pdb.yaml + - equal: + path: spec.maxUnavailable + value: 1 + template: templates/supervisor-pdb.yaml + documentSelector: + path: kind + value: PodDisruptionBudget + - it: enables the supervisor PodDisruptionBudget by default + template: templates/supervisor-pdb.yaml + set: + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + asserts: + - hasDocuments: + count: 1 + - isKind: + of: PodDisruptionBudget + - equal: + path: spec.maxUnavailable + value: 1 + - it: keeps the PodDisruptionBudget enabled when older reused values lack the setting + template: templates/supervisor-pdb.yaml + set: + supervisor: + podDisruptionBudget: null + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + asserts: + - hasDocuments: + count: 1 + - equal: + path: spec.maxUnavailable + value: 1 + - it: renders only the PodDisruptionBudget away when it is disabled + set: + supervisor: + replicas: 3 + podDisruptionBudget: + enabled: false + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + asserts: + - hasDocuments: + count: 0 + template: templates/supervisor-pdb.yaml + - hasDocuments: + count: 1 + template: templates/supervisor.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-service.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-networkpolicy.yaml + - hasDocuments: + count: 1 + template: templates/supervisor-token-secret.yaml + - equal: + path: spec.replicas + value: 3 + template: templates/supervisor.yaml + - notExists: + path: spec.template.metadata.annotations + template: templates/supervisor.yaml + - matchRegex: + path: data["gateway.toml"] + pattern: 'shared_supervisor = \{ runtime_port = 5500, service_name = "openshell-supervisor", service_namespace = "gateway", token_secret_name = "openshell-supervisor-token" \}' + template: templates/gateway-config.yaml + - notMatchRegex: + path: data["gateway.toml"] + pattern: (?i)disruption + template: templates/gateway-config.yaml + - it: renders no PodDisruptionBudget without the gateway topology + template: templates/supervisor-pdb.yaml + set: + supervisor: + podDisruptionBudget: + enabled: true + asserts: + - hasDocuments: + count: 0 + - it: rejects a non-boolean PodDisruptionBudget toggle + template: templates/supervisor.yaml + set: + supervisor: + podDisruptionBudget: + enabled: "false" + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + asserts: + - failedTemplate: + errorMessage: supervisor.podDisruptionBudget.enabled must be a boolean + - it: renders the pool's recovery deadline in the gateway table by default + template: templates/gateway-config.yaml + set: + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: 'shared_supervisor = \{ runtime_port = 5500, service_name = "openshell-supervisor", service_namespace = "gateway", token_secret_name = "openshell-supervisor-token" \}' + - matchRegex: + path: data["gateway.toml"] + pattern: '\[openshell\.gateway\.supervisor_pool\]\nrecovery_timeout_secs = 60\n' + - notMatchRegex: + path: data["gateway.toml"] + pattern: 'supervisor_pool\]\n[^\[]*(enabled|high_availability)' + - it: renders no pool tuning when the recovery timeout is null, leaving the gateway's defaults + template: templates/gateway-config.yaml + set: + supervisor: + recoveryTimeoutSeconds: null + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: 'supervisor_topology = "gateway"' + - notMatchRegex: + path: data["gateway.toml"] + pattern: supervisor_pool|recovery_timeout_secs + - it: renders a configured recovery timeout into the supervisor pool table + template: templates/gateway-config.yaml + set: + supervisor: + replicas: 2 + recoveryTimeoutSeconds: 45 + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: '\[openshell\.gateway\.supervisor_pool\]\nrecovery_timeout_secs = 45\n' + - matchRegex: + path: data["gateway.toml"] + pattern: 'shared_supervisor = \{ runtime_port = 5500, service_name = "openshell-supervisor", service_namespace = "gateway", token_secret_name = "openshell-supervisor-token" \}' + - it: renders no supervisor pool without the gateway topology + template: templates/gateway-config.yaml + set: + supervisor: + replicas: 2 + recoveryTimeoutSeconds: 120 + asserts: + - notMatchRegex: + path: data["gateway.toml"] + pattern: supervisor_pool|recovery_timeout_secs|shared_supervisor + - it: rejects a hand-written supervisor pool table + template: templates/supervisor.yaml + set: + supervisor: + replicas: 2 + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + openshell.gateway.supervisor_pool: + enabled: true + asserts: + - failedTemplate: + errorMessage: supervisor_pool is rendered by the chart; set supervisor.replicas and supervisor.recoveryTimeoutSeconds + - it: rejects a hand-written supervisor pool inside the gateway table + template: templates/supervisor.yaml + set: + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + openshell.gateway: + supervisor_pool: + enabled: true + asserts: + - failedTemplate: + errorMessage: supervisor_pool is rendered by the chart; set supervisor.replicas and supervisor.recoveryTimeoutSeconds + - it: rejects a hand-written supervisor pool without the gateway topology + template: templates/supervisor.yaml + set: + gatewayConfig: + openshell.gateway.supervisor_pool: + enabled: true + asserts: + - failedTemplate: + errorMessage: supervisor_pool is rendered by the chart; set supervisor.replicas and supervisor.recoveryTimeoutSeconds + - it: rejects zero supervisor replicas + template: templates/supervisor.yaml + set: + supervisor: + replicas: 0 + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + asserts: + - failedTemplate: + errorMessage: supervisor.replicas must be an integer of at least 1 + - it: rejects fractional supervisor replicas + template: templates/supervisor.yaml + set: + supervisor: + replicas: 1.5 + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + asserts: + - failedTemplate: + errorMessage: supervisor.replicas must be an integer of at least 1 + - it: rejects supervisor replicas given as a string + template: templates/supervisor.yaml + set: + supervisor: + replicas: "2" + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + asserts: + - failedTemplate: + errorMessage: supervisor.replicas must be an integer of at least 1 + - it: rejects a recovery timeout below 30 seconds + template: templates/supervisor.yaml + set: + supervisor: + recoveryTimeoutSeconds: 29 + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + asserts: + - failedTemplate: + errorMessage: supervisor.recoveryTimeoutSeconds must be an integer between 30 and 300, or null for the default + - it: rejects a recovery timeout above 300 seconds + template: templates/supervisor.yaml + set: + supervisor: + replicas: 2 + recoveryTimeoutSeconds: 301 + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + asserts: + - failedTemplate: + errorMessage: supervisor.recoveryTimeoutSeconds must be an integer between 30 and 300, or null for the default + - it: rejects a fractional recovery timeout + template: templates/supervisor.yaml + set: + supervisor: + recoveryTimeoutSeconds: 30.5 + gatewayConfig: + openshell.drivers.kubernetes: + supervisor_topology: gateway + asserts: + - failedTemplate: + errorMessage: supervisor.recoveryTimeoutSeconds must be an integer between 30 and 300, or null for the default + - it: ignores supervisor pool values without the gateway topology + template: templates/supervisor.yaml + set: + supervisor: + replicas: 0 + recoveryTimeoutSeconds: 5 + asserts: + - hasDocuments: + count: 0 + template: templates/supervisor.yaml + - hasDocuments: + count: 0 + template: templates/supervisor-service.yaml + - hasDocuments: + count: 0 + template: templates/supervisor-networkpolicy.yaml + - hasDocuments: + count: 0 + template: templates/supervisor-token-secret.yaml + - hasDocuments: + count: 0 + template: templates/supervisor-pdb.yaml - it: rejects a hand-written shared supervisor connection template: templates/supervisor.yaml set: diff --git a/charts/openshell/values.yaml b/charts/openshell/values.yaml index cffc6a4ce..473e9fb6e 100644 --- a/charts/openshell/values.yaml +++ b/charts/openshell/values.yaml @@ -69,6 +69,51 @@ supervisor: tag: "" # -- Supervisor image digest. When set, this takes precedence over tag. digest: "" + # -- Shared supervisor pool members, used only when gatewayConfig selects + # supervisor_topology = "gateway". The gateway always coordinates the shared + # supervisor as a high-availability pool: when a member fails or leaves, + # another member takes over its sandboxes without restarting their + # workloads. 1 has no failover; use 2 or more to survive losing a member. + # Only the supervisor Deployment's replica count depends on this value, so + # changing it neither restarts the gateway nor replaces running members. + # Ignored when supervisor.autoscaling.enabled is true. + replicas: 1 + autoscaling: + # -- Enable memory-based autoscaling of the shared supervisor Deployment. + # Requires supervisor_topology = "gateway" and the Kubernetes resource + # metrics API (usually Metrics Server). When enabled, the HPA owns the + # replica count and supervisor.replicas is ignored. + enabled: false + # -- Minimum supervisor replicas. Keep at least 2 for failover capacity. + minReplicas: 2 + # -- Maximum supervisor replicas. + maxReplicas: 4 + # -- Target average memory working set per supervisor Pod, as a positive + # Kubernetes quantity. This is an absolute value, not a percentage of + # requests. Scaling up adds capacity for new sandboxes; it does not move + # existing sandboxes away from a busy member. + targetMemoryAverageValue: 512Mi + # -- HPA scaling behavior. Remove at most one member every two minutes, + # after a five-minute stabilization window, to allow ownership handoff. + behavior: + scaleDown: + stabilizationWindowSeconds: 300 + policies: + - type: Pods + value: 1 + periodSeconds: 120 + podDisruptionBudget: + # -- Render a PodDisruptionBudget with maxUnavailable 1 for the shared + # supervisor, used only when gatewayConfig selects supervisor_topology = + # "gateway". Disable it only when maintenance must evict more than one + # supervisor Pod at a time, or evict a member while another is unavailable. + enabled: true + # -- Seconds a sandbox workload stays frozen after losing its pool member's + # supervision before the sandbox runtime terminates it, rendered as + # [openshell.gateway.supervisor_pool] recovery_timeout_secs. Must be an + # integer from 30 to 300. Null leaves the gateway's pool default of 60 + # seconds. + recoveryTimeoutSeconds: 60 # Pod security context for the shared supervisor Deployment, rendered only # when gatewayConfig selects supervisor_topology = "gateway". On OpenShift, # set the user and group IDs to null so the restricted-v2 SCC can assign the @@ -85,6 +130,13 @@ supervisor: seccompProfile: # -- Seccomp profile applied to the shared supervisor pod. type: RuntimeDefault + # -- Affinity rules for the shared supervisor pod, rendered only when + # gatewayConfig selects supervisor_topology = "gateway". Top-level affinity + # applies only to the gateway pod. + affinity: {} + network: + # -- Name of an operator-owned ConfigMap with a `ca.crt` key containing additional destination CA certificates for sandbox supervisors. Empty disables additional destination trust. + additionalCaConfigMapName: "" sandboxRuntime: # -- Workload boundary TLS listener port. boundaryPort: 5500 @@ -470,6 +522,13 @@ server: # -- Enable plaintext HTTP routing for loopback sandbox service URLs on # TLS-enabled gateways. enableLoopbackServiceHttp: true + # -- Hostname format for exposed sandbox services. Empty or "readable" + # keeps readable `--[--]` hostnames; "hashed" + # mints opaque one-label hostnames discriminated by the gateway + # installation name (server.name, defaulting to the release name). The + # name must be unique among all gateways sharing the service base domain, + # and renaming the installation changes every minted hostname. + serviceHostnameFormat: "" # -- Enable the WebSocket tunnel used by CLI/SDK clients behind an # authenticated edge proxy. Leave disabled for direct gateway installs. enableWebsocketTunnel: false diff --git a/components/pr-test/e2e-openshell-roks.sh b/components/pr-test/e2e-openshell-roks.sh index 77764c497..d6e3c22a7 100755 --- a/components/pr-test/e2e-openshell-roks.sh +++ b/components/pr-test/e2e-openshell-roks.sh @@ -64,8 +64,8 @@ GW_OIDC_CLIENT_ID="" # Mirrored gateway images (ROKS nodes can only pull the internal registry). Used # only when this run has to CREATE the gateway; ignored when it already exists. REG_MIRROR="${REG_MIRROR:-image-registry.openshift-image-registry.svc:5000/openshift}" -GW_IMAGE="${GW_IMAGE:-${REG_MIRROR}/openshell-gateway:0.1.5-rhaiv.3}" -GW_SUPERVISOR_IMAGE="${GW_SUPERVISOR_IMAGE:-${REG_MIRROR}/openshell-supervisor:0.1.5-rhaiv.3}" +GW_IMAGE="${GW_IMAGE:-${REG_MIRROR}/openshell-gateway:0.1.6-rhaiv.0}" +GW_SUPERVISOR_IMAGE="${GW_SUPERVISOR_IMAGE:-${REG_MIRROR}/openshell-supervisor:0.1.6-rhaiv.0}" PASS=0 FAIL=0 diff --git a/deploy/base/platform-resources/controller.yaml b/deploy/base/platform-resources/controller.yaml index d4554c6b9..777e49654 100644 --- a/deploy/base/platform-resources/controller.yaml +++ b/deploy/base/platform-resources/controller.yaml @@ -61,7 +61,7 @@ spec: - name: HYPERSHELL_API_SERVER_URL value: "http://hypershell-api-server:8000" - name: HYPERSHELL_MANAGED_CLUSTER_CAPABILITIES - value: "provider/aws,network/public,gateway-version/v0.1.5-rhaiv.3" + value: "provider/aws,network/public,gateway-version/v0.1.6-rhaiv.0" - name: HYPERSHELL_NAMESPACE valueFrom: fieldRef: @@ -75,11 +75,11 @@ spec: value: "0.0.0.0:9443" # Tags must match OPENSHELL_VERSION when bumping the OpenShell release. - name: GATEWAY_IMAGE - value: quay.io/opendatahub/odh-openshell-gateway:v0.1.5-rhaiv.3 + value: quay.io/opendatahub/odh-openshell-gateway:v0.1.6-rhaiv.0 - name: GATEWAY_SUPERVISOR_IMAGE - value: quay.io/opendatahub/odh-openshell-supervisor:v0.1.5-rhaiv.3 + value: quay.io/opendatahub/odh-openshell-supervisor:v0.1.6-rhaiv.0 - name: GATEWAY_SANDBOX_RUNTIME_IMAGE - value: quay.io/opendatahub/odh-openshell-sandbox:v0.1.5-rhaiv.3 + value: quay.io/opendatahub/odh-openshell-sandbox:v0.1.6-rhaiv.0 # Per-gateway console (OpenShell dashboard), pinned by digest. Required: # the controller refuses to start without it. Stamped from # OPENSHELL_CONSOLE_IMAGE and OPENSHELL_CONSOLE_DIGEST by diff --git a/deploy/gitops/platform/kustomization.yaml b/deploy/gitops/platform/kustomization.yaml index 36bea5bce..c764e7c06 100644 --- a/deploy/gitops/platform/kustomization.yaml +++ b/deploy/gitops/platform/kustomization.yaml @@ -30,4 +30,4 @@ patches: - name: HYPERSHELL_MANAGED_CLUSTER_NAME value: "hub" - name: HYPERSHELL_MANAGED_CLUSTER_CAPABILITIES - value: "provider/aws,network/public,gateway-version/v0.1.5-rhaiv.3" + value: "provider/aws,network/public,gateway-version/v0.1.6-rhaiv.0" diff --git a/deploy/ibm/kustomization.yaml b/deploy/ibm/kustomization.yaml index 0ffb10884..243296ff4 100644 --- a/deploy/ibm/kustomization.yaml +++ b/deploy/ibm/kustomization.yaml @@ -65,7 +65,7 @@ patches: - name: controller env: - name: HYPERSHELL_MANAGED_CLUSTER_CAPABILITIES - value: "provider/ibm,network/public,gateway-version/v0.1.5-rhaiv.3" + value: "provider/ibm,network/public,gateway-version/v0.1.6-rhaiv.0" - name: GATEWAY_INGRESS_MODE value: route - name: GATEWAY_API_BASE_DOMAIN @@ -73,8 +73,8 @@ patches: # hysh-ibm-01--0000.us-east.containers.appdomain.cloud value: openshell.ibm.example.com - name: GATEWAY_IMAGE - value: image-registry.openshift-image-registry.svc:5000/openshift/openshell-gateway:v0.1.5-rhaiv.3 + value: image-registry.openshift-image-registry.svc:5000/openshift/openshell-gateway:v0.1.6-rhaiv.0 - name: GATEWAY_SUPERVISOR_IMAGE - value: image-registry.openshift-image-registry.svc:5000/openshift/openshell-supervisor:v0.1.5-rhaiv.3 + value: image-registry.openshift-image-registry.svc:5000/openshift/openshell-supervisor:v0.1.6-rhaiv.0 - name: GATEWAY_SANDBOX_RUNTIME_IMAGE - value: image-registry.openshift-image-registry.svc:5000/openshift/openshell-sandbox-runtime:v0.1.5-rhaiv.3 + value: image-registry.openshift-image-registry.svc:5000/openshift/openshell-sandbox-runtime:v0.1.6-rhaiv.0 diff --git a/deploy/kind/kustomization.yaml b/deploy/kind/kustomization.yaml index 8539bfb80..0ee12ca6b 100644 --- a/deploy/kind/kustomization.yaml +++ b/deploy/kind/kustomization.yaml @@ -69,7 +69,7 @@ patches: - name: HYPERSHELL_MANAGED_CLUSTER_NAME value: "local-kind" - name: HYPERSHELL_MANAGED_CLUSTER_CAPABILITIES - value: "provider/kind,network/public,environment/kind,gateway-version/v0.1.5-rhaiv.3" + value: "provider/kind,network/public,environment/kind,gateway-version/v0.1.6-rhaiv.0" - name: OIDC_ISSUER value: "http://keycloak-service.keycloak.svc.cluster.local:8080/realms/hypershell" - name: OIDC_CLIENT_ID diff --git a/deploy/openshift/kustomization.yaml b/deploy/openshift/kustomization.yaml index d15651539..232130c9a 100644 --- a/deploy/openshift/kustomization.yaml +++ b/deploy/openshift/kustomization.yaml @@ -54,7 +54,7 @@ patches: - name: HYPERSHELL_MANAGED_CLUSTER_NAME value: "local-openshift" - name: HYPERSHELL_MANAGED_CLUSTER_CAPABILITIES - value: "provider/aws,network/public,gateway-version/v0.1.5-rhaiv.3" + value: "provider/aws,network/public,gateway-version/v0.1.6-rhaiv.0" - name: OIDC_ISSUER valueFrom: secretKeyRef: diff --git a/skills/deploy/deploy-cluster/SKILL.md b/skills/deploy/deploy-cluster/SKILL.md index 9cb1f447a..9903bb15b 100644 --- a/skills/deploy/deploy-cluster/SKILL.md +++ b/skills/deploy/deploy-cluster/SKILL.md @@ -229,9 +229,9 @@ reconciliation fails immediately: ```yaml env: - name: GATEWAY_IMAGE - value: quay.io/opendatahub/odh-openshell-gateway:v0.1.5-rhaiv.3@sha256:... + value: quay.io/opendatahub/odh-openshell-gateway:v0.1.6-rhaiv.0@sha256:... - name: GATEWAY_SUPERVISOR_IMAGE - value: quay.io/opendatahub/odh-openshell-supervisor:v0.1.5-rhaiv.3@sha256:... + value: quay.io/opendatahub/odh-openshell-supervisor:v0.1.6-rhaiv.0@sha256:... ``` When a gateway is created, the controller reads these environment variables to determine diff --git a/skills/deploy/gcp-cluster/SKILL.md b/skills/deploy/gcp-cluster/SKILL.md index d3f1d0317..57e6a4cff 100644 --- a/skills/deploy/gcp-cluster/SKILL.md +++ b/skills/deploy/gcp-cluster/SKILL.md @@ -412,7 +412,7 @@ GATEWAY=$(curl -sk -X POST "$API/gateways" -H 'Content-Type: application/json' \ \"name\": \"gcp-test-gw\", \"placement\": {\"selector\": [\"network/public\", \"provider/aws\"]}, \"namespace\": \"openshell-gcptest\", - \"image\": \"quay.io/opendatahub/odh-openshell-gateway:v0.1.5-rhaiv.3\", + \"image\": \"quay.io/opendatahub/odh-openshell-gateway:v0.1.6-rhaiv.0\", \"route\": \"{\\\"enabled\\\": true}\" }") echo "$GATEWAY" | python3 -m json.tool diff --git a/skills/deploy/ibm-cluster/SKILL.md b/skills/deploy/ibm-cluster/SKILL.md index 2a7791687..82a7364ef 100644 --- a/skills/deploy/ibm-cluster/SKILL.md +++ b/skills/deploy/ibm-cluster/SKILL.md @@ -384,17 +384,17 @@ secrets are unnecessary): skopeo copy --remove-signatures --dest-tls-verify=false --dest-creds "pusher:$(oc -n hypershell create token pusher)" \ docker://docker.io/library/postgres:18 docker://$REG/openshift/postgres:18 # Source images are normally pinned by digest (resolved from the Quay tag API). -# The v0.1.5-rhaiv.3 bump could not reach quay.io from the agent runtime that +# The v0.1.6-rhaiv.0 bump could not reach quay.io from the agent runtime that # made this change, so these fall back to a tag-only reference; resolve and pin # the digest before mirroring with # `skopeo inspect --format '{{.Digest}}' docker://`. A bare tag is a # mutable reference and only a last-resort fallback. skopeo copy --dest-tls-verify=false --dest-creds "pusher:$(oc -n hypershell create token pusher)" \ - docker://quay.io/opendatahub/odh-openshell-gateway:v0.1.5-rhaiv.3 docker://$REG/openshift/openshell-gateway:v0.1.5-rhaiv.3 + docker://quay.io/opendatahub/odh-openshell-gateway:v0.1.6-rhaiv.0 docker://$REG/openshift/openshell-gateway:v0.1.6-rhaiv.0 skopeo copy --dest-tls-verify=false --dest-creds "pusher:$(oc -n hypershell create token pusher)" \ - docker://quay.io/opendatahub/odh-openshell-supervisor:v0.1.5-rhaiv.3 docker://$REG/openshift/openshell-supervisor:v0.1.5-rhaiv.3 + docker://quay.io/opendatahub/odh-openshell-supervisor:v0.1.6-rhaiv.0 docker://$REG/openshift/openshell-supervisor:v0.1.6-rhaiv.0 skopeo copy --dest-tls-verify=false --dest-creds "pusher:$(oc -n hypershell create token pusher)" \ - docker://quay.io/opendatahub/odh-openshell-sandbox:v0.1.5-rhaiv.3 docker://$REG/openshift/openshell-sandbox-runtime:v0.1.5-rhaiv.3 + docker://quay.io/opendatahub/odh-openshell-sandbox:v0.1.6-rhaiv.0 docker://$REG/openshift/openshell-sandbox-runtime:v0.1.6-rhaiv.0 oc -n openshift get is # expect openshell-gateway, openshell-supervisor, openshell-sandbox-runtime, postgres ``` diff --git a/skills/tooling/update-openshell/SKILL.md b/skills/tooling/update-openshell/SKILL.md index ee57db3b2..99bce1cd2 100644 --- a/skills/tooling/update-openshell/SKILL.md +++ b/skills/tooling/update-openshell/SKILL.md @@ -181,6 +181,12 @@ facts rather than the active pin: illustrative values, not pins. - the arbitrary tag strings in `scripts/test_sync_openshell_version.py` and the capability fixtures in `components/api-server/plugins/` tests. +- the matched example version in + `components/control-plane/internal/otel/metrics_test.go` (`gateway-version/v0.1.5-rhaiv.2`, + `odh-openshell-gateway:v0.1.5-rhaiv.2`) and the mirrored scenario fixture in + `specs/platform/control-plane-observability.spec.md` (OBS-CPL-09.01) - both + exercise the metrics-export behavior with an arbitrary capability/image string, + not the deployed version. Found during the v0.1.6-rhaiv.0 run. If unsure whether an occurrence is a pin or a fixture, treat it as a pin and note the ambiguity in the [Learnings log](#learnings-log). @@ -516,6 +522,98 @@ If a run produced no new lessons, that is itself worth a one-line log entry Newest first. Each entry: version, date, what happened, what changed in the repo. +- **v0.1.6-rhaiv.0 (2026-10-29, v0.1.5-rhaiv.3 -> v0.1.6-rhaiv.0, no-arg scheduled run):** + Pins and chart bumped. Python policy checks that make up `make check` all pass run + directly (same sandbox limits as the 2026-10-23/27 entries: no Go toolchain, and + `make` itself cannot spawn `/bin/sh` here - every step below used the underlying + `git clone`/`cp`/`perl -pi`/`python3 scripts/*.py`/`python3 -m unittest scripts/*` + commands directly instead of `make vendor-openshell-chart` / `make check`). + `go build/vet/test` and `check-cli-drift` could not run; only one `.go` file + changed (`tests/e2e/cli_test.go`, a string-literal tag bump), same recorded gap + as prior Go-toolchain-less runs. + - **Step 0:** #809/#760/#752 have zero comments (no human reply yet, not + malformed - just unaddressed); skipped per "waiting for human direction." #366's + latest non-bot comment ("Reopening...") still names no version; skipped as + malformed, consistent with the 2026-10-27 disposition. + - **Found an existing unmerged PR for an intermediate version:** #810 + (`chore(deps): bump OpenShell to v0.1.5-rhaiv.4`, branch + `implementer/update-openshell-v0.1.5-rhaiv.4`) was already open from a prior + run and is now `CONFLICTING` with `main`. Since the latest midstream tag + (`v0.1.6-rhaiv.0`) is newer than that PR's target, this run branched fresh from + `main` (still at `v0.1.5-rhaiv.3`) and bumped straight to `v0.1.6-rhaiv.0` + rather than rebasing #810, carrying its triage findings forward instead of + rediscovering them. Per the "never merge, close, or force-push" rule, #810 was + left open for a human to close as superseded; did not duplicate its content, + only linked the new PR from issue #809. + - **Auth-gate finding carried forward, not re-triaged from scratch:** the + `v0.1.5-rhaiv.4` upgrade note (NVIDIA #4821, gateways now refuse to start + without OIDC/mTLS/operator-auth/`gateway_jwt` or an explicit + `allow_unauthenticated_users = true`) is already filed as + openshift-online/hypershell#809 with the same disposition PR #810 used: ship + the mechanical bump, do not change `ValidateRenderedGatewayConfig` without a + maintainer decision. Still open, still unresolved as of this run; this PR does + not re-litigate it, only links to it. + - **New needs-decision/verify-live item: sandbox<->supervisor control-connection + direction reversed for new sandboxes** (NVIDIA #4656, Kubernetes/Docker/Podman/VM + drivers): previously the supervisor dialed the sandbox; now the sandbox dials + out to its supervisor (existing running sandboxes keep the old direction across + the upgrade). HyperShell's own `openshell-gateway-allow-sandbox` NetworkPolicy + (`deploy/base/platform-resources/networkpolicies.yaml`) already permits ingress + from sandbox pods (`agents.x-k8s.io/sandbox-name-hash` selector) to the gateway + pod on port 8080, and the vendored chart's `*-sandbox-ssh` policies only ever + restricted *ingress to* sandbox pods on port 2222 (SSH), never egress - so + nothing in the existing NetworkPolicy set should block a sandbox dialing out. + But the exact port/protocol the reversed control connection itself uses was not + identified from the release notes or the vendored chart, so this is **not + independently verified** - flag for live Kind/ROKS validation on the next + deploy (same disposition as the 0.1.3-rhaiv.3 entry's readiness-probe-port + item). No NetworkPolicy change applied in this PR. + - **Chart vendor diff (27 files, not the usual ~48 - this crosses a minor NVIDIA + base bump 0.1.5->0.1.6):** the new/changed templates + (`supervisor-hpa.yaml`, `supervisor-pdb.yaml`, `supervisor-networkpolicy.yaml`, + `supervisor-service.yaml`, `supervisor-token-secret.yaml`, plus `supervisor.yaml`, + `_helpers.tpl`, `_toml.tpl`, `clusterrole.yaml`, `role.yaml`, `rolebinding.yaml` + diffs) all implement the new experimental shared-supervisor HA pool + (replicas/autoscaling/PDB) and an operator-owned additional-destination-CA + trust bundle for sandbox supervisors. Read every new `{{- if ... }}` gate + directly rather than assuming from the release notes: every one of them is + gated on `gatewayConfig["openshell.drivers.kubernetes"].supervisor_topology == + "gateway"` (`openshell.supervisorEnabled`) or + `supervisor.network.additionalCaConfigMapName` being non-empty, and HyperShell + sets neither - confirmed inert, consistent with the "not adopted" disposition + already recorded for this feature in the 0.1.5-rhaiv.2/rhaiv.3 entries. The new + unconditional piece is a `gateway-state` `emptyDir` volume + `XDG_STATE_HOME` + env var on the gateway pod itself (content-addressed supervisor trust-artifact + cache) - harmless, no RBAC/SecurityContext/NetworkPolicy interaction. + - **RBAC:** the only new grants (`configmaps` get/create, gated on + `additionalCaConfigMapName`) never render for HyperShell's config; no + `controller-rbac.yaml` change needed. + - **`values_test.go`:** the TOML serialization/spacing logic itself is unchanged + this bump (only new, gated keys were added to the schema); no assertion update + needed, unlike the 0.1.3-rhaiv.3 chart-format rewrite. + - **Console:** unchanged, still pinned to dashboard release v0.1.1. Still blocked + on `quay.io` being unreachable from this runtime (same gap as every entry since + 2026-10-22); tracked in #760/#752, not duplicated. + - **`v1beta1` citation bumped, not just reconfirmed:** the + `global-architecture.spec.md` PLAT-ARCH sentence had cited "`v1beta1` for + gateway 0.1.2-rhaiv.0" unchanged across three prior bumps (0.1.3, 0.1.4, 0.1.5) + even though the skill's own footprint table says to re-check this claim on + every run. Confirmed `agents.x-k8s.io/v1beta1` is still the chart's primary + served API version (`v1alpha1` is preflight-fallback only, per + `charts/openshell/templates/agent-sandbox-preflight.yaml`) and, per the + skill's own "treat an ambiguous occurrence as a pin" rule, updated the cited + version number to `0.1.6-rhaiv.0` this time instead of leaving it stale again. + - **Additive, no action:** `ResumeBoundaryRequest` field 3 rename (external + compute-driver protocol only; HyperShell uses the built-in Kubernetes driver), + gateway-authored OCSF audit events for auth/credential events (gateway-internal, + no config HyperShell renders), new opt-in `server.serviceHostnameFormat` + (default `""` keeps today's readable hostnames), new optional Helm + `server.extraEnv` / `pkiInitJob.podLabels`/`podAnnotations` (not adopted). + - **Footprint:** no new pinned file found missing from the table this run; added + one new do-not-change fixture pair (`otel/metrics_test.go` + + `control-plane-observability.spec.md`'s OBS-CPL-09.01 scenario, both sharing an + arbitrary example tag unrelated to the live pin). + - **v0.1.5-rhaiv.3 (2026-10-27, v0.1.5-rhaiv.2 -> v0.1.5-rhaiv.3, no-arg scheduled run):** Clean mechanical bump. Pins and chart updated; the Python policy checks that make up `make check` (`check_forbidden_terms`, `check_dependency_pins`, `check_ci_components`, diff --git a/specs/platform/data-model.spec.md b/specs/platform/data-model.spec.md index a766ce175..1763afac4 100644 --- a/specs/platform/data-model.spec.md +++ b/specs/platform/data-model.spec.md @@ -392,7 +392,7 @@ selected per sandbox through OpenShell; they are not Gateway resource fields. | Field | Type | Description | |---|---|---| -| `image` | string | Gateway container image reference (e.g., `quay.io/opendatahub/odh-openshell-gateway:v0.1.5-rhaiv.3@sha256:...`) | +| `image` | string | Gateway container image reference (e.g., `quay.io/opendatahub/odh-openshell-gateway:v0.1.6-rhaiv.0@sha256:...`) | | `supervisor_image` | string | Supervisor sidecar container image (default supplied by `GATEWAY_SUPERVISOR_IMAGE` env var on the control-plane deployment; see `deploy/base/platform-resources/controller.yaml`) | | `server_dns_names` | string[] | DNS names for TLS certificate SANs | | `oidc` | JSONB | OIDC authentication config: `{issuer, audience, jwks_ttl, roles_claim, admin_role, user_role, scopes_claim}` | diff --git a/specs/platform/global-architecture.spec.md b/specs/platform/global-architecture.spec.md index d01d3a860..f48444e9c 100644 --- a/specs/platform/global-architecture.spec.md +++ b/specs/platform/global-architecture.spec.md @@ -891,7 +891,7 @@ NOT install the CRD/controller, which is a cluster prerequisite on par with cert-manager. The gateway's Kubernetes compute driver watches this CRD; when it is absent the sandbox RPCs surface as gRPC `Unimplemented` and the driver logs `no supported Agent Sandbox API version is available`. The version installed SHALL -serve the API version the gateway requires (`v1beta1` for gateway 0.1.2-rhaiv.0; upstream +serve the API version the gateway requires (`v1beta1` for gateway 0.1.6-rhaiv.0; upstream `v0.5.x`). ### Requirement: PLAT-ARCH-17 -- Sandbox Base Image Supports an In-Cluster Registry diff --git a/specs/platform/openshell-gateway.spec.md b/specs/platform/openshell-gateway.spec.md index c26b026bb..029b34fde 100644 --- a/specs/platform/openshell-gateway.spec.md +++ b/specs/platform/openshell-gateway.spec.md @@ -386,7 +386,7 @@ The GatewayReconciler SHALL validate Gateway resource fields before applying K8s - THEN validation SHALL fail with a descriptive error - AND the Gateway SHALL not be reconciled until the configuration is corrected -> **Image tag convention:** OpenShell gateway and supervisor images are published on `quay.io/opendatahub/` with semver tags (e.g., `v0.1.5-rhaiv.3`) and pinned by digest for reproducibility. The GatewayReconciler continuously reconciles the image field, so the gitops overlay must be the source of truth for the image tag - manual image changes on the Deployment will be reverted. +> **Image tag convention:** OpenShell gateway and supervisor images are published on `quay.io/opendatahub/` with semver tags (e.g., `v0.1.6-rhaiv.0`) and pinned by digest for reproducibility. The GatewayReconciler continuously reconciles the image field, so the gitops overlay must be the source of truth for the image tag - manual image changes on the Deployment will be reverted. #### Scenario: OSG-CORE-07.03 -- Invalid DNS name @@ -959,8 +959,8 @@ Control Plane | Variable | Default | Description | |---|---|---| -| `GATEWAY_IMAGE` | *(required)* | Gateway container image reference with digest (e.g., `quay.io/opendatahub/odh-openshell-gateway:v0.1.5-rhaiv.3@sha256:...`). Sets the default when a Gateway resource does not specify `image`. | -| `GATEWAY_SUPERVISOR_IMAGE` | *(required)* | Supervisor sidecar container image reference with digest (e.g., `quay.io/opendatahub/odh-openshell-supervisor:v0.1.5-rhaiv.3@sha256:...`). Sets the default when a Gateway resource does not specify `supervisor_image`. | +| `GATEWAY_IMAGE` | *(required)* | Gateway container image reference with digest (e.g., `quay.io/opendatahub/odh-openshell-gateway:v0.1.6-rhaiv.0@sha256:...`). Sets the default when a Gateway resource does not specify `image`. | +| `GATEWAY_SUPERVISOR_IMAGE` | *(required)* | Supervisor sidecar container image reference with digest (e.g., `quay.io/opendatahub/odh-openshell-supervisor:v0.1.6-rhaiv.0@sha256:...`). Sets the default when a Gateway resource does not specify `supervisor_image`. | | `GATEWAY_SANDBOX_RUNTIME_CLASS` | *(unset - cluster default RuntimeClass)* | Kubernetes `RuntimeClass` name applied to every sandbox pod on the cluster (e.g. `kata`, `kata-remote`). Passed to the chart as `server.defaultRuntimeClassName`. Omitted from chart values when unset, so sandbox pods carry no `runtimeClassName` field and the cluster default (runc) is used. Requires the named `RuntimeClass` to exist on the cluster. | | `GATEWAY_RESOURCES` | *(unset - requests `cpu: 100m`, `memory: 512Mi`; limits `cpu: 500m`, `memory: 1Gi`)* | Gateway container requests and limits as a JSON Kubernetes `ResourceRequirements` object, e.g. `{"requests":{"cpu":"100m","memory":"512Mi"},"limits":{"cpu":"500m","memory":"1Gi"}}`. Replaces the defaults entirely (not merged). MUST set `limits.memory`; no request may exceed its limit; `claims` is not supported. An invalid value fails controller startup. Applied to every gateway on the cluster on its next reconcile (Helm upgrade, which restarts the gateway pod). The Kind overlay (`deploy/kind`) sets lower requests (`cpu: 50m`, `memory: 128Mi`) with the default limits, so more gateways fit on the single Kind node. | | `GATEWAY_API_GATEWAY_NAME` | *(required)* | Name of the pre-existing Gateway resource that tenant GRPCRoutes attach to | diff --git a/specs/platform/openshell-image-auto-update.spec.md b/specs/platform/openshell-image-auto-update.spec.md index 7c222732c..55e9b2df8 100644 --- a/specs/platform/openshell-image-auto-update.spec.md +++ b/specs/platform/openshell-image-auto-update.spec.md @@ -38,7 +38,7 @@ merge only on green - preventing both silent drift and unvalidated upgrades. All OpenShell image references are pinned using the `tag@digest` format: ``` -quay.io/opendatahub/odh-openshell-gateway:v0.1.5-rhaiv.3@sha256: +quay.io/opendatahub/odh-openshell-gateway:v0.1.6-rhaiv.0@sha256: ``` - **Tag** provides human readability and version ordering. diff --git a/tests/e2e/cli_test.go b/tests/e2e/cli_test.go index 122bf4370..fb37ef6b1 100644 --- a/tests/e2e/cli_test.go +++ b/tests/e2e/cli_test.go @@ -40,7 +40,7 @@ func TestOpenShiftCLICommandSharesCallerConfig(t *testing.T) { "/home/runner", 1001, 1001, - "quay.io/opendatahub/odh-openshell-cli:v0.1.5-rhaiv.3", + "quay.io/opendatahub/odh-openshell-cli:v0.1.6-rhaiv.0", "openshell-example-openshell", false, "status", @@ -51,7 +51,7 @@ func TestOpenShiftCLICommandSharesCallerConfig(t *testing.T) { "--user", "1001:1001", "-e", "HOME=/home/cli", "-v", "/home/runner/.config/openshell:/home/cli/.config/openshell:z", - "quay.io/opendatahub/odh-openshell-cli:v0.1.5-rhaiv.3", + "quay.io/opendatahub/odh-openshell-cli:v0.1.6-rhaiv.0", "-g", "openshell-example-openshell", "status", } if !reflect.DeepEqual(got, want) {