Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion OPENSHELL_VERSION
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# OpenShell upstream coordinates. Gateway, supervisor, sandbox runtime, and CLI image tags
# and the chart ref use OPENSHELL_TAG. The console image is pinned by digest
# (it uses its own tagging scheme). Update this file when bumping versions.
OPENSHELL_TAG=v0.1.5-rhaiv.3
OPENSHELL_TAG=v0.1.6-rhaiv.0
OPENSHELL_CHART_REPO=https://github.com/opendatahub-io/openshell.git
OPENSHELL_GATEWAY_IMAGE=quay.io/opendatahub/odh-openshell-gateway
OPENSHELL_SUPERVISOR_IMAGE=quay.io/opendatahub/odh-openshell-supervisor
Expand Down
96 changes: 96 additions & 0 deletions charts/openshell/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,9 +207,81 @@ See [`values.yaml`](values.yaml) for source defaults. Selected overlays:
- [`ci/values-keycloak.yaml`](ci/values-keycloak.yaml) - Keycloak OIDC integration
- [`ci/values-high-availability.yaml`](ci/values-high-availability.yaml) - CI overlay for multi-replica external PostgreSQL testing
- [`ci/values-autoscaling.yaml`](ci/values-autoscaling.yaml) - CI overlay for rendering the optional gateway HorizontalPodAutoscaler
- [`ci/values-supervisor-autoscaling.yaml`](ci/values-supervisor-autoscaling.yaml) - CI overlay for rendering the optional shared supervisor memory HorizontalPodAutoscaler
- [`ci/values-spire.yaml`](ci/values-spire.yaml) - SPIFFE/SPIRE provider token grants
- [`ci/values-spire-stack.yaml`](ci/values-spire-stack.yaml) - SPIRE hardened chart values for local development

### Additional destination CAs

Create an operator-owned ConfigMap with a `ca.crt` key when sandboxes must reach
TLS destinations signed by a private CA:

```shell
kubectl create configmap private-destination-ca -n openshell \
--from-file=ca.crt=/path/to/ca.crt
```

Then set the source ConfigMap name:

```yaml
supervisor:
network:
additionalCaConfigMapName: private-destination-ca
```

The chart mounts the source into the gateway and renders
`[openshell.supervisor.network].additional_ca_bundle`. The gateway validates
and normalizes the one certificate-only PEM bundle at startup. The Kubernetes
driver creates a separate managed ConfigMap in each target sandbox namespace
and mounts it only in the dedicated supervisor Pod at
`/etc/openshell/tls/additional/ca-bundle.pem`. That supervisor receives the
path through `--additional-ca-bundle` and folds the bundle into system roots.
The configured roots augment default destination trust. They do not configure
the corporate proxy, OIDC, or gateway callback mTLS trust.

Changing only the source ConfigMap does not roll or reload the gateway. Restart
the gateway, then recreate or restart affected sandboxes to use the new roots.
Removing `additionalCaConfigMapName` removes the global TOML setting; newly
created or recreated sandbox pods then have no additional-CA volume or mount.
Existing running sandbox supervisors retain their startup trust until they are
restarted or recreated.

The driver creates one immutable, content-addressed ConfigMap generation in
each target namespace. Names begin with
`openshell-additional-ca-` and include DNS-safe hashes of the effective
gateway ID and normalized bundle generation. Existing objects are accepted only
when their ownership labels, generation annotation, immutability, and exact
`ca.crt` data match; the driver never patches or updates them.

Kubernetes cannot restrict `create` by `resourceNames`, and `resourceNames`
cannot express the digest-bearing name prefix needed for `get`. Enabling this
feature therefore gives the gateway/driver service account namespace-wide
ConfigMap `get` and `create` permissions in shared mode. In managed/operator
modes, the chart's ClusterRoleBinding makes those permissions cluster-wide for
the bound service account. The chart does not grant ConfigMap `list`, `watch`,
`patch`, `update`, or `delete`. Use a dedicated gateway service account and
namespace-scoped bindings when this boundary matters. If you render the TOML
setting outside this Helm value, grant `get` and `create` in every target
sandbox namespace.

Disabling the setting intentionally does not delete driver-managed generations:
the gateway has no ConfigMap list/delete permissions, and an older object may
still be referenced by a running or stopped sandbox. An operator with separate
list/delete authority can inventory the gateway's generations by label:

```shell
kubectl -n <sandbox-namespace> get configmap \
-l 'openshell.ai/managed-by=openshell,openshell.ai/gateway-id=<effective-gateway-id>'
```

Before deleting a named generation, verify that no Sandbox resource or pod
volume in that namespace references it. Delete only the confirmed unreferenced
name; removing a referenced generation can make a later pod start or reschedule
fail closed.

A configured CA can authenticate every policy-permitted TLS endpoint with a
matching certificate chain and hostname. Scope private CAs accordingly.

### Database backend

By default, OpenShell uses SQLite and runs the gateway as a StatefulSet so the
Expand Down Expand Up @@ -377,6 +449,19 @@ Disabling autoscaling renders `spec.replicas` from `replicaCount` again,
which defaults to 1, so set `replicaCount` to the replica count you want in
that upgrade.

Set `supervisor.autoscaling.enabled=true` with the shared supervisor topology
to autoscale its Deployment independently of the gateway. It uses average
memory per Pod (`supervisor.autoscaling.targetMemoryAverageValue`, default
`512Mi`) through the resource metrics API, usually provided by Metrics Server.
Defaults keep two to four members and remove at most one member every two
minutes after a five-minute stabilization window. The HPA owns replicas, so
`supervisor.replicas` is ignored. Enabling it on an existing Deployment has
the same temporary reset-to-one risk described above; pick up new defaults
with `--reset-then-reuse-values`. When disabling it, set `supervisor.replicas`
explicitly. Existing sandboxes stay on their owners when scaling up, and
memory-based scaling does not guarantee spare capacity for handoff. See the
[shared supervisor autoscaling guide](https://docs.nvidia.com/openshell/latest/kubernetes/high-availability#autoscale-shared-supervisors-by-memory).

## Secret bootstrap

By default, a pre-install/pre-upgrade hook Job runs `openshell-gateway generate-certs`
Expand Down Expand Up @@ -586,6 +671,7 @@ discovery endpoint or its TLS CA.
| server.sandboxJwt.ttlSecs | int | `3600` | Token TTL in seconds. Defaults to 3600 (1h). |
| server.sandboxNamespace | string | `""` | Namespace where sandbox pods are created. Defaults to the Helm release namespace (.Release.Namespace) when left empty. |
| server.sandboxUid | string | `""` | UID for sandbox pods (`sandbox_uid`). Empty (default) = use the OpenShift SCC namespace annotation if present, otherwise the driver default. Must be an integer between 1 and 4294967294. |
| server.serviceHostnameFormat | string | `""` | Hostname format for exposed sandbox services. Empty or "readable" keeps readable `<workspace>--<sandbox>[--<service>]` hostnames; "hashed" mints opaque one-label hostnames discriminated by the gateway installation name (server.name, defaulting to the release name). The name must be unique among all gateways sharing the service base domain, and renaming the installation changes every minted hostname. |
| server.telemetryEnabled | bool | `true` | Enable anonymous OpenShell telemetry from the gateway and the sandbox supervisors it launches. |
| server.tls.certSecretName | string | `"openshell-server-tls"` | K8s secret (type kubernetes.io/tls) with tls.crt and tls.key for the server. |
| server.tls.clientCaSecretName | string | `"openshell-server-client-ca"` | K8s secret with ca.crt for client certificate verification (mTLS). Only used when enableMtls is true. Set to "" to disable client certificate verification for HTTPS-only mode. |
Expand All @@ -600,16 +686,26 @@ discovery endpoint or its TLS CA.
| serviceAccount.annotations | object | `{}` | Annotations to add to the generated service account. |
| serviceAccount.create | bool | `true` | Create a service account for the gateway. |
| serviceAccount.name | string | `""` | Existing service account name to use when serviceAccount.create is false. |
| supervisor.affinity | object | `{}` | Affinity rules for the shared supervisor pod, rendered only when gatewayConfig selects supervisor_topology = "gateway". Top-level affinity applies only to the gateway pod. |
| supervisor.autoscaling.behavior | object | `{"scaleDown":{"policies":[{"periodSeconds":120,"type":"Pods","value":1}],"stabilizationWindowSeconds":300}}` | HPA scaling behavior. Remove at most one member every two minutes, after a five-minute stabilization window, to allow ownership handoff. |
| supervisor.autoscaling.enabled | bool | `false` | Enable memory-based autoscaling of the shared supervisor Deployment. Requires supervisor_topology = "gateway" and the Kubernetes resource metrics API (usually Metrics Server). When enabled, the HPA owns the replica count and supervisor.replicas is ignored. |
| supervisor.autoscaling.maxReplicas | int | `4` | Maximum supervisor replicas. |
| supervisor.autoscaling.minReplicas | int | `2` | Minimum supervisor replicas. Keep at least 2 for failover capacity. |
| supervisor.autoscaling.targetMemoryAverageValue | string | `"512Mi"` | Target average memory working set per supervisor Pod, as a positive Kubernetes quantity. This is an absolute value, not a percentage of requests. Scaling up adds capacity for new sandboxes; it does not move existing sandboxes away from a busy member. |
| supervisor.image.digest | string | `""` | Supervisor image digest. When set, this takes precedence over tag. |
| supervisor.image.pullPolicy | string | `nil` | Supervisor image pull policy. Empty uses global.image.pullPolicy. Prefer always, if_not_present, or never; the chart also accepts legacy Kubernetes spellings Always, IfNotPresent, and Never. |
| supervisor.image.registry | string | `""` | Supervisor image registry. Empty uses global.image.registry. |
| supervisor.image.repository | string | `"openshell/supervisor"` | Supervisor image repository. |
| supervisor.image.tag | string | `""` | Supervisor image tag. Defaults to the chart appVersion when empty. |
| supervisor.network.additionalCaConfigMapName | string | `""` | Name of an operator-owned ConfigMap with a `ca.crt` key containing additional destination CA certificates for sandbox supervisors. Empty disables additional destination trust. |
| supervisor.podDisruptionBudget.enabled | bool | `true` | Render a PodDisruptionBudget with maxUnavailable 1 for the shared supervisor, used only when gatewayConfig selects supervisor_topology = "gateway". Disable it only when maintenance must evict more than one supervisor Pod at a time, or evict a member while another is unavailable. |
| supervisor.podSecurityContext.fsGroup | int | `65534` | fsGroup assigned to the shared supervisor pod. |
| supervisor.podSecurityContext.runAsGroup | int | `65534` | Primary GID assigned to the shared supervisor pod. |
| supervisor.podSecurityContext.runAsNonRoot | bool | `true` | Require the shared supervisor to run as a non-root user. |
| supervisor.podSecurityContext.runAsUser | int | `65534` | UID assigned to the shared supervisor pod. |
| supervisor.podSecurityContext.seccompProfile.type | string | `"RuntimeDefault"` | Seccomp profile applied to the shared supervisor pod. |
| supervisor.recoveryTimeoutSeconds | int | `60` | Seconds a sandbox workload stays frozen after losing its pool member's supervision before the sandbox runtime terminates it, rendered as [openshell.gateway.supervisor_pool] recovery_timeout_secs. Must be an integer from 30 to 300. Null leaves the gateway's pool default of 60 seconds. |
| supervisor.replicas | int | `1` | Shared supervisor pool members, used only when gatewayConfig selects supervisor_topology = "gateway". The gateway always coordinates the shared supervisor as a high-availability pool: when a member fails or leaves, another member takes over its sandboxes without restarting their workloads. 1 has no failover; use 2 or more to survive losing a member. Only the supervisor Deployment's replica count depends on this value, so changing it neither restarts the gateway nor replaces running members. Ignored when supervisor.autoscaling.enabled is true. |
| supervisor.sandboxRuntime.boundaryPort | int | `5500` | Workload boundary TLS listener port. |
| tolerations | list | `[]` | Tolerations for the gateway pod. |
| upstreamProxy | object | `{"authAllowInsecure":false,"authSecret":{"key":"","name":""},"caBundle":{"configMapName":"","key":"ca.crt"},"connectByHostname":false,"noProxy":"","url":""}` | Operator-owned corporate forward proxy for policy-approved TLS egress from Kubernetes sandboxes. The workload cannot select or override it. |
Expand Down
85 changes: 85 additions & 0 deletions charts/openshell/README.md.gotmpl
Original file line number Diff line number Diff line change
Expand Up @@ -208,9 +208,81 @@ See [`values.yaml`](values.yaml) for source defaults. Selected overlays:
- [`ci/values-keycloak.yaml`](ci/values-keycloak.yaml) - Keycloak OIDC integration
- [`ci/values-high-availability.yaml`](ci/values-high-availability.yaml) - CI overlay for multi-replica external PostgreSQL testing
- [`ci/values-autoscaling.yaml`](ci/values-autoscaling.yaml) - CI overlay for rendering the optional gateway HorizontalPodAutoscaler
- [`ci/values-supervisor-autoscaling.yaml`](ci/values-supervisor-autoscaling.yaml) - CI overlay for rendering the optional shared supervisor memory HorizontalPodAutoscaler
- [`ci/values-spire.yaml`](ci/values-spire.yaml) - SPIFFE/SPIRE provider token grants
- [`ci/values-spire-stack.yaml`](ci/values-spire-stack.yaml) - SPIRE hardened chart values for local development

### Additional destination CAs

Create an operator-owned ConfigMap with a `ca.crt` key when sandboxes must reach
TLS destinations signed by a private CA:

```shell
kubectl create configmap private-destination-ca -n openshell \
--from-file=ca.crt=/path/to/ca.crt
```

Then set the source ConfigMap name:

```yaml
supervisor:
network:
additionalCaConfigMapName: private-destination-ca
```

The chart mounts the source into the gateway and renders
`[openshell.supervisor.network].additional_ca_bundle`. The gateway validates
and normalizes the one certificate-only PEM bundle at startup. The Kubernetes
driver creates a separate managed ConfigMap in each target sandbox namespace
and mounts it only in the dedicated supervisor Pod at
`/etc/openshell/tls/additional/ca-bundle.pem`. That supervisor receives the
path through `--additional-ca-bundle` and folds the bundle into system roots.
The configured roots augment default destination trust. They do not configure
the corporate proxy, OIDC, or gateway callback mTLS trust.

Changing only the source ConfigMap does not roll or reload the gateway. Restart
the gateway, then recreate or restart affected sandboxes to use the new roots.
Removing `additionalCaConfigMapName` removes the global TOML setting; newly
created or recreated sandbox pods then have no additional-CA volume or mount.
Existing running sandbox supervisors retain their startup trust until they are
restarted or recreated.

The driver creates one immutable, content-addressed ConfigMap generation in
each target namespace. Names begin with
`openshell-additional-ca-` and include DNS-safe hashes of the effective
gateway ID and normalized bundle generation. Existing objects are accepted only
when their ownership labels, generation annotation, immutability, and exact
`ca.crt` data match; the driver never patches or updates them.

Kubernetes cannot restrict `create` by `resourceNames`, and `resourceNames`
cannot express the digest-bearing name prefix needed for `get`. Enabling this
feature therefore gives the gateway/driver service account namespace-wide
ConfigMap `get` and `create` permissions in shared mode. In managed/operator
modes, the chart's ClusterRoleBinding makes those permissions cluster-wide for
the bound service account. The chart does not grant ConfigMap `list`, `watch`,
`patch`, `update`, or `delete`. Use a dedicated gateway service account and
namespace-scoped bindings when this boundary matters. If you render the TOML
setting outside this Helm value, grant `get` and `create` in every target
sandbox namespace.

Disabling the setting intentionally does not delete driver-managed generations:
the gateway has no ConfigMap list/delete permissions, and an older object may
still be referenced by a running or stopped sandbox. An operator with separate
list/delete authority can inventory the gateway's generations by label:

```shell
kubectl -n <sandbox-namespace> get configmap \
-l 'openshell.ai/managed-by=openshell,openshell.ai/gateway-id=<effective-gateway-id>'
```

Before deleting a named generation, verify that no Sandbox resource or pod
volume in that namespace references it. Delete only the confirmed unreferenced
name; removing a referenced generation can make a later pod start or reschedule
fail closed.

A configured CA can authenticate every policy-permitted TLS endpoint with a
matching certificate chain and hostname. Scope private CAs accordingly.

### Database backend

By default, OpenShell uses SQLite and runs the gateway as a StatefulSet so the
Expand Down Expand Up @@ -378,6 +450,19 @@ Disabling autoscaling renders `spec.replicas` from `replicaCount` again,
which defaults to 1, so set `replicaCount` to the replica count you want in
that upgrade.

Set `supervisor.autoscaling.enabled=true` with the shared supervisor topology
to autoscale its Deployment independently of the gateway. It uses average
memory per Pod (`supervisor.autoscaling.targetMemoryAverageValue`, default
`512Mi`) through the resource metrics API, usually provided by Metrics Server.
Defaults keep two to four members and remove at most one member every two
minutes after a five-minute stabilization window. The HPA owns replicas, so
`supervisor.replicas` is ignored. Enabling it on an existing Deployment has
the same temporary reset-to-one risk described above; pick up new defaults
with `--reset-then-reuse-values`. When disabling it, set `supervisor.replicas`
explicitly. Existing sandboxes stay on their owners when scaling up, and
memory-based scaling does not guarantee spare capacity for handoff. See the
[shared supervisor autoscaling guide](https://docs.nvidia.com/openshell/latest/kubernetes/high-availability#autoscale-shared-supervisors-by-memory).

## Secret bootstrap

By default, a pre-install/pre-upgrade hook Job runs `openshell-gateway generate-certs`
Expand Down
8 changes: 8 additions & 0 deletions charts/openshell/ci/values-supervisor-autoscaling.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
gatewayConfig:
openshell.drivers.kubernetes:
supervisor_topology: gateway
supervisor:
autoscaling:
enabled: true
14 changes: 14 additions & 0 deletions charts/openshell/ci/values-supervisor-ha-scaling.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# E2E overlay: supervise new sandbox generations with a shared supervisor pool
# that starts with one member, so the test can scale it to two members and
# back with `helm upgrade --set supervisor.replicas=N` without changing the
# gateway configuration or the supervisor Pod template.
gatewayConfig:
openshell.drivers.kubernetes:
supervisor_topology: gateway

supervisor:
replicas: 1
recoveryTimeoutSeconds: 60
14 changes: 14 additions & 0 deletions charts/openshell/ci/values-supervisor-ha.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# E2E overlay: supervise new sandbox generations with a high-availability pool
# of two shared supervisors, so a failed or leaving member's sandboxes move to
# the other member. Layer it on a workspace-mode overlay to cover managed or
# operator namespaces.
gatewayConfig:
openshell.drivers.kubernetes:
supervisor_topology: gateway

supervisor:
replicas: 2
recoveryTimeoutSeconds: 60
4 changes: 2 additions & 2 deletions charts/openshell/ci/values-supervisor.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# E2E overlay: supervise new sandbox generations with one supervisor process
# shared by the gateway. Layer it on a workspace-mode overlay
# E2E overlay: supervise new sandbox generations with a one-member shared
# supervisor pool. Layer it on a workspace-mode overlay
# to cover managed or operator namespaces.
gatewayConfig:
openshell.drivers.kubernetes:
Expand Down
Loading
Loading