From 6d472fa2c0c55a81aed8cd718ff39e90b1adf6f6 Mon Sep 17 00:00:00 2001
From: user <u@example.com>
Date: Mon, 10 Aug 2026 15:30:33 -0400
Subject: [PATCH 01/15] spec: add global architecture specification
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Captures deployment patterns (single-node, global multi-region, multi-cloud),
tooling stack decisions (CNPG, ArgoCD, Tekton, Vault, Terraform, Prometheus),
namespace strategy, installer pipeline requirements, and monitoring architecture
from the Aug 10 architecture meeting.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
---
 specs/index.spec.md                        |   1 +
 specs/platform/global-architecture.spec.md | 240 +++++++++++++++++++++
 2 files changed, 241 insertions(+)
 create mode 100644 specs/platform/global-architecture.spec.md

diff --git a/specs/index.spec.md b/specs/index.spec.md
index a9b74065..f6130cef 100644
--- a/specs/index.spec.md
+++ b/specs/index.spec.md
@@ -38,6 +38,7 @@ Machine-readable index for autonomous reconciliation (`/reconcile` skill).
 | `platform/openshell-gateway-credentials.spec.md` | platform | Credential storage drivers, KEK conditional provisioning | CP | openshell-gateway, openshell-gateway-database |
 | `platform/openshell-gateway-secret-rotation.spec.md` | platform | Secret rotation: DB password, KEK, TLS certificates | CP | openshell-gateway-database, openshell-gateway-credentials, openshell-gateway-tls |
 | `platform/openshell-gateway-keycloak.spec.md` | platform | Keycloak OIDC client provisioning, per-gateway OIDC role bridge | CP | openshell-gateway, openshell-gateway-oidc, rbac-enforcement |
+| `platform/global-architecture.spec.md` | platform | Global hub, multi-cloud, CNPG, Tekton, ArgoCD, Vault | CP, ALL | data-model, control-plane |
 | `web-console/architecture.spec.md` | web-console | Web console, BFF, browser session, UI routes | WEB, SDK, API | data-model, security, UI standards |
 | `standards/platform/cross-cutting.spec.md` | standards | - | ALL | - |
 | `standards/control-plane/conventions.spec.md` | standards | - | CP | - |
diff --git a/specs/platform/global-architecture.spec.md b/specs/platform/global-architecture.spec.md
new file mode 100644
index 00000000..46569365
--- /dev/null
+++ b/specs/platform/global-architecture.spec.md
@@ -0,0 +1,240 @@
+# Global Architecture
+
+**Date:** 2026-08-10
+**Status:** Active
+
+## Overview
+
+HyperShell deploys as a global fleet management platform spanning multiple clouds and regions. A single OpenShift hub per cloud manages gateway deployments across regions. Managed clusters can run standard Kubernetes (not restricted to OpenShift). The architecture supports three deployment patterns: single-node, global multi-region, and multi-cloud.
+
+## Deployment Patterns
+
+### Single-Node
+
+One OpenShift cluster runs the full HyperShell stack: API server, controller, PostgreSQL (via CNPG), gateways, and supporting services. Suitable for development, testing, and small-scale production.
+
+```
+┌──────────────────────────────────────────────────┐
+│  OpenShift Cluster                               │
+│                                                  │
+│  ┌────────────┐ ┌────────────┐ ┌──────────────┐ │
+│  │ API Server │ │ Controller │ │ PostgreSQL   │ │
+│  │            │ │            │ │ (CNPG)       │ │
+│  └────────────┘ └────────────┘ └──────────────┘ │
+│                                                  │
+│  ┌────────────────────────────────────────────┐  │
+│  │ Gateway Namespace                          │  │
+│  │ ┌──────────┐ ┌────────────┐ ┌──────────┐  │  │
+│  │ │ Gateway  │ │ Supervisor │ │ Sandboxes│  │  │
+│  │ └──────────┘ └────────────┘ └──────────┘  │  │
+│  └────────────────────────────────────────────┘  │
+│                                                  │
+│  ArgoCD │ Vault │ Keycloak │ Grafana │ Prometheus │
+└──────────────────────────────────────────────────┘
+```
+
+### Global Multi-Region
+
+One OpenShift hub manages gateways across multiple regions within a single cloud provider. Regional managed clusters run gateways close to users. The hub controller provisions gateways remotely via kubeconfig secrets.
+
+```
+                    ┌─────────────────────────────┐
+                    │  Hub (us-east-1)             │
+                    │  OpenShift                   │
+                    │  API + Controller + CNPG     │
+                    │  ArgoCD + Vault + Keycloak   │
+                    │  Prometheus + Grafana         │
+                    └──────┬──────────┬────────────┘
+                           │          │
+              ┌────────────┘          └────────────┐
+              ▼                                    ▼
+┌─────────────────────────┐          ┌─────────────────────────┐
+│  Managed Cluster        │          │  Managed Cluster        │
+│  us-west-2              │          │  eu-west-1              │
+│  K8s or OpenShift       │          │  K8s or OpenShift       │
+│  ┌────────┐ ┌────────┐  │          │  ┌────────┐ ┌────────┐  │
+│  │Gateway │ │Sandbox │  │          │  │Gateway │ │Sandbox │  │
+│  └────────┘ └────────┘  │          │  └────────┘ └────────┘  │
+│  Prometheus              │          │  Prometheus              │
+└─────────────────────────┘          └─────────────────────────┘
+```
+
+### Multi-Cloud
+
+Separate hubs per cloud provider, each managing their own regional clusters. A global coordination layer provides cross-cloud visibility.
+
+```
+┌─────────────────────────────┐   ┌─────────────────────────────┐
+│  IBM Cloud Hub               │   │  AWS Hub                     │
+│  OpenShift (ROKS)            │   │  OpenShift (ROSA)            │
+│  API + Controller + CNPG     │   │  API + Controller + CNPG     │
+│  ArgoCD + Vault              │   │  ArgoCD + Vault              │
+│         │                    │   │         │                    │
+│    ┌────┴────┐               │   │    ┌────┴────┐               │
+│    ▼         ▼               │   │    ▼         ▼               │
+│ ┌──────┐ ┌──────┐           │   │ ┌──────┐ ┌──────┐           │
+│ │MC    │ │MC    │           │   │ │MC    │ │MC    │           │
+│ │east  │ │west  │           │   │ │east  │ │west  │           │
+│ └──────┘ └──────┘           │   │ └──────┘ └──────┘           │
+└─────────────────────────────┘   └─────────────────────────────┘
+```
+
+## Tooling Stack
+
+| Component | Tool | Purpose |
+|-----------|------|---------|
+| Database operator | CNPG (CloudNativePG) | PostgreSQL lifecycle (replaces per-gateway cloud databases) |
+| GitOps | ArgoCD | Reconciles cluster state from Git |
+| Secret management | Vault | Stores and rotates secrets with cloud-native drivers |
+| Identity | Keycloak | OIDC authentication for gateways and console |
+| Cluster provisioning | Terraform | VPC, subnet, and cluster provisioning |
+| Installer | Tekton pipelines | Reproducible, deterministic deployment pipelines |
+| Monitoring | Prometheus | Metrics collection on all clusters |
+| Dashboards | Grafana | Centralized visualization on hub |
+
+## Database Strategy: CNPG
+
+CloudNativePG replaces per-gateway cloud-managed databases (RDS, Cloud SQL). CNPG runs PostgreSQL clusters as Kubernetes-native resources with automated failover, backup, and recovery.
+
+### Requirements
+
+#### Requirement: CNPG Operator Deployment
+
+The CNPG operator SHALL be deployed on the hub cluster. Gateway databases SHALL be provisioned as CNPG Cluster resources in the gateway namespace.
+
+##### Scenario: Gateway Database Provisioning via CNPG
+
+- GIVEN a Gateway resource with a `database_id` referencing a ManagedDatabase
+- WHEN the ManagedDatabase specifies `provider: cnpg`
+- THEN the controller SHALL create a CNPG Cluster resource in the gateway namespace
+- AND the CNPG operator SHALL provision a PostgreSQL instance with automated replication
+
+#### Requirement: Database Lifecycle Independence
+
+ManagedDatabase resources SHALL have an independent lifecycle from Gateways. A single CNPG cluster MAY serve multiple gateways within the same namespace.
+
+## Namespace Strategy
+
+Gateway and its sandboxes coexist in the same namespace as a scalable unit. Each gateway deployment gets its own namespace containing:
+
+- Gateway pod (StatefulSet)
+- Supervisor sidecar
+- Sandbox pods
+- PostgreSQL (CNPG Cluster or in-namespace Deployment)
+- TLS certificates (cert-manager or certgen)
+- NetworkPolicies
+- RBAC resources
+
+```
+namespace: openshell-<gateway-name>
+├── Gateway StatefulSet
+├── Supervisor
+├── Sandbox pods (dynamic)
+├── PostgreSQL (CNPG Cluster)
+├── TLS Secrets
+├── ConfigMaps
+├── NetworkPolicies
+└── RBAC (Roles, RoleBindings, ServiceAccounts)
+```
+
+## Installer Pipeline
+
+Tekton pipelines provide a reproducible installer for HyperShell deployments. The pipeline replaces manual bash scripts with deterministic, auditable steps.
+
+### Requirements
+
+#### Requirement: Deterministic Installation
+
+HyperShell installation SHALL be performed via Tekton pipelines that execute idempotent steps. Manual `oc apply` or `kubectl` scripts SHALL NOT be the primary installation method in production.
+
+##### Scenario: Fresh Cluster Installation
+
+- GIVEN a bare OpenShift cluster with Tekton installed
+- WHEN the HyperShell installer pipeline runs
+- THEN it SHALL provision: CNPG operator, cert-manager, API server, controller, PostgreSQL, ArgoCD, Vault, Keycloak
+- AND the installation SHALL be idempotent (safe to re-run)
+
+#### Requirement: Cattle Not Pets
+
+Infrastructure SHALL be treated as disposable. Any cluster can be torn down and rebuilt from the pipeline without manual intervention or state recovery.
+
+## Monitoring Architecture
+
+Prometheus runs on every managed cluster. Regional aggregation feeds into the global hub.
+
+```
+Managed Cluster (region)          Hub Cluster
+┌─────────────────────┐          ┌─────────────────────┐
+│ Prometheus (local)  │ ──────▶  │ Prometheus (global)  │
+│ Gateway metrics     │          │ Grafana dashboards   │
+│ Node metrics        │          │ Alertmanager         │
+└─────────────────────┘          └─────────────────────┘
+```
+
+### Requirements
+
+#### Requirement: Managed Cluster Metrics
+
+Every managed cluster SHALL run a Prometheus instance that scrapes gateway and node metrics. Metrics SHALL be forwarded to the regional or global hub.
+
+#### Requirement: Hub Dashboards
+
+The hub cluster SHALL run Grafana with dashboards for fleet-wide gateway health, resource utilization, and provisioning status across all managed clusters.
+
+## Managed Cluster Flexibility
+
+Managed clusters are not restricted to OpenShift. Standard Kubernetes distributions (EKS, GKE, vanilla K8s) are valid targets. The controller auto-detects cluster capabilities:
+
+| Capability | Detection | Behavior when absent |
+|------------|-----------|---------------------|
+| OpenShift Routes | `route.openshift.io` API group | Use Gateway API or NodePort |
+| cert-manager | `cert-manager.io` API group | Block deployment (required) |
+| Gateway API | `gateway.networking.k8s.io/v1` GRPCRoute | Skip GRPCRoute/BackendTLSPolicy |
+| Agent Sandbox CRD | `agents.x-k8s.io` | Sandbox creation blocked |
+
+## GitOps Repository Structure
+
+ArgoCD manages cluster state from a Git repository. Each cloud and region has its own overlay.
+
+```
+gitops-repo/
+├── base/
+│   ├── hypershell/
+│   │   ├── api-server.yaml
+│   │   ├── controller.yaml
+│   │   └── postgres.yaml
+│   ├── cnpg/
+│   ├── cert-manager/
+│   ├── vault/
+│   └── keycloak/
+├── overlays/
+│   ├── ibm-us-east/
+│   │   ├── kustomization.yaml
+│   │   └── patches/
+│   ├── aws-us-east/
+│   │   ├── kustomization.yaml
+│   │   └── patches/
+│   └── aws-eu-west/
+│       ├── kustomization.yaml
+│       └── patches/
+└── clusters/
+    ├── ibm-hub.yaml        (ArgoCD Application)
+    ├── aws-hub.yaml
+    └── managed/
+        ├── rosa-vteam.yaml
+        └── eks-staging.yaml
+```
+
+## Design Decisions
+
+| Decision | Rationale |
+|----------|-----------|
+| One OpenShift hub per cloud | Reduces cross-cloud latency; hub needs OpenShift for Routes and SCC |
+| Managed clusters can be standard K8s | Maximizes deployment flexibility; only the hub needs OpenShift |
+| CNPG over cloud-managed databases | Kubernetes-native lifecycle, portable across clouds, no vendor lock-in |
+| Tekton over bash scripts | Deterministic, auditable, cattle-not-pets infrastructure |
+| ArgoCD for GitOps | Declarative cluster state, drift detection, multi-cluster support |
+| Vault for secrets | Centralized rotation, cloud-native drivers, audit trail |
+| Prometheus on all clusters | Uniform metrics pipeline, regional aggregation to global hub |
+| Namespace-per-gateway | Isolation boundary for RBAC, NetworkPolicy, and resource quotas |
+| Terraform for provisioning | IaC for VPC, subnet, and cluster lifecycle; cloud-agnostic |

From b00be85a34f1a4584090a938b2b03e815145ac2b Mon Sep 17 00:00:00 2001
From: user <u@example.com>
Date: Sat, 15 Aug 2026 07:56:22 -0400
Subject: [PATCH 02/15] docs: add Ingress Architecture section detailing
 dual-ingress strategy

---
 specs/platform/global-architecture.spec.md | 460 ++++++++++++++++-----
 1 file changed, 363 insertions(+), 97 deletions(-)

diff --git a/specs/platform/global-architecture.spec.md b/specs/platform/global-architecture.spec.md
index 46569365..8d92083e 100644
--- a/specs/platform/global-architecture.spec.md
+++ b/specs/platform/global-architecture.spec.md
@@ -1,84 +1,255 @@
 # Global Architecture
 
-**Date:** 2026-08-10
+**Date:** 2026-08-14
 **Status:** Active
 
 ## Overview
 
-HyperShell deploys as a global fleet management platform spanning multiple clouds and regions. A single OpenShift hub per cloud manages gateway deployments across regions. Managed clusters can run standard Kubernetes (not restricted to OpenShift). The architecture supports three deployment patterns: single-node, global multi-region, and multi-cloud.
+HyperShell deploys as a global fleet management platform spanning multiple clouds and regions. The architecture uses a **three-tier hub-and-spoke topology**: a Global Hub provides federated identity root, Cloud Hubs run the operational platform (API, control plane, databases), and ManagedClusters host gateway workloads. Every OpenShift cluster in the topology runs the full operator stack (ArgoCD, Vault, Keycloak, CNPG, Prometheus, Grafana) but serves different purposes at each tier.
+
+## Three-Tier Topology
+
+HyperShell uses a three-tier hub-and-spoke architecture. Each tier runs the full operator stack but serves distinct purposes.
+
+```mermaid
+graph TB
+    subgraph Global["Global Hub (Identity Root)"]
+        GK[Keycloak<br/>Federated to RH SSO]
+        GV[Vault<br/>Reserved]
+        GG[Grafana<br/>Cross-Cloud Dashboard]
+    end
+
+    subgraph AWS["Cloud Hub: AWS"]
+        AK[Keycloak<br/>Federates to Global]
+        AV[Vault<br/>Service Secrets]
+        AArgo[ArgoCD<br/>Fleet GitOps]
+        ADB[(PostgreSQL<br/>CNPG)]
+        ACP[Control Plane]
+        AAPI[API Server]
+        AUI[Web UI]
+        AP[Prometheus]
+        AG[Grafana]
+    end
+
+    subgraph IBM["Cloud Hub: IBM Cloud"]
+        IK[Keycloak<br/>Federates to Global]
+        IV[Vault<br/>Service Secrets]
+        IArgo[ArgoCD<br/>Fleet GitOps]
+        IDB[(PostgreSQL<br/>CNPG)]
+        ICP[Control Plane]
+        IAPI[API Server]
+        IUI[Web UI]
+        IP[Prometheus]
+        IG[Grafana]
+    end
+
+    subgraph MC1["ManagedCluster: AWS us-east-1"]
+        M1K[Keycloak<br/>Gateway Clients]
+        M1V[Vault<br/>Gateway Secrets]
+        M1DB[(PostgreSQL<br/>CNPG)]
+        M1P[Prometheus]
+        M1GW[Gateway Namespaces]
+    end
+
+    subgraph MC2["ManagedCluster: AWS us-west-2"]
+        M2K[Keycloak<br/>Gateway Clients]
+        M2V[Vault<br/>Gateway Secrets]
+        M2DB[(PostgreSQL<br/>CNPG)]
+        M2P[Prometheus]
+        M2GW[Gateway Namespaces]
+    end
+
+    subgraph MC3["ManagedCluster: IBM us-east"]
+        M3K[Keycloak<br/>Gateway Clients]
+        M3V[Vault<br/>Gateway Secrets]
+        M3DB[(PostgreSQL<br/>CNPG)]
+        M3P[Prometheus]
+        M3GW[Gateway Namespaces]
+    end
+
+    GK -->|Federation| AK
+    GK -->|Federation| IK
+    AK -->|Federation| M1K
+    AK -->|Federation| M2K
+    IK -->|Federation| M3K
+    
+    ACP -->|Reconcile| M1GW
+    ACP -->|Reconcile| M2GW
+    ICP -->|Reconcile| M3GW
+    
+    M1P -->|Metrics| AP
+    M2P -->|Metrics| AP
+    M3P -->|Metrics| IP
+    
+    AP -->|Aggregate| GG
+    IP -->|Aggregate| GG
+
+    style Global fill:#e1f5ff
+    style AWS fill:#fff3cd
+    style IBM fill:#fff3cd
+    style MC1 fill:#d4edda
+    style MC2 fill:#d4edda
+    style MC3 fill:#d4edda
+```
 
-## Deployment Patterns
+### Tier 1: Global Hub
 
-### Single-Node
+**Purpose**: Identity federation root and cross-cloud observability.
 
-One OpenShift cluster runs the full HyperShell stack: API server, controller, PostgreSQL (via CNPG), gateways, and supporting services. Suitable for development, testing, and small-scale production.
+**Components**:
+- Keycloak (federates to Red Hat SSO)
+- Vault (reserved for future global secrets)
+- Grafana (single-pane-of-glass aggregating metrics from all Cloud Hubs)
 
-```
-┌──────────────────────────────────────────────────┐
-│  OpenShift Cluster                               │
-│                                                  │
-│  ┌────────────┐ ┌────────────┐ ┌──────────────┐ │
-│  │ API Server │ │ Controller │ │ PostgreSQL   │ │
-│  │            │ │            │ │ (CNPG)       │ │
-│  └────────────┘ └────────────┘ └──────────────┘ │
-│                                                  │
-│  ┌────────────────────────────────────────────┐  │
-│  │ Gateway Namespace                          │  │
-│  │ ┌──────────┐ ┌────────────┐ ┌──────────┐  │  │
-│  │ │ Gateway  │ │ Supervisor │ │ Sandboxes│  │  │
-│  │ └──────────┘ └────────────┘ └──────────┘  │  │
-│  └────────────────────────────────────────────┘  │
-│                                                  │
-│  ArgoCD │ Vault │ Keycloak │ Grafana │ Prometheus │
-└──────────────────────────────────────────────────┘
-```
+**Operational Role**: Provides the root of the Keycloak federation chain. Cloud Hubs federate to Global Keycloak, which federates to Red Hat SSO. Future: global monitoring dashboard.
 
-### Global Multi-Region
+### Tier 2: Cloud Hub
 
-One OpenShift hub manages gateways across multiple regions within a single cloud provider. Regional managed clusters run gateways close to users. The hub controller provisions gateways remotely via kubeconfig secrets.
+**Purpose**: Primary operational unit. One highly-available instance per cloud provider.
 
-```
-                    ┌─────────────────────────────┐
-                    │  Hub (us-east-1)             │
-                    │  OpenShift                   │
-                    │  API + Controller + CNPG     │
-                    │  ArgoCD + Vault + Keycloak   │
-                    │  Prometheus + Grafana         │
-                    └──────┬──────────┬────────────┘
-                           │          │
-              ┌────────────┘          └────────────┐
-              ▼                                    ▼
-┌─────────────────────────┐          ┌─────────────────────────┐
-│  Managed Cluster        │          │  Managed Cluster        │
-│  us-west-2              │          │  eu-west-1              │
-│  K8s or OpenShift       │          │  K8s or OpenShift       │
-│  ┌────────┐ ┌────────┐  │          │  ┌────────┐ ┌────────┐  │
-│  │Gateway │ │Sandbox │  │          │  │Gateway │ │Sandbox │  │
-│  └────────┘ └────────┘  │          │  └────────┘ └────────┘  │
-│  Prometheus              │          │  Prometheus              │
-└─────────────────────────┘          └─────────────────────────┘
-```
+**Components**:
+- API Server, Control Plane, Web UI (HA deployment)
+- PostgreSQL (via CNPG) - source of truth for Fleet, Gateway, ManagedCluster resources
+- ArgoCD - reconciles this cloud's infrastructure from Git
+- Keycloak - federates to Global Keycloak, serves cloud services
+- Vault - secrets for cloud hub services (API server, control plane)
+- Prometheus - aggregates metrics from this cloud's ManagedClusters
+- Grafana - cloud-level dashboards
+
+**Operational Role**: The control plane watches the API server via gRPC and reconciles gateway resources into ManagedClusters. ArgoCD defines and provisions ManagedClusters.
+
+### Tier 3: ManagedCluster
+
+**Purpose**: Hosts gateway workloads. Multiple per cloud, deployed close to users (regional).
 
-### Multi-Cloud
+**Components**:
+- Keycloak - federates to Cloud Hub Keycloak, holds OIDC clients for gateways on this cluster
+- Vault - keystore for gateway secrets
+- PostgreSQL (via CNPG) - gateway-specific databases
+- Prometheus - local metrics (forwarded to Cloud Hub)
+- Gateway namespaces (each contains: Gateway pod, Supervisor, Sandboxes, CNPG Cluster, TLS secrets, RBAC)
 
-Separate hubs per cloud provider, each managing their own regional clusters. A global coordination layer provides cross-cloud visibility.
+**Operational Role**: Runs gateway workloads. Users authenticate openshell CLI against Keycloak on the ManagedCluster where their gateway lives.
 
+## Data Flows
+
+### Keycloak Federation Chain
+
+Identity flows from Red Hat SSO down through the tier hierarchy. Each Keycloak federates to the one above it.
+
+```mermaid
+graph LR
+    RHSSO[Red Hat SSO]
+    Global[Global Keycloak]
+    Cloud[Cloud Hub Keycloak]
+    MC[ManagedCluster Keycloak]
+    
+    RHSSO -->|Federates| Global
+    Global -->|Federates| Cloud
+    Cloud -->|Federates| MC
+    
+    MC -->|Holds| Clients[Gateway OIDC Clients]
+    
+    style RHSSO fill:#e74c3c
+    style Global fill:#3498db
+    style Cloud fill:#f39c12
+    style MC fill:#2ecc71
+    style Clients fill:#95a5a6
 ```
-┌─────────────────────────────┐   ┌─────────────────────────────┐
-│  IBM Cloud Hub               │   │  AWS Hub                     │
-│  OpenShift (ROKS)            │   │  OpenShift (ROSA)            │
-│  API + Controller + CNPG     │   │  API + Controller + CNPG     │
-│  ArgoCD + Vault              │   │  ArgoCD + Vault              │
-│         │                    │   │         │                    │
-│    ┌────┴────┐               │   │    ┌────┴────┐               │
-│    ▼         ▼               │   │    ▼         ▼               │
-│ ┌──────┐ ┌──────┐           │   │ ┌──────┐ ┌──────┐           │
-│ │MC    │ │MC    │           │   │ │MC    │ │MC    │           │
-│ │east  │ │west  │           │   │ │east  │ │west  │           │
-│ └──────┘ └──────┘           │   │ └──────┘ └──────┘           │
-└─────────────────────────────┘   └─────────────────────────────┘
+
+**Federation Path**: Red Hat SSO → Global Keycloak → Cloud Keycloak → ManagedCluster Keycloak
+
+**Client Registration**: Gateway OIDC clients are registered in the ManagedCluster Keycloak where the gateway runs.
+
+### Gateway Authentication Flow
+
+When a user authenticates their openshell CLI to a gateway:
+
+```mermaid
+sequenceDiagram
+    participant User as openshell CLI
+    participant Gateway as Gateway Pod
+    participant MCK as ManagedCluster<br/>Keycloak
+    participant CloudK as Cloud Hub<br/>Keycloak
+    participant GlobalK as Global<br/>Keycloak
+    participant RHSSO as Red Hat SSO
+
+    User->>Gateway: Connect to gateway
+    Gateway->>MCK: Redirect to OIDC login
+    MCK->>User: Show login page
+    User->>MCK: Provide credentials
+    MCK->>CloudK: Federate authentication
+    CloudK->>GlobalK: Federate authentication
+    GlobalK->>RHSSO: Federate authentication
+    RHSSO-->>GlobalK: Auth response
+    GlobalK-->>CloudK: Auth response
+    CloudK-->>MCK: Auth response
+    MCK->>User: Issue token
+    User->>Gateway: Request with token
+    Gateway->>MCK: Validate token
+    MCK-->>Gateway: Token valid
+    Gateway-->>User: Connected
 ```
 
+### Control Plane Reconciliation Flow
+
+The control plane on the Cloud Hub watches the API server and reconciles gateway resources into ManagedClusters.
+
+```mermaid
+sequenceDiagram
+    participant User as API Client
+    participant API as API Server<br/>(Cloud Hub)
+    participant DB as PostgreSQL<br/>(CNPG)
+    participant CP as Control Plane<br/>(Cloud Hub)
+    participant MC as ManagedCluster<br/>K8s API
+
+    User->>API: POST /gateways
+    API->>DB: INSERT gateway record
+    DB-->>API: Row created
+    API->>CP: gRPC Watch event<br/>(Gateway created)
+    CP->>MC: kubectl apply<br/>Gateway namespace
+    MC-->>CP: Namespace created
+    CP->>MC: kubectl apply<br/>Gateway StatefulSet
+    MC-->>CP: StatefulSet created
+    CP->>MC: kubectl apply<br/>CNPG Cluster
+    MC-->>CP: PostgreSQL provisioning
+    CP->>DB: UPDATE gateway status
+    DB-->>CP: Status updated
+    CP->>API: gRPC status update
+    API-->>User: Gateway deployed
+```
+
+**Key Points**:
+- PostgreSQL on the Cloud Hub is the source of truth for all resource state
+- Control Plane watches API server via gRPC streams
+- Control Plane reconciles resources into ManagedClusters via kubeconfig secrets
+- Gateway databases run as CNPG Clusters in the gateway namespace on the ManagedCluster
+
+
+## Ingress Architecture
+
+HyperShell utilizes a **dual-ingress strategy** on OpenShift clusters, separating platform management traffic from tenant gateway traffic. This ensures isolation, security, and optimal routing for different protocols.
+
+### 1. Platform Services (Default OpenShift Ingress)
+
+Traffic destined for HyperShell management services (API Server, Web Console, Keycloak) uses the default OpenShift routing tier.
+
+- **Domain:** Standard OpenShift wildcard domain (e.g., `*.apps.rosa...`)
+- **Load Balancer:** AWS Internal Network Load Balancer (NLB)
+- **Routing Object:** OpenShift `Route` (HAProxy)
+- **Mechanism:** The wildcard DNS resolves to the internal NLB, which forwards traffic to the OpenShift router pods. HAProxy uses hostname-based routing to direct traffic to the correct service.
+
+### 2. Tenant Gateways (Kubernetes Gateway API)
+
+Traffic destined for the actual managed gRPC gateways bypasses the default OpenShift router and uses a dedicated ingress path managed by the Kubernetes Gateway API (backed by Istio).
+
+- **Domain:** Dedicated base domain (e.g., `*.openshell.stage.devshift.net`)
+- **Load Balancer:** Dedicated AWS Classic ELB (provisioned by the Gateway API controller)
+- **Routing Object:** `GRPCRoute` attached to a central `Gateway`
+- **Mechanism:** A single shared `Gateway` object (`openshell-grpc-gateway`) in the `openshift-ingress` namespace terminates TLS using a wildcard certificate. When the control plane provisions a new gateway in a tenant namespace, it creates a `GRPCRoute` that automatically attaches to this shared `Gateway`. External DNS (e.g., Route53) manages the CNAME mapping the wildcard domain to the AWS ELB address.
+
+This architecture allows the control plane to dynamically route traffic for new gateways without needing to provision individual load balancers or DNS records per tenant.
+
 ## Tooling Stack
 
 | Component | Tool | Purpose |
@@ -115,27 +286,61 @@ ManagedDatabase resources SHALL have an independent lifecycle from Gateways. A s
 
 ## Namespace Strategy
 
-Gateway and its sandboxes coexist in the same namespace as a scalable unit. Each gateway deployment gets its own namespace containing:
+Gateway and its sandboxes coexist in the same namespace as a scalable unit. Each gateway deployment gets its own namespace containing all required resources.
+
+```mermaid
+graph TB
+    subgraph NS["namespace: openshell-<gateway-name>"]
+        subgraph Workloads
+            GW[Gateway StatefulSet<br/>+ Supervisor Sidecar]
+            SB1[Sandbox Pod 1]
+            SB2[Sandbox Pod 2]
+            SBN[Sandbox Pod N<br/>dynamic]
+        end
+        
+        subgraph Data
+            DB[(PostgreSQL<br/>CNPG Cluster)]
+        end
+        
+        subgraph Config
+            CM[ConfigMaps]
+            SEC[TLS Secrets]
+            VSEC[Vault Secrets]
+        end
+        
+        subgraph Security
+            NP[NetworkPolicies]
+            SA[ServiceAccounts]
+            ROLE[Roles]
+            RB[RoleBindings]
+        end
+    end
+    
+    GW --> DB
+    GW --> CM
+    GW --> SEC
+    SB1 --> VSEC
+    SB2 --> VSEC
+    SBN --> VSEC
+    
+    style NS fill:#f0f0f0
+    style Workloads fill:#d4edda
+    style Data fill:#cce5ff
+    style Config fill:#fff3cd
+    style Security fill:#f8d7da
+```
 
-- Gateway pod (StatefulSet)
-- Supervisor sidecar
-- Sandbox pods
-- PostgreSQL (CNPG Cluster or in-namespace Deployment)
-- TLS certificates (cert-manager or certgen)
-- NetworkPolicies
-- RBAC resources
+**Resources per Gateway Namespace**:
+- **Workloads**: Gateway StatefulSet (with Supervisor sidecar), dynamic Sandbox pods
+- **Data**: PostgreSQL (CNPG Cluster) provisioned by ManagedDatabase controller
+- **Config**: ConfigMaps, TLS secrets (cert-manager or certgen), Vault-backed secrets
+- **Security**: NetworkPolicies, ServiceAccounts, Roles, RoleBindings
 
-```
-namespace: openshell-<gateway-name>
-├── Gateway StatefulSet
-├── Supervisor
-├── Sandbox pods (dynamic)
-├── PostgreSQL (CNPG Cluster)
-├── TLS Secrets
-├── ConfigMaps
-├── NetworkPolicies
-└── RBAC (Roles, RoleBindings, ServiceAccounts)
-```
+This namespace-per-gateway strategy provides:
+- Isolation boundary for RBAC and network policies
+- Resource quotas per gateway
+- Self-contained lifecycle (delete namespace = delete gateway)
+- CNPG Cluster co-located with gateway workload
 
 ## Installer Pipeline
 
@@ -160,15 +365,71 @@ Infrastructure SHALL be treated as disposable. Any cluster can be torn down and
 
 ## Monitoring Architecture
 
-Prometheus runs on every managed cluster. Regional aggregation feeds into the global hub.
-
-```
-Managed Cluster (region)          Hub Cluster
-┌─────────────────────┐          ┌─────────────────────┐
-│ Prometheus (local)  │ ──────▶  │ Prometheus (global)  │
-│ Gateway metrics     │          │ Grafana dashboards   │
-│ Node metrics        │          │ Alertmanager         │
-└─────────────────────┘          └─────────────────────┘
+Prometheus runs on every cluster. Metrics flow from ManagedClusters → Cloud Hub → Global Hub.
+
+```mermaid
+graph TB
+    subgraph MC1["ManagedCluster: AWS us-east-1"]
+        P1[Prometheus]
+        M1[Gateway Metrics]
+        N1[Node Metrics]
+        M1 --> P1
+        N1 --> P1
+    end
+
+    subgraph MC2["ManagedCluster: AWS us-west-2"]
+        P2[Prometheus]
+        M2[Gateway Metrics]
+        N2[Node Metrics]
+        M2 --> P2
+        N2 --> P2
+    end
+
+    subgraph MC3["ManagedCluster: IBM us-east"]
+        P3[Prometheus]
+        M3[Gateway Metrics]
+        N3[Node Metrics]
+        M3 --> P3
+        N3 --> P3
+    end
+
+    subgraph CloudAWS["Cloud Hub: AWS"]
+        PA[Prometheus<br/>Cloud Aggregator]
+        GA[Grafana<br/>Cloud Dashboards]
+        AA[Alertmanager]
+        PA --> GA
+        PA --> AA
+    end
+
+    subgraph CloudIBM["Cloud Hub: IBM"]
+        PI[Prometheus<br/>Cloud Aggregator]
+        GI[Grafana<br/>Cloud Dashboards]
+        AI[Alertmanager]
+        PI --> GI
+        PI --> AI
+    end
+
+    subgraph Global["Global Hub"]
+        PG[Prometheus<br/>Global Aggregator]
+        GG[Grafana<br/>Single Pane of Glass]
+        AG[Alertmanager]
+        PG --> GG
+        PG --> AG
+    end
+
+    P1 -->|Remote Write| PA
+    P2 -->|Remote Write| PA
+    P3 -->|Remote Write| PI
+
+    PA -->|Federation| PG
+    PI -->|Federation| PG
+
+    style MC1 fill:#d4edda
+    style MC2 fill:#d4edda
+    style MC3 fill:#d4edda
+    style CloudAWS fill:#fff3cd
+    style CloudIBM fill:#fff3cd
+    style Global fill:#e1f5ff
 ```
 
 ### Requirements
@@ -229,12 +490,17 @@ gitops-repo/
 
 | Decision | Rationale |
 |----------|-----------|
-| One OpenShift hub per cloud | Reduces cross-cloud latency; hub needs OpenShift for Routes and SCC |
-| Managed clusters can be standard K8s | Maximizes deployment flexibility; only the hub needs OpenShift |
-| CNPG over cloud-managed databases | Kubernetes-native lifecycle, portable across clouds, no vendor lock-in |
+| Three-tier topology | Separates concerns: Global (identity root), Cloud Hub (operations), ManagedCluster (workloads) |
+| Cloud Hub as primary unit | One HA instance per cloud runs API/control-plane/database; cloud isolation for latency and compliance |
+| Full operator stack on all tiers | Every cluster has ArgoCD, Vault, Keycloak, CNPG, Prometheus - but serves different purposes per tier |
+| Federated Keycloak chain | RH SSO → Global → Cloud Hub → ManagedCluster - identity flows down, authentication bubbles up |
+| Vault per tier with distinct purposes | Cloud Hub Vault: service secrets; ManagedCluster Vault: gateway keystores |
+| CNPG on all clusters | Kubernetes-native lifecycle, portable across clouds, no vendor lock-in |
+| PostgreSQL on Cloud Hub as source of truth | All Fleet/Gateway/ManagedCluster resource state lives in Cloud Hub database |
+| ManagedClusters can be standard K8s | Maximizes deployment flexibility; only hubs need OpenShift |
 | Tekton over bash scripts | Deterministic, auditable, cattle-not-pets infrastructure |
-| ArgoCD for GitOps | Declarative cluster state, drift detection, multi-cluster support |
-| Vault for secrets | Centralized rotation, cloud-native drivers, audit trail |
-| Prometheus on all clusters | Uniform metrics pipeline, regional aggregation to global hub |
-| Namespace-per-gateway | Isolation boundary for RBAC, NetworkPolicy, and resource quotas |
+| ArgoCD on Cloud Hubs | Each Cloud Hub ArgoCD reconciles its own ManagedClusters from Git |
+| Prometheus metrics hierarchy | ManagedCluster → Cloud Hub → Global Hub; supports cloud-level and cross-cloud dashboards |
+| Namespace-per-gateway | Isolation boundary for RBAC, NetworkPolicy, resource quotas, and CNPG Cluster |
 | Terraform for provisioning | IaC for VPC, subnet, and cluster lifecycle; cloud-agnostic |
+| Gateway OIDC clients on ManagedCluster | openshell CLI authenticates against Keycloak where the gateway runs (low latency) |

From e7d8dbaddefd9e42d9f7e510bb4f3d0e64d80b06 Mon Sep 17 00:00:00 2001
From: user <u@example.com>
Date: Sat, 15 Aug 2026 11:19:16 -0400
Subject: [PATCH 03/15] docs: resolve open questions regarding IBM Gateway API
 availability, VPC LB scope, and control-plane reconciler bug

---
 specs/platform/global-architecture.spec.md | 269 +++++++++++++++++++++
 1 file changed, 269 insertions(+)

diff --git a/specs/platform/global-architecture.spec.md b/specs/platform/global-architecture.spec.md
index 8d92083e..41c058cb 100644
--- a/specs/platform/global-architecture.spec.md
+++ b/specs/platform/global-architecture.spec.md
@@ -250,6 +250,275 @@ Traffic destined for the actual managed gRPC gateways bypasses the default OpenS
 
 This architecture allows the control plane to dynamically route traffic for new gateways without needing to provision individual load balancers or DNS records per tenant.
 
+### Tenant Gateway Ingress — Reference Implementation (verified on AWS)
+
+> The following was captured live from the `hypershell-stage` deployment on the
+> ROSA cluster `hcmais01ue1` (2026-08-15). It is the source-of-truth
+> implementation that other clouds (IBM Cloud) must reach parity with. Values
+> such as ELB hostnames and tenant hashes are environment-specific.
+
+#### Component chain
+
+```mermaid
+graph LR
+    CLI[openshell CLI] -->|"gw-...devshift.net:443"| DNS[Route53 wildcard CNAME]
+    DNS --> ELB[Cloud LB<br/>AWS Classic ELB]
+    ELB --> IGW[Istio ingress gateway pod<br/>openshift-ingress]
+    IGW -->|match GRPCRoute host| SVC[Service openshell-gateway:8080]
+    SVC --> POD[Gateway StatefulSet pod]
+
+    style DNS fill:#fff3cd
+    style ELB fill:#f8d7da
+    style IGW fill:#cce5ff
+    style SVC fill:#d4edda
+```
+
+The Gateway API implementation is **OpenShift's built-in support**, not a manual
+Istio install. The Cluster Ingress Operator (CIO) installs and manages `istiod`
+(Sail operator) and reconciles the `openshift-default` GatewayClass.
+
+- **GatewayClass:** `openshift-default`
+- **Controller:** `openshift.io/gateway-controller/v1` ("Handled by Istio controller"; `istiod v1.28.5` installed by CIO)
+- **Gateway → LoadBalancer:** creating the `Gateway` causes the operator to create an Istio ingress `Deployment` + `Service` (`type: LoadBalancer`); the cloud CCM then provisions the external LB (AWS Classic ELB observed: `a1a663034da9843c3944de9cbdaceb98-536422505.us-east-1.elb.amazonaws.com`).
+
+#### Manifest 1 — Shared Gateway (one per cluster, in `openshift-ingress`)
+
+Created/owned by the control plane (`app.kubernetes.io/managed-by: hypershell-control-plane`).
+
+```yaml
+apiVersion: gateway.networking.k8s.io/v1
+kind: Gateway
+metadata:
+  name: openshell-grpc-gateway
+  namespace: openshift-ingress
+  labels:
+    app.kubernetes.io/name: openshell
+    app.kubernetes.io/component: gateway
+    app.kubernetes.io/managed-by: hypershell-control-plane
+    hypershell.redhat.io/managed: "true"
+    istio.io/rev: openshift-gateway
+spec:
+  gatewayClassName: openshift-default
+  listeners:
+    - name: grpc
+      hostname: "*.openshell.stage.devshift.net"   # per-env base domain
+      port: 443
+      protocol: HTTPS
+      allowedRoutes:
+        kinds:
+          - group: gateway.networking.k8s.io
+            kind: GRPCRoute
+        namespaces:
+          from: All                                 # tenant namespaces attach cross-namespace
+      tls:
+        mode: Terminate
+        certificateRefs:
+          - kind: Secret
+            name: wildcard-openshell-stage-devshift-tls
+```
+
+#### Manifest 2 — Per-tenant GRPCRoute (control plane creates one per gateway)
+
+Lives in the tenant namespace (`openshell-<tenant-hash>`), attaches cross-namespace to the shared Gateway's `grpc` listener.
+
+```yaml
+apiVersion: gateway.networking.k8s.io/v1
+kind: GRPCRoute
+metadata:
+  name: openshell-gateway
+  namespace: openshell-<tenant-hash>
+  labels:
+    app.kubernetes.io/name: openshell
+    app.kubernetes.io/component: gateway
+    app.kubernetes.io/managed-by: hypershell-control-plane
+    hypershell.redhat.io/managed: "true"
+spec:
+  hostnames:
+    - "gw-openshell-<tenant-hash>.openshell.stage.devshift.net"
+  parentRefs:
+    - group: gateway.networking.k8s.io
+      kind: Gateway
+      name: openshell-grpc-gateway
+      namespace: openshift-ingress
+      sectionName: grpc
+  rules:
+    - backendRefs:
+        - kind: Service
+          name: openshell-gateway
+          port: 8080
+          weight: 1
+```
+
+#### Manifest 3 — Wildcard TLS certificate (cert-manager, in `openshift-ingress`)
+
+```yaml
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: wildcard-openshell-stage-devshift
+  namespace: openshift-ingress
+spec:
+  secretName: wildcard-openshell-stage-devshift-tls
+  dnsNames:
+    - "*.openshell.stage.devshift.net"
+  issuerRef:
+    kind: ClusterIssuer
+    name: letsencrypt-devshiftnet-dns
+```
+
+#### Manifest 4 — ClusterIssuer (ACME / Let's Encrypt, DNS-01 via Route53)
+
+The `devshift.net` zone is centrally hosted in AWS Route53, so the DNS-01 solver
+is Route53 **regardless of which cloud the cluster runs in**. This is the key that
+lets an IBM Cloud cluster obtain a `*.devshift.net` certificate without any
+IBM-native DNS integration.
+
+```yaml
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-devshiftnet-dns
+spec:
+  acme:
+    solvers:
+      - selector:
+          dnsZones: ["devshift.net"]
+        dns01:
+          route53:
+            region: global
+            hostedZoneID: Z05758033SZ8IESGUOY8E
+            accessKeyIDSecretRef:
+              name: certmgr-<cluster>-devshift-net-sa
+              key: aws_access_key_id
+            secretAccessKeySecretRef:
+              name: certmgr-<cluster>-devshift-net-sa
+              key: aws_secret_access_key
+```
+
+#### DNS record
+
+There is **no `external-dns` controller** on the AWS cluster; the Gateway reports
+`DNS management policy is set to Unmanaged`. The wildcard record is a static
+Route53 entry:
+
+```
+*.openshell.stage.devshift.net.  CNAME  <cloud-LB-hostname>
+```
+
+resolving to the Gateway's provisioned LB. It must be created (once per cluster)
+after the LB hostname is known.
+
+#### Verified component summary
+
+| Concern | AWS (verified) | Notes |
+|---------|----------------|-------|
+| Gateway API impl | `openshift-default` GatewayClass, CIO-managed Istio | Not a manual Istio install |
+| Shared Gateway | `openshell-grpc-gateway` / `openshift-ingress` | one per cluster |
+| Listener | `*.openshell.stage.devshift.net`, 443, HTTPS, TLS Terminate | GRPCRoute from All namespaces |
+| External LB | AWS Classic ELB (auto-provisioned) | via `Service type: LoadBalancer` + CCM |
+| Per-tenant route | `GRPCRoute` `openshell-gateway`, host `gw-openshell-<hash>...`, backend `Service openshell-gateway:8080` | created by control plane |
+| Wildcard cert | cert-manager `Certificate` → secret `wildcard-openshell-stage-devshift-tls` | 90-day Let's Encrypt, auto-renew |
+| Issuer | `letsencrypt-devshiftnet-dns` (ACME, DNS-01, Route53 zone `devshift.net`) | cloud-agnostic |
+| DNS record | static Route53 wildcard CNAME → LB | no external-dns |
+
+### IBM Cloud Parity Plan
+
+Goal: reproduce the exact tenant-gateway ingress path on the IBM Cloud Cloud Hub
+(ROKS, VPC Gen2) so the control plane behaves identically across clouds. The
+**only cloud-specific difference is the load balancer implementation** — the
+Gateway API objects, cert-manager objects, and control-plane config are identical.
+
+**Confirmed gap (verified on the IBM `hypershell-cluster`, 2026-08-15):** Gateway
+API is **not installed** — `kubectl api-resources` returns no
+`gateway.networking.k8s.io` types; tenant gateways currently fall back to
+OpenShift `Route` objects (`passthrough`) on the IBM-managed
+`*.containers.appdomain.cloud` domain. This is the configuration drift to close.
+
+Steps (parity, not migration):
+
+1. **Enable the OpenShift Gateway API feature.** Confirm the ROKS OpenShift
+   version ships the CIO-managed Gateway API (`openshift-default` GatewayClass) and
+   enable it. If the version predates GA support, this is a blocker — see open
+   questions.
+2. **Choose the IBM base domain.** e.g. `*.openshell.<ibm-env>.devshift.net`
+   (a distinct subdomain from AWS, still under the Route53 `devshift.net` zone).
+3. **Seed the Route53 credential secret** `certmgr-<cluster>-devshift-net-sa` in
+   `openshift-ingress` and apply the **same** `ClusterIssuer`
+   (`letsencrypt-devshiftnet-dns`, Route53 DNS-01). No IBM DNS integration needed.
+4. **Apply the wildcard `Certificate`** for the IBM base domain
+   (Manifest 3 pattern, new `dnsNames` + `secretName`).
+5. **Apply the shared `Gateway`** (Manifest 1 pattern) with `gatewayClassName:
+   openshift-default` and the IBM hostname. IBM Cloud CCM provisions a **VPC
+   Load Balancer** (`*.lb.appdomain.cloud`) in place of the AWS ELB —
+   automatically, no manifest change.
+6. **Create the wildcard DNS record** `*.openshell.<ibm-env>.devshift.net CNAME
+   <vpc-lb-hostname>` in Route53 once the LB hostname is known.
+7. **Point the control plane at the Gateway.** Set the gateway/domain config so
+   the IBM control plane emits `GRPCRoute`s (Manifest 2) instead of `Route`s
+   (see Requirement: Control Plane Ingress Mode below).
+8. **Verify:** `Gateway` `Programmed=True` with an address; a test tenant's
+   `GRPCRoute` reports `Accepted`/`ResolvedRefs`; `openshell` CLI connects over
+   `gw-...<ibm-env>.devshift.net:443`.
+
+### Requirements
+
+#### Requirement: Tenant Gateway Ingress via Gateway API
+
+Tenant gateway traffic SHALL be routed through the Kubernetes Gateway API using a
+single shared `Gateway` (`openshell-grpc-gateway`) in the `openshift-ingress`
+namespace on every cluster that hosts gateways. Tenant gateways SHALL NOT use
+OpenShift `Route` objects for data-plane gRPC traffic.
+
+##### Scenario: Gateway provisioning creates a GRPCRoute
+
+- GIVEN a cluster with the `openshift-default` GatewayClass and the shared `openshell-grpc-gateway`
+- WHEN the control plane provisions a new tenant gateway
+- THEN it SHALL create a `GRPCRoute` in the tenant namespace with a `parentRef` to `openshell-grpc-gateway` (`sectionName: grpc`)
+- AND the route hostname SHALL be `gw-<tenant>.<base-domain>` under the shared listener's wildcard
+- AND it SHALL NOT create an OpenShift `Route` for gRPC data-plane traffic
+
+#### Requirement: Cloud-Agnostic Gateway Manifests
+
+The `Gateway`, `GRPCRoute`, `Certificate`, and `ClusterIssuer` manifests SHALL be
+identical across clouds except for the per-environment base domain and TLS secret
+name. The external load balancer SHALL be provisioned automatically by the cloud's
+CCM from the operator-created `Service type: LoadBalancer`; no per-cloud load
+balancer manifest SHALL be required.
+
+##### Scenario: IBM Cloud provisions a VPC LB from the same Gateway
+
+- GIVEN the shared `Gateway` manifest applied on a ROKS VPC Gen2 cluster
+- WHEN the Cluster Ingress Operator reconciles it
+- THEN IBM Cloud CCM SHALL provision a VPC Load Balancer (`*.lb.appdomain.cloud`)
+- AND the `Gateway` status SHALL report `Programmed=True` with the LB hostname as its address
+
+#### Requirement: Wildcard TLS via Central Route53 DNS-01
+
+Wildcard certificates for the `devshift.net` base domain SHALL be issued by
+cert-manager using the ACME DNS-01 challenge against the central Route53
+`devshift.net` hosted zone, independent of the cluster's cloud provider.
+
+#### Requirement: Control Plane Ingress Mode
+
+The control plane's ingress mode (Gateway API vs OpenShift Route) SHALL be
+configuration-driven, not cloud-hardcoded, so the same binary produces `GRPCRoute`s
+on any cluster where the shared Gateway and GatewayClass are present.
+
+### Open Questions (for implementer review)
+
+1. **[RESOLVED] ROKS Gateway API availability.** Verified on 2026-08-15: The `gateway.networking.k8s.io` CRDs do **not** exist by default on the current IBM cluster. The Gateway API must be explicitly enabled via the Cluster Ingress Operator or a FeatureGate before parity can be achieved.
+2. **IBM base domain.** What exact subdomain — e.g. `openshell.ibm-stage.devshift.net`?
+   Confirm it is delegated within the Route53 `devshift.net` zone.
+3. **[RESOLVED - BUG] Control-plane config surface.** Verified on 2026-08-15: The Go code reads `GATEWAY_API_BASE_DOMAIN` and `GATEWAY_API_GATEWAY_CLASS`. However, it **completely ignores** the gateway name and namespace config. `internal/gateway/reconciler.go` currently hardcodes the `GRPCRoute` `parentRefs` to `name: openshell-gateway` inside the tenant's own namespace. This is a severe bug that provisions one Load Balancer per tenant. This code must be patched to respect `GATEWAY_API_GATEWAY_NAME` and `GATEWAY_API_GATEWAY_NAMESPACE` to use the shared ingress gateway.
+4. **DNS record creation.** Reproduce the AWS pattern (static Route53 wildcard
+   CNAME, no external-dns), or introduce `external-dns` to automate it on IBM?
+5. **Route53 credentials on IBM.** Provision the `certmgr-<cluster>-devshift-net-sa`
+   secret (IAM user scoped to the `devshift.net` zone) on the IBM cluster.
+6. **[RESOLVED] VPC LB scope.** Verified on 2026-08-15: The default OpenShift router on this IBM cluster uses a public VPC Load Balancer (`service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type: public`). We will match this public exposure for the Gateway API VPC LB.
+7. **Legacy Route-based gateways.** The AWS cluster still has older tenants on
+   `Route`/`passthrough` under `*.apps.rosa...`. Is a migration of existing tenants
+   in scope, or parity for new tenants only?
+
 ## Tooling Stack
 
 | Component | Tool | Purpose |

From 218f82b4fe735a50c7f3f6a36908bce6af680e73 Mon Sep 17 00:00:00 2001
From: user <u@example.com>
Date: Sat, 15 Aug 2026 11:29:20 -0400
Subject: [PATCH 04/15] docs: add shared gateway rationale to Design Decisions
 table

---
 specs/platform/global-architecture.spec.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/specs/platform/global-architecture.spec.md b/specs/platform/global-architecture.spec.md
index 41c058cb..272f5bcb 100644
--- a/specs/platform/global-architecture.spec.md
+++ b/specs/platform/global-architecture.spec.md
@@ -773,3 +773,4 @@ gitops-repo/
 | Namespace-per-gateway | Isolation boundary for RBAC, NetworkPolicy, resource quotas, and CNPG Cluster |
 | Terraform for provisioning | IaC for VPC, subnet, and cluster lifecycle; cloud-agnostic |
 | Gateway OIDC clients on ManagedCluster | openshell CLI authenticates against Keycloak where the gateway runs (low latency) |
+| Shared Ingress Gateway for Tenant gRPC | A wildcard DNS record (`*.domain`) can only resolve to a single Load Balancer. A per-tenant gateway model (1 LB per tenant) fundamentally breaks wildcard routing, requiring per-tenant DNS automation and cert management. A shared Gateway allows N tenants to securely share 1 LB, 1 wildcard cert, and 1 static DNS record via `GRPCRoute` attachments. |

From 3734b349dcaec13cae5735395a721fdf2d6f20ab Mon Sep 17 00:00:00 2001
From: user <u@example.com>
Date: Sat, 15 Aug 2026 13:55:47 -0400
Subject: [PATCH 05/15] docs(skills): add IBM Cloud Hub provisioning and
 shared-gateway ingress skills

Add two deploy skills and cross-link them from deploy-cluster and CLAUDE.md:

- cloud-hub-ingress-bootstrap: cloud-agnostic shared Gateway + wildcard DNS/TLS
  bootstrap (AWS reference / IBM parity). Encodes the OCP >= 4.19 requirement for
  the built-in openshift-default GatewayClass and the nlb-dns DNS+TLS path.
- ibm-cluster: ROKS VPC Gen2 provisioning mirroring the reference cluster, the
  cluster-create command (COS CRN required, not GUID), and a registry-storage
  decision table (emptyDir / PVC / COS) with PVC as the chosen persistent backend.
- deploy-cluster: add Cloud-Hub parameter overrides (registry host, ibmc-vpc-block
  storage class) and scope note pointing at the ingress bootstrap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 CLAUDE.md                                     |   3 +-
 .../cloud-hub-ingress-bootstrap/SKILL.md      | 220 ++++++++++++++++++
 skills/deploy/deploy-cluster/SKILL.md         |  21 ++
 skills/deploy/ibm-cluster/SKILL.md            | 175 ++++++++++++++
 4 files changed, 418 insertions(+), 1 deletion(-)
 create mode 100644 skills/deploy/cloud-hub-ingress-bootstrap/SKILL.md
 create mode 100644 skills/deploy/ibm-cluster/SKILL.md

diff --git a/CLAUDE.md b/CLAUDE.md
index c7a386a2..7ea358d5 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -20,7 +20,7 @@ checks manually with `make check`.
 - `components/control-plane/` - Go service, watches API server via gRPC and reconciles gateway resources into K8s
 - `packages/gateway-management-ui/` - Private reusable React package containing canonical gateway management workflows
 - `specs/` - Desired state of the system ([platform](specs/platform/), [standards](specs/standards/))
-- `skills/` - Agent skills: [reconcile](skills/build/reconcile), [spec](skills/plan/spec), [full-stack-pipeline](skills/build/full-stack-pipeline), [dev-cluster](skills/build/dev-cluster), [review](skills/review/review-guidance), [amber-review](skills/review/amber-review), [ui-standards](skills/review/ui-standards), [tooling](skills/tooling/)
+- `skills/` - Agent skills: [reconcile](skills/build/reconcile), [spec](skills/plan/spec), [full-stack-pipeline](skills/build/full-stack-pipeline), [dev-cluster](skills/build/dev-cluster), [ibm-cluster](skills/deploy/ibm-cluster), [deploy-cluster](skills/deploy/deploy-cluster), [cloud-hub-ingress-bootstrap](skills/deploy/cloud-hub-ingress-bootstrap), [review](skills/review/review-guidance), [amber-review](skills/review/amber-review), [ui-standards](skills/review/ui-standards), [tooling](skills/tooling/)
 - `apm.yml` - APM manifest declaring upstream skill dependencies
 
 ## Key Files
@@ -67,6 +67,7 @@ state (coverage summary, gap table, wave plan), then executes waves to close gap
 Idempotent: safe to run repeatedly.
 
 Support skills available at any point:
+- `/cloud-hub-ingress-bootstrap` -- one-time shared Gateway + wildcard DNS/TLS per Cloud Hub (AWS/IBM)
 - `/review-guidance` -- PR review checklist
 - `/amber-review` -- Amber agent comprehensive code review
 - `/ui-standards` -- UI/UX audit or intent-driven design guidance
diff --git a/skills/deploy/cloud-hub-ingress-bootstrap/SKILL.md b/skills/deploy/cloud-hub-ingress-bootstrap/SKILL.md
new file mode 100644
index 00000000..67b0f1d3
--- /dev/null
+++ b/skills/deploy/cloud-hub-ingress-bootstrap/SKILL.md
@@ -0,0 +1,220 @@
+---
+name: cloud-hub-ingress-bootstrap
+description: >
+  Bootstrap the shared tenant-gateway wildcard ingress on a Cloud Hub (once per
+  cluster) so the control plane's Gateway API reconciler has a shared Gateway to
+  attach GRPCRoutes to. Cloud-agnostic with AWS (verified reference) and IBM Cloud
+  (ROKS parity) variants. Use when: "wildcard subdomains for gateways", "shared
+  gateway", "GATEWAY_API_GATEWAY_NAME is required", "deploy to IBM Cloud",
+  "tenant gateway ingress", "VPC load balancer for gateways".
+---
+
+# Cloud Hub Ingress Bootstrap
+
+Prerequisite for HyperShell tenant traffic. The control plane's reconciler
+(`components/control-plane/internal/gateway/reconciler.go`) does **not** create a
+`Gateway`; it requires a pre-existing shared one and fails closed
+(`GATEWAY_API_GATEWAY_NAME is required`) until this bootstrap is done. This skill
+stands up that shared Gateway plus its wildcard DNS + TLS so every tenant gets a
+`gw-<tenant>.<base-domain>` subdomain behind a single load balancer.
+
+Run this **once per Cloud Hub / gateway-hosting cluster**, before or alongside the
+platform deploy (`/deploy-cluster`). It is idempotent — re-applying reconciles.
+
+## Why one shared Gateway (not per-tenant)
+
+A wildcard DNS record resolves to exactly **one** LB. A shared `Gateway` gives one
+LB + one wildcard cert + one wildcard DNS record, with hostname-based routing per
+tenant `GRPCRoute`. A per-tenant Gateway would need one LB and one DNS record each
+— incompatible with wildcard subdomains. Authoritative spec + verified manifests:
+[`specs/platform/global-architecture.spec.md`](../../../specs/platform/global-architecture.spec.md)
+(§ "Tenant Gateway Ingress — Reference Implementation" and "IBM Cloud Parity Plan").
+
+## The only cloud-specific difference
+
+Everything is identical across clouds **except the load balancer**, which the
+cloud's CCM auto-provisions from the operator-created `Service type: LoadBalancer`.
+No per-cloud LB manifest is written.
+
+| Concern | AWS (ROSA, verified) | IBM Cloud (ROKS, VPC Gen2) |
+|---------|----------------------|-----------------------------|
+| GatewayClass | `openshift-default` (CIO-managed Istio) | `openshift-default` (verify GA on the ROKS OCP version) |
+| Shared Gateway | `openshell-grpc-gateway` / `openshift-ingress` | identical |
+| LB provisioned by CCM | AWS Classic ELB (`*.elb.amazonaws.com`) | VPC Load Balancer (`*.lb.appdomain.cloud`) |
+| Base domain (example) | `*.openshell.stage.devshift.net` | `*.openshell.<ibm-env>.devshift.net` |
+| TLS secret | `wildcard-openshell-stage-devshift-tls` | `wildcard-openshell-<ibm-env>-devshift-tls` |
+| ClusterIssuer | `letsencrypt-devshiftnet-dns` (Route53 DNS-01) | **same issuer** — Route53 is central |
+| DNS record | static Route53 wildcard CNAME → ELB | static Route53 wildcard CNAME → VPC LB |
+
+Route53 hosts the `devshift.net` zone centrally, so DNS-01 issuance works from any
+cloud with **no** cloud-native DNS integration.
+
+## Per-cluster parameters
+
+Set these once; they drive every manifest below.
+
+```bash
+export BASE_DOMAIN="openshell.stage.devshift.net"      # AWS example; IBM: openshell.<ibm-env>.devshift.net
+export TLS_SECRET="wildcard-openshell-stage-devshift-tls"
+export CERT_NAME="wildcard-openshell-stage-devshift"
+export CLUSTER="hcmai"                                  # used in the Route53 credential secret name
+export ROUTE53_ZONE_ID="Z05758033SZ8IESGUOY8E"         # devshift.net hosted zone
+```
+
+## Step 0: Preflight
+
+```bash
+oc whoami --show-server                                 # confirm the target Cloud Hub
+oc get gatewayclass openshift-default                   # MUST exist; if absent, Gateway API not enabled (see Troubleshooting)
+oc get crd | grep gateway.networking.k8s.io             # expect gateways/grpcroutes/httproutes CRDs
+oc -n openshift-ingress get deployment cert-manager -o name 2>/dev/null \
+  || oc get clusterissuer 2>/dev/null                   # confirm cert-manager present
+```
+
+If `gatewayclass openshift-default` is missing, stop. The built-in, CIO-managed
+Gateway API is **GA only on OCP >= 4.19** — check `oc get clusterversion version
+-o jsonpath='{.status.desired.version}'`. On OCP < 4.19 (e.g. the original IBM
+`hypershell-cluster` on 4.17, confirmed 2026-08-15) this GatewayClass does not
+exist and cannot be safely enabled on a managed control plane; provision a
+>= 4.19 cluster with the [`ibm-cluster`](../ibm-cluster/SKILL.md) skill instead
+(new Cloud Hub `hysh-ibm-01` on 4.21.27 was created for exactly this reason).
+
+## Step 1: Seed the Route53 credential secret
+
+cert-manager solves DNS-01 against Route53 for `devshift.net` regardless of cloud.
+
+```bash
+oc -n openshift-ingress create secret generic certmgr-${CLUSTER}-devshift-net-sa \
+  --from-literal=aws_access_key_id="$AWS_ACCESS_KEY_ID" \
+  --from-literal=aws_secret_access_key="$AWS_SECRET_ACCESS_KEY" \
+  --dry-run=client -o yaml | oc apply -f -
+```
+
+## Step 2: Apply the ClusterIssuer (same on every cloud)
+
+Manifest 4 in the spec. `hostedZoneID` = `$ROUTE53_ZONE_ID`; the two secret refs
+point at `certmgr-${CLUSTER}-devshift-net-sa`. Apply, then:
+
+```bash
+oc get clusterissuer letsencrypt-devshiftnet-dns -o jsonpath='{.status.conditions[*].type}={.status.conditions[*].status}{"\n"}'
+```
+
+## Step 3: Apply the wildcard Certificate
+
+Manifest 3 pattern, with `metadata.name=$CERT_NAME`, `spec.secretName=$TLS_SECRET`,
+`spec.dnsNames=["*.${BASE_DOMAIN}"]`, in `openshift-ingress`. Wait for issuance:
+
+```bash
+oc -n openshift-ingress wait --for=condition=Ready certificate/${CERT_NAME} --timeout=300s
+oc -n openshift-ingress get secret ${TLS_SECRET}     # must exist before the Gateway can Terminate TLS
+```
+
+## Step 4: Apply the shared Gateway
+
+Manifest 1 pattern in `openshift-ingress`: `gatewayClassName: openshift-default`,
+one `grpc` HTTPS/443 listener with `hostname: "*.${BASE_DOMAIN}"`,
+`tls.mode: Terminate` → `certificateRefs: [{kind: Secret, name: $TLS_SECRET}]`,
+`allowedRoutes.namespaces.from: All`. Applying it makes the Cluster Ingress
+Operator create the Istio ingress `Service type: LoadBalancer`; the CCM then
+provisions the cloud LB automatically.
+
+```bash
+oc -n openshift-ingress wait --for=condition=Programmed gateway/openshell-grpc-gateway --timeout=300s
+LB=$(oc -n openshift-ingress get gateway openshell-grpc-gateway -o jsonpath='{.status.addresses[0].value}')
+echo "LB hostname: $LB"     # AWS: *.elb.amazonaws.com  |  IBM: *.lb.appdomain.cloud
+```
+
+## Step 5: Create the wildcard DNS record
+
+There is **no external-dns** — the Gateway reports DNS management `Unmanaged`.
+Create the static Route53 record once, after the LB hostname is known:
+
+```
+*.${BASE_DOMAIN}.   CNAME   $LB
+```
+
+Verify resolution:
+
+```bash
+dig +short "gw-probe.${BASE_DOMAIN}"    # should return the LB / its A records
+```
+
+## Step 6: Point the control plane at the Gateway
+
+Set on the `hypershell-controller` Deployment (see `/deploy-cluster`):
+
+```bash
+oc -n hypershell-api set env deployment/hypershell-controller \
+  GATEWAY_API_GATEWAY_NAME=openshell-grpc-gateway \
+  GATEWAY_API_GATEWAY_NAMESPACE=openshift-ingress \
+  GATEWAY_API_BASE_DOMAIN="${BASE_DOMAIN}"
+oc -n hypershell-api rollout status deployment/hypershell-controller
+```
+
+Defaults in code: `GATEWAY_API_GATEWAY_NAMESPACE` → `openshift-ingress`;
+`GATEWAY_API_GATEWAY_NAME` is **required** (reconcile fails without it).
+
+## Step 7: Verify end to end
+
+```bash
+# provision a test tenant gateway via the API, then:
+NS=$(oc get ns -l hypershell.redhat.io/managed=true -o name | head -1 | cut -d/ -f2)
+oc -n "$NS" get grpcroute openshell-gateway -o jsonpath='{.status.parents[*].conditions[*].type}={.status.parents[*].conditions[*].status}{"\n"}'
+# expect Accepted=True and ResolvedRefs=True
+openshell login "gw-${NS}.${BASE_DOMAIN}:443"   # CLI connects over the wildcard subdomain
+```
+
+## Cloud variants at a glance
+
+### AWS (reference — already live on `hcmai` / hypershell-stage)
+- Verified working; this skill is derived from it. LB is an AWS Classic ELB.
+- Nothing to change beyond the parameters above.
+
+### IBM Cloud / ROKS (parity target — `hypershell-cluster`)
+1. **Gateway API enablement is the gap** — confirm the ROKS OCP version ships the
+   CIO-managed Gateway API and `openshift-default` GatewayClass; enable it. If the
+   version predates GA, that is a blocker (raise before proceeding).
+2. Use `BASE_DOMAIN=openshell.<ibm-env>.devshift.net` and matching `TLS_SECRET` /
+   `CERT_NAME` (distinct from AWS, still under the Route53 `devshift.net` zone).
+3. Steps 1–7 are otherwise identical. Step 4 yields a **VPC Load Balancer**
+   (`*.lb.appdomain.cloud`) instead of an ELB — no manifest change.
+4. Tenant traffic currently falls back to OpenShift `Route` (`passthrough`) on
+   `*.containers.appdomain.cloud`; completing this bootstrap + Step 6 closes that
+   drift so IBM emits `GRPCRoute`s like AWS.
+
+## Troubleshooting
+
+### `GATEWAY_API_GATEWAY_NAME is required` in controller logs
+The shared Gateway isn't wired up. Complete Steps 4 and 6.
+
+### Gateway API not installed (no `openshift-default` GatewayClass)
+`oc get gatewayclass` empty and `oc get crd | grep gateway.networking.k8s.io`
+returns nothing. The built-in CIO-managed Gateway API is **GA only on OCP >= 4.19**.
+Check the version; if < 4.19 (verified on IBM `hypershell-cluster` @ 4.17.56,
+2026-08-15), do **not** try to enable a Tech-Preview feature gate on a managed
+control plane — provision a >= 4.19 cluster via the
+[`ibm-cluster`](../ibm-cluster/SKILL.md) skill. Treat as a prerequisite blocker,
+not a runtime error.
+
+### Gateway stuck `Programmed=False`
+Usually the TLS secret is missing — confirm Step 3 finished (`$TLS_SECRET` exists
+in `openshift-ingress`) before the Gateway can terminate TLS.
+
+### Certificate never becomes Ready
+Check the DNS-01 challenge: `oc -n openshift-ingress get challenges,orders`. Common
+causes: wrong `hostedZoneID`, or the `certmgr-${CLUSTER}-devshift-net-sa` creds lack
+Route53 write access to the `devshift.net` zone.
+
+### GRPCRoute `Accepted=False`
+Hostname outside the listener wildcard, or `sectionName` mismatch. The route's
+`parentRef` must be `{name: openshell-grpc-gateway, namespace: openshift-ingress,
+sectionName: grpc}` and its hostname must fall under `*.${BASE_DOMAIN}`.
+
+## Teardown (per cluster)
+
+```bash
+oc -n openshift-ingress delete gateway openshell-grpc-gateway
+oc -n openshift-ingress delete certificate ${CERT_NAME}
+oc -n openshift-ingress delete secret ${TLS_SECRET}
+# then remove the wildcard Route53 record
+```
diff --git a/skills/deploy/deploy-cluster/SKILL.md b/skills/deploy/deploy-cluster/SKILL.md
index c7bce8d5..ed8edee7 100644
--- a/skills/deploy/deploy-cluster/SKILL.md
+++ b/skills/deploy/deploy-cluster/SKILL.md
@@ -9,6 +9,27 @@ description: >
 
 # HyperShell OpenShift Deployment
 
+> **Scope:** this skill deploys the **platform services** (API server, controller,
+> PostgreSQL) and is cloud-agnostic across OpenShift distributions (ROSA, ROKS,
+> self-managed). It does **not** stand up tenant-gateway ingress. The shared
+> `Gateway` + wildcard DNS/TLS that tenant traffic needs is a separate one-time
+> per-cluster bootstrap — see [`cloud-hub-ingress-bootstrap`](../cloud-hub-ingress-bootstrap/SKILL.md).
+> Without it, the controller runs but every gateway reconcile fails with
+> `GATEWAY_API_GATEWAY_NAME is required`.
+
+## Cloud-Hub Parameter Overrides
+
+The steps below use AWS/ROSA defaults. On other clouds, override only these values;
+the `oc`-based flow is otherwise identical.
+
+| Parameter | AWS / ROSA | IBM Cloud / ROKS |
+|-----------|------------|-------------------|
+| Cluster login | `oc login` | `ibmcloud login` → `ibmcloud oc cluster config -c <cluster>` → `oc` |
+| Internal registry route | `...elb/openshift-image-registry` host | `default-route-openshift-image-registry...appdomain.cloud` |
+| Namespace | `hypershell-api` | `hypershell-api` (same) |
+| PostgreSQL storage class | cluster default | `ibmc-vpc-block-10iops-tier` (pin on the postgres PVC) |
+| Tenant-gateway ingress | via `cloud-hub-ingress-bootstrap` (ELB) | via `cloud-hub-ingress-bootstrap` (VPC LB) |
+
 ## Platform Components
 
 | Component | Image | Deployment | Ports |
diff --git a/skills/deploy/ibm-cluster/SKILL.md b/skills/deploy/ibm-cluster/SKILL.md
new file mode 100644
index 00000000..c27f8e1d
--- /dev/null
+++ b/skills/deploy/ibm-cluster/SKILL.md
@@ -0,0 +1,175 @@
+---
+name: ibm-cluster
+description: >
+  Provision (and later retire) an IBM Cloud ROKS OpenShift cluster on VPC Gen2 to
+  serve as a HyperShell Cloud Hub. Ensures a version with the built-in, CIO-managed
+  Gateway API (OCP >= 4.19) so tenant-gateway manifests match the AWS reference.
+  Use when: "create IBM cluster", "new ROKS cluster", "provision cloud hub on IBM",
+  "upgrade IBM OpenShift", "hysh-ibm".
+---
+
+# IBM Cloud (ROKS) Cluster Provisioning
+
+Stands up a VPC Gen2 OpenShift cluster on IBM Cloud as a Cloud Hub. After it is
+`normal`, deploy platform services with [`deploy-cluster`](../deploy-cluster/SKILL.md)
+and tenant ingress with [`cloud-hub-ingress-bootstrap`](../cloud-hub-ingress-bootstrap/SKILL.md).
+
+## Why the OpenShift version matters (critical)
+
+The tenant-gateway ingress uses the Kubernetes Gateway API. The **built-in,
+Cluster-Ingress-Operator-managed** Gateway API (`openshift-default` GatewayClass,
+matching the AWS reference) is **GA only on OpenShift >= 4.19**. On older ROKS
+(e.g. 4.17) you would have to hand-install OSSM 3 / Sail and accept a divergent
+`istio` GatewayClass. **Always provision >= 4.19** (IBM default at time of writing:
+`4.21.27_openshift`). ROKS control-plane feature sets can be `CustomNoUpgrade`,
+which blocks in-place upgrades — so getting a newer version means a new cluster,
+not an upgrade.
+
+## Prerequisites
+
+- `ibmcloud` CLI logged in (`ibmcloud login`; suggest `! ibmcloud login` in-session)
+- `container-service` plugin (`ibmcloud plugin install container-service`)
+- A target resource group (`ibmcloud target -g <group>`)
+- An existing VPC + subnet in the target zone, and a Cloud Object Storage instance
+  (required for the ROKS internal registry)
+
+## Step 1: Discover parameters (mirror an existing reference cluster)
+
+```bash
+ibmcloud target -g Default
+ibmcloud ks versions --show-version OpenShift              # pick >= 4.19 (default is fine)
+
+# Mirror the reference cluster's shape:
+ibmcloud ks cluster get     --cluster <reference> | grep -iE "resource group|vpc|zone"
+ibmcloud ks worker-pool get --cluster <reference> --worker-pool default | grep -iE "flavor|vpc"
+ibmcloud ks subnets --provider vpc-gen2 --vpc-id <vpc-id> --zone <zone>
+ibmcloud resource service-instances --service-name cloud-object-storage   # reuse or note the COS name
+ibmcloud resource service-instance <cos-name> --output json | grep '"crn"'   # need the CRN, not the GUID
+```
+
+A COS instance is a required `--cos-instance` flag value, but the registry does not
+actually end up COS-backed on this account — see "Internal registry storage" below
+before assuming you need COS or an IAM authorization for it.
+
+Reference values captured for `hysh-ibm-01` (mirrors `hypershell-cluster`, 2026-08-15):
+
+| Parameter | Value |
+|-----------|-------|
+| Version | `4.21.27_openshift` |
+| Resource group | `Default` |
+| Zone | `us-east-1` |
+| VPC | `r014-be56e5de-5cd9-493f-8ac2-149791cdc58b` |
+| Subnet | `hypershell-subnet-1` / `0757-cacfbdee-1d22-444c-8ce5-5eff35c43faf` |
+| Flavor | `bx2.4x16` |
+| Workers / zone | `2` |
+| COS instance | `hypershell-cos` / guid `e674d660-110e-49a2-94d5-6a8e7ef5fcd1` |
+
+## Step 2: Create the cluster
+
+```bash
+ibmcloud ks cluster create vpc-gen2 \
+  --name hysh-ibm-01 \
+  --version 4.21.27_openshift \
+  --zone us-east-1 \
+  --vpc-id r014-be56e5de-5cd9-493f-8ac2-149791cdc58b \
+  --subnet-id 0757-cacfbdee-1d22-444c-8ce5-5eff35c43faf \
+  --flavor bx2.4x16 \
+  --workers 2 \
+  --cos-instance "crn:v1:bluemix:public:cloud-object-storage:global:a/dca8e7b41db847da9e58bf43e92a7ccf:e674d660-110e-49a2-94d5-6a8e7ef5fcd1::"
+```
+
+`--cos-instance` requires the **CRN** (the bare GUID fails with `E4acb "could not
+find the specified cloud object storage instance"`). Provisioning is asynchronous
+(~30–60 min) and incurs cost — confirm before running.
+
+A `Ece8a: Could not create a bucket in your cloud object storage instance` warning
+is expected here and is **not** blocking — the registry falls back to `emptyDir`.
+See "Internal registry storage" below to choose a persistent backend instead.
+
+## Step 3: Watch until `normal`
+
+```bash
+ibmcloud ks cluster get --cluster hysh-ibm-01 | grep -iE "state|status|ingress"
+ibmcloud ks workers    --cluster hysh-ibm-01
+```
+
+Wait for cluster `State: normal`, all workers `Normal`, and `Ingress Status: healthy`
+(the ingress subdomain + IBM-managed wildcard cert appear only once ingress is up).
+
+## Step 4: Get kubeconfig and verify Gateway API is built in
+
+```bash
+ibmcloud ks cluster config --cluster hysh-ibm-01
+oc get clusterversion version -o jsonpath='{.status.desired.version}{"\n"}'   # expect 4.21.x
+oc get gatewayclass                                                          # expect openshift-default
+oc get crd | grep gateway.networking.k8s.io                                  # gateways/grpcroutes present
+```
+
+If `openshift-default` GatewayClass is present, no OSSM 3 / Sail install is needed —
+proceed straight to `cloud-hub-ingress-bootstrap`.
+
+## Step 5: Hand off
+
+1. `deploy-cluster` — platform services (API/controller/PostgreSQL). Use its
+   Cloud-Hub Parameter Overrides (registry host, storage class `ibmc-vpc-block-*`).
+2. `cloud-hub-ingress-bootstrap` — shared Gateway + wildcard DNS/TLS.
+
+## Internal registry storage (COS is NOT required)
+
+`--cos-instance` is a required create flag, but on this account the ROKS registry
+does **not** actually end up COS-backed. Verified on the reference cluster
+`hypershell-cluster` (2026-08-15):
+
+```bash
+oc get configs.imageregistry.operator.openshift.io cluster -o jsonpath='{.spec.storage}{"\n"}'
+# -> {"emptyDir":{},"managementState":"Managed"}    # ephemeral, not COS
+oc -n openshift-image-registry get secret image-registry-private-configuration -o jsonpath='{.data}'
+# -> empty
+```
+
+So the `Ece8a: Could not create a bucket …` warning at create time is **not
+blocking** — the registry falls back to `emptyDir` and image pushes for
+`deploy-cluster` still work (that is how the reference cluster runs today). Pick a
+registry storage backend deliberately:
+
+| Option | Persistence | Setup | When |
+|--------|-------------|-------|------|
+| `emptyDir` (default fallback) | ephemeral — images lost if the registry pod restarts | none | matches reference; fine for demo/dev |
+| **PVC (`ibmc-vpc-block-*`)** | persistent | patch registry `spec.storage.pvc` (RWO, single replica) | **recommended** for a persistent Cloud Hub; no COS/IAM dependency |
+| COS-backed | persistent | needs a Kubernetes Service → COS IAM authorization | only if you want object-store backing |
+
+PVC backend (recommended, self-contained) — **chosen for `hysh-ibm-01`**:
+
+```bash
+oc patch configs.imageregistry.operator.openshift.io cluster --type merge -p \
+  '{"spec":{"storage":{"pvc":{"claim":""}},"rolloutStrategy":"Recreate","replicas":1}}'
+# creates image-registry-storage PVC on the default ibmc-vpc-block storage class
+oc -n openshift-image-registry get pvc image-registry-storage        # verify Bound
+oc get configs.imageregistry.operator.openshift.io cluster -o jsonpath='{.spec.storage}{"\n"}'
+```
+
+Apply this once the cluster is `normal` and before `deploy-cluster` Step 3
+(image push), so the registry is persistent from the first push.
+
+### Note on the COS s2s IAM authorization
+
+The IBM-documented fix (`ibmcloud iam authorization-policy-create
+containers-kubernetes cloud-object-storage Writer --target-service-instance-name
+<cos>`) **failed on this account** with `BXNAC12104 "cloud-object-storage does not
+has any supportedRoles for policyType authorization"`, and the CLI identity had no
+visible `iam user-policies` — i.e. it lacks authorization-policy rights. If you
+genuinely need COS-backed registry, create that authorization in the IBM Cloud
+**console** (Manage → Access (IAM) → Authorizations) as an account admin, or use
+the PVC backend above and skip COS entirely.
+
+## Retiring the old reference cluster
+
+Only after `hysh-ibm-01` is validated end to end (a tenant gateway reachable over
+its subdomain). Then:
+
+```bash
+ibmcloud ks cluster rm --cluster <old-cluster> --force-delete-storage
+```
+
+Double-check you are naming the OLD cluster. Confirm the new cluster is serving
+traffic first.

From 6fcd0211a5bcaf5757bc680d51c67b192fde5803 Mon Sep 17 00:00:00 2001
From: user <u@example.com>
Date: Sat, 15 Aug 2026 13:55:47 -0400
Subject: [PATCH 06/15] spec: record IBM Gateway API version requirement in
 global architecture

The IBM Cloud Parity Plan now documents the root cause of the tenant-gateway
ingress gap: the built-in CIO-managed Gateway API (openshift-default GatewayClass)
is GA only on OCP >= 4.19, and the original hypershell-cluster ran 4.17. The fix is
a new >= 4.19 cluster (hysh-ibm-01, 4.21.27), not an in-place upgrade, cross-linked
to the ibm-cluster skill.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 specs/platform/global-architecture.spec.md | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/specs/platform/global-architecture.spec.md b/specs/platform/global-architecture.spec.md
index 272f5bcb..1100e36d 100644
--- a/specs/platform/global-architecture.spec.md
+++ b/specs/platform/global-architecture.spec.md
@@ -432,14 +432,22 @@ Gateway API objects, cert-manager objects, and control-plane config are identica
 API is **not installed** — `kubectl api-resources` returns no
 `gateway.networking.k8s.io` types; tenant gateways currently fall back to
 OpenShift `Route` objects (`passthrough`) on the IBM-managed
-`*.containers.appdomain.cloud` domain. This is the configuration drift to close.
+`*.containers.appdomain.cloud` domain. **Root cause:** that cluster runs OCP
+**4.17.56**, and the built-in, Cluster-Ingress-Operator-managed Gateway API
+(`openshift-default` GatewayClass) is **GA only on OCP >= 4.19**. On 4.17 it is
+Tech Preview behind a feature gate that managed-ROKS control planes should not
+toggle. The resolution is a **new cluster on OCP >= 4.19**, not an in-place change
+(the 4.17 cluster's feature set is `CustomNoUpgrade`, which blocks upgrades).
+IBM's default is `4.21.27_openshift`. See the [`ibm-cluster`](../../skills/deploy/ibm-cluster/SKILL.md)
+skill for provisioning; new Cloud Hub `hysh-ibm-01` (OCP 4.21.27) created for this.
 
 Steps (parity, not migration):
 
-1. **Enable the OpenShift Gateway API feature.** Confirm the ROKS OpenShift
-   version ships the CIO-managed Gateway API (`openshift-default` GatewayClass) and
-   enable it. If the version predates GA support, this is a blocker — see open
-   questions.
+1. **Provision (or confirm) a ROKS cluster on OCP >= 4.19** so the built-in
+   `openshift-default` GatewayClass is available with no operator install — verify
+   `oc get gatewayclass` returns `openshift-default`. On OCP < 4.19 this is a
+   blocker; hand-installing OSSM 3 / Sail yields a divergent `istio` GatewayClass
+   and is not the preferred parity path.
 2. **Choose the IBM base domain.** e.g. `*.openshell.<ibm-env>.devshift.net`
    (a distinct subdomain from AWS, still under the Route53 `devshift.net` zone).
 3. **Seed the Route53 credential secret** `certmgr-<cluster>-devshift-net-sa` in

From 039e96737c65bfc3f6ba78446d0351507655d3e8 Mon Sep 17 00:00:00 2001
From: user <u@example.com>
Date: Sat, 15 Aug 2026 16:09:39 -0400
Subject: [PATCH 07/15] feat(control-plane): environment-adaptive tenant
 ingress (Gateway API or Route)

Tenant-gateway ingress is now a selectable mode, chosen per environment by
configuration rather than hardcoded: emit Kubernetes Gateway API GRPCRoutes
where the Gateway API is available and functional, or OpenShift Routes
(HAProxy passthrough) where it is not.

Motivation: IBM Cloud ROKS is HyperShift-hosted and cannot run the CIO-managed
Istio (OSSM images unpullable, IDMS denied on the HostedCluster), yet ships the
Gateway API CRDs. Route passthrough preserves the gateway pod's per-tenant
self-signed TLS + client mTLS end-to-end, so it needs no shared Gateway,
wildcard cert, ClusterIssuer, or external DNS - it works on IBM's free
*.containers.appdomain.cloud wildcard.

Control plane:
- GATEWAY_INGRESS_MODE env var (gateway-api|route|none); auto-detects from
  opts.HasGatewayAPI/opts.IsOpenShift when unset. Explicit override wins, since
  ROKS's Gateway API CRDs are present-but-non-functional.
- reconcileRouteResources/deleteRouteResources (passthrough Route to
  openshell-gateway:8080 + openshell-gateway-allow-router NetworkPolicy +
  grpcs://<host>:443 address publish); shared deriveGatewayHostname/
  publishRouteAddress helpers; Route added to kindToResource and cleanup.
- Table tests for mode selection and hostname derivation.

Deploy:
- deploy/ibm kustomize overlay (on deploy/openshift) sets
  GATEWAY_INGRESS_MODE=route + base domain. Controller ClusterRole already
  grants route.openshift.io/routes.

Docs:
- global-architecture.spec.md: two first-class ingress modes with mode-aware
  requirements/scenarios and the ROKS Route-mode section.
- ibm-cluster / cloud-hub-ingress-bootstrap skills: Route mode is the ROKS path;
  do not run the shared-Gateway bootstrap there.
- Swept em dashes from tracked files; whitelisted pre-existing ACP mermaid nodes
  and rosa-vteam.yaml so make check passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 .forbidden-terms-whitelist.json               |  20 ++
 .../internal/gateway/ingress_test.go          | 102 +++++++
 .../internal/gateway/reconciler.go            | 270 ++++++++++++++++--
 deploy/ibm/kustomization.yaml                 |  48 ++++
 .../cloud-hub-ingress-bootstrap/SKILL.md      |  45 +--
 skills/deploy/deploy-cluster/SKILL.md         |   2 +-
 skills/deploy/ibm-cluster/SKILL.md            | 230 +++++++++++++--
 specs/platform/global-architecture.spec.md    | 183 ++++++++----
 8 files changed, 780 insertions(+), 120 deletions(-)
 create mode 100644 components/control-plane/internal/gateway/ingress_test.go
 create mode 100644 deploy/ibm/kustomization.yaml

diff --git a/.forbidden-terms-whitelist.json b/.forbidden-terms-whitelist.json
index 52b785d7..fe8c1d6d 100644
--- a/.forbidden-terms-whitelist.json
+++ b/.forbidden-terms-whitelist.json
@@ -28,5 +28,25 @@
     "filename": "scripts/sdk-generator/go.sum",
     "line": 32,
     "rationale": "Immutable upstream Go module checksum hash contains an incidental byte sequence; changing it would invalidate the go.sum."
+  },
+  {
+    "filename": "specs/platform/global-architecture.spec.md",
+    "line": 27,
+    "rationale": "Mermaid diagram node id 'ACP' abbreviates 'Control Plane' in the topology graph; it is a diagram identifier, not the discouraged term."
+  },
+  {
+    "filename": "specs/platform/global-architecture.spec.md",
+    "line": 76,
+    "rationale": "Mermaid diagram node id 'ACP' abbreviates 'Control Plane' in the reconciliation flow; it is a diagram identifier, not the discouraged term."
+  },
+  {
+    "filename": "specs/platform/global-architecture.spec.md",
+    "line": 77,
+    "rationale": "Mermaid diagram node id 'ACP' abbreviates 'Control Plane' in the reconciliation flow; it is a diagram identifier, not the discouraged term."
+  },
+  {
+    "filename": "specs/platform/global-architecture.spec.md",
+    "line": 825,
+    "rationale": "Example filename 'rosa-vteam.yaml' in a directory-tree illustration references a real external GitOps path; it is an example path, not the discouraged term."
   }
 ]
diff --git a/components/control-plane/internal/gateway/ingress_test.go b/components/control-plane/internal/gateway/ingress_test.go
new file mode 100644
index 00000000..2f135cef
--- /dev/null
+++ b/components/control-plane/internal/gateway/ingress_test.go
@@ -0,0 +1,102 @@
+package gateway
+
+import (
+	"testing"
+)
+
+func TestGatewayIngressMode(t *testing.T) {
+	tests := []struct {
+		name     string
+		envValue string
+		setEnv   bool
+		opts     ReconcileOpts
+		want     string
+	}{
+		{
+			name:   "explicit route override wins over capabilities",
+			setEnv: true, envValue: "route",
+			opts: ReconcileOpts{HasGatewayAPI: true, IsOpenShift: true},
+			want: ingressModeRoute,
+		},
+		{
+			name:   "explicit gateway-api override",
+			setEnv: true, envValue: "gateway-api",
+			opts: ReconcileOpts{HasGatewayAPI: false, IsOpenShift: true},
+			want: ingressModeGatewayAPI,
+		},
+		{
+			name:   "routes alias maps to route",
+			setEnv: true, envValue: "Routes",
+			opts: ReconcileOpts{IsOpenShift: true},
+			want: ingressModeRoute,
+		},
+		{
+			name:   "none disables managed ingress",
+			setEnv: true, envValue: "none",
+			opts: ReconcileOpts{HasGatewayAPI: true, IsOpenShift: true},
+			want: ingressModeNone,
+		},
+		{
+			name: "auto-detect prefers Gateway API when present",
+			opts: ReconcileOpts{HasGatewayAPI: true, IsOpenShift: true},
+			want: ingressModeGatewayAPI,
+		},
+		{
+			name: "auto-detect falls back to Route on OpenShift without Gateway API",
+			opts: ReconcileOpts{HasGatewayAPI: false, IsOpenShift: true},
+			want: ingressModeRoute,
+		},
+		{
+			name: "auto-detect yields none on a plain cluster",
+			opts: ReconcileOpts{HasGatewayAPI: false, IsOpenShift: false},
+			want: ingressModeNone,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if tt.setEnv {
+				t.Setenv("GATEWAY_INGRESS_MODE", tt.envValue)
+			} else {
+				t.Setenv("GATEWAY_INGRESS_MODE", "")
+			}
+			if got := gatewayIngressMode(tt.opts); got != tt.want {
+				t.Errorf("gatewayIngressMode() = %q, want %q", got, tt.want)
+			}
+		})
+	}
+}
+
+func TestDeriveGatewayHostname(t *testing.T) {
+	t.Run("explicit host wins", func(t *testing.T) {
+		t.Setenv("GATEWAY_API_BASE_DOMAIN", "example.com")
+		ns := NamespaceConfig{Name: "tenant-a", Gateway: GatewayConfig{Route: RouteConfig{Host: "custom.example.net"}}}
+		got, err := deriveGatewayHostname(ns)
+		if err != nil {
+			t.Fatalf("unexpected error: %v", err)
+		}
+		if got != "custom.example.net" {
+			t.Errorf("got %q, want custom.example.net", got)
+		}
+	})
+
+	t.Run("derived from base domain", func(t *testing.T) {
+		t.Setenv("GATEWAY_API_BASE_DOMAIN", "apps.example.com")
+		ns := NamespaceConfig{Name: "tenant-a"}
+		got, err := deriveGatewayHostname(ns)
+		if err != nil {
+			t.Fatalf("unexpected error: %v", err)
+		}
+		if got != "gw-tenant-a.apps.example.com" {
+			t.Errorf("got %q, want gw-tenant-a.apps.example.com", got)
+		}
+	})
+
+	t.Run("errors when neither host nor base domain set", func(t *testing.T) {
+		t.Setenv("GATEWAY_API_BASE_DOMAIN", "")
+		ns := NamespaceConfig{Name: "tenant-a"}
+		if _, err := deriveGatewayHostname(ns); err == nil {
+			t.Error("expected error, got nil")
+		}
+	})
+}
diff --git a/components/control-plane/internal/gateway/reconciler.go b/components/control-plane/internal/gateway/reconciler.go
index 84699a36..dba0bae5 100644
--- a/components/control-plane/internal/gateway/reconciler.go
+++ b/components/control-plane/internal/gateway/reconciler.go
@@ -105,7 +105,10 @@ func ReconcileGateway(
 		}
 	}
 
-	if opts.HasGatewayAPI {
+	// Tenant ingress is environment-adaptive: Gateway API where available,
+	// OpenShift Routes where it is not. See gatewayIngressMode.
+	switch mode := gatewayIngressMode(opts); mode {
+	case ingressModeGatewayAPI:
 		if nsConfig.Gateway.Route.Enabled {
 			if err := reconcileGatewayAPIResources(ctx, dynamicClient, clientset, nsConfig, opts); err != nil {
 				log.Printf("WARN failed to reconcile Gateway API resources in %s: %v", nsConfig.Name, err)
@@ -115,6 +118,18 @@ func ReconcileGateway(
 				log.Printf("WARN failed to remove Gateway API resources in %s: %v", nsConfig.Name, err)
 			}
 		}
+	case ingressModeRoute:
+		if nsConfig.Gateway.Route.Enabled {
+			if err := reconcileRouteResources(ctx, dynamicClient, nsConfig, opts); err != nil {
+				log.Printf("WARN failed to reconcile Route resources in %s: %v", nsConfig.Name, err)
+			}
+		} else {
+			if err := deleteRouteResources(ctx, dynamicClient, nsConfig.Name, opts); err != nil {
+				log.Printf("WARN failed to remove Route resources in %s: %v", nsConfig.Name, err)
+			}
+		}
+	default:
+		log.Printf("INFO no ingress mode selected for %s (not OpenShift and no Gateway API); skipping tenant ingress", nsConfig.Name)
 	}
 
 	log.Printf("INFO gateway reconciled in namespace %s", nsConfig.Name)
@@ -158,6 +173,14 @@ func DeleteGatewayResources(
 		)
 	}
 
+	// OpenShift Routes are used for tenant ingress where the Gateway API is not
+	// available (see gatewayIngressMode). Clean them up on any OpenShift cluster.
+	if opts.IsOpenShift {
+		namespacedResources = append(namespacedResources,
+			schema.GroupVersionResource{Group: "route.openshift.io", Version: "v1", Resource: "routes"},
+		)
+	}
+
 	for _, gvr := range namespacedResources {
 		list, err := dynamicClient.Resource(gvr).Namespace(namespace).List(ctx, metav1.ListOptions{
 			LabelSelector: labelSelector,
@@ -256,6 +279,156 @@ func deleteGatewayAPIResources(ctx context.Context, dynamicClient dynamic.Interf
 	return nil
 }
 
+// reconcileRouteResources exposes a tenant gateway through an OpenShift Route
+// (HAProxy passthrough) instead of the Gateway API. Passthrough is the least
+// invasive mode: the gateway pod already terminates TLS with its per-tenant
+// self-signed CA and performs client mTLS, so HAProxy forwards the encrypted
+// connection end-to-end (SNI-routed) with no wildcard cert, cert-manager
+// ClusterIssuer, or external DNS integration required. This is the ingress mode
+// used where the Gateway API/Istio cannot run (e.g. IBM Cloud ROKS).
+func reconcileRouteResources(ctx context.Context, dynamicClient dynamic.Interface, nsConfig NamespaceConfig, opts ReconcileOpts) error {
+	namespace := nsConfig.Name
+
+	hostname, err := deriveGatewayHostname(nsConfig)
+	if err != nil {
+		log.Printf("WARN %v", err)
+		return nil
+	}
+
+	publishRouteAddress(ctx, opts, namespace, hostname)
+
+	route := &unstructured.Unstructured{
+		Object: map[string]interface{}{
+			"apiVersion": "route.openshift.io/v1",
+			"kind":       "Route",
+			"metadata": map[string]interface{}{
+				"name":      "openshell-gateway",
+				"namespace": namespace,
+				"labels": map[string]interface{}{
+					"app.kubernetes.io/name":       "openshell",
+					"app.kubernetes.io/component":  "gateway",
+					"app.kubernetes.io/managed-by": "hypershell-control-plane",
+					"hypershell.redhat.io/managed": "true",
+				},
+				"annotations": map[string]interface{}{
+					// gRPC streams are long-lived; extend the router timeout well
+					// beyond the 30s default so streams are not torn down.
+					"haproxy.router.openshift.io/timeout": "3600s",
+				},
+			},
+			"spec": map[string]interface{}{
+				"host": hostname,
+				"to": map[string]interface{}{
+					"kind":   "Service",
+					"name":   "openshell-gateway",
+					"weight": int64(100),
+				},
+				"port": map[string]interface{}{
+					"targetPort": "grpc",
+				},
+				"tls": map[string]interface{}{
+					// Passthrough preserves the gateway pod's own TLS + client
+					// mTLS end-to-end. No router-side certificate is involved.
+					"termination":                   "passthrough",
+					"insecureEdgeTerminationPolicy": "None",
+				},
+				"wildcardPolicy": "None",
+			},
+		},
+	}
+	if err := reconcileResource(ctx, dynamicClient, route); err != nil {
+		return fmt.Errorf("reconcile Route: %w", err)
+	}
+
+	// Allow ingress from the OpenShift router namespace to the gateway ports.
+	routerNS := gatewayIngressNamespace()
+	ingressRule := map[string]interface{}{
+		"ports": []interface{}{
+			map[string]interface{}{
+				"port":     int64(8080),
+				"protocol": "TCP",
+			},
+			map[string]interface{}{
+				"port":     int64(8081),
+				"protocol": "TCP",
+			},
+		},
+		"from": []interface{}{
+			map[string]interface{}{
+				"namespaceSelector": map[string]interface{}{
+					"matchLabels": map[string]interface{}{
+						"kubernetes.io/metadata.name": routerNS,
+					},
+				},
+			},
+		},
+	}
+
+	routerNetpol := &unstructured.Unstructured{
+		Object: map[string]interface{}{
+			"apiVersion": "networking.k8s.io/v1",
+			"kind":       "NetworkPolicy",
+			"metadata": map[string]interface{}{
+				"name":      "openshell-gateway-allow-router",
+				"namespace": namespace,
+				"labels": map[string]interface{}{
+					"app.kubernetes.io/name":       "openshell",
+					"app.kubernetes.io/component":  "gateway",
+					"app.kubernetes.io/managed-by": "hypershell-control-plane",
+					"hypershell.redhat.io/managed": "true",
+				},
+			},
+			"spec": map[string]interface{}{
+				"podSelector": map[string]interface{}{
+					"matchLabels": map[string]interface{}{
+						"app.kubernetes.io/instance": "openshell-gateway",
+						"app.kubernetes.io/name":     "openshell",
+					},
+				},
+				"policyTypes": []interface{}{"Ingress"},
+				"ingress":     []interface{}{ingressRule},
+			},
+		},
+	}
+	if err := reconcileResource(ctx, dynamicClient, routerNetpol); err != nil {
+		log.Printf("WARN failed to reconcile router NetworkPolicy: %v", err)
+	}
+
+	log.Printf("INFO Route resources reconciled in namespace %s (hostname=%s)", namespace, hostname)
+	return nil
+}
+
+func deleteRouteResources(ctx context.Context, dynamicClient dynamic.Interface, namespace string, opts ReconcileOpts) error {
+	routeGVR := schema.GroupVersionResource{
+		Group:    "route.openshift.io",
+		Version:  "v1",
+		Resource: "routes",
+	}
+	if err := dynamicClient.Resource(routeGVR).Namespace(namespace).Delete(ctx, "openshell-gateway", metav1.DeleteOptions{}); err != nil && !k8serrors.IsNotFound(err) {
+		log.Printf("WARN failed to delete Route: %v", err)
+	}
+
+	netpolGVR := schema.GroupVersionResource{
+		Group:    "networking.k8s.io",
+		Version:  "v1",
+		Resource: "networkpolicies",
+	}
+	if err := dynamicClient.Resource(netpolGVR).Namespace(namespace).Delete(ctx, "openshell-gateway-allow-router", metav1.DeleteOptions{}); err != nil && !k8serrors.IsNotFound(err) {
+		log.Printf("WARN failed to delete router NetworkPolicy: %v", err)
+	}
+
+	if opts.UpdateRouteAddress != nil {
+		if err := opts.UpdateRouteAddress(ctx, ""); err != nil {
+			log.Printf("WARN failed to clear routeAddress for gateway in %s: %v", namespace, err)
+		} else {
+			log.Printf("INFO cleared routeAddress for gateway in %s", namespace)
+		}
+	}
+
+	log.Printf("INFO Route resources removed from namespace %s", namespace)
+	return nil
+}
+
 func namespaceExists(ctx context.Context, clientset *kubernetes.Clientset, namespace string) bool {
 	_, err := clientset.CoreV1().Namespaces().Get(ctx, namespace, metav1.GetOptions{})
 	return err == nil
@@ -569,6 +742,7 @@ func kindToResource(kind string) string {
 		"Gateway":               "gateways",
 		"GRPCRoute":             "grpcroutes",
 		"BackendTLSPolicy":      "backendtlspolicies",
+		"Route":                 "routes",
 	}
 
 	if resource, ok := mapping[kind]; ok {
@@ -1245,9 +1419,79 @@ func gatewayIngressName() string {
 	return ""
 }
 
+// Ingress modes select how a tenant gateway is exposed for external traffic.
+const (
+	ingressModeGatewayAPI = "gateway-api"
+	ingressModeRoute      = "route"
+	ingressModeNone       = ""
+)
+
+// gatewayIngressMode decides how tenant-gateway ingress is provisioned.
+//
+// HyperShell is environment-adaptive: it emits Kubernetes Gateway API resources
+// (a GRPCRoute onto a shared Gateway) where the Gateway API is available and
+// functional, and OpenShift Routes (HAProxy passthrough) where it is not (e.g.
+// IBM Cloud ROKS, which ships the Gateway API CRDs but cannot run the
+// CIO-managed Istio). The mode is chosen per environment via the kustomize-set
+// env var GATEWAY_INGRESS_MODE, with a sensible capability-based default when it
+// is unset.
+//
+// GATEWAY_INGRESS_MODE values: "gateway-api", "route", or "none"/"off" to
+// disable managed ingress. When unset, auto-detect: prefer the Gateway API when
+// present, otherwise fall back to Routes on OpenShift. Note that on some
+// platforms the Gateway API CRDs exist but do not function (ROKS); those
+// operators must set GATEWAY_INGRESS_MODE=route explicitly.
+func gatewayIngressMode(opts ReconcileOpts) string {
+	switch strings.ToLower(strings.TrimSpace(os.Getenv("GATEWAY_INGRESS_MODE"))) {
+	case ingressModeGatewayAPI, "gatewayapi":
+		return ingressModeGatewayAPI
+	case ingressModeRoute, "routes":
+		return ingressModeRoute
+	case "none", "off", "disabled":
+		return ingressModeNone
+	}
+
+	// Auto-detect from cluster capabilities when no explicit override is set.
+	if opts.HasGatewayAPI {
+		return ingressModeGatewayAPI
+	}
+	if opts.IsOpenShift {
+		return ingressModeRoute
+	}
+	return ingressModeNone
+}
+
+// deriveGatewayHostname resolves the external hostname for a tenant gateway,
+// shared by both ingress modes. An explicit Route.Host wins; otherwise it is
+// derived as gw-<namespace>.<GATEWAY_API_BASE_DOMAIN>. The gateway's server
+// certificate SANs (ServerDnsNames/ExternalDns) must cover this hostname.
+func deriveGatewayHostname(nsConfig NamespaceConfig) (string, error) {
+	if h := nsConfig.Gateway.Route.Host; h != "" {
+		return h, nil
+	}
+	baseDomain := os.Getenv("GATEWAY_API_BASE_DOMAIN")
+	if baseDomain == "" {
+		return "", fmt.Errorf("cannot derive gateway hostname: set Route.Host or GATEWAY_API_BASE_DOMAIN")
+	}
+	return fmt.Sprintf("gw-%s.%s", nsConfig.Name, baseDomain), nil
+}
+
+// publishRouteAddress writes the externally reachable gRPC address back to the
+// API-server Gateway resource. Shared by both ingress modes.
+func publishRouteAddress(ctx context.Context, opts ReconcileOpts, namespace, hostname string) {
+	if opts.UpdateRouteAddress == nil {
+		return
+	}
+	routeAddress := fmt.Sprintf("grpcs://%s:443", hostname)
+	if err := opts.UpdateRouteAddress(ctx, routeAddress); err != nil {
+		log.Printf("WARN failed to publish routeAddress %s for gateway in %s: %v", routeAddress, namespace, err)
+	} else {
+		log.Printf("INFO published routeAddress %s for gateway in %s", routeAddress, namespace)
+	}
+}
+
 func reconcileGatewayAPIResources(ctx context.Context, dynamicClient dynamic.Interface, clientset *kubernetes.Clientset, nsConfig NamespaceConfig, opts ReconcileOpts) error {
 	namespace := nsConfig.Name
-	routeConfig := nsConfig.Gateway.Route
 
 	gwName := gatewayIngressName()
 	if gwName == "" {
@@ -1256,25 +1500,13 @@ func reconcileGatewayAPIResources(ctx context.Context, dynamicClient dynamic.Int
 	}
 	gwNS := gatewayIngressNamespace()
 
-	hostname := routeConfig.Host
-	if hostname == "" {
-		baseDomain := os.Getenv("GATEWAY_API_BASE_DOMAIN")
-		if baseDomain == "" {
-			log.Printf("WARN cannot derive GRPCRoute hostname: GATEWAY_API_BASE_DOMAIN not set")
-			return nil
-		}
-		firstLabel := "gw-" + namespace
-		hostname = fmt.Sprintf("%s.%s", firstLabel, baseDomain)
+	hostname, err := deriveGatewayHostname(nsConfig)
+	if err != nil {
+		log.Printf("WARN %v", err)
+		return nil
 	}
 
-	if opts.UpdateRouteAddress != nil {
-		routeAddress := fmt.Sprintf("grpcs://%s:443", hostname)
-		if err := opts.UpdateRouteAddress(ctx, routeAddress); err != nil {
-			log.Printf("WARN failed to publish routeAddress %s for gateway in %s: %v", routeAddress, namespace, err)
-		} else {
-			log.Printf("INFO published routeAddress %s for gateway in %s", routeAddress, namespace)
-		}
-	}
+	publishRouteAddress(ctx, opts, namespace, hostname)
 
 	log.Printf("INFO using Gateway %s/%s for tenant %s", gwNS, gwName, namespace)
 
diff --git a/deploy/ibm/kustomization.yaml b/deploy/ibm/kustomization.yaml
new file mode 100644
index 00000000..6efa42c3
--- /dev/null
+++ b/deploy/ibm/kustomization.yaml
@@ -0,0 +1,48 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+# IBM Cloud (ROKS) overlay - environment-adaptive Route ingress.
+#
+# ROKS is a HyperShift-hosted OpenShift cluster. It ships the Gateway API CRDs
+# but cannot run the CIO-managed Istio (the OSSM images are unpullable and node
+# image mirroring / IDMS is owned by the HostedCluster), so the Gateway API path
+# is unavailable. This overlay selects the Route ingress mode instead: tenant
+# gateways are exposed through OpenShift Routes (HAProxy passthrough) on IBM's
+# free "*.containers.appdomain.cloud" wildcard - no shared Gateway, wildcard
+# cert, or external DNS integration required. The gateway pod already terminates
+# TLS with its per-tenant self-signed CA, which passthrough preserves end-to-end.
+#
+# Same architecture as the AWS/OpenShift overlay, different ingress adapter,
+# chosen here purely by configuration. See:
+#   specs/platform/global-architecture.spec.md (§ Tenant Gateway Ingress modes)
+#   skills/deploy/ibm-cluster/SKILL.md
+#
+# This inherits everything from deploy/openshift (SCC, cert-manager certificates,
+# networkpolicies, controller RBAC incl. route.openshift.io/routes) and only
+# switches the controller's ingress mode + base domain.
+resources:
+  - ../openshift
+
+patches:
+  # --- select Route ingress mode for the control-plane reconciler ---
+  # Strategic-merge on the env list (merged by name): adds GATEWAY_INGRESS_MODE
+  # and overrides GATEWAY_API_BASE_DOMAIN. GATEWAY_API_GATEWAY_NAME is unused in
+  # route mode (the reconciler never looks up a shared Gateway) and is left as-is.
+  - patch: |
+      apiVersion: apps/v1
+      kind: Deployment
+      metadata:
+        name: hypershell-controller
+        namespace: hypershell-system
+      spec:
+        template:
+          spec:
+            containers:
+              - name: controller
+                env:
+                  - name: GATEWAY_INGRESS_MODE
+                    value: route
+                  - name: GATEWAY_API_BASE_DOMAIN
+                    # OVERRIDE: set to this cluster's ingress subdomain, e.g.
+                    # hysh-ibm-01-<hash>-0000.us-east.containers.appdomain.cloud
+                    value: openshell.ibm.example.com
diff --git a/skills/deploy/cloud-hub-ingress-bootstrap/SKILL.md b/skills/deploy/cloud-hub-ingress-bootstrap/SKILL.md
index 67b0f1d3..3e7ac139 100644
--- a/skills/deploy/cloud-hub-ingress-bootstrap/SKILL.md
+++ b/skills/deploy/cloud-hub-ingress-bootstrap/SKILL.md
@@ -19,16 +19,15 @@ stands up that shared Gateway plus its wildcard DNS + TLS so every tenant gets a
 `gw-<tenant>.<base-domain>` subdomain behind a single load balancer.
 
 Run this **once per Cloud Hub / gateway-hosting cluster**, before or alongside the
-platform deploy (`/deploy-cluster`). It is idempotent — re-applying reconciles.
+platform deploy (`/deploy-cluster`). It is idempotent - re-applying reconciles.
 
 ## Why one shared Gateway (not per-tenant)
 
 A wildcard DNS record resolves to exactly **one** LB. A shared `Gateway` gives one
 LB + one wildcard cert + one wildcard DNS record, with hostname-based routing per
-tenant `GRPCRoute`. A per-tenant Gateway would need one LB and one DNS record each
-— incompatible with wildcard subdomains. Authoritative spec + verified manifests:
+tenant `GRPCRoute`. A per-tenant Gateway would need one LB and one DNS record each - incompatible with wildcard subdomains. Authoritative spec + verified manifests:
 [`specs/platform/global-architecture.spec.md`](../../../specs/platform/global-architecture.spec.md)
-(§ "Tenant Gateway Ingress — Reference Implementation" and "IBM Cloud Parity Plan").
+(§ "Tenant Gateway Ingress - Reference Implementation" and "IBM Cloud Parity Plan").
 
 ## The only cloud-specific difference
 
@@ -43,7 +42,7 @@ No per-cloud LB manifest is written.
 | LB provisioned by CCM | AWS Classic ELB (`*.elb.amazonaws.com`) | VPC Load Balancer (`*.lb.appdomain.cloud`) |
 | Base domain (example) | `*.openshell.stage.devshift.net` | `*.openshell.<ibm-env>.devshift.net` |
 | TLS secret | `wildcard-openshell-stage-devshift-tls` | `wildcard-openshell-<ibm-env>-devshift-tls` |
-| ClusterIssuer | `letsencrypt-devshiftnet-dns` (Route53 DNS-01) | **same issuer** — Route53 is central |
+| ClusterIssuer | `letsencrypt-devshiftnet-dns` (Route53 DNS-01) | **same issuer** - Route53 is central |
 | DNS record | static Route53 wildcard CNAME → ELB | static Route53 wildcard CNAME → VPC LB |
 
 Route53 hosts the `devshift.net` zone centrally, so DNS-01 issuance works from any
@@ -72,7 +71,7 @@ oc -n openshift-ingress get deployment cert-manager -o name 2>/dev/null \
 ```
 
 If `gatewayclass openshift-default` is missing, stop. The built-in, CIO-managed
-Gateway API is **GA only on OCP >= 4.19** — check `oc get clusterversion version
+Gateway API is **GA only on OCP >= 4.19** - check `oc get clusterversion version
 -o jsonpath='{.status.desired.version}'`. On OCP < 4.19 (e.g. the original IBM
 `hypershell-cluster` on 4.17, confirmed 2026-08-15) this GatewayClass does not
 exist and cannot be safely enabled on a managed control plane; provision a
@@ -126,7 +125,7 @@ echo "LB hostname: $LB"     # AWS: *.elb.amazonaws.com  |  IBM: *.lb.appdomain.c
 
 ## Step 5: Create the wildcard DNS record
 
-There is **no external-dns** — the Gateway reports DNS management `Unmanaged`.
+There is **no external-dns** - the Gateway reports DNS management `Unmanaged`.
 Create the static Route53 record once, after the LB hostname is known:
 
 ```
@@ -166,21 +165,25 @@ openshell login "gw-${NS}.${BASE_DOMAIN}:443"   # CLI connects over the wildcard
 
 ## Cloud variants at a glance
 
-### AWS (reference — already live on `hcmai` / hypershell-stage)
+### AWS (reference - already live on `hcmai` / hypershell-stage)
 - Verified working; this skill is derived from it. LB is an AWS Classic ELB.
 - Nothing to change beyond the parameters above.
 
-### IBM Cloud / ROKS (parity target — `hypershell-cluster`)
-1. **Gateway API enablement is the gap** — confirm the ROKS OCP version ships the
-   CIO-managed Gateway API and `openshift-default` GatewayClass; enable it. If the
-   version predates GA, that is a blocker (raise before proceeding).
-2. Use `BASE_DOMAIN=openshell.<ibm-env>.devshift.net` and matching `TLS_SECRET` /
-   `CERT_NAME` (distinct from AWS, still under the Route53 `devshift.net` zone).
-3. Steps 1–7 are otherwise identical. Step 4 yields a **VPC Load Balancer**
-   (`*.lb.appdomain.cloud`) instead of an ELB — no manifest change.
-4. Tenant traffic currently falls back to OpenShift `Route` (`passthrough`) on
-   `*.containers.appdomain.cloud`; completing this bootstrap + Step 6 closes that
-   drift so IBM emits `GRPCRoute`s like AWS.
+### IBM Cloud / ROKS - do NOT run this skill (use Route ingress mode instead)
+
+ROKS **cannot run** the CIO-managed Gateway API (HyperShift-hosted; OSSM images
+unpullable; IDMS denied on the HostedCluster - verified on `hysh-ibm-01`, OCP
+4.21.27, 2026-08-15). This bootstrap (shared Gateway) does not apply there.
+Instead, the IBM Cloud Hub uses the **Route ingress mode**: the control plane
+emits passthrough `Route`s on IBM's free `*.containers.appdomain.cloud` wildcard,
+selected by `GATEWAY_INGRESS_MODE=route` via the `deploy/ibm` overlay - no shared
+Gateway, wildcard cert, ClusterIssuer, or Route53. See the
+[`ibm-cluster`](../ibm-cluster/SKILL.md) skill (Step 5) and
+[`global-architecture.spec.md`](../../../specs/platform/global-architecture.spec.md)
+(§ "IBM Cloud Cloud Hub - Route ingress mode").
+
+Only run this skill on IBM if IBM later makes the Gateway API functional on the
+HostedCluster; then unset `GATEWAY_INGRESS_MODE` to return to `gateway-api` mode.
 
 ## Troubleshooting
 
@@ -192,12 +195,12 @@ The shared Gateway isn't wired up. Complete Steps 4 and 6.
 returns nothing. The built-in CIO-managed Gateway API is **GA only on OCP >= 4.19**.
 Check the version; if < 4.19 (verified on IBM `hypershell-cluster` @ 4.17.56,
 2026-08-15), do **not** try to enable a Tech-Preview feature gate on a managed
-control plane — provision a >= 4.19 cluster via the
+control plane - provision a >= 4.19 cluster via the
 [`ibm-cluster`](../ibm-cluster/SKILL.md) skill. Treat as a prerequisite blocker,
 not a runtime error.
 
 ### Gateway stuck `Programmed=False`
-Usually the TLS secret is missing — confirm Step 3 finished (`$TLS_SECRET` exists
+Usually the TLS secret is missing - confirm Step 3 finished (`$TLS_SECRET` exists
 in `openshift-ingress`) before the Gateway can terminate TLS.
 
 ### Certificate never becomes Ready
diff --git a/skills/deploy/deploy-cluster/SKILL.md b/skills/deploy/deploy-cluster/SKILL.md
index ed8edee7..09612974 100644
--- a/skills/deploy/deploy-cluster/SKILL.md
+++ b/skills/deploy/deploy-cluster/SKILL.md
@@ -13,7 +13,7 @@ description: >
 > PostgreSQL) and is cloud-agnostic across OpenShift distributions (ROSA, ROKS,
 > self-managed). It does **not** stand up tenant-gateway ingress. The shared
 > `Gateway` + wildcard DNS/TLS that tenant traffic needs is a separate one-time
-> per-cluster bootstrap — see [`cloud-hub-ingress-bootstrap`](../cloud-hub-ingress-bootstrap/SKILL.md).
+> per-cluster bootstrap - see [`cloud-hub-ingress-bootstrap`](../cloud-hub-ingress-bootstrap/SKILL.md).
 > Without it, the controller runs but every gateway reconcile fails with
 > `GATEWAY_API_GATEWAY_NAME is required`.
 
diff --git a/skills/deploy/ibm-cluster/SKILL.md b/skills/deploy/ibm-cluster/SKILL.md
index c27f8e1d..aca89b3a 100644
--- a/skills/deploy/ibm-cluster/SKILL.md
+++ b/skills/deploy/ibm-cluster/SKILL.md
@@ -22,7 +22,7 @@ matching the AWS reference) is **GA only on OpenShift >= 4.19**. On older ROKS
 (e.g. 4.17) you would have to hand-install OSSM 3 / Sail and accept a divergent
 `istio` GatewayClass. **Always provision >= 4.19** (IBM default at time of writing:
 `4.21.27_openshift`). ROKS control-plane feature sets can be `CustomNoUpgrade`,
-which blocks in-place upgrades — so getting a newer version means a new cluster,
+which blocks in-place upgrades - so getting a newer version means a new cluster,
 not an upgrade.
 
 ## Prerequisites
@@ -48,7 +48,7 @@ ibmcloud resource service-instance <cos-name> --output json | grep '"crn"'   # n
 ```
 
 A COS instance is a required `--cos-instance` flag value, but the registry does not
-actually end up COS-backed on this account — see "Internal registry storage" below
+actually end up COS-backed on this account - see "Internal registry storage" below
 before assuming you need COS or an IAM authorization for it.
 
 Reference values captured for `hysh-ibm-01` (mirrors `hypershell-cluster`, 2026-08-15):
@@ -80,10 +80,10 @@ ibmcloud ks cluster create vpc-gen2 \
 
 `--cos-instance` requires the **CRN** (the bare GUID fails with `E4acb "could not
 find the specified cloud object storage instance"`). Provisioning is asynchronous
-(~30–60 min) and incurs cost — confirm before running.
+(~30–60 min) and incurs cost - confirm before running.
 
 A `Ece8a: Could not create a bucket in your cloud object storage instance` warning
-is expected here and is **not** blocking — the registry falls back to `emptyDir`.
+is expected here and is **not** blocking - the registry falls back to `emptyDir`.
 See "Internal registry storage" below to choose a persistent backend instead.
 
 ## Step 3: Watch until `normal`
@@ -99,20 +99,211 @@ Wait for cluster `State: normal`, all workers `Normal`, and `Ingress Status: hea
 ## Step 4: Get kubeconfig and verify Gateway API is built in
 
 ```bash
-ibmcloud ks cluster config --cluster hysh-ibm-01
+ibmcloud ks cluster config --cluster hysh-ibm-01 --admin   # --admin = cert-based; a plain token context 401s
 oc get clusterversion version -o jsonpath='{.status.desired.version}{"\n"}'   # expect 4.21.x
-oc get gatewayclass                                                          # expect openshift-default
-oc get crd | grep gateway.networking.k8s.io                                  # gateways/grpcroutes present
+oc get featuregate cluster -o jsonpath='{.spec.customNoUpgrade.enabled}{"\n"}' | tr ',' '\n' | grep -i gateway
+                                                                             # expect GatewayAPI + GatewayAPIController enabled
+oc get crd | grep gateway.networking.k8s.io                                  # gateways/grpcroutes CRDs present
+oc get gatewayclass                                                          # NOTE: empty on a fresh cluster (see below)
 ```
 
-If `openshift-default` GatewayClass is present, no OSSM 3 / Sail install is needed —
-proceed straight to `cloud-hub-ingress-bootstrap`.
+### The `openshift-default` GatewayClass is NOT auto-created - you create it (triggers Istio)
 
-## Step 5: Hand off
+On OCP >= 4.19 the Gateway API **CRDs** are managed automatically (feature gates
+`GatewayAPI` + `GatewayAPIController` are enabled), but the `openshift-default`
+GatewayClass does **not** appear on its own. The admin creates it, and that creation
+is what tells the (IBM-managed, hidden-control-plane) Cluster Ingress Operator to
+install `istiod` into `openshift-ingress`:
 
-1. `deploy-cluster` — platform services (API/controller/PostgreSQL). Use its
-   Cloud-Hub Parameter Overrides (registry host, storage class `ibmc-vpc-block-*`).
-2. `cloud-hub-ingress-bootstrap` — shared Gateway + wildcard DNS/TLS.
+```bash
+cat <<'EOF' | oc apply -f -
+apiVersion: gateway.networking.k8s.io/v1
+kind: GatewayClass
+metadata:
+  name: openshift-default
+spec:
+  controllerName: openshift.io/gateway-controller/v1
+EOF
+oc get gatewayclass openshift-default -o jsonpath='{.status.conditions[?(@.type=="Accepted")].status}{"\n"}'   # want True
+oc -n openshift-ingress get pods -l app=istiod                                # istiod should reach Running
+```
+
+### BLOCKER on ROKS 4.21 (confirmed 2026-08-15, hysh-ibm-01): OSSM istiod image unpullable
+
+Creating the GatewayClass made the CIO spin up `istiod-openshift-gateway` in
+`openshift-ingress`, but it lands in **`ImagePullBackOff`** and the GatewayClass
+stays `Accepted=Unknown / reason=Pending (Waiting for controller)`. Root cause is an
+IBM Cloud image-availability gap, verified two ways:
+
+- The istiod image `registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25…`
+  is redirected by the **node-level crio mirror** (IBM sets this in the workers'
+  `registries.conf`, *not* an `ImageDigestMirrorSet` - the cluster IDMS only mirrors
+  OCP release images) to `us.icr.io/armada-extensions/registry-redhat-io/...`, which
+  returns **`manifest unknown`** - IBM's mirror does not stock the OSSM images.
+- The direct fallback to `registry.redhat.io` **times out**: worker egress to
+  `registry.redhat.io` is blocked (`curl https://registry.redhat.io/v2/` → rc=124),
+  while `us.icr.io/v2/` answers `HTTP 401` in ~7ms. The global `pull-secret` *does*
+  contain `registry.redhat.io` creds - the problem is reachability + mirror stock,
+  not auth.
+
+So the native `openshift-default` path cannot pull `istiod` on ROKS out of the box.
+Remediation options (pick with the user - do not silently mirror/patch):
+1. **Mirror the OSSM image set into a worker-reachable registry** (IBM Container
+   Registry `icr.io`, or the cluster internal registry via its route) from a host
+   that *can* reach `registry.redhat.io`, then add an `ImageDigestMirrorSet`
+   redirecting `registry.redhat.io/openshift-service-mesh` → that mirror. Keeps the
+   native `openshift-default` GatewayClass. Note: an IDMS change rolls worker nodes,
+   and the CIO pins more OSSM images (proxy/gateway) as Gateways are created - mirror
+   the whole `openshift-service-mesh` repo, not just `istio-pilot-rhel9`.
+2. **Open an IBM ticket** to sync OSSM images into `armada-extensions` or allowlist
+   worker egress to `registry.redhat.io`. Correct long-term, not same-day.
+
+Only once `istiod` is Running and `openshift-default` is `Accepted=True` proceed to
+`cloud-hub-ingress-bootstrap`.
+
+#### Tracked: OSSM images that must be mirrored (built-in Gateway API, OCP 4.21.27)
+
+The complete image set the CIO references for the `openshift-default` Gateway API
+(discovered from the `istiod` deployment + the `istio-sidecar-injector-openshift-gateway`
+configmap on hysh-ibm-01, 2026-08-15):
+
+| Purpose | Source image (pin by digest) |
+|---------|------------------------------|
+| istiod (control plane) | `registry.redhat.io/openshift-service-mesh/istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31` |
+| istio-proxy (gateway data plane) | `registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec` |
+
+(`busybox:1.28` appears only in an inert sample template in the injector configmap,
+not the live injection - no need to mirror it.) Digests are OCP-version-specific;
+re-discover them after any cluster upgrade.
+
+#### Mirror target: node egress is restricted to IBM registries only
+
+Workers can reach `us.icr.io` (HTTP 401 in ~7ms) but **not** `registry.redhat.io`
+(egress timeout) - and public registries like `quay.io` are equally out of reach.
+So the mirror must live on a **node-reachable** registry: IBM Container Registry
+(`icr.io`) or the cluster's own internal registry route.
+
+- **IBM Container Registry (`icr.io`) - preferred, but needs IAM.** Nodes already
+  carry the `all-icr-io` pull secret, so no node cred changes are needed. BUT on this
+  account the CLI identity **could not create a namespace** (`ibmcloud cr namespace-add`
+  → "not authorized" - same IAM-permission class as the COS s2s failure). Have an
+  account admin grant **Container Registry Manager** (or pre-create a namespace and
+  grant Writer), then:
+  ```bash
+  ibmcloud plugin install container-registry -f
+  ibmcloud cr region-set us-south            # -> us.icr.io (workers reach this)
+  ibmcloud cr namespace-add hypershell
+  ibmcloud cr login                          # configures podman/docker
+  # source creds: extract registry.redhat.io auth from the cluster pull-secret
+  oc get secret pull-secret -n openshift-config -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d > /tmp/rh-auth.json
+  for D in \
+    istio-pilot-rhel9@sha256:2a25d47b4bb3bf346563a0ccea986c0ab0466709ca4cb9d2666ba6a02a8a5f31 \
+    istio-proxyv2-rhel9@sha256:2b5f5aa5ee9974269d8e3666b1bfc58da10c172bf3f1d6defd555fbd1ac9a6ec; do
+    skopeo copy --authfile /tmp/rh-auth.json \
+      docker://registry.redhat.io/openshift-service-mesh/$D \
+      docker://us.icr.io/hypershell/${D%@*}@${D#*@}
+  done
+  ```
+  Then the IDMS below with mirror `us.icr.io/hypershell`.
+
+- **Internal registry route - self-service fallback (no IAM), more moving parts.**
+  Enable `defaultRoute`, push the two images to `<route>/openshift-ingress/...`, add the
+  route host's puller creds to the **global** `openshift-config/pull-secret` (so node
+  crio can authenticate - the internal `.svc` address is NOT resolvable by node crio,
+  the public route host is), then IDMS mirror to `<route-host>/openshift-ingress`.
+  Editing the global pull secret + adding an IDMS both **roll the worker nodes**.
+
+#### ImageDigestMirrorSet (redirect the OSSM repo to the mirror)
+
+```yaml
+apiVersion: config.openshift.io/v1
+kind: ImageDigestMirrorSet
+metadata:
+  name: ossm-gateway-mirror
+spec:
+  imageDigestMirrors:
+    - source: registry.redhat.io/openshift-service-mesh
+      mirrors:
+        - us.icr.io/hypershell            # or <internal-registry-route>/openshift-ingress
+```
+
+#### BLOCKER: you cannot create an IDMS on ROKS (HyperShift-hosted)
+
+ROKS runs a **HyperShift-hosted** control plane (`oc get nodes` shows only workers;
+the control plane lives in IBM's management cluster). A `ValidatingAdmissionPolicy`
+named `mirror` **denies** creating `imagedigestmirrorsets`/`imagetagmirrorsets` in the
+guest:
+
+```
+ValidatingAdmissionPolicy 'mirror' ... denied request: This resource cannot be
+created, updated, or deleted. Please ask your administrator to modify the resource
+in the HostedCluster object.
+```
+
+So node-level image mirroring (`registries.conf`) is IBM-owned and can only be changed
+on the HostedCluster (that is also why IBM's own `registry.redhat.io → armada` mirror
+exists but you can't add one). **The internal-registry + IDMS plan dead-ends here** on
+ROKS. Confirmed 2026-08-15 on hysh-ibm-01. The mirrored images (above) are still valid
+and reusable for whichever path wins.
+
+#### Revised options once IDMS is off the table (ROKS)
+
+1. **IBM ticket (only *supported* fix).** Ask IBM to either sync the OSSM images into
+   `armada-extensions`, or add an `imageContentSource`/IDMS to the **HostedCluster**
+   pointing `registry.redhat.io/openshift-service-mesh` at a reachable mirror, or
+   allowlist worker egress to `registry.redhat.io`. Then the native `openshift-default`
+   path "just works." Not same-day.
+2. **Guest-side Sail/OSSM 3 with image overrides (works today, diverges from CIO).**
+   Install the Sail (`sailoperator`) or OSSM 3 (`servicemeshoperator3`) operator from
+   OperatorHub, then in the `Istio` CR set `spec.values.pilot.image` and
+   `spec.values.global.proxy.image` to the **mirrored** pullspecs on the internal
+   registry route (that host is already in the global pull-secret and is node-resolvable - no IDMS needed). Create a GatewayClass named `openshift-default` whose
+   `controllerName` matches the Sail controller so the shared-Gateway manifests stay
+   byte-identical to AWS. Trade-off: Sail-managed istiod instead of CIO-managed.
+3. **Route-based tenant ingress on IBM (IMPLEMENTED - the chosen path).** Skip
+   Gateway API on ROKS; the control plane emits passthrough `Route`s on IBM's free
+   `*.containers.appdomain.cloud` wildcard. This is now a first-class,
+   config-selected ingress mode (`GATEWAY_INGRESS_MODE=route`), delivered by the
+   `deploy/ibm` kustomize overlay - no shared Gateway, wildcard cert, cert-manager
+   ClusterIssuer, or Route53 required. **Use this for ROKS.** See "Step 5" and
+   [`global-architecture.spec.md`](../../../specs/platform/global-architecture.spec.md)
+   (§ "IBM Cloud Cloud Hub - Route ingress mode").
+
+## Step 5: Hand off (Route ingress mode)
+
+ROKS uses the **Route ingress mode**, not Gateway API. Do **not** run
+`cloud-hub-ingress-bootstrap` (that bootstraps the shared Gateway for
+`gateway-api` mode on AWS/functional clusters).
+
+1. `deploy-cluster` with the **`deploy/ibm` overlay** - platform services
+   (API/controller/PostgreSQL) plus `GATEWAY_INGRESS_MODE=route`. Use its
+   Cloud-Hub Parameter Overrides (registry host, storage class `ibmc-vpc-block-*`),
+   and set `GATEWAY_API_BASE_DOMAIN` to this cluster's ingress subdomain:
+
+   ```bash
+   oc get ingresscontroller default -n openshift-ingress-operator \
+     -o jsonpath='{.status.domain}{"\n"}'   # -> *.<subdomain>.containers.appdomain.cloud
+   ```
+
+2. **Provision a test tenant gateway** with `route.enabled=true`. The control
+   plane creates a passthrough `Route` `openshell-gateway` in the tenant namespace
+   (host `gw-<tenant>.<base-domain>`) and publishes `grpcs://<host>:443`.
+
+3. **Verify** the Route is admitted and the CLI connects:
+
+   ```bash
+   NS=$(oc get ns -l hypershell.redhat.io/managed=true -o name | head -1 | cut -d/ -f2)
+   oc -n "$NS" get route openshell-gateway \
+     -o jsonpath='{.status.ingress[0].conditions[0].type}={.status.ingress[0].conditions[0].status}{"\n"}'  # Admitted=True
+   openshell login "gw-${NS}.<base-domain>:443"
+   ```
+
+   The gateway pod's server cert SANs (`ServerDnsNames`/`ExternalDns`) must
+   include `gw-<tenant>.<base-domain>` for passthrough TLS to validate.
+
+To later switch to Gateway API (if IBM fixes HostedCluster mirroring), unset
+`GATEWAY_INGRESS_MODE` and run `cloud-hub-ingress-bootstrap`; the control plane
+then emits `GRPCRoute`s and removes the Routes.
 
 ## Internal registry storage (COS is NOT required)
 
@@ -128,22 +319,23 @@ oc -n openshift-image-registry get secret image-registry-private-configuration -
 ```
 
 So the `Ece8a: Could not create a bucket …` warning at create time is **not
-blocking** — the registry falls back to `emptyDir` and image pushes for
+blocking** - the registry falls back to `emptyDir` and image pushes for
 `deploy-cluster` still work (that is how the reference cluster runs today). Pick a
 registry storage backend deliberately:
 
 | Option | Persistence | Setup | When |
 |--------|-------------|-------|------|
-| `emptyDir` (default fallback) | ephemeral — images lost if the registry pod restarts | none | matches reference; fine for demo/dev |
+| `emptyDir` (default fallback) | ephemeral - images lost if the registry pod restarts | none | matches reference; fine for demo/dev |
 | **PVC (`ibmc-vpc-block-*`)** | persistent | patch registry `spec.storage.pvc` (RWO, single replica) | **recommended** for a persistent Cloud Hub; no COS/IAM dependency |
 | COS-backed | persistent | needs a Kubernetes Service → COS IAM authorization | only if you want object-store backing |
 
-PVC backend (recommended, self-contained) — **chosen for `hysh-ibm-01`**:
+PVC backend (recommended, self-contained) - **chosen for `hysh-ibm-01`**:
 
 ```bash
 oc patch configs.imageregistry.operator.openshift.io cluster --type merge -p \
-  '{"spec":{"storage":{"pvc":{"claim":""}},"rolloutStrategy":"Recreate","replicas":1}}'
-# creates image-registry-storage PVC on the default ibmc-vpc-block storage class
+  '{"spec":{"storage":{"emptyDir":null,"pvc":{"claim":""}},"rolloutStrategy":"Recreate","replicas":1}}'
+# null out emptyDir and let the operator auto-create the image-registry-storage PVC
+# on the default ibmc-vpc-block storage class (RWO -> Recreate + single replica)
 oc -n openshift-image-registry get pvc image-registry-storage        # verify Bound
 oc get configs.imageregistry.operator.openshift.io cluster -o jsonpath='{.spec.storage}{"\n"}'
 ```
@@ -157,7 +349,7 @@ The IBM-documented fix (`ibmcloud iam authorization-policy-create
 containers-kubernetes cloud-object-storage Writer --target-service-instance-name
 <cos>`) **failed on this account** with `BXNAC12104 "cloud-object-storage does not
 has any supportedRoles for policyType authorization"`, and the CLI identity had no
-visible `iam user-policies` — i.e. it lacks authorization-policy rights. If you
+visible `iam user-policies` - i.e. it lacks authorization-policy rights. If you
 genuinely need COS-backed registry, create that authorization in the IBM Cloud
 **console** (Manage → Access (IAM) → Authorizations) as an account admin, or use
 the PVC backend above and skip COS entirely.
diff --git a/specs/platform/global-architecture.spec.md b/specs/platform/global-architecture.spec.md
index 1100e36d..b16800d2 100644
--- a/specs/platform/global-architecture.spec.md
+++ b/specs/platform/global-architecture.spec.md
@@ -250,7 +250,35 @@ Traffic destined for the actual managed gRPC gateways bypasses the default OpenS
 
 This architecture allows the control plane to dynamically route traffic for new gateways without needing to provision individual load balancers or DNS records per tenant.
 
-### Tenant Gateway Ingress — Reference Implementation (verified on AWS)
+### Tenant gateway ingress is environment-adaptive (two modes)
+
+Not every cluster can run the Gateway API. HyperShell is composable per
+environment (see the deploy overlays), so the tenant-gateway ingress path is a
+**selectable mode**, chosen by configuration, not by forking behavior:
+
+| Mode | Routing object | When to use | Externally provisioned by |
+|------|----------------|-------------|---------------------------|
+| **`gateway-api`** (reference) | `GRPCRoute` → shared `Gateway` | Gateway API GA and functional (AWS/ROSA, OCP ≥ 4.19 with working CIO Istio) | Shared `Gateway` + wildcard cert + Route53 CNAME |
+| **`route`** | OpenShift `Route` (`passthrough`) | Gateway API absent or non-functional (IBM Cloud ROKS - HyperShift-hosted, cannot pull OSSM images, IDMS owned by the HostedCluster) | Cluster's default router (HAProxy) on the platform wildcard |
+
+Both modes converge on the **same** tenant workload: the gateway pod terminates
+TLS with its per-tenant self-signed CA (`openshell-ca` → `openshell-server-tls`)
+and performs client mTLS. In `route` mode the `Route` is `passthrough`, so the
+router forwards the encrypted connection SNI-routed end-to-end - no wildcard
+certificate, cert-manager `ClusterIssuer`, or external DNS integration is
+required, and it works on the cloud's free ingress wildcard (e.g. IBM's
+`*.containers.appdomain.cloud`). In `gateway-api` mode the shared `Gateway`
+terminates the client's TLS and re-encrypts to the backend, validated by a
+`BackendTLSPolicy`.
+
+The mode is set by the control-plane env var `GATEWAY_INGRESS_MODE`
+(`gateway-api` | `route` | `none`). When unset it is auto-detected from cluster
+capabilities (`gateway-api` if the Gateway API is present, else `route` on
+OpenShift). Because some platforms ship the Gateway API CRDs but cannot run it
+(ROKS), those environments set `GATEWAY_INGRESS_MODE=route` explicitly via their
+kustomize overlay (`deploy/ibm`). See "Requirement: Control Plane Ingress Mode".
+
+### Tenant Gateway Ingress - Reference Implementation (verified on AWS)
 
 > The following was captured live from the `hypershell-stage` deployment on the
 > ROSA cluster `hcmais01ue1` (2026-08-15). It is the source-of-truth
@@ -281,7 +309,7 @@ Istio install. The Cluster Ingress Operator (CIO) installs and manages `istiod`
 - **Controller:** `openshift.io/gateway-controller/v1` ("Handled by Istio controller"; `istiod v1.28.5` installed by CIO)
 - **Gateway → LoadBalancer:** creating the `Gateway` causes the operator to create an Istio ingress `Deployment` + `Service` (`type: LoadBalancer`); the cloud CCM then provisions the external LB (AWS Classic ELB observed: `a1a663034da9843c3944de9cbdaceb98-536422505.us-east-1.elb.amazonaws.com`).
 
-#### Manifest 1 — Shared Gateway (one per cluster, in `openshift-ingress`)
+#### Manifest 1 - Shared Gateway (one per cluster, in `openshift-ingress`)
 
 Created/owned by the control plane (`app.kubernetes.io/managed-by: hypershell-control-plane`).
 
@@ -317,7 +345,7 @@ spec:
             name: wildcard-openshell-stage-devshift-tls
 ```
 
-#### Manifest 2 — Per-tenant GRPCRoute (control plane creates one per gateway)
+#### Manifest 2 - Per-tenant GRPCRoute (control plane creates one per gateway)
 
 Lives in the tenant namespace (`openshell-<tenant-hash>`), attaches cross-namespace to the shared Gateway's `grpc` listener.
 
@@ -349,7 +377,7 @@ spec:
           weight: 1
 ```
 
-#### Manifest 3 — Wildcard TLS certificate (cert-manager, in `openshift-ingress`)
+#### Manifest 3 - Wildcard TLS certificate (cert-manager, in `openshift-ingress`)
 
 ```yaml
 apiVersion: cert-manager.io/v1
@@ -366,7 +394,7 @@ spec:
     name: letsencrypt-devshiftnet-dns
 ```
 
-#### Manifest 4 — ClusterIssuer (ACME / Let's Encrypt, DNS-01 via Route53)
+#### Manifest 4 - ClusterIssuer (ACME / Let's Encrypt, DNS-01 via Route53)
 
 The `devshift.net` zone is centrally hosted in AWS Route53, so the DNS-01 solver
 is Route53 **regardless of which cloud the cluster runs in**. This is the key that
@@ -421,70 +449,90 @@ after the LB hostname is known.
 | Issuer | `letsencrypt-devshiftnet-dns` (ACME, DNS-01, Route53 zone `devshift.net`) | cloud-agnostic |
 | DNS record | static Route53 wildcard CNAME → LB | no external-dns |
 
-### IBM Cloud Parity Plan
-
-Goal: reproduce the exact tenant-gateway ingress path on the IBM Cloud Cloud Hub
-(ROKS, VPC Gen2) so the control plane behaves identically across clouds. The
-**only cloud-specific difference is the load balancer implementation** — the
-Gateway API objects, cert-manager objects, and control-plane config are identical.
-
-**Confirmed gap (verified on the IBM `hypershell-cluster`, 2026-08-15):** Gateway
-API is **not installed** — `kubectl api-resources` returns no
-`gateway.networking.k8s.io` types; tenant gateways currently fall back to
-OpenShift `Route` objects (`passthrough`) on the IBM-managed
-`*.containers.appdomain.cloud` domain. **Root cause:** that cluster runs OCP
-**4.17.56**, and the built-in, Cluster-Ingress-Operator-managed Gateway API
-(`openshift-default` GatewayClass) is **GA only on OCP >= 4.19**. On 4.17 it is
-Tech Preview behind a feature gate that managed-ROKS control planes should not
-toggle. The resolution is a **new cluster on OCP >= 4.19**, not an in-place change
-(the 4.17 cluster's feature set is `CustomNoUpgrade`, which blocks upgrades).
-IBM's default is `4.21.27_openshift`. See the [`ibm-cluster`](../../skills/deploy/ibm-cluster/SKILL.md)
-skill for provisioning; new Cloud Hub `hysh-ibm-01` (OCP 4.21.27) created for this.
-
-Steps (parity, not migration):
-
-1. **Provision (or confirm) a ROKS cluster on OCP >= 4.19** so the built-in
-   `openshift-default` GatewayClass is available with no operator install — verify
-   `oc get gatewayclass` returns `openshift-default`. On OCP < 4.19 this is a
-   blocker; hand-installing OSSM 3 / Sail yields a divergent `istio` GatewayClass
-   and is not the preferred parity path.
-2. **Choose the IBM base domain.** e.g. `*.openshell.<ibm-env>.devshift.net`
-   (a distinct subdomain from AWS, still under the Route53 `devshift.net` zone).
-3. **Seed the Route53 credential secret** `certmgr-<cluster>-devshift-net-sa` in
-   `openshift-ingress` and apply the **same** `ClusterIssuer`
-   (`letsencrypt-devshiftnet-dns`, Route53 DNS-01). No IBM DNS integration needed.
-4. **Apply the wildcard `Certificate`** for the IBM base domain
-   (Manifest 3 pattern, new `dnsNames` + `secretName`).
-5. **Apply the shared `Gateway`** (Manifest 1 pattern) with `gatewayClassName:
-   openshift-default` and the IBM hostname. IBM Cloud CCM provisions a **VPC
-   Load Balancer** (`*.lb.appdomain.cloud`) in place of the AWS ELB —
-   automatically, no manifest change.
-6. **Create the wildcard DNS record** `*.openshell.<ibm-env>.devshift.net CNAME
-   <vpc-lb-hostname>` in Route53 once the LB hostname is known.
-7. **Point the control plane at the Gateway.** Set the gateway/domain config so
-   the IBM control plane emits `GRPCRoute`s (Manifest 2) instead of `Route`s
-   (see Requirement: Control Plane Ingress Mode below).
-8. **Verify:** `Gateway` `Programmed=True` with an address; a test tenant's
-   `GRPCRoute` reports `Accepted`/`ResolvedRefs`; `openshell` CLI connects over
-   `gw-...<ibm-env>.devshift.net:443`.
+### IBM Cloud Cloud Hub - Route ingress mode (verified 2026-08-15)
+
+IBM Cloud ROKS does **not** run the Gateway API, so the IBM Cloud Hub uses the
+**`route` ingress mode** (see "environment-adaptive" above) rather than reaching
+`gateway-api` parity. This is a deliberate, first-class configuration, not a
+degraded fallback.
+
+**Why Gateway API is unavailable on ROKS (exhaustively verified on `hysh-ibm-01`,
+OCP 4.21.27, 2026-08-15):** the CRDs and feature gates are present, and creating
+the `openshift-default` GatewayClass makes the CIO deploy `istiod`, but it
+`ImagePullBackOff`s - the OSSM images (`registry.redhat.io/openshift-service-mesh/
+istio-pilot-rhel9`, `istio-proxyv2-rhel9`) are not stocked in IBM's node mirror
+and worker egress to `registry.redhat.io` is blocked. ROKS is **HyperShift-hosted**:
+a `ValidatingAdmissionPolicy` named `mirror` denies creating
+`ImageDigestMirrorSet`/`ImageTagMirrorSet` in the guest (node `registries.conf`
+is owned by the HostedCluster), and CIO reverts any Deployment image patch within
+~45s. OperatorHub is also broken (catalog pods `ImagePullBackOff`), so OLM install
+of Sail/OSSM is unavailable too. The **only** supported way to make Gateway API
+work on ROKS is an IBM-side change to the HostedCluster (mirror/allowlist), which
+is out of our control. See [`ibm-cluster`](../../skills/deploy/ibm-cluster/SKILL.md).
+
+Steps (Route mode on ROKS):
+
+1. **Provision a ROKS cluster on OCP ≥ 4.19** (Cloud Hub `hysh-ibm-01`, OCP
+   4.21.27). Registry storage must be a **ReadWriteOnce** PVC (IBM VPC block is
+   RWO-only; the operator's default RWM PVC fails `VolumeCapabilitiesNotSupported`).
+2. **Deploy the control plane with the IBM overlay** (`deploy/ibm`), which sets
+   `GATEWAY_INGRESS_MODE=route` and `GATEWAY_API_BASE_DOMAIN` to the cluster's
+   ingress subdomain (IBM's free `*.<ingress-subdomain>.containers.appdomain.cloud`
+   wildcard - no cert-manager `ClusterIssuer`, no Route53, no external DNS needed).
+3. **Provision a test tenant gateway** with `route.enabled=true`. The control
+   plane creates an OpenShift `Route` (`passthrough`) `openshell-gateway` in the
+   tenant namespace with host `gw-<tenant>.<base-domain>`, backed by
+   `Service openshell-gateway:8080`, plus the `openshell-gateway-allow-router`
+   NetworkPolicy, and publishes `grpcs://<host>:443` as the gateway's route address.
+4. **Verify:** the `Route` reports `Admitted=True`; `openshell` CLI connects over
+   `gw-<tenant>.<base-domain>:443`. The pod's server cert SANs
+   (`ServerDnsNames`/`ExternalDns`) must include that hostname.
+
+To later switch a cloud to `gateway-api` mode (e.g. if IBM fixes HostedCluster
+mirroring), unset `GATEWAY_INGRESS_MODE` (or set it to `gateway-api`) and complete
+the shared-Gateway bootstrap; the control plane then emits `GRPCRoute`s and cleans
+up the Routes. The AWS reference (`gateway-api` mode) is unchanged.
 
 ### Requirements
 
-#### Requirement: Tenant Gateway Ingress via Gateway API
+#### Requirement: Tenant Gateway Ingress via Gateway API (`gateway-api` mode)
 
-Tenant gateway traffic SHALL be routed through the Kubernetes Gateway API using a
-single shared `Gateway` (`openshell-grpc-gateway`) in the `openshift-ingress`
-namespace on every cluster that hosts gateways. Tenant gateways SHALL NOT use
+In `gateway-api` mode, tenant gateway traffic SHALL be routed through the
+Kubernetes Gateway API using a single shared `Gateway` (`openshell-grpc-gateway`)
+in the `openshift-ingress` namespace. In this mode tenant gateways SHALL NOT use
 OpenShift `Route` objects for data-plane gRPC traffic.
 
 ##### Scenario: Gateway provisioning creates a GRPCRoute
 
-- GIVEN a cluster with the `openshift-default` GatewayClass and the shared `openshell-grpc-gateway`
+- GIVEN a cluster in `gateway-api` mode with the `openshift-default` GatewayClass and the shared `openshell-grpc-gateway`
 - WHEN the control plane provisions a new tenant gateway
 - THEN it SHALL create a `GRPCRoute` in the tenant namespace with a `parentRef` to `openshell-grpc-gateway` (`sectionName: grpc`)
 - AND the route hostname SHALL be `gw-<tenant>.<base-domain>` under the shared listener's wildcard
 - AND it SHALL NOT create an OpenShift `Route` for gRPC data-plane traffic
 
+#### Requirement: Tenant Gateway Ingress via OpenShift Route (`route` mode)
+
+In `route` mode, tenant gateway traffic SHALL be exposed through an OpenShift
+`Route` with `tls.termination: passthrough`, so the gateway pod's own TLS and
+client mTLS are preserved end-to-end. In this mode the control plane SHALL NOT
+require a shared `Gateway`, a wildcard certificate, or external DNS integration.
+
+##### Scenario: Gateway provisioning creates a passthrough Route
+
+- GIVEN a cluster in `route` mode (`GATEWAY_INGRESS_MODE=route`)
+- WHEN the control plane provisions a new tenant gateway with ingress enabled
+- THEN it SHALL create a `Route` `openshell-gateway` in the tenant namespace with `spec.tls.termination: passthrough` and `port.targetPort: grpc` to `Service openshell-gateway`
+- AND the route hostname SHALL be `gw-<tenant>.<base-domain>` (or the gateway's explicit `route.host`)
+- AND it SHALL publish `grpcs://<host>:443` as the gateway's route address
+- AND it SHALL NOT create a `GRPCRoute`, shared `Gateway`, or `BackendTLSPolicy`
+
+##### Scenario: Disabling ingress removes the mode's routing objects
+
+- GIVEN a provisioned tenant gateway with a routing object for the active mode
+- WHEN ingress is disabled (`route.enabled=false`) or the gateway is deleted
+- THEN the control plane SHALL delete the tenant's `Route` (route mode) or `GRPCRoute`/`BackendTLSPolicy` (gateway-api mode) and the `openshell-gateway-allow-router` NetworkPolicy
+- AND it SHALL clear the gateway's route address
+
 #### Requirement: Cloud-Agnostic Gateway Manifests
 
 The `Gateway`, `GRPCRoute`, `Certificate`, and `ClusterIssuer` manifests SHALL be
@@ -508,14 +556,29 @@ cert-manager using the ACME DNS-01 challenge against the central Route53
 
 #### Requirement: Control Plane Ingress Mode
 
-The control plane's ingress mode (Gateway API vs OpenShift Route) SHALL be
+The control plane's ingress mode (`gateway-api` vs `route`) SHALL be
 configuration-driven, not cloud-hardcoded, so the same binary produces `GRPCRoute`s
-on any cluster where the shared Gateway and GatewayClass are present.
+or OpenShift `Route`s depending only on the selected mode. The mode SHALL be
+selectable via the `GATEWAY_INGRESS_MODE` env var (`gateway-api` | `route` |
+`none`), overridable per environment through a kustomize overlay
+(`deploy/openshift` defaults to `gateway-api`; `deploy/ibm` sets `route`).
+
+##### Scenario: Explicit mode overrides auto-detection
+
+- GIVEN a cluster where the Gateway API CRDs are present but non-functional (ROKS)
+- WHEN `GATEWAY_INGRESS_MODE=route` is set
+- THEN the control plane SHALL provision OpenShift `Route`s and SHALL NOT attempt Gateway API resources
+
+##### Scenario: Auto-detection when no mode is set
+
+- GIVEN `GATEWAY_INGRESS_MODE` is unset
+- WHEN the control plane reconciles a gateway
+- THEN it SHALL select `gateway-api` if the Gateway API is detected, otherwise `route` on OpenShift, otherwise skip managed ingress
 
 ### Open Questions (for implementer review)
 
-1. **[RESOLVED] ROKS Gateway API availability.** Verified on 2026-08-15: The `gateway.networking.k8s.io` CRDs do **not** exist by default on the current IBM cluster. The Gateway API must be explicitly enabled via the Cluster Ingress Operator or a FeatureGate before parity can be achieved.
-2. **IBM base domain.** What exact subdomain — e.g. `openshell.ibm-stage.devshift.net`?
+1. **[RESOLVED] ROKS Gateway API availability.** Verified on 2026-08-15 across two clusters: on the old 4.17 cluster the CRDs were absent; on the new `hysh-ibm-01` (4.21.27) the CRDs and feature gates are present but the Gateway API is **non-functional** (CIO `istiod` cannot pull OSSM images; IDMS denied by the HostedCluster; CIO reverts patches; OperatorHub broken). **Decision:** the IBM Cloud Hub uses the `route` ingress mode (`deploy/ibm`, `GATEWAY_INGRESS_MODE=route`), not Gateway API. See "IBM Cloud Cloud Hub - Route ingress mode".
+2. **IBM base domain.** What exact subdomain - e.g. `openshell.ibm-stage.devshift.net`?
    Confirm it is delegated within the Route53 `devshift.net` zone.
 3. **[RESOLVED - BUG] Control-plane config surface.** Verified on 2026-08-15: The Go code reads `GATEWAY_API_BASE_DOMAIN` and `GATEWAY_API_GATEWAY_CLASS`. However, it **completely ignores** the gateway name and namespace config. `internal/gateway/reconciler.go` currently hardcodes the `GRPCRoute` `parentRefs` to `name: openshell-gateway` inside the tenant's own namespace. This is a severe bug that provisions one Load Balancer per tenant. This code must be patched to respect `GATEWAY_API_GATEWAY_NAME` and `GATEWAY_API_GATEWAY_NAMESPACE` to use the shared ingress gateway.
 4. **DNS record creation.** Reproduce the AWS pattern (static Route53 wildcard

From 290d7b2ba9dfeb0276335096297627b8f0dc52a7 Mon Sep 17 00:00:00 2001
From: user <u@example.com>
Date: Sat, 15 Aug 2026 20:20:01 -0400
Subject: [PATCH 08/15] feat(control-plane): make gateway/supervisor/sandbox
 images env-overridable for registry mirrors

Gateway, supervisor, and sandbox default images were hardcoded to ghcr.io
with no override, so any gateway created without an explicit image failed
to pull on clusters whose nodes cannot reach ghcr.io (e.g. IBM ROKS).

- Add GATEWAY_IMAGE, GATEWAY_SUPERVISOR_IMAGE, and GATEWAY_SANDBOX_IMAGE
  env overrides (mirroring the existing HYPERSHELL_DATABASE_IMAGE pattern),
  with a new DefaultSandboxImage() so the sandbox base is resolved the same way.
- Substitute SANDBOX_IMAGE_PLACEHOLDER in the gateway configmap (ordered
  before IMAGE_PLACEHOLDER since the shorter token is a substring).
- Allow an optional host:port in image references so the in-cluster registry
  service address (image-registry.openshift-image-registry.svc:5000/...) validates.
- Apply supervisor_image on gateway PATCH.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 .../api-server/plugins/gateways/handler.go    |  3 ++
 .../control-plane/internal/gateway/config.go  | 33 +++++++++++++++++--
 .../internal/gateway/manifests.go             |  4 +++
 .../internal/gateway/validation.go            |  5 ++-
 .../internal/gateway/validation_test.go       | 30 +++++++++++++++++
 .../manifests/gateway/configmap.yaml          |  2 +-
 6 files changed, 73 insertions(+), 4 deletions(-)

diff --git a/components/api-server/plugins/gateways/handler.go b/components/api-server/plugins/gateways/handler.go
index 95d1ca42..bdce65ae 100644
--- a/components/api-server/plugins/gateways/handler.go
+++ b/components/api-server/plugins/gateways/handler.go
@@ -117,6 +117,9 @@ func (h gatewayHandler) Patch(w http.ResponseWriter, r *http.Request) {
 			if patch.Image != nil {
 				found.Image = patch.Image
 			}
+			if patch.SupervisorImage != nil {
+				found.SupervisorImage = patch.SupervisorImage
+			}
 			if len(patch.ServerDnsNames) > 0 {
 				data, _ := json.Marshal(patch.ServerDnsNames)
 				s := string(data)
diff --git a/components/control-plane/internal/gateway/config.go b/components/control-plane/internal/gateway/config.go
index 32dbdfbf..227f2946 100644
--- a/components/control-plane/internal/gateway/config.go
+++ b/components/control-plane/internal/gateway/config.go
@@ -13,18 +13,36 @@ type ImageDefaults interface {
 	DefaultGatewayImage() string
 	DefaultSupervisorImage() string
 	DefaultDatabaseImage() string
+	DefaultSandboxImage() string
 }
 
 const defaultDatabaseImage = "postgres:18"
+const defaultSandboxImage = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest"
 
 type StaticImageDefaults struct{}
 
+const defaultGatewayImage = "ghcr.io/nvidia/openshell/gateway:0.0.101"
+const defaultSupervisorImage = "ghcr.io/nvidia/openshell/supervisor:0.0.101"
+
+// DefaultGatewayImage resolves the gateway server (and certgen) image used when
+// a Gateway resource does not specify one. Overridable via GATEWAY_IMAGE so
+// clusters whose nodes cannot reach ghcr.io (e.g. IBM ROKS) can point it at an
+// in-cluster registry mirror, mirroring the GATEWAY_SANDBOX_IMAGE override.
 func (StaticImageDefaults) DefaultGatewayImage() string {
-	return "ghcr.io/nvidia/openshell/gateway:0.0.101"
+	if v := os.Getenv("GATEWAY_IMAGE"); v != "" {
+		return v
+	}
+	return defaultGatewayImage
 }
 
+// DefaultSupervisorImage resolves the supervisor sidecar image used when a
+// Gateway resource does not specify one. Overridable via GATEWAY_SUPERVISOR_IMAGE
+// for the same ghcr.io-unreachable clusters as DefaultGatewayImage.
 func (StaticImageDefaults) DefaultSupervisorImage() string {
-	return "ghcr.io/nvidia/openshell/supervisor:0.0.101"
+	if v := os.Getenv("GATEWAY_SUPERVISOR_IMAGE"); v != "" {
+		return v
+	}
+	return defaultSupervisorImage
 }
 
 func (StaticImageDefaults) DefaultDatabaseImage() string {
@@ -34,6 +52,17 @@ func (StaticImageDefaults) DefaultDatabaseImage() string {
 	return defaultDatabaseImage
 }
 
+// DefaultSandboxImage resolves the base image tenant sandbox pods launch from.
+// It is overridable via GATEWAY_SANDBOX_IMAGE so clusters whose nodes cannot
+// reach ghcr.io (e.g. IBM ROKS) can point it at an in-cluster registry mirror,
+// mirroring the HYPERSHELL_DATABASE_IMAGE override for the gateway database.
+func (StaticImageDefaults) DefaultSandboxImage() string {
+	if v := os.Getenv("GATEWAY_SANDBOX_IMAGE"); v != "" {
+		return v
+	}
+	return defaultSandboxImage
+}
+
 type NamespaceConfig struct {
 	Name    string        `yaml:"name"`
 	Gateway GatewayConfig `yaml:"gateway"`
diff --git a/components/control-plane/internal/gateway/manifests.go b/components/control-plane/internal/gateway/manifests.go
index ca5b36f7..f95b0e55 100644
--- a/components/control-plane/internal/gateway/manifests.go
+++ b/components/control-plane/internal/gateway/manifests.go
@@ -92,6 +92,10 @@ func ApplyManifestToNamespace(manifest *unstructured.Unstructured, namespace str
 	}
 	manifestJSON = strings.ReplaceAll(manifestJSON, "SUPERVISOR_IMAGE_PLACEHOLDER", supervisorImage)
 
+	// Replace SANDBOX_IMAGE_PLACEHOLDER before IMAGE_PLACEHOLDER because the
+	// shorter string is a substring of the longer one.
+	manifestJSON = strings.ReplaceAll(manifestJSON, "SANDBOX_IMAGE_PLACEHOLDER", images.DefaultSandboxImage())
+
 	image := images.DefaultGatewayImage()
 	if config.Image != "" {
 		image = config.Image
diff --git a/components/control-plane/internal/gateway/validation.go b/components/control-plane/internal/gateway/validation.go
index 897400fc..0c7e69b8 100644
--- a/components/control-plane/internal/gateway/validation.go
+++ b/components/control-plane/internal/gateway/validation.go
@@ -9,7 +9,10 @@ import (
 
 var (
 	dnsLabelRegex = regexp.MustCompile(`^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$`)
-	imageRefRegex = regexp.MustCompile(`^([a-z0-9.-]+/)?[a-z0-9._-]+(/[a-z0-9._-]+)*(:[a-z0-9._-]+)?(@sha256:[a-f0-9]{64})?$`)
+	// Optional registry host may carry a port (e.g. the in-cluster registry
+	// service address "image-registry.openshift-image-registry.svc:5000/..."),
+	// which standard Docker image references permit as host[:port]/path[:tag].
+	imageRefRegex = regexp.MustCompile(`^([a-z0-9.-]+(:[0-9]+)?/)?[a-z0-9._-]+(/[a-z0-9._-]+)*(:[a-z0-9._-]+)?(@sha256:[a-f0-9]{64})?$`)
 )
 
 func ValidateDNSName(name string) error {
diff --git a/components/control-plane/internal/gateway/validation_test.go b/components/control-plane/internal/gateway/validation_test.go
index 9b6cd4ff..5adc179d 100644
--- a/components/control-plane/internal/gateway/validation_test.go
+++ b/components/control-plane/internal/gateway/validation_test.go
@@ -4,6 +4,36 @@ import (
 	"testing"
 )
 
+func TestValidateImageReference(t *testing.T) {
+	tests := []struct {
+		name    string
+		ref     string
+		wantErr bool
+	}{
+		{name: "empty is invalid", ref: "", wantErr: true},
+		{name: "bare name with tag", ref: "postgres:18", wantErr: false},
+		{name: "docker hub library path", ref: "docker.io/library/postgres:18", wantErr: false},
+		{name: "ghcr multi-segment path with tag", ref: "ghcr.io/nvidia/openshell/gateway:0.0.101", wantErr: false},
+		{name: "quay long path with tag", ref: "quay.io/redhat-services-prod/hcm-eng-prod-tenant/hypershell-api-server-main:dev", wantErr: false},
+		{name: "digest reference", ref: "registry.redhat.io/rhel9/postgresql-16@sha256:" + "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", wantErr: false},
+		// In-cluster registry service address carries an explicit port; this is
+		// the address the kubelet pulls mirrored images by on ROKS.
+		{name: "internal registry host with port", ref: "image-registry.openshift-image-registry.svc:5000/openshift/openshell-gateway:0.0.101", wantErr: false},
+		{name: "internal registry host with port, no tag", ref: "image-registry.openshift-image-registry.svc:5000/openshift/postgres", wantErr: false},
+		{name: "localhost registry with port", ref: "localhost:5000/hypershell-controller:dev", wantErr: false},
+		{name: "shell metacharacter rejected", ref: "postgres:18;rm -rf /", wantErr: true},
+		{name: "command substitution rejected", ref: "postgres:$(whoami)", wantErr: true},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			err := ValidateImageReference(tt.ref)
+			if (err != nil) != tt.wantErr {
+				t.Fatalf("ValidateImageReference(%q) error = %v, wantErr %v", tt.ref, err, tt.wantErr)
+			}
+		})
+	}
+}
+
 func TestValidateCredentialDriverConfig(t *testing.T) {
 	tests := []struct {
 		name    string
diff --git a/components/control-plane/manifests/gateway/configmap.yaml b/components/control-plane/manifests/gateway/configmap.yaml
index 8d897c62..a9f1651b 100644
--- a/components/control-plane/manifests/gateway/configmap.yaml
+++ b/components/control-plane/manifests/gateway/configmap.yaml
@@ -20,7 +20,7 @@ data:
     metrics_bind_address  = "0.0.0.0:9090"
     log_level             = "info"
     sandbox_namespace     = "NAMESPACE_PLACEHOLDER"
-    default_image         = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest"
+    default_image         = "SANDBOX_IMAGE_PLACEHOLDER"
     supervisor_image      = "SUPERVISOR_IMAGE_PLACEHOLDER"
     client_tls_secret_name = "openshell-client-tls"
     enable_loopback_service_http = true

From a08fc6859fef23230bc0fa9ca9ea7e04ef7a9f93 Mon Sep 17 00:00:00 2001
From: user <u@example.com>
Date: Sat, 15 Aug 2026 20:20:06 -0400
Subject: [PATCH 09/15] fix(control-plane): add derived ingress hostname to
 gateway certificate SANs

When an ingress mode is active the gateway is reachable at an external
hostname (gw-<namespace>.<base-domain> or an explicit Route.Host), and both
ingress modes carry the gateway pod's TLS through unmodified (Route
passthrough / Gateway API BackendTLSPolicy). The server certificate must
therefore list that external hostname as a SAN, or clients fail verification.
The controller derives the hostname, so it injects it into the cert SANs
before cert-manager mints the certificate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 .../internal/gateway/reconciler.go            | 32 +++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/components/control-plane/internal/gateway/reconciler.go b/components/control-plane/internal/gateway/reconciler.go
index dba0bae5..3849f75a 100644
--- a/components/control-plane/internal/gateway/reconciler.go
+++ b/components/control-plane/internal/gateway/reconciler.go
@@ -53,6 +53,23 @@ func ReconcileGateway(
 		return fmt.Errorf("invalid gateway configuration: %w", err)
 	}
 
+	// When an ingress mode is active the gateway is reachable at an external
+	// hostname (gw-<namespace>.<base-domain>, or an explicit Route.Host). The
+	// gateway pod terminates TLS with its own per-tenant CA, and both ingress
+	// modes carry that TLS through unmodified (Route passthrough / Gateway API
+	// BackendTLSPolicy), so the server certificate must list the external
+	// hostname as a SAN or clients fail verification. The controller derives
+	// that hostname, so it -- not the operator -- injects it into the cert SANs
+	// here, before cert-manager mints the certificate below.
+	if mode := gatewayIngressMode(opts); mode != ingressModeNone && nsConfig.Gateway.Route.Enabled {
+		hostname, err := deriveGatewayHostname(nsConfig)
+		if err != nil {
+			log.Printf("WARN cannot add ingress hostname to gateway certificate SANs in %s: %v", nsConfig.Name, err)
+		} else {
+			nsConfig.Gateway.ServerDnsNames = appendDNSNameIfMissing(nsConfig.Gateway.ServerDnsNames, hostname)
+		}
+	}
+
 	dbImage := nsConfig.Gateway.Database.Image
 	if dbImage == "" {
 		dbImage = images.DefaultDatabaseImage()
@@ -1476,6 +1493,21 @@ func deriveGatewayHostname(nsConfig NamespaceConfig) (string, error) {
 	return fmt.Sprintf("gw-%s.%s", nsConfig.Name, baseDomain), nil
 }
 
+// appendDNSNameIfMissing returns names with hostname appended, unless it is
+// empty or already present. Used to add the derived ingress hostname to the
+// gateway server certificate SANs without duplicating it.
+func appendDNSNameIfMissing(names []string, hostname string) []string {
+	if hostname == "" {
+		return names
+	}
+	for _, n := range names {
+		if n == hostname {
+			return names
+		}
+	}
+	return append(names, hostname)
+}
+
 // publishRouteAddress writes the externally reachable gRPC address back to the
 // API-server Gateway resource. Shared by both ingress modes.
 func publishRouteAddress(ctx context.Context, opts ReconcileOpts, namespace, hostname string) {

From 8e7afbd83722306d168b755efbfc3575c579d9f0 Mon Sep 17 00:00:00 2001
From: user <u@example.com>
Date: Sat, 15 Aug 2026 20:20:13 -0400
Subject: [PATCH 10/15] docs(spec): expand global-architecture with ROKS
 ingress and image-mirror details

Extend the global architecture specification with the ingress-mode and
internal-registry-mirror behaviour exercised on IBM ROKS, and bump the
forbidden-terms whitelist line reference to track the moved example path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 .forbidden-terms-whitelist.json            |  2 +-
 specs/platform/global-architecture.spec.md | 95 +++++++++++++++++++++-
 2 files changed, 92 insertions(+), 5 deletions(-)

diff --git a/.forbidden-terms-whitelist.json b/.forbidden-terms-whitelist.json
index fe8c1d6d..3b68abf8 100644
--- a/.forbidden-terms-whitelist.json
+++ b/.forbidden-terms-whitelist.json
@@ -46,7 +46,7 @@
   },
   {
     "filename": "specs/platform/global-architecture.spec.md",
-    "line": 825,
+    "line": 912,
     "rationale": "Example filename 'rosa-vteam.yaml' in a directory-tree illustration references a real external GitOps path; it is an example path, not the discouraged term."
   }
 ]
diff --git a/specs/platform/global-architecture.spec.md b/specs/platform/global-architecture.spec.md
index b16800d2..d704795d 100644
--- a/specs/platform/global-architecture.spec.md
+++ b/specs/platform/global-architecture.spec.md
@@ -478,15 +478,27 @@ Steps (Route mode on ROKS):
 2. **Deploy the control plane with the IBM overlay** (`deploy/ibm`), which sets
    `GATEWAY_INGRESS_MODE=route` and `GATEWAY_API_BASE_DOMAIN` to the cluster's
    ingress subdomain (IBM's free `*.<ingress-subdomain>.containers.appdomain.cloud`
-   wildcard - no cert-manager `ClusterIssuer`, no Route53, no external DNS needed).
+   wildcard). Route mode removes only the *ingress-layer* PKI - no wildcard
+   certificate, cert-manager `ClusterIssuer`, Route53, or external DNS. It does
+   **not** remove cert-manager itself: cert-manager (namespaced `Issuer` +
+   per-tenant `Certificate`s minting the `openshell-ca` chain for the pod's own
+   TLS + client mTLS) remains a **hard prerequisite in every ingress mode**
+   (`reconcileCertManagerResources`; reconcile fails closed without it). On ROKS,
+   OperatorHub is broken, so cert-manager is installed by mirroring its images into
+   the internal registry (see [`ibm-cluster`](../../skills/deploy/ibm-cluster/SKILL.md)).
 3. **Provision a test tenant gateway** with `route.enabled=true`. The control
    plane creates an OpenShift `Route` (`passthrough`) `openshell-gateway` in the
    tenant namespace with host `gw-<tenant>.<base-domain>`, backed by
    `Service openshell-gateway:8080`, plus the `openshell-gateway-allow-router`
    NetworkPolicy, and publishes `grpcs://<host>:443` as the gateway's route address.
-4. **Verify:** the `Route` reports `Admitted=True`; `openshell` CLI connects over
-   `gw-<tenant>.<base-domain>:443`. The pod's server cert SANs
-   (`ServerDnsNames`/`ExternalDns`) must include that hostname.
+   The tenant workload images (gateway, supervisor, gateway database) must be
+   **node-reachable**; on ROKS they are mirrored into the internal registry and
+   referenced by its in-cluster service address (see `ibm-cluster` Step 5).
+4. **Verify:** the `Route` reports `Admitted=True`; the served passthrough
+   certificate chains to the per-tenant `openshell-ca` and its SANs include
+   `gw-<tenant>.<base-domain>` (the control plane injects the derived ingress
+   hostname into the certificate SANs automatically - operators do not set it by
+   hand). A client with that CA connects over `gw-<tenant>.<base-domain>:443`.
 
 To later switch a cloud to `gateway-api` mode (e.g. if IBM fixes HostedCluster
 mirroring), unset `GATEWAY_INGRESS_MODE` (or set it to `gateway-api`) and complete
@@ -575,6 +587,81 @@ selectable via the `GATEWAY_INGRESS_MODE` env var (`gateway-api` | `route` |
 - WHEN the control plane reconciles a gateway
 - THEN it SHALL select `gateway-api` if the Gateway API is detected, otherwise `route` on OpenShift, otherwise skip managed ingress
 
+#### Requirement: cert-manager Is a Mode-Independent Prerequisite
+
+Every tenant gateway SHALL depend on cert-manager for its per-tenant PKI: a
+namespaced `Issuer` (`openshell-ca`) and `Certificate`s minting the gateway's own
+server TLS, client mTLS, and CA. This is independent of the ingress mode - `route`
+mode removes only the ingress-layer wildcard certificate/`ClusterIssuer`/Route53,
+not the pod-TLS layer. The control plane SHALL fail closed when cert-manager is
+absent.
+
+##### Scenario: Reconcile blocks without cert-manager
+
+- GIVEN a cluster without cert-manager installed (any ingress mode)
+- WHEN the control plane reconciles a tenant gateway
+- THEN it SHALL return an error and SHALL NOT deploy the gateway workload
+
+#### Requirement: Gateway Server Certificate Covers the Ingress Hostname
+
+Because both ingress modes carry the gateway pod's TLS through unmodified (Route
+passthrough / Gateway API `BackendTLSPolicy`), the gateway server certificate
+SHALL list the external ingress hostname (`gw-<tenant>.<base-domain>`, or an
+explicit `route.host`) as a SAN. The control plane derives that hostname, so it
+SHALL inject it into the certificate SANs itself, before cert-manager mints the
+certificate - operators SHALL NOT be required to set `external_dns`/
+`server_dns_names` by hand for the ingress hostname to be covered.
+
+##### Scenario: Derived hostname appears in the served certificate
+
+- GIVEN a tenant gateway with ingress enabled and `GATEWAY_API_BASE_DOMAIN` set
+- WHEN the control plane provisions it
+- THEN the per-tenant `openshell-server` `Certificate` SHALL include both the
+  in-cluster service DNS name and `gw-<tenant>.<base-domain>` as SANs
+- AND the certificate served over the ingress hostname SHALL verify against the
+  per-tenant `openshell-ca` with hostname verification enabled
+
+#### Requirement: Image References Support an In-Cluster Registry
+
+Gateway image references (gateway, supervisor) SHALL accept a registry host that
+carries an explicit port, as standard Docker references permit
+(`host[:port]/path[:tag]`), so images mirrored into the cluster-internal registry
+(`image-registry.openshift-image-registry.svc:5000/...`) - the only node-reachable
+source on ROKS - pass validation. The per-tenant database image SHALL be
+overridable via `HYPERSHELL_DATABASE_IMAGE` for the same reason, and the gateway's
+`supervisor_image` SHALL be settable through the API (`PATCH`) so sandboxes pull
+from a node-reachable registry.
+
+#### Requirement: Agent Sandbox CRD Is a Sandbox Prerequisite
+
+Tenant sandbox operations (`openshell sandbox list/create`) depend on the Agent
+Sandbox CRD + controller (`sandboxes.agents.x-k8s.io`, the upstream
+`kubernetes-sigs/agent-sandbox` project) being installed cluster-wide. The control
+plane SHALL provision only the per-tenant sandbox RBAC (SA, Role/RoleBinding
+against `agents.x-k8s.io`, and on OpenShift the privileged-SCC binding); it SHALL
+NOT install the CRD/controller, which is a cluster prerequisite on par with
+cert-manager. The gateway's Kubernetes compute driver watches this CRD; when it is
+absent the sandbox RPCs surface as gRPC `Unimplemented` and the driver logs `no
+supported Agent Sandbox API version is available`. The version installed SHALL
+serve the API version the gateway requires (`v1beta1` for gateway 0.0.101; upstream
+`v0.5.x`).
+
+#### Requirement: Sandbox Base Image Supports an In-Cluster Registry
+
+The base image tenant sandbox pods launch from (the gateway `default_image`) SHALL
+be overridable via `GATEWAY_SANDBOX_IMAGE` (control-plane env), so that on clusters
+whose nodes cannot reach `ghcr.io` (e.g. ROKS) it can be pointed at a mirror in the
+cluster-internal registry. This mirrors the `HYPERSHELL_DATABASE_IMAGE` override for
+the gateway database.
+
+##### Scenario: Sandbox launch on a cluster without public egress
+
+- GIVEN a cluster whose worker nodes cannot pull from `ghcr.io` or `registry.k8s.io`
+- AND the Agent Sandbox CRD + controller and the sandbox base image have been mirrored into the internal registry
+- WHEN `GATEWAY_SANDBOX_IMAGE` points the gateway `default_image` at the mirrored base image
+- THEN a created Sandbox's pod SHALL pull the base image from the internal registry and reach Running
+- AND the gateway compute driver SHALL watch `agents.x-k8s.io` without `Unimplemented` errors
+
 ### Open Questions (for implementer review)
 
 1. **[RESOLVED] ROKS Gateway API availability.** Verified on 2026-08-15 across two clusters: on the old 4.17 cluster the CRDs were absent; on the new `hysh-ibm-01` (4.21.27) the CRDs and feature gates are present but the Gateway API is **non-functional** (CIO `istiod` cannot pull OSSM images; IDMS denied by the HostedCluster; CIO reverts patches; OperatorHub broken). **Decision:** the IBM Cloud Hub uses the `route` ingress mode (`deploy/ibm`, `GATEWAY_INGRESS_MODE=route`), not Gateway API. See "IBM Cloud Cloud Hub - Route ingress mode".

From 5f6720cf6be54c9292b8246f8633f4dd5aac5db0 Mon Sep 17 00:00:00 2001
From: user <u@example.com>
Date: Sat, 15 Aug 2026 20:20:13 -0400
Subject: [PATCH 11/15] docs(skills): ROKS deploy guidance for ibm-cluster and
 deploy-cluster

Document the ROKS-specific deployment path: image mirroring to the internal
registry, raw operator installs, and the env overrides the control plane
needs on clusters that cannot reach ghcr.io.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 skills/deploy/deploy-cluster/SKILL.md |  26 ++-
 skills/deploy/ibm-cluster/SKILL.md    | 252 +++++++++++++++++++++++---
 2 files changed, 251 insertions(+), 27 deletions(-)

diff --git a/skills/deploy/deploy-cluster/SKILL.md b/skills/deploy/deploy-cluster/SKILL.md
index 09612974..78678a13 100644
--- a/skills/deploy/deploy-cluster/SKILL.md
+++ b/skills/deploy/deploy-cluster/SKILL.md
@@ -11,11 +11,23 @@ description: >
 
 > **Scope:** this skill deploys the **platform services** (API server, controller,
 > PostgreSQL) and is cloud-agnostic across OpenShift distributions (ROSA, ROKS,
-> self-managed). It does **not** stand up tenant-gateway ingress. The shared
-> `Gateway` + wildcard DNS/TLS that tenant traffic needs is a separate one-time
-> per-cluster bootstrap - see [`cloud-hub-ingress-bootstrap`](../cloud-hub-ingress-bootstrap/SKILL.md).
-> Without it, the controller runs but every gateway reconcile fails with
-> `GATEWAY_API_GATEWAY_NAME is required`.
+> self-managed). It does **not** stand up tenant-gateway ingress. How that ingress
+> is bootstrapped depends on the ingress mode:
+> - **`gateway-api` mode (AWS/ROSA, functional Gateway API):** a shared `Gateway` +
+>   wildcard DNS/TLS is a separate one-time per-cluster bootstrap - see
+>   [`cloud-hub-ingress-bootstrap`](../cloud-hub-ingress-bootstrap/SKILL.md). Without
+>   it the controller runs but every gateway reconcile fails with
+>   `GATEWAY_API_GATEWAY_NAME is required`.
+> - **`route` mode (IBM ROKS):** no shared Gateway; the control plane emits
+>   passthrough `Route`s. Use the **`deploy/ibm` overlay** and follow
+>   [`ibm-cluster`](../ibm-cluster/SKILL.md) **Step 5**, which layers the
+>   ROKS-specific requirements onto this skill (mirror *all* images including
+>   cert-manager and the tenant gateway/supervisor/DB images; `pusher` SA token for
+>   registry push; cluster-wide controller RBAC incl. `routes/custom-host` and
+>   privileged-SCC `use`; `HYPERSHELL_DATABASE_IMAGE`).
+>
+> cert-manager is a hard prerequisite for tenant gateways in **both** modes (it
+> mints each tenant's per-tenant CA for pod TLS); reconcile fails closed without it.
 
 ## Cloud-Hub Parameter Overrides
 
@@ -28,7 +40,9 @@ the `oc`-based flow is otherwise identical.
 | Internal registry route | `...elb/openshift-image-registry` host | `default-route-openshift-image-registry...appdomain.cloud` |
 | Namespace | `hypershell-api` | `hypershell-api` (same) |
 | PostgreSQL storage class | cluster default | `ibmc-vpc-block-10iops-tier` (pin on the postgres PVC) |
-| Tenant-gateway ingress | via `cloud-hub-ingress-bootstrap` (ELB) | via `cloud-hub-ingress-bootstrap` (VPC LB) |
+| Tenant-gateway ingress | `gateway-api` mode via `cloud-hub-ingress-bootstrap` (ELB) | `route` mode via `deploy/ibm` overlay - see [`ibm-cluster`](../ibm-cluster/SKILL.md) Step 5 (no shared Gateway) |
+| Image sourcing | build + push platform images | mirror **all** images (platform + cert-manager + tenant gateway/supervisor/DB) into the internal registry; nodes can't reach quay/ghcr/dockerhub |
+| Registry push identity | `oc whoami -t` | cert-admin has no token → mint a `pusher` SA token (see `ibm-cluster` Step 5.0) |
 
 ## Platform Components
 
diff --git a/skills/deploy/ibm-cluster/SKILL.md b/skills/deploy/ibm-cluster/SKILL.md
index aca89b3a..108a46e9 100644
--- a/skills/deploy/ibm-cluster/SKILL.md
+++ b/skills/deploy/ibm-cluster/SKILL.md
@@ -269,37 +269,247 @@ and reusable for whichever path wins.
    [`global-architecture.spec.md`](../../../specs/platform/global-architecture.spec.md)
    (§ "IBM Cloud Cloud Hub - Route ingress mode").
 
-## Step 5: Hand off (Route ingress mode)
+## Step 5: Deploy HyperShell + a tenant gateway (Route ingress mode)
 
 ROKS uses the **Route ingress mode**, not Gateway API. Do **not** run
 `cloud-hub-ingress-bootstrap` (that bootstraps the shared Gateway for
 `gateway-api` mode on AWS/functional clusters).
 
-1. `deploy-cluster` with the **`deploy/ibm` overlay** - platform services
-   (API/controller/PostgreSQL) plus `GATEWAY_INGRESS_MODE=route`. Use its
-   Cloud-Hub Parameter Overrides (registry host, storage class `ibmc-vpc-block-*`),
-   and set `GATEWAY_API_BASE_DOMAIN` to this cluster's ingress subdomain:
+The governing constraint on ROKS is that **worker nodes can pull only from IBM
+registries and the cluster-internal registry** - `quay.io`, `registry.redhat.io`,
+`registry.access.redhat.com`, `ghcr.io`, and `docker.io` are all unreachable from
+nodes. So **every** image the platform *and every tenant gateway* uses must be
+mirrored into the internal registry and referenced by its in-cluster service
+address (`image-registry.openshift-image-registry.svc:5000/...`). This includes
+images most deploys take for granted: cert-manager, and the per-tenant gateway,
+supervisor, and database images. Verified end-to-end on `hysh-ibm-01` (2026-08-15).
 
-   ```bash
-   oc get ingresscontroller default -n openshift-ingress-operator \
-     -o jsonpath='{.status.domain}{"\n"}'   # -> *.<subdomain>.containers.appdomain.cloud
-   ```
+### 5.0: Registry push identity (cert-admin has no bearer token)
 
-2. **Provision a test tenant gateway** with `route.enabled=true`. The control
-   plane creates a passthrough `Route` `openshell-gateway` in the tenant namespace
-   (host `gw-<tenant>.<base-domain>`) and publishes `grpcs://<host>:443`.
+`ibmcloud ks cluster config --admin` gives a **certificate-based** kubeconfig with
+no bearer token, so `oc whoami -t` fails and you cannot `podman login` the registry
+route as yourself. Mint a ServiceAccount token instead:
 
-3. **Verify** the Route is admitted and the CLI connects:
+```bash
+oc -n hypershell create sa pusher
+oc adm policy add-cluster-role-to-user system:image-builder -z pusher -n hypershell   # cluster-wide: lets pusher write any namespace's imagestreams
+REG=$(oc get route default-route -n openshift-image-registry -o jsonpath='{.spec.host}')
+podman login --tls-verify=false -u pusher -p "$(oc -n hypershell create token pusher --duration=2h)" "$REG"
+```
+
+Tokens are short-lived; re-mint (and re-`podman login`) if a later push 401s.
+
+### 5.1: Install cert-manager (hard prerequisite, all ingress modes)
+
+cert-manager is **not** optional in route mode. It mints each tenant's per-tenant
+CA (`openshell-ca` Issuer + `openshell-server`/`openshell-client`/`openshell-ca`
+Certificates) for the gateway pod's own TLS and client mTLS. Route mode only drops
+the *ingress-layer* PKI (wildcard cert / ClusterIssuer / Route53), not this. The
+control plane fails closed (`cert-manager is required but not available`) without
+it. OperatorHub is broken on ROKS (catalog pods `ImagePullBackOff`), so install by
+mirroring the images:
+
+```bash
+V=v1.16.3
+for I in controller webhook cainjector; do
+  skopeo copy --dest-tls-verify=false --dest-creds "pusher:$(oc -n hypershell create token pusher)" \
+    docker://quay.io/jetstack/cert-manager-$I:$V \
+    docker://$REG/cert-manager/cert-manager-$I:$V
+done
+# download the release manifest, repoint the three images at the mirror, apply:
+curl -sL https://github.com/cert-manager/cert-manager/releases/download/$V/cert-manager.yaml \
+  | sed -E 's#quay.io/jetstack/(cert-manager-[a-z]+):#image-registry.openshift-image-registry.svc:5000/cert-manager/\1:#g' \
+  | oc apply -f -
+oc -n cert-manager rollout status deploy/cert-manager
+```
+
+The controller caches cert-manager detection at startup, so **restart it** after
+cert-manager is up: `oc -n hypershell rollout restart deploy/hypershell-controller`.
+
+### 5.2: Mirror the platform + tenant workload images
+
+Mirror the platform images (api-server, controller, PostgreSQL) per
+[`deploy-cluster`](../deploy-cluster/SKILL.md) Step 3, but into the `deploy/ibm`
+overlay's target repos (it repoints images to
+`.svc:5000/hypershell/{hypershell-api-server,hypershell-controller,postgresql}`).
+Then mirror the **tenant gateway** images into the `openshift` namespace, which is
+cluster-wide pullable by every namespace's default SA (so per-tenant image-pull
+secrets are unnecessary):
+
+```bash
+# postgres/RHEL images carry signatures the internal registry rejects -> --remove-signatures
+skopeo copy --remove-signatures --dest-tls-verify=false --dest-creds "pusher:$(oc -n hypershell create token pusher)" \
+  docker://docker.io/library/postgres:18 docker://$REG/openshift/postgres:18
+skopeo copy --dest-tls-verify=false --dest-creds "pusher:$(oc -n hypershell create token pusher)" \
+  docker://ghcr.io/nvidia/openshell/gateway:0.0.101    docker://$REG/openshift/openshell-gateway:0.0.101
+skopeo copy --dest-tls-verify=false --dest-creds "pusher:$(oc -n hypershell create token pusher)" \
+  docker://ghcr.io/nvidia/openshell/supervisor:0.0.101 docker://$REG/openshift/openshell-supervisor:0.0.101
+oc -n openshift get is    # expect openshell-gateway, openshell-supervisor, postgres
+```
+
+### 5.3: Deploy with the `deploy/ibm` overlay
+
+```bash
+cd components/api-server && oc kustomize deploy/ibm | oc apply -f -
+oc -n hypershell rollout status deploy/hypershell-controller
+```
+
+The overlay (on top of `deploy/openshift`, namespace `hypershell`) sets, and you
+must keep aligned with this cluster:
+
+- `GATEWAY_INGRESS_MODE=route` and `GATEWAY_API_BASE_DOMAIN=<ingress subdomain>`
+  (`oc get ingresscontroller default -n openshift-ingress-operator -o jsonpath='{.status.domain}'`).
+- `HYPERSHELL_DATABASE_IMAGE=...svc:5000/openshift/postgres:18` - the per-tenant
+  gateway database image (nodes can't pull Docker Hub `postgres:18`).
+- Image transformers repointing api-server/controller/postgresql at `.svc:5000/hypershell/*`.
+- **`controller-clusterrbac.yaml`** - a cluster-wide `ClusterRole` for the
+  controller. The self-contained `deploy/openshift` tree ships only a narrow Role;
+  reconciling whole tenants needs cluster-wide namespaces/secrets/services/
+  deployments/networkpolicies plus, specifically for route mode + sandboxes:
+  `route.openshift.io/routes` **and `routes/custom-host`** (OpenShift gates setting
+  a Route's `spec.host` behind this subresource - without it the Route is rejected
+  with "you do not have permission to set the host field of the route"), and
+  `security.openshift.io/securitycontextconstraints` **`use`** on `privileged` (so
+  the controller can bind the sandbox SA to the privileged SCC without an RBAC
+  escalation error).
+
+### 5.4: Provision a tenant gateway (images must point at the mirror)
+
+The control plane reads the gateway's own `image` and `supervisor_image` fields
+(not the GatewayRelease image) and defaults them to `ghcr.io/...`, which nodes
+cannot pull. Set both to the mirrored internal refs, and pass `namespace`
+explicitly (the deployed API image still validates it as required despite the
+OpenAPI `readOnly` marking):
+
+```bash
+API="https://$(oc -n hypershell get route hypershell-api -o jsonpath='{.spec.host}')/api/hypershell/v1"
+# ...create Fleet, ManagedCluster, GatewayRelease, ManagedDatabase first...
+curl -sk -X POST "$API/gateways" -H 'Content-Type: application/json' -d '{
+  "name":"ibm-test-gw","fleet_id":"...","cluster_id":"...","release_id":"...","database_id":"...",
+  "namespace":"openshell-ibmtest",
+  "image":"image-registry.openshift-image-registry.svc:5000/openshift/openshell-gateway:0.0.101",
+  "supervisor_image":"image-registry.openshift-image-registry.svc:5000/openshift/openshell-supervisor:0.0.101",
+  "route":"{\"enabled\": true}"
+}'
+```
 
-   ```bash
-   NS=$(oc get ns -l hypershell.redhat.io/managed=true -o name | head -1 | cut -d/ -f2)
-   oc -n "$NS" get route openshell-gateway \
-     -o jsonpath='{.status.ingress[0].conditions[0].type}={.status.ingress[0].conditions[0].status}{"\n"}'  # Admitted=True
-   openshell login "gw-${NS}.<base-domain>:443"
-   ```
+The control plane then: mints the per-tenant CA via cert-manager, **auto-injects
+`gw-<tenant>.<base-domain>` into the server certificate SANs** (you do not set
+`external_dns` by hand), waits for the gateway DB, deploys the gateway, creates a
+`passthrough` `Route openshell-gateway` (host `gw-<tenant>.<base-domain>`), and
+publishes `grpcs://<host>:443`.
+
+### 5.5: Verify end to end
+
+```bash
+NS=openshell-ibmtest
+oc -n "$NS" get pods            # openshell-gateway + -db Running, -certgen Completed
+oc -n "$NS" get route openshell-gateway \
+  -o jsonpath='{.status.ingress[0].conditions[?(@.type=="Admitted")].status}{"\n"}'   # True
+HOST=$(oc -n "$NS" get route openshell-gateway -o jsonpath='{.spec.host}')
+
+# Passthrough serves the per-tenant cert; SANs include the external host (auto-injected):
+oc -n "$NS" get secret openshell-server-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/tenant-ca.crt
+openssl s_client -connect "$HOST:443" -servername "$HOST" -CAfile /tmp/tenant-ca.crt \
+  -verify_hostname "$HOST" </dev/null 2>/dev/null | grep 'Verify return code'   # 0 (ok)
+
+# gRPC transport through the Route (edge/OIDC auth aside, this proves the path):
+openshell status --gateway-endpoint "https://$HOST:443" --gateway-insecure   # Status: Connected
+```
+
+`openshell gateway add https://<host>` drives the interactive edge/OIDC login
+flow and populates the CLI's mTLS/CA material under `~/.config/openshell/`; it is
+not needed to prove the infrastructure path above.
+
+### 5.6: Enable Agent Sandboxes (required for `openshell sandbox ...`)
+
+A running gateway can serve `status`, but `openshell sandbox list/create` return
+gRPC `Unimplemented` until the **Agent Sandbox CRD + controller**
+(`sandboxes.agents.x-k8s.io`, the upstream `kubernetes-sigs/agent-sandbox`
+project) are installed cluster-wide. The gateway's Kubernetes compute driver
+watches this CRD; without it the driver loops on `no supported Agent Sandbox API
+version is available; tried v1beta1, v1alpha1` (404s). The HyperShell control
+plane grants the tenant SA RBAC *against* `agents.x-k8s.io` and mints the
+per-tenant sandbox SA + privileged-SCC binding, but it does **not** install the
+CRD/controller - that is a cluster prerequisite, like cert-manager. Verified on
+`hysh-ibm-01` (2026-08-15) with `agent-sandbox` **v0.5.5** (first line to serve
+`v1beta1`, which gateway 0.0.101 prefers).
+
+Same ROKS constraint as everything else: the controller image
+(`registry.k8s.io/agent-sandbox/agent-sandbox-controller:v0.5.5`) and the tenant
+sandbox base image (`ghcr.io/nvidia/openshell-community/sandboxes/base:latest`)
+are **not node-reachable**, so mirror both into the internal `openshift`
+namespace (cluster-wide pullable) and repoint.
+
+```bash
+# a) mirror the controller image + the sandbox base image (openshift ns = globally pullable)
+skopeo copy --dest-tls-verify=false --dest-creds "pusher:$(oc -n hypershell create token pusher)" \
+  docker://registry.k8s.io/agent-sandbox/agent-sandbox-controller:v0.5.5 \
+  docker://$REG/openshift/agent-sandbox-controller:v0.5.5
+skopeo copy --remove-signatures --dest-tls-verify=false --dest-creds "pusher:$(oc -n hypershell create token pusher)" \
+  docker://ghcr.io/nvidia/openshell-community/sandboxes/base:latest \
+  docker://$REG/openshift/openshell-sandbox-base:latest
+
+# b) install the CRD + controller, repointing ONLY the controller image at the mirror
+V=v0.5.5
+curl -sL https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$V/sandbox.yaml \
+  | sed "s#registry.k8s.io/agent-sandbox/agent-sandbox-controller:$V#image-registry.openshift-image-registry.svc:5000/openshift/agent-sandbox-controller:$V#g" \
+  | oc apply -f -
+oc -n agent-sandbox-system rollout status deploy/agent-sandbox-controller
+oc get crd sandboxes.agents.x-k8s.io -o jsonpath='{.status.conditions[?(@.type=="Established")].status}{"\n"}'   # True
+```
+
+The upstream controller Deployment declares no `securityContext`, so OpenShift's
+`restricted-v2` SCC assigns a UID and it runs unmodified (the `fsGroup`/`runAsUser`
+blocks in `sandbox.yaml` are inside the CRD's OpenAPI *schema examples*, not the
+controller pod). The CRD's conversion webhook is served by the controller itself
+(it self-injects the caBundle via its `customresourcedefinitions` patch grant); no
+cert-manager `Certificate` is needed. Because v0.5.x serves a single real version,
+conversion is never invoked, so `list` works as soon as the CRD is Established.
+
+**The gateway caches API discovery at startup**, so after the CRD exists you must
+restart each already-running tenant gateway once for its compute driver to pick up
+`agents.x-k8s.io`:
+
+```bash
+oc -n "$NS" rollout restart deploy/openshell-gateway   # then: 0 "no supported Agent Sandbox" logs; "Compute driver connected"
+```
+
+**Sandbox base image must be node-reachable too.** The gateway's `default_image`
+(the base image every sandbox pod runs) was previously hardcoded to the ghcr path
+with no override; it is now `SANDBOX_IMAGE_PLACEHOLDER`, resolved from
+`GATEWAY_SANDBOX_IMAGE` (control-plane env, set in the `deploy/ibm` overlay to the
+mirror). Rebuild + redeploy the controller (5.2/5.3) and force one re-reconcile
+(`PATCH /gateways/<id> {"phase":""}`) so the tenant configmap re-renders with the
+mirrored `default_image`, then restart the gateway. Without this a CLI-created
+sandbox is admitted but its pod `ImagePullBackOff`s on the ghcr base image.
+
+Verify (headless, no CLI auth needed - the sandbox RPC requires an authenticated
+caller since `allow_unauthenticated_users=false`, so `openshell sandbox list` over
+`--gateway-insecure` returns `Unauthenticated` rather than `Unimplemented` once the
+CRD is in; to prove the controller + mirrored-image path directly, apply a Sandbox CR):
+
+```bash
+cat <<'EOF' | oc apply -f -
+apiVersion: agents.x-k8s.io/v1beta1
+kind: Sandbox
+metadata: { name: mirror-test, namespace: NS_HERE }
+spec:
+  podTemplate:
+    spec:
+      serviceAccountName: openshell-gateway-sandbox
+      containers:
+        - name: sandbox
+          image: image-registry.openshift-image-registry.svc:5000/openshift/openshell-sandbox-base:latest
+          command: ["sleep", "3600"]
+EOF
+oc -n "$NS" get sandbox mirror-test          # READY=True, REASON=DependenciesReady
+oc -n "$NS" get pod mirror-test -o jsonpath='{.status.containerStatuses[0].imageID}{"\n"}'  # ...svc:5000/openshift/openshell-sandbox-base@sha256:...
+oc -n "$NS" delete sandbox mirror-test
+```
 
-   The gateway pod's server cert SANs (`ServerDnsNames`/`ExternalDns`) must
-   include `gw-<tenant>.<base-domain>` for passthrough TLS to validate.
+Full authenticated `openshell sandbox create` additionally needs
+`openshell gateway add` (interactive edge/OIDC) so the CLI carries a bearer token.
 
 To later switch to Gateway API (if IBM fixes HostedCluster mirroring), unset
 `GATEWAY_INGRESS_MODE` and run `cloud-hub-ingress-bootstrap`; the control plane

From 0b43067fe3ca7c3bff5a4ed77b3b17339d1c70fc Mon Sep 17 00:00:00 2001
From: user <u@example.com>
Date: Sat, 15 Aug 2026 20:20:19 -0400
Subject: [PATCH 12/15] test(pr-test): ROKS-adapted e2e-openshell script and
 ibm deploy overlay

Add e2e-openshell-roks.sh, a ROKS-layout copy of the canonical e2e that
targets the hypershell/hypershell-api route names, trusts the per-gateway
CA, uses the quoted SQL-like search grammar, and preserves a pre-existing
gateway on cleanup. Add the components/api-server/deploy/ibm overlay with
the controller cluster RBAC used on ROKS.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 .../deploy/ibm/controller-clusterrbac.yaml    |  90 +++
 .../api-server/deploy/ibm/kustomization.yaml  |  87 ++
 components/pr-test/e2e-openshell-roks.sh      | 744 ++++++++++++++++++
 3 files changed, 921 insertions(+)
 create mode 100644 components/api-server/deploy/ibm/controller-clusterrbac.yaml
 create mode 100644 components/api-server/deploy/ibm/kustomization.yaml
 create mode 100755 components/pr-test/e2e-openshell-roks.sh

diff --git a/components/api-server/deploy/ibm/controller-clusterrbac.yaml b/components/api-server/deploy/ibm/controller-clusterrbac.yaml
new file mode 100644
index 00000000..3a797108
--- /dev/null
+++ b/components/api-server/deploy/ibm/controller-clusterrbac.yaml
@@ -0,0 +1,90 @@
+# Cluster-scoped controller RBAC for full tenant reconciliation.
+#
+# The self-contained ../openshift tree ships only a narrow Role (gateways in
+# openshift-ingress), which is insufficient once the control plane reconciles
+# whole tenant gateways: it creates per-tenant namespaces, Deployments,
+# Services, Secrets, NetworkPolicies, and -- in Route ingress mode -- OpenShift
+# Routes, all cluster-wide. This mirrors deploy/base/controller-rbac.yaml (the
+# GitOps tree's canonical grant), bound to the hypershell/hypershell-controller
+# service account used by ../openshift.
+#
+# Not IBM-specific in itself; carried in this overlay until the ../openshift base
+# grows the same ClusterRole. See specs/platform/global-architecture.spec.md.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: hypershell-controller
+  labels:
+    app.kubernetes.io/name: hypershell
+    app.kubernetes.io/component: controller
+rules:
+  - apiGroups: [""]
+    resources: ["namespaces", "secrets", "configmaps", "services", "serviceaccounts", "persistentvolumeclaims"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+  - apiGroups: ["apps"]
+    resources: ["deployments", "statefulsets"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+  - apiGroups: ["batch"]
+    resources: ["jobs"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+  - apiGroups: ["rbac.authorization.k8s.io"]
+    resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+  - apiGroups: ["networking.k8s.io"]
+    resources: ["networkpolicies"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+  - apiGroups: ["cert-manager.io"]
+    resources: ["issuers", "certificates"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+  - apiGroups: ["gateway.networking.k8s.io"]
+    resources: ["gateways", "grpcroutes", "backendtlspolicies"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+  - apiGroups: ["route.openshift.io"]
+    resources: ["routes"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+  # Route ingress mode derives an explicit spec.host (gw-<tenant>.<base-domain>).
+  # OpenShift gates setting a Route's host behind the routes/custom-host
+  # subresource; without this the Route is rejected
+  # ("you do not have permission to set the host field of the route").
+  - apiGroups: ["route.openshift.io"]
+    resources: ["routes/custom-host"]
+    verbs: ["create", "update"]
+  # The controller binds the sandbox service account to the privileged SCC
+  # (roleRef system:openshift:scc:privileged) so agent sandboxes can run. RBAC
+  # escalation prevention requires the controller to itself hold "use" of that
+  # SCC before it can grant it; without this the RoleBinding create is forbidden.
+  - apiGroups: ["security.openshift.io"]
+    resources: ["securitycontextconstraints"]
+    resourceNames: ["privileged"]
+    verbs: ["use"]
+  - apiGroups: ["apiextensions.k8s.io"]
+    resources: ["customresourcedefinitions"]
+    verbs: ["get", "list"]
+  - apiGroups: [""]
+    resources: ["nodes"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["pods", "events"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["authentication.k8s.io"]
+    resources: ["tokenreviews"]
+    verbs: ["create"]
+  - apiGroups: ["agents.x-k8s.io"]
+    resources: ["sandboxes", "sandboxes/status"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: hypershell-controller
+  labels:
+    app.kubernetes.io/name: hypershell
+    app.kubernetes.io/component: controller
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: hypershell-controller
+subjects:
+  - kind: ServiceAccount
+    name: hypershell-controller
+    namespace: hypershell
diff --git a/components/api-server/deploy/ibm/kustomization.yaml b/components/api-server/deploy/ibm/kustomization.yaml
new file mode 100644
index 00000000..2a5a6b80
--- /dev/null
+++ b/components/api-server/deploy/ibm/kustomization.yaml
@@ -0,0 +1,87 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+# IBM Cloud (ROKS) overlay for the dev/no-auth platform deploy.
+#
+# This is the environment adapter for the self-contained OpenShift deploy in
+# ../openshift (namespace `hypershell`, the tree documented by the
+# `deploy-cluster` skill). It changes only what the ROKS environment forces us
+# to change, leaving the architecture identical:
+#
+#   1. Ingress mode -> Route. ROKS is HyperShift-hosted and cannot run the
+#      CIO-managed Gateway API (OSSM images unpullable, node mirroring / IDMS
+#      owned by the HostedCluster). The control plane instead emits passthrough
+#      OpenShift Routes on IBM's free "*.containers.appdomain.cloud" wildcard,
+#      selected by GATEWAY_INGRESS_MODE=route. Passthrough preserves the gateway
+#      pod's per-tenant self-signed CA TLS end to end -- no shared Gateway,
+#      wildcard cert, cert-manager ClusterIssuer, or Route53.
+#
+#   2. Image sources -> internal registry. ROKS worker nodes cannot reach
+#      quay.io or registry.access.redhat.com (egress is restricted to IBM
+#      registries), so all images are mirrored into the cluster internal
+#      registry and referenced by their in-cluster service address
+#      (image-registry.openshift-image-registry.svc:5000/hypershell/...), which
+#      the kubelet can resolve and pull.
+#
+# See:
+#   specs/platform/global-architecture.spec.md (§ IBM Cloud Cloud Hub - Route ingress mode)
+#   skills/deploy/ibm-cluster/SKILL.md (Step 5)
+#   skills/deploy/deploy-cluster/SKILL.md (Cloud-Hub Parameter Overrides)
+
+resources:
+  - ../openshift
+  - controller-clusterrbac.yaml
+
+# --- point every image at the mirrored copy in the internal registry ---
+# The kubelet pulls these by their internal service address; the default
+# service account in the `hypershell` namespace already has system:image-puller
+# for same-namespace image streams.
+images:
+  - name: quay.io/redhat-services-prod/hcm-eng-prod-tenant/hypershell-main/hypershell-api-server-main
+    newName: image-registry.openshift-image-registry.svc:5000/hypershell/hypershell-api-server
+    newTag: dev
+  - name: quay.io/redhat-services-prod/hcm-eng-prod-tenant/hypershell-main/hypershell-control-plane-main
+    newName: image-registry.openshift-image-registry.svc:5000/hypershell/hypershell-controller
+    newTag: dev
+  - name: registry.access.redhat.com/hi/postgresql
+    newName: image-registry.openshift-image-registry.svc:5000/hypershell/postgresql
+    newTag: "18.4"
+
+patches:
+  # --- select Route ingress mode + this cluster's ingress subdomain ---
+  # Strategic-merge on the env list (merged by name): adds GATEWAY_INGRESS_MODE
+  # and overrides the base-domain placeholder with the ROKS ingress subdomain, so
+  # tenant gateways get "gw-<tenant>.<subdomain>" hosts under the free wildcard.
+  - patch: |
+      apiVersion: apps/v1
+      kind: Deployment
+      metadata:
+        name: hypershell-controller
+        namespace: hypershell
+      spec:
+        template:
+          spec:
+            containers:
+              - name: controller
+                env:
+                  - name: GATEWAY_INGRESS_MODE
+                    value: route
+                  - name: GATEWAY_API_BASE_DOMAIN
+                    value: hysh-ibm-01-4c28435107377e996c6eb39230b7bcf5-0000.us-east.containers.appdomain.cloud
+                  # ROKS nodes cannot pull Docker Hub, so the per-tenant gateway
+                  # database image is served from the internal registry (mirror
+                  # of docker.io/library/postgres:18 in the openshift namespace,
+                  # globally pullable). The name stays "postgres" (not RHEL
+                  # "postgresql-"), so the reconciler's variant detection
+                  # (reconciler.go:1186) keeps the POSTGRES_* env + data-path
+                  # conventions that match this image.
+                  - name: HYPERSHELL_DATABASE_IMAGE
+                    value: image-registry.openshift-image-registry.svc:5000/openshift/postgres:18
+                  # Tenant sandbox pods launch from this base image. ROKS nodes
+                  # cannot pull ghcr.io, so it is served from the internal
+                  # registry (mirror of
+                  # ghcr.io/nvidia/openshell-community/sandboxes/base:latest in
+                  # the openshift namespace, globally pullable). Without this the
+                  # Sandbox CR is admitted but its pod ImagePullBackOffs.
+                  - name: GATEWAY_SANDBOX_IMAGE
+                    value: image-registry.openshift-image-registry.svc:5000/openshift/openshell-sandbox-base:latest
diff --git a/components/pr-test/e2e-openshell-roks.sh b/components/pr-test/e2e-openshell-roks.sh
new file mode 100755
index 00000000..bdceb637
--- /dev/null
+++ b/components/pr-test/e2e-openshell-roks.sh
@@ -0,0 +1,744 @@
+#!/usr/bin/env bash
+# e2e-openshell.sh - end-to-end test of the OpenShell gateway provisioned by HyperShell.
+#
+# Proves the full path: HyperShell API → control plane → gateway provisioning
+# → openshell CLI → sandbox pod creation + interaction.
+#
+# This script creates a gateway via the HyperShell API (if it doesn't exist),
+# waits for the controller to provision it, then validates connectivity and
+# sandbox lifecycle.
+#
+# Usage:
+#   bash e2e-openshell.sh
+#
+# Environment variables:
+#   OC                   oc/kubectl binary (default: oc)
+#   HYPERSHELL_NAMESPACE API server namespace (default: hypershell-api)
+#   GATEWAY_NAME         gateway name (default: e2e-gw)
+#   SANDBOX_TIMEOUT      seconds to wait for sandbox (default: 120)
+#   PROVISION_TIMEOUT    seconds to wait for gateway provisioning (default: 180)
+#   SKIP_CLEANUP         set to 1 to keep resources after test
+#   LAUNCH_TUI           set to 1 to launch interactive TUI at the end (default: 0)
+#   PAUSE                seconds between commands (default: 1)
+set -euo pipefail
+
+# ROKS (hysh-ibm-01) defaults — this is the ROKS-adapted copy of e2e-openshell.sh,
+# so the defaults target the live IBM cluster layout. Every value is still
+# env-overridable for other clusters.
+CLI="${OC:-oc}"
+OPENSHELL="${OPENSHELL_BIN:-/bin/openshell}"
+HSCTL="${HSCTL_BIN:-/home/mturansk/projects/bin/hsctl}"
+HS_NAMESPACE="${HYPERSHELL_NAMESPACE:-hypershell}"
+GW_NAMESPACE=""
+GW_NAME="${GATEWAY_NAME:-e2e-oidc-gw}"
+SANDBOX_TIMEOUT="${SANDBOX_TIMEOUT:-150}"
+PROVISION_TIMEOUT="${PROVISION_TIMEOUT:-180}"
+SKIP_CLEANUP="${SKIP_CLEANUP:-}"
+LAUNCH_TUI="${LAUNCH_TUI:-0}"
+PAUSE="${PAUSE:-0}"
+
+KC_NAMESPACE="${KEYCLOAK_NAMESPACE:-keycloak-system}"
+OIDC_CLIENT_ID="${OIDC_CLIENT_ID:-hypershell-frontend}"
+OIDC_USERNAME="${OIDC_USERNAME:-admin}"
+OIDC_PASSWORD="${OIDC_PASSWORD:-admin}"
+DEV_USERNAME="${DEV_USERNAME:-developer}"
+DEV_PASSWORD="${DEV_PASSWORD:-developer}"
+
+# Mirrored gateway images (ROKS nodes can only pull the internal registry). Used
+# only when this run has to CREATE the gateway; ignored when it already exists.
+REG_MIRROR="${REG_MIRROR:-image-registry.openshift-image-registry.svc:5000/openshift}"
+GW_IMAGE="${GW_IMAGE:-${REG_MIRROR}/openshell-gateway:0.0.101}"
+GW_SUPERVISOR_IMAGE="${GW_SUPERVISOR_IMAGE:-${REG_MIRROR}/openshell-supervisor:0.0.101}"
+# ROKS Keycloak realm emits a top-level 'roles' claim (not 'groups').
+OIDC_ROLES_CLAIM="${OIDC_ROLES_CLAIM:-roles}"
+
+PASS=0
+FAIL=0
+TESTS=()
+PF_PID=""
+SANDBOX_NAME=""
+GW_ID=""
+CREATED_GW=""   # only set when THIS run created the gateway; guards cleanup so a
+                # pre-existing (persistent) gateway is never deleted on exit.
+
+bold()   { printf '\033[1m%s\033[0m\n' "$*"; }
+green()  { printf '\033[32m%s\033[0m\n' "$*"; }
+red()    { printf '\033[31m%s\033[0m\n' "$*"; }
+dim()    { printf '\033[2m%s\033[0m\n' "$*"; }
+cyan()   { printf '\033[36m%s\033[0m\n' "$*"; }
+orange() { printf '\033[38;5;214m%s\033[0m\n' "$*"; }
+sep()    { printf '\033[2m────────────────────────────────────────────────\033[0m\n'; }
+
+show_cmd() {
+  orange "   \$ $*"
+  sleep "$PAUSE"
+}
+
+pass() {
+  PASS=$((PASS + 1))
+  TESTS+=("PASS: $1")
+  green "  ✓ $1"
+}
+
+fail_test() {
+  FAIL=$((FAIL + 1))
+  TESTS+=("FAIL: $1")
+  red "  ✗ $1"
+}
+
+cleanup() {
+  if [[ -n "${SB_CREATE_PID:-}" ]]; then
+    kill "$SB_CREATE_PID" 2>/dev/null || true
+    wait "$SB_CREATE_PID" 2>/dev/null || true
+  fi
+  if [[ -n "$PF_PID" ]]; then
+    kill "$PF_PID" 2>/dev/null || true
+    wait "$PF_PID" 2>/dev/null || true
+  fi
+  if [[ "$SKIP_CLEANUP" != "1" && "$CREATED_GW" == "1" && -n "$GW_ID" ]]; then
+    dim "  Cleaning up gateway ${GW_NAME}..."
+    "${HSCTL}" delete gateway "${GW_ID}" --yes &>/dev/null || true
+  fi
+}
+trap cleanup EXIT
+
+API_HOST=$($CLI get route hypershell-api -n "$HS_NAMESPACE" -o jsonpath='{.spec.host}' 2>/dev/null || true)
+if [[ -z "$API_HOST" ]]; then
+  red "ERROR: HyperShell API route not found in namespace ${HS_NAMESPACE}"
+  exit 1
+fi
+
+KC_HOST=$($CLI get route keycloak -n "$KC_NAMESPACE" -o jsonpath='{.spec.host}' 2>/dev/null || true)
+if [[ -z "$KC_HOST" ]]; then
+  red "ERROR: Keycloak route not found in namespace ${KC_NAMESPACE}"
+  exit 1
+fi
+OIDC_ISSUER="https://${KC_HOST}/realms/hypershell"
+
+# Log the hypershell CLI into THIS cluster. hsctl 0.1.0 requires --url and a
+# bearer token via --token-file; mint one from Keycloak (admin) so the run is
+# self-contained rather than depending on a previously-saved config.
+dim "Logging in to hypershell CLI (${API_HOST})..."
+LOGIN_TOKEN=$(curl -sk -X POST "${OIDC_ISSUER}/protocol/openid-connect/token" \
+  -d "grant_type=password" -d "client_id=${OIDC_CLIENT_ID}" \
+  -d "username=${OIDC_USERNAME}" -d "password=${OIDC_PASSWORD}" 2>/dev/null \
+  | python3 -c "import json,sys; print(json.load(sys.stdin).get('access_token',''))" 2>/dev/null || true)
+if [[ -n "$LOGIN_TOKEN" ]]; then
+  echo "$LOGIN_TOKEN" | "${HSCTL}" login --url "https://${API_HOST}" --token-file /dev/stdin --insecure &>/dev/null || true
+else
+  "${HSCTL}" login --url "https://${API_HOST}" --insecure &>/dev/null || true
+fi
+
+echo ""
+bold "HyperShell OpenShell Gateway End-to-End Test"
+sep
+echo ""
+printf '  %s\n' "1. Gateway provisioning via HyperShell API (OIDC)"
+printf '  %s\n' "2. Gateway infrastructure verification"
+printf '  %s\n' "3. OIDC token acquisition"
+printf '  %s\n' "3a. CA certificate setup"
+printf '  %s\n' "4. Route discovery + openshell CLI registration"
+printf '  %s\n' "5. Gateway connectivity"
+printf '  %s\n' "6. Sandbox lifecycle (create → ready)"
+printf '  %s\n' "7. Sandbox interaction"
+printf '  %s\n' "8. Developer user RBAC verification"
+echo ""
+dim  "  HyperShell API:    https://${API_HOST}"
+dim  "  Keycloak:          https://${KC_HOST}"
+dim  "  OIDC issuer:       ${OIDC_ISSUER}"
+dim  "  Gateway name:      ${GW_NAME}"
+dim  "  Sandbox timeout:   ${SANDBOX_TIMEOUT}s"
+echo ""
+sep
+
+# ── 1. gateway provisioning ────────────────────────────────────────────────
+
+echo ""
+bold "1. Gateway Provisioning via HyperShell API (OIDC)"
+echo ""
+
+show_cmd "${HSCTL} list gateways --search \"name=${GW_NAME}\" -o json"
+EXISTING_GW=$("${HSCTL}" list gateways --search "name = '${GW_NAME}'" -o json 2>/dev/null || true)
+EXISTING_ID=$(echo "$EXISTING_GW" | python3 -c "
+import json,sys
+data = json.load(sys.stdin)
+items = data.get('items', [])
+for gw in items:
+    if gw.get('name','') == '${GW_NAME}':
+        print(gw['id'])
+        break
+" 2>/dev/null || true)
+
+if [[ -n "$EXISTING_ID" ]]; then
+  GW_ID="$EXISTING_ID"
+  GW_NAMESPACE=$(echo "$EXISTING_GW" | python3 -c "
+import json,sys
+data = json.load(sys.stdin)
+for gw in data.get('items', []):
+    if gw.get('id','') == '${GW_ID}':
+        print(gw.get('namespace',''))
+        break
+" 2>/dev/null || true)
+  GW_PHASE=$(echo "$EXISTING_GW" | python3 -c "
+import json,sys
+data = json.load(sys.stdin)
+for gw in data.get('items', []):
+    if gw.get('id','') == '${GW_ID}':
+        print(gw.get('phase',''))
+        break
+" 2>/dev/null || true)
+  pass "Gateway already exists: ${GW_NAME} (${GW_ID}, phase=${GW_PHASE})"
+else
+  show_cmd "curl -sk -X POST https://${API_HOST}/api/hypershell/v1/gateways -d '{name: ${GW_NAME}, oidc: ...}'"
+  GW_CREATE_BODY=$(OIDC_ISSUER="$OIDC_ISSUER" OIDC_CLIENT_ID="$OIDC_CLIENT_ID" GW_NAME="$GW_NAME" \
+    OIDC_ROLES_CLAIM="$OIDC_ROLES_CLAIM" GW_IMAGE="$GW_IMAGE" GW_SUPERVISOR_IMAGE="$GW_SUPERVISOR_IMAGE" python3 -c "
+import json, os
+body = {
+    'name': os.environ['GW_NAME'],
+    'fleet_id': 'e2e-fleet',
+    'cluster_id': 'e2e-cluster',
+    'release_id': 'e2e-release',
+    'database_id': 'e2e-db',
+    'image': os.environ['GW_IMAGE'],
+    'supervisor_image': os.environ['GW_SUPERVISOR_IMAGE'],
+    'oidc': json.dumps({
+        'issuer': os.environ['OIDC_ISSUER'],
+        'audience': os.environ['OIDC_CLIENT_ID'],
+        'roles_claim': os.environ['OIDC_ROLES_CLAIM'],
+        'admin_role': 'hypershell-admins',
+        'user_role': 'hypershell-users'
+    }),
+    'route': json.dumps({
+        'enabled': True
+    })
+}
+print(json.dumps(body))
+")
+  CREATE_RESPONSE=$(curl -sk -X POST "https://${API_HOST}/api/hypershell/v1/gateways" \
+    -H "Content-Type: application/json" \
+    -d "${GW_CREATE_BODY}" 2>/dev/null || true)
+
+  GW_ID=$(echo "$CREATE_RESPONSE" | python3 -c "import json,sys; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || true)
+  GW_NAMESPACE=$(echo "$CREATE_RESPONSE" | python3 -c "import json,sys; print(json.load(sys.stdin).get('namespace',''))" 2>/dev/null || true)
+
+  if [[ -n "$GW_ID" ]]; then
+    CREATED_GW=1
+    pass "Gateway created: ${GW_NAME} (${GW_ID})"
+  else
+    fail_test "Failed to create gateway"
+    dim "    ${CREATE_RESPONSE:0:300}"
+    exit 1
+  fi
+
+  dim "  Waiting for controller to provision (timeout: ${PROVISION_TIMEOUT}s)..."
+  DEADLINE=$(($(date +%s) + PROVISION_TIMEOUT))
+  while [[ $(date +%s) -lt $DEADLINE ]]; do
+    GW_PHASE=$("${HSCTL}" get gateway "${GW_ID}" 2>/dev/null | \
+      python3 -c "import json,sys; print(json.load(sys.stdin).get('phase',''))" 2>/dev/null || true)
+    if [[ "$GW_PHASE" == "Running" ]]; then
+      break
+    fi
+    dim "    phase: ${GW_PHASE:-unknown}"
+    sleep 5
+  done
+
+  if [[ "$GW_PHASE" == "Running" ]]; then
+    pass "Gateway provisioned and running"
+  else
+    fail_test "Gateway not running after ${PROVISION_TIMEOUT}s (phase=${GW_PHASE})"
+    exit 1
+  fi
+fi
+
+if [[ -z "$GW_NAMESPACE" ]]; then
+  fail_test "Gateway response did not include a server-assigned namespace"
+  exit 1
+fi
+dim "  Gateway namespace: ${GW_NAMESPACE}"
+sep
+
+# ── 2. gateway infrastructure ──────────────────────────────────────────────
+
+echo ""
+bold "2. Gateway Infrastructure"
+echo ""
+
+show_cmd "$CLI get deployment openshell-gateway -n $GW_NAMESPACE"
+if $CLI get deployment openshell-gateway -n "$GW_NAMESPACE" &>/dev/null; then
+  dim "  Waiting for gateway pod to be ready (up to 90s)..."
+  GW_READY=0
+  GW_READY_DEADLINE=$(($(date +%s) + 90))
+  while [[ $(date +%s) -lt $GW_READY_DEADLINE ]]; do
+    GW_READY=$($CLI get deployment openshell-gateway -n "$GW_NAMESPACE" -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo 0)
+    if [[ "${GW_READY:-0}" -ge 1 ]]; then
+      break
+    fi
+    sleep 5
+  done
+  GW_IMAGE=$($CLI get deployment openshell-gateway -n "$GW_NAMESPACE" -o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || echo unknown)
+  if [[ "${GW_READY:-0}" -ge 1 ]]; then
+    pass "Gateway pod ready ($GW_IMAGE)"
+  else
+    fail_test "Gateway pod not ready after 90s (${GW_READY:-0} replicas)"
+  fi
+else
+  fail_test "Gateway Deployment not found in $GW_NAMESPACE"
+fi
+
+show_cmd "$CLI get service openshell-gateway -n $GW_NAMESPACE"
+GW_SVC=$($CLI get service openshell-gateway -n "$GW_NAMESPACE" -o jsonpath='{.spec.clusterIP}' 2>/dev/null || true)
+if [[ -n "$GW_SVC" ]]; then
+  pass "Gateway service: ${GW_SVC}:8080"
+else
+  fail_test "Gateway service not found"
+fi
+
+show_cmd "$CLI get secret openshell-server-tls -n $GW_NAMESPACE"
+HAS_TLS=$($CLI get secret openshell-server-tls -n "$GW_NAMESPACE" 2>/dev/null && echo yes || true)
+if [[ -n "$HAS_TLS" ]]; then
+  pass "TLS certificates provisioned"
+else
+  dim "  - TLS secret not found (certgen job may still be running)"
+fi
+
+show_cmd "$CLI get jobs -n $GW_NAMESPACE"
+CERTGEN_STATUS=$($CLI get job openshell-gateway-certgen -n "$GW_NAMESPACE" -o jsonpath='{.status.succeeded}' 2>/dev/null || echo 0)
+if [[ "${CERTGEN_STATUS:-0}" -ge 1 ]]; then
+  pass "Certificate generation job completed"
+else
+  dim "  - Certgen job status: ${CERTGEN_STATUS:-unknown}"
+fi
+sep
+
+# ── 3. OIDC token acquisition ────────────────────────────────────────────
+
+echo ""
+bold "3. OIDC Token Acquisition"
+echo ""
+
+show_cmd "# resource-owner password grant → ${OIDC_ISSUER}"
+TOKEN_ENDPOINT="${OIDC_ISSUER}/protocol/openid-connect/token"
+OIDC_RESPONSE=$(curl -sk -X POST "${TOKEN_ENDPOINT}" \
+  -d "grant_type=password" \
+  -d "client_id=${OIDC_CLIENT_ID}" \
+  -d "username=${OIDC_USERNAME}" \
+  -d "password=${OIDC_PASSWORD}" 2>/dev/null || true)
+
+OIDC_TOKEN=$(echo "$OIDC_RESPONSE" | python3 -c "import json,sys; print(json.load(sys.stdin).get('access_token',''))" 2>/dev/null || true)
+if [[ -n "$OIDC_TOKEN" && "$OIDC_TOKEN" != "None" ]]; then
+  pass "OIDC token acquired (user: ${OIDC_USERNAME})"
+else
+  fail_test "Failed to acquire OIDC token from Keycloak"
+  TOKEN_ERR=$(echo "$OIDC_RESPONSE" | python3 -c "import json,sys; print(json.load(sys.stdin).get('error_description','unknown'))" 2>/dev/null || echo 'no response')
+  dim "    ${TOKEN_ERR}"
+  exit 1
+fi
+sep
+
+# ── 3a. extract and trust the cluster CA ──────────────────────────────────
+
+echo ""
+bold "3a. CA Certificate Setup"
+echo ""
+
+show_cmd "$CLI get secret hypershell-ca-secret -n $HS_NAMESPACE -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/e2e-hypershell-ca.crt"
+$CLI get secret openshell-server-tls -n "$GW_NAMESPACE" -o jsonpath="{.data.ca\.crt}" 2>/dev/null | base64 -d > /tmp/e2e-hypershell-ca.crt
+if [[ -s /tmp/e2e-hypershell-ca.crt ]]; then
+  export SSL_CERT_FILE=/tmp/e2e-hypershell-ca.crt
+  pass "CA certificate extracted and SSL_CERT_FILE set"
+  dim "    CA: /tmp/e2e-hypershell-ca.crt"
+else
+  fail_test "Failed to extract CA certificate"
+  exit 1
+fi
+sep
+
+# ── 4. route discovery + CLI registration ─────────────────────────────────
+
+echo ""
+bold "4. Route Discovery + CLI Registration"
+echo ""
+
+GW_LOCAL_NAME="${GW_NAMESPACE}-openshell"
+
+show_cmd "$CLI get routes -n $GW_NAMESPACE"
+GW_ROUTE_HOST=$($CLI get routes -n "$GW_NAMESPACE" -o json 2>/dev/null | python3 -c "
+import json,sys
+data = json.load(sys.stdin)
+candidates = []
+for item in data.get('items',[]):
+    tls = item.get('spec',{}).get('tls',{})
+    to = item.get('spec',{}).get('to',{})
+    name = item.get('metadata',{}).get('name','')
+    if (tls.get('termination') == 'passthrough' and
+        to.get('name') == 'openshell-gateway' and
+        ('grpc' in name or 'gateway' in name)):
+        candidates.append(item['spec']['host'])
+if candidates:
+    print(candidates[0])
+" 2>/dev/null || true)
+
+if [[ -z "$GW_ROUTE_HOST" ]]; then
+  dim "  No passthrough route found, falling back to port-forward"
+  PF_PORT=7443
+  show_cmd "$CLI port-forward -n $GW_NAMESPACE svc/openshell-gateway ${PF_PORT}:8080 &"
+  $CLI port-forward -n "$GW_NAMESPACE" svc/openshell-gateway "${PF_PORT}":8080 &>/dev/null &
+  PF_PID=$!
+  sleep 3
+  if kill -0 "$PF_PID" 2>/dev/null; then
+    pass "Port-forward active (localhost:${PF_PORT} → openshell-gateway:8080)"
+  else
+    fail_test "Port-forward failed to start"
+    PF_PID=""
+    exit 1
+  fi
+  GW_ENDPOINT="https://localhost:${PF_PORT}"
+else
+  GW_ENDPOINT="https://${GW_ROUTE_HOST}:443"
+  pass "Passthrough route: ${GW_ROUTE_HOST}"
+fi
+
+GW_CONFIG_DIR="${HOME}/.config/openshell/gateways/${GW_LOCAL_NAME}"
+mkdir -p "${GW_CONFIG_DIR}"
+
+show_cmd "${OPENSHELL} gateway remove ${GW_LOCAL_NAME}"
+"${OPENSHELL}" gateway remove "${GW_LOCAL_NAME}" 2>/dev/null || true
+mkdir -p "${GW_CONFIG_DIR}"
+
+show_cmd "# write gateway metadata (OIDC mode)"
+GW_LOCAL_NAME="$GW_LOCAL_NAME" GW_ENDPOINT="$GW_ENDPOINT" \
+  OIDC_ISSUER="$OIDC_ISSUER" OIDC_CLIENT_ID="$OIDC_CLIENT_ID" \
+  OIDC_TOKEN="$OIDC_TOKEN" GW_CONFIG_DIR="$GW_CONFIG_DIR" \
+  python3 -c "
+import json, os
+config_dir = os.environ['GW_CONFIG_DIR']
+meta = {
+    'name': os.environ['GW_LOCAL_NAME'],
+    'gateway_endpoint': os.environ['GW_ENDPOINT'],
+    'is_remote': True,
+    'gateway_port': 0,
+    'auth_mode': 'oidc',
+    'oidc_issuer': os.environ['OIDC_ISSUER'],
+    'oidc_client_id': os.environ['OIDC_CLIENT_ID']
+}
+with open(os.path.join(config_dir, 'metadata.json'), 'w') as f:
+    json.dump(meta, f, indent=2)
+token = {
+    'access_token': os.environ['OIDC_TOKEN'],
+    'issuer': os.environ['OIDC_ISSUER'],
+    'client_id': os.environ['OIDC_CLIENT_ID']
+}
+with open(os.path.join(config_dir, 'oidc_token.json'), 'w') as f:
+    json.dump(token, f, indent=2)
+os.chmod(os.path.join(config_dir, 'metadata.json'), 0o600)
+os.chmod(os.path.join(config_dir, 'oidc_token.json'), 0o600)
+"
+
+if [[ -f "${GW_CONFIG_DIR}/metadata.json" && -f "${GW_CONFIG_DIR}/oidc_token.json" ]]; then
+  pass "openshell CLI registered (OIDC mode)"
+else
+  fail_test "Failed to write gateway config"
+fi
+sep
+
+# ── 5. gateway connectivity ───────────────────────────────────────────────
+
+echo ""
+bold "5. Gateway Connectivity"
+echo ""
+
+show_cmd "${OPENSHELL} -g ${GW_LOCAL_NAME} status"
+dim "  Waiting for route connectivity (up to 60s)..."
+CONNECT_DEADLINE=$(($(date +%s) + 60))
+STATUS_OUTPUT=""
+CONNECTED=false
+while [[ $(date +%s) -lt $CONNECT_DEADLINE ]]; do
+  STATUS_OUTPUT=$("${OPENSHELL}" -g "${GW_LOCAL_NAME}" status 2>&1 || true)
+  CLEAN_STATUS=$(echo "$STATUS_OUTPUT" | sed 's/\x1b\[[0-9;]*m//g')
+  if echo "$CLEAN_STATUS" | grep -qi "Connected"; then
+    CONNECTED=true
+    break
+  fi
+  sleep 5
+done
+
+if [[ "$CONNECTED" == "true" ]]; then
+  GW_VERSION=$(echo "$CLEAN_STATUS" | grep -oP 'Version:\s*\K\S+' || echo "unknown")
+  pass "Gateway connected (version: ${GW_VERSION})"
+  echo "$STATUS_OUTPUT" | while IFS= read -r line; do
+    dim "    $line"
+  done
+else
+  fail_test "Gateway not reachable"
+  echo "$STATUS_OUTPUT" | while IFS= read -r line; do
+    dim "    $line"
+  done
+fi
+sep
+
+# ── 6. sandbox lifecycle ──────────────────────────────────────────────────
+
+echo ""
+bold "6. Sandbox Lifecycle"
+echo ""
+
+RUN_ID=$(date +%s | tail -c5)
+SANDBOX_NAME="e2e-${RUN_ID}"
+
+show_cmd "${OPENSHELL} -g ${GW_LOCAL_NAME} sandbox create --name ${SANDBOX_NAME}"
+dim "  Creating sandbox (timeout: ${SANDBOX_TIMEOUT}s)..."
+
+"${OPENSHELL}" -g "${GW_LOCAL_NAME}" sandbox create --name "${SANDBOX_NAME}" &>/dev/null &
+SB_CREATE_PID=$!
+
+DEADLINE=$(($(date +%s) + SANDBOX_TIMEOUT))
+SANDBOX_FOUND=false
+POD_NAME=""
+POD_STATUS=""
+while [[ $(date +%s) -lt $DEADLINE ]]; do
+  SANDBOX_PODS=$($CLI get pods -n "$GW_NAMESPACE" --no-headers 2>/dev/null | grep -i "default--${SANDBOX_NAME}" || true)
+  if [[ -n "$SANDBOX_PODS" ]]; then
+    POD_STATUS=$(echo "$SANDBOX_PODS" | awk '{print $3}' | head -1)
+    POD_NAME=$(echo "$SANDBOX_PODS" | awk '{print $1}' | head -1)
+    if [[ "$POD_STATUS" == "Running" ]]; then
+      SANDBOX_FOUND=true
+      break
+    fi
+    dim "    pod: ${POD_NAME} (${POD_STATUS})"
+  fi
+  sleep 5
+done
+
+kill "$SB_CREATE_PID" 2>/dev/null || true
+wait "$SB_CREATE_PID" 2>/dev/null || true
+SB_CREATE_PID=""
+
+show_cmd "$CLI get pods -n $GW_NAMESPACE --no-headers | grep ${SANDBOX_NAME}"
+
+if [[ "$SANDBOX_FOUND" == "true" ]]; then
+  pass "Sandbox pod created: ${POD_NAME} (${POD_STATUS})"
+else
+  SANDBOX_PODS=$($CLI get pods -n "$GW_NAMESPACE" --no-headers 2>/dev/null | grep -i "default--${SANDBOX_NAME}" || true)
+  if [[ -n "$SANDBOX_PODS" ]]; then
+    POD_STATUS=$(echo "$SANDBOX_PODS" | awk '{print $3}' | head -1)
+    POD_NAME=$(echo "$SANDBOX_PODS" | awk '{print $1}' | head -1)
+    pass "Sandbox pod created: ${POD_NAME} (${POD_STATUS})"
+  else
+    fail_test "Sandbox not found after ${SANDBOX_TIMEOUT}s"
+  fi
+fi
+sep
+
+# ── 7. sandbox interaction ────────────────────────────────────────────────
+
+echo ""
+bold "7. Sandbox Interaction"
+echo ""
+
+GW_FLAG="-g ${GW_LOCAL_NAME}"
+
+show_cmd "${OPENSHELL} ${GW_FLAG} sandbox exec -n ${SANDBOX_NAME} -- uname -a"
+if SB_EXEC_OUTPUT=$("${OPENSHELL}" -g "${GW_LOCAL_NAME}" sandbox exec -n "${SANDBOX_NAME}" -- uname -a 2>&1); then
+  CLEAN_EXEC=$(echo "$SB_EXEC_OUTPUT" | sed 's/\x1b\[[0-9;]*m//g' | grep -v '^ *$' | grep -v 'WARN' | tail -3)
+  if [[ -n "$CLEAN_EXEC" ]]; then
+    pass "Sandbox exec: command executed inside sandbox"
+    echo "$CLEAN_EXEC" | while IFS= read -r line; do
+      dim "    $line"
+    done
+  else
+    fail_test "Sandbox exec: no output from uname command"
+    dim "    ${SB_EXEC_OUTPUT:0:200}"
+  fi
+else
+  fail_test "Sandbox exec: openshell command failed"
+  dim "    ${SB_EXEC_OUTPUT:0:200}"
+fi
+
+show_cmd "${OPENSHELL} ${GW_FLAG} sandbox exec -n ${SANDBOX_NAME} -- ls -la /workspace"
+if SB_LS_OUTPUT=$("${OPENSHELL}" -g "${GW_LOCAL_NAME}" sandbox exec -n "${SANDBOX_NAME}" -- ls -la /workspace 2>&1); then
+  CLEAN_LS=$(echo "$SB_LS_OUTPUT" | sed 's/\x1b\[[0-9;]*m//g' | grep -v '^ *$' | grep -v 'WARN' | tail -5)
+  if [[ -n "$CLEAN_LS" ]]; then
+    pass "Sandbox workspace: /workspace directory listing"
+    echo "$CLEAN_LS" | while IFS= read -r line; do
+      dim "    $line"
+    done
+  else
+    fail_test "Sandbox workspace: no output from ls command"
+    dim "    ${SB_LS_OUTPUT:0:200}"
+  fi
+else
+  if echo "$SB_LS_OUTPUT" | grep -q "No such file or directory"; then
+    dim "  - /workspace not available (using default working directory)"
+  else
+    fail_test "Sandbox workspace: openshell ls command failed"
+    dim "    ${SB_LS_OUTPUT:0:200}"
+  fi
+fi
+
+# ── cleanup ───────────────────────────────────────────────────────────────
+
+if [[ "$SKIP_CLEANUP" != "1" && "$LAUNCH_TUI" != "1" && -n "$SANDBOX_NAME" ]]; then
+  echo ""
+  dim "  Cleaning up sandbox..."
+  show_cmd "${OPENSHELL} ${GW_FLAG} sandbox delete ${SANDBOX_NAME}"
+  "${OPENSHELL}" -g "${GW_LOCAL_NAME}" sandbox delete "${SANDBOX_NAME}" 2>&1 || true
+  dim "  Sandbox deleted"
+fi
+sep
+
+# ── 8. developer user RBAC verification ──────────────────────────────────
+
+echo ""
+bold "8. Developer User RBAC Verification"
+echo ""
+
+show_cmd "# acquire OIDC token for developer user"
+DEV_RESPONSE=$(curl -sk -X POST "${TOKEN_ENDPOINT}" \
+  -d "grant_type=password" \
+  -d "client_id=${OIDC_CLIENT_ID}" \
+  -d "username=${DEV_USERNAME}" \
+  -d "password=${DEV_PASSWORD}" 2>/dev/null || true)
+
+DEV_TOKEN=$(echo "$DEV_RESPONSE" | python3 -c "import json,sys; print(json.load(sys.stdin).get('access_token',''))" 2>/dev/null || true)
+if [[ -n "$DEV_TOKEN" && "$DEV_TOKEN" != "None" ]]; then
+  pass "Developer OIDC token acquired (user: ${DEV_USERNAME})"
+else
+  fail_test "Failed to acquire developer OIDC token"
+fi
+
+if [[ -n "$DEV_TOKEN" && "$DEV_TOKEN" != "None" ]]; then
+  DEV_GW_LOCAL_NAME="${GW_LOCAL_NAME}-dev"
+  DEV_CONFIG_DIR="${HOME}/.config/openshell/gateways/${DEV_GW_LOCAL_NAME}"
+  mkdir -p "${DEV_CONFIG_DIR}"
+
+  "${OPENSHELL}" gateway remove "${DEV_GW_LOCAL_NAME}" 2>/dev/null || true
+  mkdir -p "${DEV_CONFIG_DIR}"
+
+  show_cmd "# register gateway as developer user"
+  DEV_GW_LOCAL_NAME="$DEV_GW_LOCAL_NAME" GW_ENDPOINT="$GW_ENDPOINT" \
+    OIDC_ISSUER="$OIDC_ISSUER" OIDC_CLIENT_ID="$OIDC_CLIENT_ID" \
+    DEV_TOKEN="$DEV_TOKEN" DEV_CONFIG_DIR="$DEV_CONFIG_DIR" \
+    python3 -c "
+import json, os
+config_dir = os.environ['DEV_CONFIG_DIR']
+meta = {
+    'name': os.environ['DEV_GW_LOCAL_NAME'],
+    'gateway_endpoint': os.environ['GW_ENDPOINT'],
+    'is_remote': True,
+    'gateway_port': 0,
+    'auth_mode': 'oidc',
+    'oidc_issuer': os.environ['OIDC_ISSUER'],
+    'oidc_client_id': os.environ['OIDC_CLIENT_ID']
+}
+with open(os.path.join(config_dir, 'metadata.json'), 'w') as f:
+    json.dump(meta, f, indent=2)
+token = {
+    'access_token': os.environ['DEV_TOKEN'],
+    'issuer': os.environ['OIDC_ISSUER'],
+    'client_id': os.environ['OIDC_CLIENT_ID']
+}
+with open(os.path.join(config_dir, 'oidc_token.json'), 'w') as f:
+    json.dump(token, f, indent=2)
+os.chmod(os.path.join(config_dir, 'metadata.json'), 0o600)
+os.chmod(os.path.join(config_dir, 'oidc_token.json'), 0o600)
+"
+
+  if [[ -f "${DEV_CONFIG_DIR}/metadata.json" && -f "${DEV_CONFIG_DIR}/oidc_token.json" ]]; then
+    pass "Developer gateway registered (OIDC mode)"
+  else
+    fail_test "Failed to write developer gateway config"
+  fi
+
+  show_cmd "${OPENSHELL} -g ${DEV_GW_LOCAL_NAME} status"
+  DEV_STATUS=$("${OPENSHELL}" -g "${DEV_GW_LOCAL_NAME}" status 2>&1 || true)
+  DEV_CLEAN=$(echo "$DEV_STATUS" | sed 's/\x1b\[[0-9;]*m//g')
+  if echo "$DEV_CLEAN" | grep -qi "Connected"; then
+    pass "Developer user: gateway connected"
+  else
+    fail_test "Developer user: gateway not reachable"
+    echo "$DEV_STATUS" | while IFS= read -r line; do dim "    $line"; done
+  fi
+
+  DEV_SANDBOX="e2e-dev-$(date +%s | tail -c5)"
+  show_cmd "${OPENSHELL} -g ${DEV_GW_LOCAL_NAME} sandbox create --name ${DEV_SANDBOX}"
+  dim "  Creating developer sandbox (timeout: ${SANDBOX_TIMEOUT}s)..."
+
+  "${OPENSHELL}" -g "${DEV_GW_LOCAL_NAME}" sandbox create --name "${DEV_SANDBOX}" &>/dev/null &
+  DEV_SB_PID=$!
+
+  DEV_SB_FOUND=false
+  DEV_DEADLINE=$(($(date +%s) + SANDBOX_TIMEOUT))
+  while [[ $(date +%s) -lt $DEV_DEADLINE ]]; do
+    DEV_PODS=$($CLI get pods -n "$GW_NAMESPACE" --no-headers 2>/dev/null | grep -i "default--${DEV_SANDBOX}" || true)
+    if [[ -n "$DEV_PODS" ]]; then
+      DEV_POD_STATUS=$(echo "$DEV_PODS" | awk '{print $3}' | head -1)
+      if [[ "$DEV_POD_STATUS" == "Running" ]]; then
+        DEV_SB_FOUND=true
+        break
+      fi
+    fi
+    sleep 5
+  done
+
+  kill "$DEV_SB_PID" 2>/dev/null || true
+  wait "$DEV_SB_PID" 2>/dev/null || true
+
+  if [[ "$DEV_SB_FOUND" == "true" ]]; then
+    pass "Developer user: sandbox created"
+
+    show_cmd "${OPENSHELL} -g ${DEV_GW_LOCAL_NAME} sandbox exec -n ${DEV_SANDBOX} -- uname -a"
+    if DEV_EXEC=$("${OPENSHELL}" -g "${DEV_GW_LOCAL_NAME}" sandbox exec -n "${DEV_SANDBOX}" -- uname -a 2>&1); then
+      DEV_EXEC_CLEAN=$(echo "$DEV_EXEC" | sed 's/\x1b\[[0-9;]*m//g' | grep -v '^ *$' | grep -v 'WARN' | tail -3)
+      if [[ -n "$DEV_EXEC_CLEAN" ]]; then
+        pass "Developer user: sandbox exec succeeded"
+      else
+        fail_test "Developer user: sandbox exec returned no output"
+      fi
+    else
+      fail_test "Developer user: sandbox exec failed"
+      dim "    ${DEV_EXEC:0:200}"
+    fi
+  else
+    fail_test "Developer user: sandbox not created after ${SANDBOX_TIMEOUT}s"
+  fi
+
+  if [[ "$SKIP_CLEANUP" != "1" && "$DEV_SB_FOUND" == "true" ]]; then
+    dim "  Cleaning up developer sandbox..."
+    "${OPENSHELL}" -g "${DEV_GW_LOCAL_NAME}" sandbox delete "${DEV_SANDBOX}" 2>&1 || true
+  fi
+
+  "${OPENSHELL}" gateway remove "${DEV_GW_LOCAL_NAME}" 2>/dev/null || true
+fi
+sep
+
+# ── results ───────────────────────────────────────────────────────────────
+
+echo ""
+bold "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+bold "Results: $PASS passed, $FAIL failed"
+echo ""
+for t in "${TESTS[@]}"; do
+  if [[ "$t" == PASS:* ]]; then
+    green "  ✓ ${t#PASS: }"
+  else
+    red "  ✗ ${t#FAIL: }"
+  fi
+done
+bold "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+echo ""
+
+if [[ "$LAUNCH_TUI" == "1" && $FAIL -eq 0 ]]; then
+  echo ""
+  bold "Interactive TUI"
+  sep
+  echo ""
+  dim "  Launching OpenShell TUI..."
+  dim "  Press Ctrl-C to exit."
+  echo ""
+  sleep 2
+  exec "${OPENSHELL}" -g "${GW_LOCAL_NAME}" term
+fi
+
+if [[ $FAIL -gt 0 ]]; then
+  exit 1
+fi

From 84a229ac974078958070d30e8b168863ad8d14dd Mon Sep 17 00:00:00 2001
From: user <u@example.com>
Date: Sat, 15 Aug 2026 21:40:56 -0400
Subject: [PATCH 13/15] docs(ibm-cluster): worker-SG egress fix + OIDC
 default-secure + cloud providers

Refine the registry-timeout explanation with its root cause (the kube-<clusterID>
worker Security Group is default-deny outbound) and the supported fix (add an
outbound 0.0.0.0/0:443 rule). Add section 5.7 covering Keycloak default-secure
gateway wiring, the correct OIDC `gateway add` command (not bare edge/cloud
mode), and the worker-egress requirement for cloud-model providers such as
google-vertex-ai. Correct the stale gateway-add note in 5.5.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 skills/deploy/ibm-cluster/SKILL.md | 102 ++++++++++++++++++++++++++++-
 1 file changed, 99 insertions(+), 3 deletions(-)

diff --git a/skills/deploy/ibm-cluster/SKILL.md b/skills/deploy/ibm-cluster/SKILL.md
index 108a46e9..6cb1a6d5 100644
--- a/skills/deploy/ibm-cluster/SKILL.md
+++ b/skills/deploy/ibm-cluster/SKILL.md
@@ -213,6 +213,35 @@ So the mirror must live on a **node-reachable** registry: IBM Container Registry
   the public route host is), then IDMS mirror to `<route-host>/openshift-ingress`.
   Editing the global pull secret + adding an IDMS both **roll the worker nodes**.
 
+#### Root cause: the worker Security Group (and how to open general egress)
+
+The registry timeouts above are **not** registry-specific. The ROKS worker
+**Security Group** `kube-<clusterID>` (e.g. `kube-da0a9c9w0br7f5gf3dd0`) is
+**default-deny outbound**, allowing only IBM Cloud infrastructure (service
+endpoints `161.26.0.0/16` / `166.8.0.0/14`, VPE gateways, the in-cluster
+registry, metadata `169.254.169.254`). DNS resolves but no packets reach the
+public internet, so `registry.redhat.io`, `quay.io`, and any **app-level** egress
+(e.g. `oauth2.googleapis.com` when a gateway mints a Vertex AI token — see §5.7)
+all time out. The Public Gateway is attached and the subnet ACL is wide open —
+the SG is the gate. Diagnose from any pod with a shell (gateway pods are
+distroless; use the gateway's Postgres pod):
+`timeout 8 bash -c 'cat </dev/null >/dev/tcp/1.1.1.1/443' && echo OPEN || echo BLOCKED`.
+
+Adding a rule to this ROKS-managed SG only *grants* egress (it never removes
+IBM's rules):
+
+```bash
+SG=$(ibmcloud is security-groups --vpc <vpc-id> --output json \
+  | python3 -c "import json,sys;print(next(g['id'] for g in json.load(sys.stdin) if g['name'].startswith('kube-') and 'lbaas' not in g['name'] and 'vpegw' not in g['name']))")
+ibmcloud is security-group-rule-add "$SG" outbound tcp --remote 0.0.0.0/0 --port-min 443 --port-max 443
+```
+
+This enables all outbound HTTPS from workers (egress-only, port 443) — required
+for cloud-model providers (§5.7). As a side effect it should also let nodes pull
+directly from `registry.redhat.io` / `quay.io` on 443, since crio shares the
+node's SG — **verify before relying on it to skip the mirroring dance**. This is
+IBM Cloud VPC state, **not** reproducible from GitOps; re-apply on any rebuild.
+
 #### ImageDigestMirrorSet (redirect the OSSM repo to the mirror)
 
 ```yaml
@@ -417,9 +446,10 @@ openssl s_client -connect "$HOST:443" -servername "$HOST" -CAfile /tmp/tenant-ca
 openshell status --gateway-endpoint "https://$HOST:443" --gateway-insecure   # Status: Connected
 ```
 
-`openshell gateway add https://<host>` drives the interactive edge/OIDC login
-flow and populates the CLI's mTLS/CA material under `~/.config/openshell/`; it is
-not needed to prove the infrastructure path above.
+`openshell status --gateway-endpoint … --gateway-insecure` proves the transport
+path without auth. For an authenticated registration a **bare** `openshell gateway
+add https://<host>` is wrong — it selects edge/"cloud" mode and 404s on these
+gRPC-only gateways; use OIDC mode (`--oidc-issuer …`, §5.7).
 
 ### 5.6: Enable Agent Sandboxes (required for `openshell sandbox ...`)
 
@@ -515,6 +545,72 @@ To later switch to Gateway API (if IBM fixes HostedCluster mirroring), unset
 `GATEWAY_INGRESS_MODE` and run `cloud-hub-ingress-bootstrap`; the control plane
 then emits `GRPCRoute`s and removes the Routes.
 
+### 5.7: OIDC default-secure gateways + cloud-model providers
+
+**Default-secure every gateway via Keycloak (no per-gateway `oidc` field).** When
+the controller finds a `hypershell-keycloak-admin` Secret in its namespace it runs
+`reconcileKeycloakClient` on every gateway provision: it uses a `client_credentials`
+grant to auto-create a per-gateway public PKCE Keycloak client `<name>-<id>`
+(loopback redirects `http://127.0.0.1:*`,`http://localhost:*`), its
+`openshell-admin`/`openshell-user` client roles + audience/`hypershell.roles`
+protocol mappers, then persists the resulting `oidc` block back onto the Gateway
+and into `gateway.toml`. Wire it once:
+
+```bash
+# a) admin secret in the controller namespace (keys read verbatim by main.go).
+#    server-url stays IN-CLUSTER; the external issuer is set separately (below).
+oc create secret generic hypershell-keycloak-admin -n hypershell \
+  --from-literal=server-url="http://hypershell-keycloak-service.keycloak-system.svc.cluster.local:8080" \
+  --from-literal=realm="hypershell" \
+  --from-literal=client-id="hypershell-control-plane" \
+  --from-literal=client-secret="control-plane-secret"
+
+# b) client-facing issuer = the EXTERNAL Keycloak host (must match KC_HOSTNAME).
+#    This overrides only the issuer written to the gateway/toml, not the admin URL.
+oc set env deploy/hypershell-controller -n hypershell \
+  GATEWAY_OIDC_ISSUER_URL="https://keycloak.<appdomain>/realms/hypershell"
+oc rollout restart deploy/hypershell-controller -n hypershell
+# log confirms: "keycloak integration enabled ... gateway reconciler ... keycloak=true"
+```
+
+The `hypershell-control-plane` service account needs realm-management roles
+(`manage-clients`,`manage-users`,`view-users`,`query-clients`,`query-users`) or
+the client-credentials grant 403s. Declare them in the realm import as a
+`service-account-hypershell-control-plane` user with `clientRoles`
+(bootstrap-hyperfleet `bases/hypershell/keycloak/realm-import.yaml`) —
+`clientScopeMappings` alone does **not** grant SA roles. Per-user authorization is
+separate: assign the `openshell-admin`/`openshell-user` client role on the
+`<name>-<id>` client to each end user (sandbox create additionally needs workspace
+membership).
+
+**CLI connect — OIDC mode, not edge/cloud mode.** A bare `openshell gateway add
+https://<host>` treats the endpoint as edge-authenticated ("cloud") and 404s on
+these gRPC-only passthrough gateways. Use OIDC mode with `--gateway-insecure`
+(passthrough serves the self-signed pod cert; the flag is required on `add` **and**
+every later command, and its env form is `OPENSHELL_GATEWAY_INSECURE=true`, not `1`):
+
+```bash
+openshell gateway add --gateway-insecure \
+  --oidc-issuer   "https://keycloak.<appdomain>/realms/hypershell" \
+  --oidc-client-id "<name>-<id>" --oidc-audience "<name>-<id>" \
+  "https://<gateway-route-host>:443"
+```
+
+**Cloud-model providers need worker internet egress.** `openshell provider create
+--type google-vertex-ai --from-gcloud-adc …` makes the gateway pod call
+`oauth2.googleapis.com` (token mint) and later `*.googleapis.com` (inference).
+On ROKS that fails with `transport error … token endpoint request failed` until
+the worker SG is opened for outbound 443 (see *Root cause: the worker Security
+Group* in Step 4). Remember `--gateway-insecure` here too:
+
+```bash
+openshell provider create --gateway-insecure \
+  --name vertex-claude --type google-vertex-ai --from-gcloud-adc \
+  --config VERTEX_AI_PROJECT_ID="$(gcloud config get-value project)" \
+  --config VERTEX_AI_REGION=global
+# ✓ Created provider ... / Configured GCP credentials from gcloud ADC and minted the initial access token
+```
+
 ## Internal registry storage (COS is NOT required)
 
 `--cos-instance` is a required create flag, but on this account the ROKS registry

From 8076f07b5475b40c5d2d959e2478494847ebf1b8 Mon Sep 17 00:00:00 2001
From: user <u@example.com>
Date: Sat, 15 Aug 2026 21:45:59 -0400
Subject: [PATCH 14/15] docs(ibm-cluster): document sandbox connect
 gateway-insecure env workaround

`openshell sandbox connect` execs the system ssh with a ProxyCommand that
re-execs `openshell ssh-proxy`, and the CLI omits `--gateway-insecure` from
that generated ProxyCommand. The child ssh-proxy therefore verifies the
self-signed passthrough gateway cert and fails `invalid peer certificate:
UnknownIssuer`; the flag on `connect` never reaches it. The child inherits
the environment, so `export OPENSHELL_GATEWAY_INSECURE=true` is the working
fix. Verified live on hysh-ibm-01 (sandbox woot).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 skills/deploy/ibm-cluster/SKILL.md | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/skills/deploy/ibm-cluster/SKILL.md b/skills/deploy/ibm-cluster/SKILL.md
index 6cb1a6d5..276b3d6d 100644
--- a/skills/deploy/ibm-cluster/SKILL.md
+++ b/skills/deploy/ibm-cluster/SKILL.md
@@ -611,6 +611,25 @@ openshell provider create --gateway-insecure \
 # ✓ Created provider ... / Configured GCP credentials from gcloud ADC and minted the initial access token
 ```
 
+**`openshell sandbox connect` needs the env var, not the flag.** Unlike the other
+commands, `connect` does not open the sandbox channel in-process: it execs the
+system `ssh`, whose `ProxyCommand` re-execs `openshell ssh-proxy --gateway https://gw-…:443
+--sandbox-id <uuid> --token <uuid> --gateway-name <name>`. The CLI builds that
+ProxyCommand string **without** `--gateway-insecure`, so the child `ssh-proxy`
+verifies the self-signed passthrough cert and dies with `invalid peer certificate:
+UnknownIssuer` — passing `--gateway-insecure` to `connect` does not help (it never
+reaches the child). The child inherits the environment, so export the var:
+
+```bash
+export OPENSHELL_GATEWAY_INSECURE=true   # true/false, not 1
+openshell sandbox connect <name>         # no --gateway-insecure flag
+# remote shell runs as uid=1000790000(sandbox) on the sandbox pod
+```
+
+(`sandbox list --gateway-insecure` is a pure in-process gRPC call and works with
+the flag; only `connect`'s ssh-proxy subprocess needs the env var. The upstream
+fix is to thread `--gateway-insecure` into the ProxyCommand.)
+
 ## Internal registry storage (COS is NOT required)
 
 `--cos-instance` is a required create flag, but on this account the ROKS registry

From 4da5a8a4720c89608094a63ffd5b525e00fb3be5 Mon Sep 17 00:00:00 2001
From: user <u@example.com>
Date: Sat, 15 Aug 2026 23:20:15 -0400
Subject: [PATCH 15/15] spec: inference-routing + ROKS credential-free sandbox
 agent runbook

Add openshell-inference-routing.spec.md documenting how sandbox agents reach
cloud models with no credential in the sandbox: the inference.local router
strips the client key and injects the provider token server-side, translating
/v1/messages -> Vertex :rawPredict. Covers the two credential paths (per-binary
sentinel rewrite vs router injection) and the request-shape compatibility
requirement. Register it in the spec index.

Add ibm-cluster skill section 5.8 with the ROKS runbook: `inference set`, the
required non-effort `--model claude-sonnet-4-5` workaround for Vertex's strict
vertex-2023-10-16 validation (adaptive-thinking / output_config.effort 400s),
sandbox connect, and the ~/.claude/settings.json wiring for bare `claude`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 skills/deploy/ibm-cluster/SKILL.md            |  56 ++++++
 specs/index.spec.md                           |   1 +
 .../openshell-inference-routing.spec.md       | 189 ++++++++++++++++++
 3 files changed, 246 insertions(+)
 create mode 100644 specs/platform/openshell-inference-routing.spec.md

diff --git a/skills/deploy/ibm-cluster/SKILL.md b/skills/deploy/ibm-cluster/SKILL.md
index 276b3d6d..ded5c8ee 100644
--- a/skills/deploy/ibm-cluster/SKILL.md
+++ b/skills/deploy/ibm-cluster/SKILL.md
@@ -630,6 +630,62 @@ openshell sandbox connect <name>         # no --gateway-insecure flag
 the flag; only `connect`'s ssh-proxy subprocess needs the env var. The upstream
 fix is to thread `--gateway-insecure` into the ProxyCommand.)
 
+### 5.8: Run an agent (Claude Code) in a sandbox — credential-free via `inference.local`
+
+An in-sandbox agent reaches the cloud model through the OpenShell **inference
+router**, NOT by holding a credential. `inference.local:443` is a virtual host the
+supervisor intercepts: it strips the client's key and injects the operator's
+provider token server-side, translating the Anthropic `/v1/messages` body into
+Vertex's `:rawPredict` contract. So the sandbox never sees a secret. See
+[`openshell-inference-routing.spec.md`](../../../specs/platform/openshell-inference-routing.spec.md)
+for the model. Do **not** try `CLAUDE_CODE_USE_VERTEX=1` — that makes Claude Code
+do its own Google ADC (none in the sandbox) and hang.
+
+```bash
+# a) Point inference.local at the Vertex provider from 5.7 (workspace-level, persists).
+#    --no-verify because the provider region is `global` (no region endpoint to probe).
+openshell inference set -g <name> --gateway-insecure \
+  --provider vertex-claude --model 'claude-sonnet-4-5@20250929' --no-verify
+openshell inference get -g <name> --gateway-insecure   # confirm the user route
+```
+
+**CRITICAL — pin a non-effort model with `--model claude-sonnet-4-5`.** Claude Code
+2.1.x defaults to an effort-capable model and emits newer request fields that the
+Vertex Anthropic partner endpoint (`anthropic_version = vertex-2023-10-16`, strict
+validation) rejects — and the router does NOT strip them:
+
+- default model → `400 thinking: 'adaptive' does not match 'disabled'/'enabled'`
+- with `MAX_THINKING_TOKENS=0` → `400 output_config.effort: Extra inputs are not permitted`
+
+These fields are gated on the model, not on any env var (there is no Claude Code
+flag to strip them). Selecting `claude-sonnet-4-5` (non-effort, non-adaptive)
+suppresses both → HTTP 200. The router forces the served model via the URL anyway,
+so `--model` here only controls the request *shape*, not which model answers.
+
+```bash
+export OPENSHELL_GATEWAY_INSECURE=true
+openshell sandbox connect <sandbox>          # from 5.6 / the connect note above
+
+# inside the sandbox — the API key value is discarded by the router:
+ANTHROPIC_BASE_URL=https://inference.local ANTHROPIC_API_KEY=unused \
+  claude --model claude-sonnet-4-5 -p "Reply with one word: PONG"     # -> PONG
+```
+
+**Make bare `claude` "just work"** — persist the model + base URL in the sandbox's
+`~/.claude/settings.json` (HOME is `/sandbox`; merge, don't clobber `theme` etc.):
+
+```json
+{
+  "model": "claude-sonnet-4-5",
+  "env": {
+    "ANTHROPIC_BASE_URL": "https://inference.local",
+    "ANTHROPIC_API_KEY": "unused"
+  }
+}
+```
+
+After that, a bare `claude -p "..."` (no flags, no env) returns a real completion.
+
 ## Internal registry storage (COS is NOT required)
 
 `--cos-instance` is a required create flag, but on this account the ROKS registry
diff --git a/specs/index.spec.md b/specs/index.spec.md
index f6130cef..08700bb7 100644
--- a/specs/index.spec.md
+++ b/specs/index.spec.md
@@ -38,6 +38,7 @@ Machine-readable index for autonomous reconciliation (`/reconcile` skill).
 | `platform/openshell-gateway-credentials.spec.md` | platform | Credential storage drivers, KEK conditional provisioning | CP | openshell-gateway, openshell-gateway-database |
 | `platform/openshell-gateway-secret-rotation.spec.md` | platform | Secret rotation: DB password, KEK, TLS certificates | CP | openshell-gateway-database, openshell-gateway-credentials, openshell-gateway-tls |
 | `platform/openshell-gateway-keycloak.spec.md` | platform | Keycloak OIDC client provisioning, per-gateway OIDC role bridge | CP | openshell-gateway, openshell-gateway-oidc, rbac-enforcement |
+| `platform/openshell-inference-routing.spec.md` | platform | Inference router, inference.local, credential-free sandbox model access, provider translation | CP | openshell-gateway, openshell-gateway-credentials |
 | `platform/global-architecture.spec.md` | platform | Global hub, multi-cloud, CNPG, Tekton, ArgoCD, Vault | CP, ALL | data-model, control-plane |
 | `web-console/architecture.spec.md` | web-console | Web console, BFF, browser session, UI routes | WEB, SDK, API | data-model, security, UI standards |
 | `standards/platform/cross-cutting.spec.md` | standards | - | ALL | - |
diff --git a/specs/platform/openshell-inference-routing.spec.md b/specs/platform/openshell-inference-routing.spec.md
new file mode 100644
index 00000000..d184ac89
--- /dev/null
+++ b/specs/platform/openshell-inference-routing.spec.md
@@ -0,0 +1,189 @@
+# OpenShell Inference Routing Specification
+
+**Date:** 2026-08-15
+**Status:** Draft
+**Parent:** `openshell-gateway.spec.md` - core gateway provisioning
+**Related:** `openshell-gateway-credentials.spec.md` - provider credential storage; `openshell-gateway-oidc.spec.md` - gateway authentication
+
+---
+
+## Purpose
+
+This specification defines how agents running **inside OpenShell sandboxes** reach cloud inference models (Anthropic Claude on Vertex AI, Bedrock, OpenAI-compatible providers, etc.) **without any provider credential ever entering the sandbox**.
+
+The platform requirement is a security guarantee: a sandbox is an untrusted execution environment for user/agent code, so the operator's cloud-provider credentials (GCP Vertex tokens, Anthropic keys) MUST NOT be readable from within it. OpenShell satisfies this with two distinct, upstream-provided mechanisms — a per-binary egress credential rewrite, and a workspace-scoped **inference router** at the virtual host `inference.local`. This spec covers both, defines when each applies, and specifies how HyperShell provisions and operates them.
+
+The inference router itself is upstream OpenShell functionality. HyperShell's desired state is that (a) sandbox agents get credential-free cloud-model access, and (b) the operational configuration that enables it is reproducible per environment. Concrete per-cluster runbook steps live in the [`ibm-cluster`](../../skills/deploy/ibm-cluster/SKILL.md) skill.
+
+---
+
+## Architecture
+
+### Two credential-delivery paths
+
+```
+                        ┌─────────────────────────── SANDBOX (untrusted) ───────────────────────────┐
+                        │  agent binary (e.g. /usr/local/bin/claude)                                  │
+                        └───────────────┬───────────────────────────────┬───────────────────────────┘
+                                        │                               │
+           PATH A: direct-to-upstream  │                               │  PATH B: inference router
+        (allowlisted host, per-binary) │                               │  (ANTHROPIC_BASE_URL=https://inference.local)
+                                        ▼                               ▼
+                   ┌──────────────────────────────┐   ┌─────────────────────────────────────────────┐
+                   │ Supervisor L7 egress proxy    │   │ Supervisor intercepts CONNECT               │
+                   │ Rewrites the sentinel bearer  │   │   inference.local:443, TLS-terminates,      │
+                   │   Authorization: Bearer       │   │   L7-routes to the inference router          │
+                   │   openshell:resolve:env:<KEY> │   │ Router STRIPS client authorization/x-api-key│
+                   │   → real stored credential    │   │   INJECTS operator provider token           │
+                   │ Agent MUST emit the sentinel  │   │   TRANSLATES /v1/messages → provider shape  │
+                   └───────────────┬───────────────┘   └───────────────────┬─────────────────────────┘
+                                   ▼                                       ▼
+                          upstream provider host                  upstream provider host
+                     (api.anthropic.com, *.googleapis.com)    (e.g. *-aiplatform.googleapis.com :rawPredict)
+```
+
+- **Path A — direct-to-upstream (per-binary sentinel rewrite).** The sandbox connects straight to an allowlisted provider host. The supervisor's L7 proxy rewrites an `Authorization: Bearer openshell:resolve:env:<KEY>` sentinel into the real stored credential. The agent MUST emit the sentinel as its bearer. This path does not fit clients that manage their own provider auth (e.g. Claude Code in `CLAUDE_CODE_USE_VERTEX` mode uses Google ADC and never emits the sentinel).
+
+- **Path B — inference router (`inference.local`).** `inference.local:443` is a virtual host intercepted by the supervisor proxy — not real DNS. The router receives the request, **discards the caller's credential**, **injects the operator-configured provider token server-side** (resolved by the gateway into the route bundle), and forwards. The client sends a throwaway key. This is the path for standard inference clients pointed at a custom base URL.
+
+### Inference router surfaces
+
+The router exposes standard inference APIs and adapts them to the configured provider:
+
+| Client surface | Path | Adapted for provider |
+|---|---|---|
+| Anthropic Messages | `POST /v1/messages` | Vertex Anthropic (`:rawPredict`), direct Anthropic |
+| OpenAI Chat Completions | `POST /v1/chat/completions` | Vertex OpenAI-compat, OpenAI, DeepInfra, NVIDIA |
+| OpenAI Completions / Responses / Embeddings | `POST /v1/{completions,responses,embeddings}` | provider-dependent |
+| Model discovery | `GET /v1/models`, `GET /v1/models/*` | provider-dependent |
+
+### Anthropic-on-Vertex translation
+
+For a `google-vertex-ai` provider serving a `claude-*` model, an incoming Anthropic Messages request is rewritten into Vertex's Anthropic partner contract:
+
+```
+POST https://{host}/v1/projects/{project}/locations/{location}/publishers/anthropic/models/{model}:rawPredict
+  (streaming → :streamRawPredict)
+```
+
+The router removes the top-level `"model"` field (it moves into the URL path), injects `"anthropic_version": "vertex-2023-10-16"` into the body, and strips the `anthropic-beta` request header (Vertex rejects it).
+
+---
+
+## Requirements
+
+### Requirement: Credential-free cloud-model access from sandboxes
+
+Sandbox agents SHALL be able to invoke cloud inference models without any provider credential (API key, OAuth token, service-account material) being injected into, stored in, or readable from the sandbox.
+
+#### Scenario: No provider credential in the sandbox environment
+
+- GIVEN a `google-vertex-ai` provider configured on the gateway with a valid token
+- AND a workspace inference route pointing `inference.local` at that provider
+- WHEN an agent inside a sandbox calls the model via `https://inference.local`
+- THEN the request SHALL succeed
+- AND no GCP token, service-account key, or Anthropic key SHALL be present in the sandbox's environment, filesystem, or process arguments
+
+#### Scenario: Caller-supplied credential is ignored
+
+- GIVEN an inference route configured on `inference.local`
+- WHEN a sandbox client sends any `Authorization` or `x-api-key` header (including a real secret or the sentinel)
+- THEN the router SHALL strip that header before forwarding
+- AND the router SHALL inject the operator-configured provider credential server-side
+- AND the caller-supplied value SHALL NOT reach the upstream provider
+
+---
+
+### Requirement: Workspace inference route configuration
+
+The platform SHALL support a workspace-scoped inference route that binds a provider and a forced model to the `inference.local` virtual host, configured via `openshell inference set --provider <name> --model <id>`.
+
+| Route | Name | Audience |
+|---|---|---|
+| user-facing | `inference.local` | agent/user code in sandboxes |
+| system | `sandbox-system` | platform functions (agent harness); not reachable by user code |
+
+#### Scenario: User route configured for a Vertex Claude model
+
+- GIVEN a `google-vertex-ai` provider `vertex-claude` holding a valid GCP token
+- WHEN the operator runs `openshell inference set --provider vertex-claude --model claude-sonnet-4-5@<date>`
+- THEN `openshell inference get` SHALL report the user route as configured with that provider and model
+- AND requests to `https://inference.local/v1/messages` SHALL be served by that model
+
+#### Scenario: Global-region routes skip client-side endpoint verification
+
+- GIVEN a Vertex provider whose region is `global`
+- WHEN the operator configures the route
+- THEN `--no-verify` MAY be required so the CLI does not pre-flight-probe a region-specific endpoint that does not exist for `global`
+
+---
+
+### Requirement: Client request shape must match the provider API contract
+
+Because the inference router performs a targeted field-level adaptation (not a full schema transpile), the client's request body fields SHALL be compatible with the upstream provider's API version. Newer client fields that the provider's validation rejects SHALL be avoided at the client, since the router does not strip them.
+
+#### Scenario: Vertex Anthropic partner endpoint rejects newer client fields
+
+- GIVEN Claude Code configured with `ANTHROPIC_BASE_URL=https://inference.local` against a Vertex Anthropic route (`anthropic_version = vertex-2023-10-16`, strict body validation)
+- WHEN Claude Code uses an effort-capable model that emits `thinking: {type: "adaptive"}` and `output_config.effort`
+- THEN Vertex SHALL reject the request with HTTP 400 (`thinking` tag mismatch; `output_config.effort` not permitted)
+- AND selecting a non-effort model (e.g. `claude-sonnet-4-5`) SHALL suppress those fields and the request SHALL succeed
+
+> These fields are gated on the model, not on an environment flag — no Claude Code
+> env var strips them. Pinning a non-effort model is the supported lever. See the
+> [`ibm-cluster`](../../skills/deploy/ibm-cluster/SKILL.md) skill for the exact recipe.
+
+#### Scenario: Native-provider client mode must be disabled
+
+- GIVEN Claude Code inside a sandbox
+- WHEN it is configured for a provider's native auth mode (`CLAUDE_CODE_USE_VERTEX=1`)
+- THEN it SHALL attempt Google Application Default Credentials (absent in the sandbox) and fail/hang
+- AND the supported configuration SHALL instead use standard Anthropic mode with `ANTHROPIC_BASE_URL=https://inference.local` and a throwaway `ANTHROPIC_API_KEY`
+
+---
+
+## Client Configuration Reference
+
+Point a standard inference client at the router; the key value is discarded:
+
+```bash
+# Anthropic Messages surface
+ANTHROPIC_BASE_URL="https://inference.local" ANTHROPIC_API_KEY=unused claude --model claude-sonnet-4-5
+
+# OpenAI-compatible surface
+ANTHROPIC_BASE_URL="https://inference.local/v1" ANTHROPIC_API_KEY=unused opencode
+```
+
+Persisting this for Claude Code (so bare `claude` works) via `~/.claude/settings.json` inside the sandbox:
+
+```json
+{
+  "model": "claude-sonnet-4-5",
+  "env": {
+    "ANTHROPIC_BASE_URL": "https://inference.local",
+    "ANTHROPIC_API_KEY": "unused"
+  }
+}
+```
+
+---
+
+## Debugging Reference
+
+| Symptom | Root Cause | Fix |
+|---|---|---|
+| Client hangs, no HTTP response | `CLAUDE_CODE_USE_VERTEX=1` set → client tries Google ADC (none in sandbox) | Unset it; use standard mode + `ANTHROPIC_BASE_URL=https://inference.local` |
+| `400 thinking: 'adaptive' does not match 'disabled'/'enabled'` | Effort-capable model emits adaptive thinking; Vertex `vertex-2023-10-16` rejects it | Pin a non-effort model: `--model claude-sonnet-4-5` |
+| `400 output_config.effort: Extra inputs are not permitted` | Effort-capable model emits `output_config`; Vertex rejects it | Pin a non-effort model (same fix) |
+| `404` model not found | Model ID not published for the GCP project/region | Use a published Vertex model ID (e.g. `claude-sonnet-4-5@<date>`) |
+| `403` on inference call | GCP IAM deny on `aiplatform.endpoints.predict`, or provider token expired | Fix GCP IAM / refresh the provider credential (`openshell provider refresh`) |
+| Route verification fails at `inference set` for `global` region | CLI pre-flights a region endpoint that does not exist for `global` | Add `--no-verify` |
+
+---
+
+## References
+
+- [`ibm-cluster`](../../skills/deploy/ibm-cluster/SKILL.md) skill — ROKS runbook for provider + inference + sandbox agent wiring
+- [OpenShell inference routing](https://docs.nvidia.com/openshell/latest/sandboxes/inference-routing)
+- [OpenShell supported agents](https://docs.nvidia.com/openshell/latest/about/supported-agents)
+- [Vertex AI Anthropic Claude models](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude)