spec: global architecture spec & IBM ROKS impl fixes - #85
Open
markturansky wants to merge 15 commits into
Open
Conversation
Captures deployment patterns (single-node, global multi-region, multi-cloud), tooling stack decisions (CNPG, ArgoCD, Tekton, Vault, Terraform, Prometheus), namespace strategy, installer pipeline requirements, and monitoring architecture from the Aug 10 architecture meeting. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
…VPC LB scope, and control-plane reconciler bug
…ss skills Add two deploy skills and cross-link them from deploy-cluster and CLAUDE.md: - cloud-hub-ingress-bootstrap: cloud-agnostic shared Gateway + wildcard DNS/TLS bootstrap (AWS reference / IBM parity). Encodes the OCP >= 4.19 requirement for the built-in openshift-default GatewayClass and the nlb-dns DNS+TLS path. - ibm-cluster: ROKS VPC Gen2 provisioning mirroring the reference cluster, the cluster-create command (COS CRN required, not GUID), and a registry-storage decision table (emptyDir / PVC / COS) with PVC as the chosen persistent backend. - deploy-cluster: add Cloud-Hub parameter overrides (registry host, ibmc-vpc-block storage class) and scope note pointing at the ingress bootstrap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The IBM Cloud Parity Plan now documents the root cause of the tenant-gateway ingress gap: the built-in CIO-managed Gateway API (openshift-default GatewayClass) is GA only on OCP >= 4.19, and the original hypershell-cluster ran 4.17. The fix is a new >= 4.19 cluster (hysh-ibm-01, 4.21.27), not an in-place upgrade, cross-linked to the ibm-cluster skill. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
markturansky
force-pushed
the
add-global-architecture-spec
branch
from
August 15, 2026 17:56
653388b to
6fcd021
Compare
… or Route) Tenant-gateway ingress is now a selectable mode, chosen per environment by configuration rather than hardcoded: emit Kubernetes Gateway API GRPCRoutes where the Gateway API is available and functional, or OpenShift Routes (HAProxy passthrough) where it is not. Motivation: IBM Cloud ROKS is HyperShift-hosted and cannot run the CIO-managed Istio (OSSM images unpullable, IDMS denied on the HostedCluster), yet ships the Gateway API CRDs. Route passthrough preserves the gateway pod's per-tenant self-signed TLS + client mTLS end-to-end, so it needs no shared Gateway, wildcard cert, ClusterIssuer, or external DNS - it works on IBM's free *.containers.appdomain.cloud wildcard. Control plane: - GATEWAY_INGRESS_MODE env var (gateway-api|route|none); auto-detects from opts.HasGatewayAPI/opts.IsOpenShift when unset. Explicit override wins, since ROKS's Gateway API CRDs are present-but-non-functional. - reconcileRouteResources/deleteRouteResources (passthrough Route to openshell-gateway:8080 + openshell-gateway-allow-router NetworkPolicy + grpcs://<host>:443 address publish); shared deriveGatewayHostname/ publishRouteAddress helpers; Route added to kindToResource and cleanup. - Table tests for mode selection and hostname derivation. Deploy: - deploy/ibm kustomize overlay (on deploy/openshift) sets GATEWAY_INGRESS_MODE=route + base domain. Controller ClusterRole already grants route.openshift.io/routes. Docs: - global-architecture.spec.md: two first-class ingress modes with mode-aware requirements/scenarios and the ROKS Route-mode section. - ibm-cluster / cloud-hub-ingress-bootstrap skills: Route mode is the ROKS path; do not run the shared-Gateway bootstrap there. - Swept em dashes from tracked files; whitelisted pre-existing ACP mermaid nodes and rosa-vteam.yaml so make check passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…idable for registry mirrors Gateway, supervisor, and sandbox default images were hardcoded to ghcr.io with no override, so any gateway created without an explicit image failed to pull on clusters whose nodes cannot reach ghcr.io (e.g. IBM ROKS). - Add GATEWAY_IMAGE, GATEWAY_SUPERVISOR_IMAGE, and GATEWAY_SANDBOX_IMAGE env overrides (mirroring the existing HYPERSHELL_DATABASE_IMAGE pattern), with a new DefaultSandboxImage() so the sandbox base is resolved the same way. - Substitute SANDBOX_IMAGE_PLACEHOLDER in the gateway configmap (ordered before IMAGE_PLACEHOLDER since the shorter token is a substring). - Allow an optional host:port in image references so the in-cluster registry service address (image-registry.openshift-image-registry.svc:5000/...) validates. - Apply supervisor_image on gateway PATCH. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…te SANs When an ingress mode is active the gateway is reachable at an external hostname (gw-<namespace>.<base-domain> or an explicit Route.Host), and both ingress modes carry the gateway pod's TLS through unmodified (Route passthrough / Gateway API BackendTLSPolicy). The server certificate must therefore list that external hostname as a SAN, or clients fail verification. The controller derives the hostname, so it injects it into the cert SANs before cert-manager mints the certificate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rror details Extend the global architecture specification with the ingress-mode and internal-registry-mirror behaviour exercised on IBM ROKS, and bump the forbidden-terms whitelist line reference to track the moved example path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document the ROKS-specific deployment path: image mirroring to the internal registry, raw operator installs, and the env overrides the control plane needs on clusters that cannot reach ghcr.io. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add e2e-openshell-roks.sh, a ROKS-layout copy of the canonical e2e that targets the hypershell/hypershell-api route names, trusts the per-gateway CA, uses the quoted SQL-like search grammar, and preserves a pre-existing gateway on cleanup. Add the components/api-server/deploy/ibm overlay with the controller cluster RBAC used on ROKS. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
markturansky
changed the title
markturansky
changed the title
… providers Refine the registry-timeout explanation with its root cause (the kube-<clusterID> worker Security Group is default-deny outbound) and the supported fix (add an outbound 0.0.0.0/0:443 rule). Add section 5.7 covering Keycloak default-secure gateway wiring, the correct OIDC `gateway add` command (not bare edge/cloud mode), and the worker-egress requirement for cloud-model providers such as google-vertex-ai. Correct the stale gateway-add note in 5.5. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…around `openshell sandbox connect` execs the system ssh with a ProxyCommand that re-execs `openshell ssh-proxy`, and the CLI omits `--gateway-insecure` from that generated ProxyCommand. The child ssh-proxy therefore verifies the self-signed passthrough gateway cert and fails `invalid peer certificate: UnknownIssuer`; the flag on `connect` never reaches it. The child inherits the environment, so `export OPENSHELL_GATEWAY_INSECURE=true` is the working fix. Verified live on hysh-ibm-01 (sandbox woot). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add openshell-inference-routing.spec.md documenting how sandbox agents reach cloud models with no credential in the sandbox: the inference.local router strips the client key and injects the provider token server-side, translating /v1/messages -> Vertex :rawPredict. Covers the two credential paths (per-binary sentinel rewrite vs router injection) and the request-shape compatibility requirement. Register it in the spec index. Add ibm-cluster skill section 5.8 with the ROKS runbook: `inference set`, the required non-effort `--model claude-sonnet-4-5` workaround for Vertex's strict vertex-2023-10-16 validation (adaptive-thinking / output_config.effort 400s), sandbox connect, and the ~/.claude/settings.json wiring for bare `claude`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
specs/platform/global-architecture.spec.mddefining the global deployment architecture for HyperShellContext
Captures key decisions from the Aug 10 architecture meeting, including the transition from per-gateway cloud databases to CNPG, from manual bash scripts to Tekton pipelines (cattle not pets), and the hub-per-cloud deployment model.
Test plan
🤖 Generated with Claude Code