Support OpenShell per-sandbox HTTP service exposure (ExposeService) via a shared sandbox-ingress proxy

Related to #234 (and the TLS cross-reference in #220).

Design note: this body reflects the dedicated-exposure-domain variant (*.sandbox.<base_domain>) adopted after review (discussion). The SAN-to-base_domain assumption it depends on has been confirmed against OpenShell source (see the Certificates section).

Why we are doing this, in plain terms

OpenShell lets a sandbox publish an HTTP service and hands the user a unique web address for it, like https://default--my-sandbox.<domain>/. Every sandbox and every named service gets its own address. OpenShell then looks at the Host header on each incoming request to decide which sandbox it belongs to.

Today, HyperShell-provisioned gateways do not make those addresses reachable. We set up routing for exactly one address per gateway (the gRPC control-plane address, gw-<namespace>.<base_domain>). Any request to a per-sandbox address is rejected by the platform router with a bare 404 before it ever reaches the OpenShell gateway. So ExposeService simply does not work on our platform. Downstream users (for example jameswnl/lightspeed-cloud-agents) have hit this and are building gRPC-tunnel workarounds to get around it.

Why not just give every gateway its own wildcard DNS name?

The "obvious" fix is to give each gateway its own subdomain, *.gw-<namespace>.<base_domain>, with its own wildcard TLS certificate. That routes cleanly with standard Gateway API rules and needs no OpenShell change. The catch is one public TLS certificate per gateway. We use Let's Encrypt, which limits how many brand-new certificates you can get for one registered domain per week. A stable fleet is fine (renewals do not count), but high-churn or ephemeral-PR-environment gateways would burn through that budget. We do not want the number of certificates we have to mint to grow with the number of gateways.

Why a proxy instead

A single shared reverse proxy per Cloud Hub lets us keep the number of public certificates fixed regardless of how many gateways or sandboxes exist. The public hostname stays a single leftmost DNS label, so one wildcard certificate keeps covering it; the proxy reads the incoming hostname, works out which gateway the sandbox belongs to, and forwards the request there. We give exposure traffic its own domain (*.sandbox.<base_domain>) and its own listener, so it is isolated from the gRPC control plane: two fixed certificates per hub (control plane + exposure), no per-gateway growth, and it scales to any number of gateways.

The proxy needs a way to know which gateway a given sandbox address belongs to. Near term we embed a small routing token in the hostname (a tiny, well-scoped OpenShell change). Longer term OpenShell is building a discovery API that tells an ingress exactly that, which lets us drop the token and route per-sandbox precisely.

Where OpenShell is heading, and why this is the right bet

OpenShell merged a design (RFC 0017, rfc/0017-sandbox-service-exposure/README.md, PR NVIDIA/OpenShell#4267, 2026-10-13) for the next generation of sandbox exposure. Two things matter for us:

1. OpenShell is not going to provision cluster routing for us. In both today's shipped behavior and the planned design, the hosting platform is responsible for DNS, certificates, and routing that preserves the sandbox hostname. OpenShell owns only the in-process demux and (in future) a discovery API. So the platform-side proxy work is not throwaway; it is the sanctioned integration point. OpenShell even ships a reference openshell-sandbox-proxy binary for operators to mirror.
2. The hostname scheme survives. The future design keeps {ws}--{sandbox}[--service].{domain} and the Host/SNI demux; it just moves the listener to a dedicated per-supervisor port and adds a discovery API that publishes which supervisor owns each hostname. None of it is shipped yet (latest release v0.1.5, 2026-10-20, still runs today's gateway shared-port behavior), and there is no target milestone.

So a shared proxy is aligned with the roadmap, not obsoleted by it. When the discovery API lands, our proxy swaps its routing source from "parse a token in the hostname" to "consume the discovery stream" and becomes strictly better (per-sandbox routing, no token, no OpenShell hostname change). We get a working feature now and a clean migration path.

Current state

How gateway exposure works today (gateway-api mode, the only mode now that ROKS is on OCP 4.22)

• A one-time per-Cloud-Hub bootstrap (skills/deploy/cloud-hub-ingress-bootstrap/SKILL.md) provisions:
  • a wildcard Let's Encrypt certificate *.<base_domain> (cert-manager, DNS-01 via Route53),
  • a static wildcard DNS record *.<base_domain> to the load balancer,
  • a single shared Gateway API Gateway (openshell-grpc-gateway, namespace openshift-ingress) with one HTTPS/443 listener hostname: *.<base_domain>, tls.mode: Terminate using the wildcard secret, allowedRoutes.namespaces.from: All.
• Per gateway, the control plane (components/control-plane/internal/gateway/reconciler.go, reconcileGatewayAPIResources, ~line 1499) creates:
  • a GRPCRoute named openshell-gateway in the tenant namespace, hostnames: [gw-<ns>.<base_domain>], backendRefs -> Service openshell-gateway port 8080, cross-namespace parentRef to the shared Gateway's listener (sharedGatewayListenerName(), default https);
  • a backend CA ConfigMap openshell-gateway-backend-ca (holds ca.crt from the per-tenant openshell-server-tls secret);
  • a BackendTLSPolicy openshell-gateway validating the pod cert against that CA (the re-encrypt leg to the pod).
• The per-tenant hostname is derived in one place (exposure.DeriveGatewayAPIHost, components/control-plane/internal/exposure/gatewayapi.go:83) as gw-<namespace>.<base_domain>.
• The gateway pod serves both gRPC and HTTP on a single port (8080); OpenShell peeks the first bytes, terminates TLS itself, demuxes gRPC vs HTTP by content-type, and demuxes sandboxes by Host header.
• The control plane already injects per-tenant serving-cert SANs via ServerDnsNames (reconciler.go:70, appendDNSNameIfMissing; wired through components/control-plane/internal/helm/values.go:206). The chart copies that same list into OpenShell's server_sans config (charts/openshell/templates/_toml.tpl:114-117), and OpenShell derives its service-routing base_domains from server_sans.

The gap

• No HTTPRoute is created for the sandbox data path; only the per-gateway GRPCRoute (plus the console HTTPRoute) exists.
• Both the single wildcard DNS record and the single wildcard certificate are first-level only (*.<base_domain>); they do not cover a second level (*.gw-<ns>.<base_domain>). Wildcards do not nest.
• The control plane has no knowledge of sandboxes (sandboxes are not API resources; they are created inside OpenShell over gRPC), so it cannot reactively create per-sandbox routes.
• OpenShell's sandbox hostname {ws}--{sandbox}[--service].<domain> carries no gateway/tenant discriminator, so a shared multi-tenant Gateway cannot tell which gateway pod a sandbox hostname belongs to.

Motivation

• Make OpenShell ExposeService a supported, first-class capability on HyperShell-provisioned gateways (resolve Neither route nor gateway-api ingress mode provisions routing for OpenShell's per-sandbox ExposeService HTTP virtual-hosting #234's open "is this in scope" question with "yes").
• Do it without growing the number of public TLS certificates with the number of gateways (protect the Let's Encrypt registered-domain budget, especially for ephemeral/PR-env gateways).
• Keep exposure traffic (public, higher-churn, untrusted) isolated from the gRPC control plane and console.
• Pick a design that is forward-compatible with OpenShell RFC 0017 so the investment compounds rather than being thrown away.

Proposed design: shared sandbox-ingress proxy on a dedicated exposure domain (Option B)

Topology

client (browser / HTTP + WebSocket)
  |  TLS, SNI + Host = {ws}--{sandbox}[--svc]--gw-<ns>.sandbox.<base_domain>
  v
shared Gateway, dedicated exposure listener  *.sandbox.<base_domain>   (NEW)
  |  TLS terminate, NEW wildcard cert (one per hub, fixed), allowedRoutes = proxy ns only
  |  HTTPRoute  *.sandbox.<base_domain>  (NEW, scoped to the exposure domain)  -> hypershell-sandbox-proxy Service
  v
hypershell-sandbox-proxy   (NEW, one per Cloud Hub)
  |  parse trailing "--gw-<ns>" from Host -> target namespace
  |  strip the token, forward with Host = {ws}--{sandbox}[--svc].sandbox.<base_domain>
  |  TLS re-encrypt, validate per-tenant openshell-ca (aggregated CA bundle)
  v
openshell-gateway.gw-<ns>.svc.cluster.local:8080   (existing gateway pod)
  |  OpenShell terminates TLS, demuxes by Host header, relays to sandbox loopback port
  v
sandbox service (127.0.0.1:<port>)

The existing *.<base_domain> listener and the per-gateway GRPCRoutes (control plane) are unchanged and untouched.

Why a dedicated exposure domain (vs reusing *.<base_domain>)

• Scoped catch-all. A catch-all on *.<base_domain> would send every otherwise-unmatched hostname under the whole base domain to the proxy, turning typos and unallocated names into proxy traffic instead of a router 404. *.sandbox.<base_domain> matches exposure traffic and nothing else; unmatched traffic under <base_domain> behaves exactly as today.
• Physical isolation, no precedence tricks. Exposure and control plane live on separate listeners with separate certs and separate routes, so a bad HTTPRoute, proxy bug, or cert-rotation mishap in exposure cannot affect the gRPC control plane or console. This also avoids relying on Gateway API hostname-specificity precedence between a GRPCRoute and a wildcard HTTPRoute on one listener (underspecified).
• Tighter attachment surface. The exposure listener can set allowedRoutes to the proxy namespace only, while the control-plane listener keeps from: All for tenant GRPCRoutes.
• No overlap. Single-label wildcards do not nest: *.<base_domain> matches X.<base_domain> only, not foo.sandbox.<base_domain>, so SNI selects the right listener unambiguously.
• Room to evolve. Exposure can later get its own TLS policy, HSTS, timeouts, or independent scaling without touching the control-plane front door.

Why this still routes correctly despite Gateway API limits

• The public hostname {ws}--{sandbox}[--svc]--gw-<ns>.sandbox.<base_domain> is a single leftmost DNS label under sandbox.<base_domain>, so it is covered by one wildcard DNS record and one wildcard certificate. No per-gateway DNS or certificate.
• Standard Gateway API hostname matching cannot select a gateway from a token buried inside a label, which is exactly why the proxy exists: it reads the full Host, extracts the trailing --gw-<ns> token, and forwards to that gateway's internal Service.

Certificates (no Let's Encrypt growth)

Two fixed certificates per hub, regardless of gateway count:

• Control-plane edge: unchanged. The existing *.<base_domain> Let's Encrypt wildcard on the existing listener.
• Exposure edge: one new *.sandbox.<base_domain> Let's Encrypt wildcard (cert-manager, DNS-01, same Route53 issuer) on the new listener. Minted once; renewals do not count against the LE new-certificate quota.
• Proxy -> gateway (internal re-encrypt leg): the proxy validates the per-tenant self-signed openshell-ca (free, no Let's Encrypt). The control plane maintains an aggregated CA bundle (see changeset) so the proxy trusts every gateway pod.
• Gateway pod serving cert + base_domain: add *.sandbox.<base_domain> to the per-tenant ServerDnsNames (self-signed openshell-ca, free). The chart already copies serverDnsNames into OpenShell's server_sans (_toml.tpl:114-117), and OpenShell derives base_domains from server_sans, so this single injection makes OpenShell both serve and demux *.sandbox.<base_domain>.

Confirmed against OpenShell source (the load-bearing assumption): service_routing_domain_from_server_san (crates/openshell-core/src/config.rs:1186-1190) does strip_prefix("*.") and discards non-wildcard SANs, so SAN *.sandbox.<base_domain> yields base_domain = sandbox.<base_domain> (unit test service_routing_domains_are_derived_from_wildcard_server_sans). parse_host then strips the .sandbox.<base_domain> suffix and splits the leftmost {ws}--{sandbox} label on --. Note the derivation reads the configured server_sans list, not the cert file's SAN extension, so the two must stay consistent; our chart keeps them consistent automatically via _toml.tpl.

The one OpenShell change required (small, upstream)

OpenShell must emit the routing token in the user-facing service hostname while leaving its internal Host demux unchanged:

• Add a config key, e.g. service_routing.external_hostname_suffix, appended as the final ---segment of the leftmost label in endpoint_host / endpoint_url (crates/openshell-server/src/service_routing.rs). HyperShell sets it to --gw-<ns> per gateway.
• parse_host / proxy_sandbox_service_request are unchanged: the proxy strips the suffix before forwarding, so OpenShell demuxes against its existing base_domains exactly as today.
• This is a clean, additive, operator-oriented knob that fits RFC 0017's "operator-owned ingress + routing token" direction. We should raise it upstream referencing RFC 0017 / the public_url work (feat(services): publish authoritative supervisor ingress URLs and status NVIDIA/OpenShell#4670). The certificate problem and the discriminator problem are orthogonal: a dedicated cert does not remove this change, and the only design that avoids it is Option A (per-gateway certs).

Auth, protocols, and what is NOT changed

• Auth is preserved end to end. OpenShell consumes gateway auth at its own listener and does not forward Authorization to the sandbox. The proxy is auth-transparent: it passes Authorization through to the OpenShell gateway, which still enforces it. The proxy adds no auth of its own.
• HTTP and WebSocket must both be supported (ExposeService relays both); the proxy must handle Upgrade/Connection and bidirectional streaming.
• gRPC control plane and console are untouched (separate listener, separate certs, separate routes; not mTLS-terminated by the proxy).

Changeset

HyperShell

1. New component components/sandbox-proxy/ (Go).

   • Reverse proxy. Reads Host/:authority, extracts and strips the trailing --gw-<ns> token, rewrites Host to {ws}--{sandbox}[--svc].sandbox.<base_domain>, forwards to openshell-gateway.gw-<ns>.svc.cluster.local:8080 over TLS.
   • Validates the backend against an aggregated per-tenant CA bundle (mounted ConfigMap, hot-reloaded). No InsecureSkipVerify.
   • Full HTTP/1.1 + HTTP/2 + WebSocket passthrough; streaming, no buffering of bodies.
   • Returns 404 for unparseable hosts or unknown/absent target namespaces; never a bare/empty 404 (include a diagnostic body).
   • Health, readiness, and metrics endpoints; structured slog + OTel (consistent with the api-server logging change in feat(api-server): adopt the rh-trex-ai slog logger and export logs over OTel #680).
   • Restricted SecurityContext: runAsNonRoot, drop ALL capabilities (per repo convention).
   • Config via env/flags only (no code changes for domain/listener config).

2. Dedicated exposure edge (per Cloud Hub), folded into skills/deploy/cloud-hub-ingress-bootstrap/SKILL.md.

   • One new Let's Encrypt wildcard certificate *.sandbox.<base_domain> (cert-manager, DNS-01, same Route53 issuer).
   • One new static wildcard DNS record *.sandbox.<base_domain> -> the same load balancer.
   • One new HTTPS/443 listener on the existing shared Gateway, hostname: *.sandbox.<base_domain>, tls.mode: Terminate with the new cert, allowedRoutes restricted to the proxy namespace.
   • A Deployment + Service for hypershell-sandbox-proxy (one per hub, not per gateway), in deploy/openshift/infrastructure/.
   • An HTTPRoute *.sandbox.<base_domain> attached to that listener, backendRefs -> the proxy Service (plus a ReferenceGrant if HTTPRoute and proxy are in different namespaces from the Gateway).

3. Control-plane reconciler changes (components/control-plane/internal/gateway/reconciler.go).

   • Add *.sandbox.<base_domain> to the per-tenant ServerDnsNames (extend the appendDNSNameIfMissing injection at ~line 70, derived from GATEWAY_API_BASE_DOMAIN). This feeds both the serving cert SAN and OpenShell's server_sans -> base_domains via the existing _toml.tpl wiring; no separate server-SAN plumbing needed.
   • Set the OpenShell service_routing.external_hostname_suffix to --gw-<ns> for the gateway (Helm values, alongside existing config wiring in components/control-plane/internal/helm/values.go).
   • Maintain an aggregated CA bundle ConfigMap consumed by the proxy: on gateway create/update/delete, upsert/remove that tenant's openshell-ca (ca.crt from openshell-server-tls, the same source already read for openshell-gateway-backend-ca). Must reconcile (update-or-create / prune), never create-and-ignore, and never silently swallow partial failures (repo conventions).
   • Gate the whole behavior behind a feature flag / env (e.g. GATEWAY_SANDBOX_EXPOSURE=enabled), default off initially.

4. Helm chart (charts/openshell/).

   • Thread the service_routing.external_hostname_suffix value into the gateway config template (_toml.tpl).
   • The *.sandbox.<base_domain> SAN flows through the existing pkiInitJob.serverDnsNames / certManager.serverDnsNames paths into both the pod cert and server_sans; verify both landing spots in _toml.tpl and the cert templates.

5. CI and registration.

   • Register components/sandbox-proxy with /maintain-ci (build, test, image). Add to make build-all, image overlays, and the component list in CLAUDE.md / specs.
   • Image reference must match across overlays/manifests (grep-all convention after naming).

6. Spec + docs.

   • New/updated spec under specs/platform/ (extend openshell-gateway-routing.spec.md or add openshell-sandbox-exposure.spec.md) capturing requirements, the hostname/token contract, the dedicated exposure domain + two-cert invariant, the CA-bundle contract, the SAN/server_sans consistency requirement, and the feature flag.
   • Document the OpenShell version dependency (the external_hostname_suffix knob) and the PR-env posture.
   • No em dashes in any committed text (pre-commit hook).

7. Tests.

   • Unit tests for the proxy: host parsing/token stripping, 404 paths, WebSocket upgrade, backend TLS validation against a test CA.
   • Reconciler unit tests for the CA-bundle aggregation (add/remove/prune) and value wiring.
   • An e2e that exposes a sandbox service through a Kind-provisioned gateway + proxy and asserts a 200 end to end (extend tests/e2e/).

OpenShell (upstream, small)

• Add service_routing.external_hostname_suffix (config only; appended in endpoint_host/endpoint_url, ignored by parse_host). Raise against NVIDIA/OpenShell referencing RFC 0017 and #4670. Pin the minimum OpenShell version in the installer/spec once merged.

Migration to the OpenShell discovery API (future, when RFC 0017 ships)

When OpenShell ships the gateway discovery API + supervisor app-listener (RFC 0017; children NVIDIA/OpenShell#4666/#4667/#4670/#4631, all currently open/design-stage):

• Replace the proxy's "parse --gw-<ns> from the hostname" step with a discovery adapter that watches the hostname -> owning-supervisor/app-address stream and routes per-sandbox precisely.
• Drop the OpenShell external_hostname_suffix dependency and the in-label token; hostnames return to clean {ws}--{sandbox}[--svc].sandbox.<base_domain>.
• Point the backend leg at the new per-supervisor app-listener port/domain instead of the shared gRPC port.
• Mirror OpenShell's reference openshell-sandbox-proxy where practical.

The public edge (dedicated exposure domain: wildcard DNS + single wildcard cert + dedicated listener + scoped HTTPRoute + proxy Service) is unchanged across this migration, so the near-term build is a stable foundation, not throwaway work.

Alternatives considered

• Reuse *.<base_domain> instead of a dedicated exposure domain: saves one fixed certificate per hub, but the catch-all HTTPRoute swallows all unmatched base-domain traffic into the proxy, shares the listener/cert with the control plane (no blast-radius isolation), requires from: All on the exposure routes, and leans on underspecified GRPCRoute-vs-HTTPRoute precedence. Rejected; the one extra fixed cert is worth the isolation.
• Option A - per-gateway subdomain ({ws}--{sandbox}.gw-<ns>.<base_domain>): native Gateway API routing, no OpenShell change, but one public wildcard certificate per gateway (second-level wildcard not covered by the existing cert/DNS). Simplest to ship, but grows certificates with gateways and risks the Let's Encrypt budget under churn. Kept as the fallback if the upstream external_hostname_suffix change is not accepted.
• Flat, no discriminator ({ws}--{sandbox}.<domain>): one shared cert and no OpenShell change, but cannot disambiguate which gateway owns a sandbox when more than one gateway shares a hub. Not multi-tenant safe.
• Route mode / passthrough wildcard: not applicable now that all hubs use gateway-api mode (ROKS on OCP 4.22). route mode code paths should be retired separately.

Open questions

1. Token format: --gw-<ns> reuses the existing namespace label; confirm it stays within the 63-char DNS-label budget combined with {ws}--{sandbox}--{svc}. Namespaces are RFC 1123 labels and cannot contain --, so "the final --gw-... segment is the token" is a sound parse rule; the length budget is the part that needs a concrete answer (OpenShell #4631 hash-prefixing may help long names later).
2. CA-bundle distribution: aggregated ConfigMap mounted into the proxy vs. the proxy watching per-tenant secrets directly. ConfigMap keeps the proxy simple and RBAC-narrow; preferred unless size becomes an issue.
3. Whether to scope the feature to stage/prod initially and keep it off for ephemeral PR environments regardless (public exposure + cert posture).
4. Upstream appetite for external_hostname_suffix vs. waiting for the discovery API; if upstream prefers we wait, ship Option A as an interim.

Acceptance criteria

• A sandbox HTTP service exposed via openshell service expose is reachable from an external client at its OpenShell-issued URL through a HyperShell-provisioned gateway, returning the sandbox's real response (not a platform 404).
• WebSocket upgrade through an exposed service works.
• No increase in the number of public (Let's Encrypt) certificates as gateways are added (fixed at two per hub).
• gRPC control-plane traffic and the per-gateway console are unaffected, including under exposure-side failures.
• The feature is behind a flag and documented, with the OpenShell minimum version pinned.