Support TLS-passthrough routing (TLSRoute / routing.tcp) in NebariApp

[viniciusdc]

Revised 2026-08-06. Rebuilt on the per-app ListenerSet primitive to align with ADR-0011 / #168 - the original dask-gateway-pack draft predated that and proposed upserting a listener onto the shared Gateway, the co-ownership pattern that work removes. Also reframed after confirming the gap live on an enforcing cluster (Hetzner k3s): this is not only an off-cluster limitation, it breaks the primary in-cluster dask client too (see Summary).

Summary

NebariApp reconciles HTTP only today: one hostname, one backend Service, HTTPRoutes, and per-app HTTPS Terminate listeners (api/v1/nebariapp_types.go, reconcilers/{routing,tls}/). Some workloads carry L4 traffic that is TLS end-to-end and routed by SNI - the gateway must forward the raw TLS stream without terminating it. Gateway API models this as a TLSRoute attached to a listener with tls.mode: Passthrough; Envoy Gateway supports it, but the operator cannot express it.

The impact is broader than "expose one more thing off-cluster." Without a passthrough listener, a pack's SNI/TLS traffic can only stay in-cluster - and on any cluster that enforces NetworkPolicies, staying in-cluster does not even serve in-cluster clients: the pack's internal proxy sits on a ClusterIP in a private range that hardened egress policies deny, so the intended workload cannot reach it. This is therefore a correctness gap for the primary consumer on production-grade clusters, with off-cluster access as the secondary case.

Motivating case: dask-gateway-pack

dask-gateway's client<->scheduler protocol (dask-gateway 2026.3.0):

• Every Dask cluster gets its own TLS credentials, generated by the gateway and terminated at the scheduler pod (mutual TLS).
• The dask_gateway client always connects through a proxy endpoint, dialing gateway://<proxy-address>/<cluster> and sending TLS SNI daskgateway-<namespace>.<cluster-name> (dask_gateway/comm.py); the proxy routes purely on SNI, passthrough. There is no direct-to-scheduler mode and no plain-HTTP fallback.
• Upstream ships a bundled Traefik for this (per-cluster IngressRouteTCP, tls.passthrough: true). In the Nebari pack that Traefik is kept strictly in-cluster (ClusterIP). Confirmed live on an enforcing Hetzner k3s cluster (2026-08-06), this breaks connectivity two ways:
  • Off-cluster dask clients cannot reach schedulers at all - they hit the REST API through the NebariApp route, create a cluster, then fail to connect.
  • In-cluster, on any CNI that enforces NetworkPolicies, the data-science-pack singleuser egress policy permits only the Envoy gateway (:10443) plus DNS/hub/proxy and public-minus-private ranges. Traefik's ClusterIP sits in 10/8, which that policy denies - so a JupyterHub notebook, the intended dask client, cannot reach the scheduler proxy either. "Keep Traefik in-cluster" does not serve the in-cluster notebook once policies are enforced.
  • Repro caveat: this surfaces only on an enforcing CNI. On k3d (flannel) the policy objects exist but are inert, so the notebook reaches Traefik and the gap is invisible - use k3s/Calico to see it.

SNI shape matters: the varying label (<cluster-name>, a UUID) is the second DNS label, so no single listener hostname wildcard can cover it (wildcards are leftmost-label only). A passthrough listener therefore needs either no hostname (match-all on a dedicated port) or per-route exact hostnames - which is exactly how TLSRoute works.

How this must fit the gateway rework (read first)

This depends on and builds on the ListenerSet migration - it is not the shared-Gateway-upsert design the first draft described:

• feat(tls): adopt per-app ListenerSet for TLS ownership (ADR-0011, Option 2) #168 (ADR-0011 Option 2) moves the operator's per-app listeners off NIC's shared Gateway into an operator-owned per-app ListenerSet attached via spec.parentRef. The passthrough listener introduced here MUST live in that same ListenerSet. Do not upsert it onto the shared Gateway - that is the exact pattern we are eliminating.
• A ListenerSet listener can be TLS/Passthrough just as well as Terminate, so this needs no new ownership mechanism: it is an additional listener protocol on the primitive already introduces.
• The "own only the listener, let the app attach its own routes" variant below is the same shape as nebari-infrastructure-core#403 (listener-only TLS) and the llm-serving-pack shared-listener use case. Converge into one "operator owns a listener (Terminate or Passthrough), routes optionally app-managed" primitive rather than a parallel design.

Proposed spec surface (sketch)

apiVersion: reconcilers.nebari.dev/v1
kind: NebariApp
spec:
  hostname: dask-gateway.example.com
  service: { name: api-dask-gateway-pack, port: 8000 }
  routing:
    routes: [...]
    # NEW: L4 passthrough routing (opt-in; see the TLSRoute stability note below)
    tcp:
      - name: scheduler
        listener:
          port: 8786            # dedicated port; protocol TLS, mode Passthrough
        sniHostnames:
          - "daskgateway-*"     # illustrative; exact semantics TBD
        service:
          name: some-backend    # optional when routes are app-managed
          port: 8786

Operator responsibilities (rebased on ListenerSet)

1. Add a TLS/Passthrough listener on a dedicated port to the app's per-app ListenerSet (the feat(tls): adopt per-app ListenerSet for TLS ownership (ADR-0011, Option 2) #168 primitive), with allowedRoutes scoped to the app's namespace. Never mutate NIC's Gateway.
2. Create TLSRoutes (gateway.networking.k8s.io/v1alpha2) for static backends; or - the dask case, where routes are inherently dynamic (one per running cluster) - own only the listener and let the app's own controller attach namespace-local TLSRoutes.
3. Surface readiness in a new TCPRoutingReady condition and publish the listener/port in status.

TLSRoute stability: GA upstream, version-lagged on our Envoy Gateway stack

TLSRoute graduated to the Standard channel (gateway.networking.k8s.io/v1) in Gateway API v1.6.0, so it is not an experimental API in the abstract - the earlier worry that it clashes with ADR-0011's "no unstable API in the stable contract" driver rested on stale information.

The real constraint is a version lag in Envoy Gateway, not the API's maturity: EG v1.8.x (the release the NIC upgrade currently targets) pins Gateway API v1.5.1, where TLSRoute is still v1alpha2; EG main already pins Gateway API v1.6.1, and the next release (v1.9.0, ~Aug 2026) ships it, at which point TLSRoute is v1 on our stack. So it is a timing gap that closes on its own, not a stable-vs-experimental design conflict - and a data point for targeting EG v1.9.0 over v1.8.x in the NIC upgrade.

To keep the operator's own contract off the alpha version regardless of the lag, prefer the listener-only split: the operator owns the stable v1 Passthrough listener in its ListenerSet, and the app (e.g. the dask pack) owns the TLSRoute in its own namespace. The only path that pulls v1alpha2 into the operator is the optional "operator creates TLSRoutes for static backends" (step 2 above); gate that as explicitly opt-in until the stack is on EG v1.9.0+ / Gateway API v1.6.

[viniciusdc]

I wired the dask-gateway-pack against the data-science-pack on an enforcing cluster this week (Hetzner k3s, split namespaces: gateway in dask-gateway, notebooks in jupyterhub) and hit this gap live, so a confirmation plus one wrinkle worth adding.

Confirmed as described: REST is fine through the NebariApp Envoy route (the /api publicRoute; /api/health returns 200 through Envoy). The client-to-scheduler hop (:8786, TLS/SNI, gateway://) has nowhere to go: every listener the operator provisions is HTTP:80 / HTTPS:443 Terminate, no TLSRoute/TCPRoute, so the scheduler SNI stream has no listener to land on. Matches the issue exactly.

The wrinkle: the issue frames this as off-cluster ("off-cluster dask clients cannot connect to schedulers"), but on an enforcing cluster it is also an in-cluster problem, for the primary consumer. The data-science-pack singleuser egress NetworkPolicy only permits the Envoy gateway (:10443) plus DNS/hub/proxy and public-internet-minus-private-ranges. The pack keeps its Traefik strictly in-cluster on a ClusterIP in 10/8, which is exactly the range that egress policy excludes. So a JupyterHub notebook, the intended dask client, cannot reach the in-cluster Traefik scheduler proxy either. It reaches the REST API via Envoy, creates a cluster, then fails to connect to the scheduler. So "keep Traefik strictly in-cluster" does not actually serve the in-cluster notebook once NetworkPolicies are enforced. The passthrough listener is needed for in-cluster clients too, not only off-cluster.

One repro note so nobody chases a ghost: this only shows up on a CNI that enforces NetworkPolicies. On k3d the policy objects exist but are inert (flannel does not enforce), so the notebook reaches Traefik there and the gap is invisible. You need an enforcing cluster like k3s/Hetzner to see it.

To unblock testing I added a temporary egress NetworkPolicy (singleuser to dask-gateway:8786). That is a stopgap, and it is exactly what a Passthrough listener on the per-app ListenerSet (#168 / ADR-0011) would make unnecessary. It also reinforces the "operator owns a listener, routes optionally app-managed" shape you land on here, since the in-cluster case wants the same listener the off-cluster case does.