ci: add k8s stack PR preview via labeled deploy + Cloudflare Tunnel #39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: K8s Stack Preview | |
| # Deploys the full Nebari platform stack (Keycloak + nic-operator + Envoy | |
| # Gateway, via nebari-dev/action-nebari-sandbox) plus this PR's chart into | |
| # an ephemeral kind cluster on the runner, then exposes | |
| # JupyterHub through a per-PR Cloudflare Tunnel behind Cloudflare Access | |
| # (GitHub SSO) so a reviewer can click a link and use it. | |
| # | |
| # This repo is public, so a plain shared secret posted in the PR comment | |
| # (a quick-tunnel URL, a basic-auth password) is readable by anyone who | |
| # opens the PR, not just intended reviewers. Access closes that gap by | |
| # authenticating the *person*, not a string in the comment: Cloudflare | |
| # challenges every request to *.<vars.PREVIEW_DOMAIN> with a GitHub SSO | |
| # login and only lets it through to cloudflared if the signed-in account | |
| # matches the Access application's policy (currently an explicit email | |
| # allow-list, not org membership -- see the Cloudflare Access application | |
| # for this domain for who's currently allowed). The PR comment only ever | |
| # contains a URL. | |
| # | |
| # The domain is a repo Variable (Settings -> Secrets and variables -> | |
| # Actions -> Variables -> PREVIEW_DOMAIN, currently openteams.app), not | |
| # hardcoded, so it can be repointed without editing this file. Hostnames | |
| # built from it stay single-level (pr-<n>-data-science-pack.<domain>, | |
| # never pr-<n>.data-science-pack.<domain>) deliberately: Cloudflare's | |
| # free Universal SSL only auto-covers the zone apex plus one wildcard | |
| # level (<domain> + *.<domain>); a second level needs the paid Advanced | |
| # Certificate Manager add-on, which this setup doesn't use. | |
| # | |
| # Scope: the tunnel points straight at the `proxy-public` service. The | |
| # chart deploys with nebariapp.enabled=true (nebariapp.auth.enabled=true | |
| # by chart default), so login goes through the real operator-provisioned | |
| # Keycloak OIDC client, exercising the same auth path production deploys | |
| # use. Getting this working required: (1) not blocking `helm upgrade | |
| # --install` with `--wait`, since the hub pod crash-loops on | |
| # FileNotFoundError until the operator's async client-provisioning | |
| # Secret exists -- a separate step now polls for that Secret and force- | |
| # restarts hub once it appears; (2) labelling the release namespace | |
| # `nebari.dev/managed=true` before install, since nic-operator otherwise | |
| # never reconciles the NebariApp at all (condition NamespaceNotOptedIn), | |
| # which is a permanent gate, not a slow-provisioning race. | |
| # | |
| # The link only lives for the run's duration (bounded by timeout-minutes | |
| # below) — it is not a persistent per-PR environment. Each run creates its | |
| # own Cloudflare Tunnel + DNS record (so concurrent previews on different | |
| # PRs don't collide on the same route) and deletes both on cleanup. | |
| # | |
| # One-time setup this workflow assumes already exists (Cloudflare Zero | |
| # Trust dashboard, done by a repo admin, not scripted here). The tunnel, | |
| # the PREVIEW_DOMAIN zone, and Zero Trust/Access all live in ONE | |
| # Cloudflare account (OpenTeams Account) — not the account behind | |
| # CLOUDFLARE_API_TOKEN / CLOUDFLARE_ACCOUNT_ID, which docs.yml uses for | |
| # Pages: | |
| # - Variable PREVIEW_DOMAIN (repo Settings -> Actions -> Variables): | |
| # the zone name, e.g. openteams.app. | |
| # - That zone in the Cloudflare account, with an Access self-hosted | |
| # application for `*.<PREVIEW_DOMAIN>`, GitHub as identity provider, | |
| # policy scoped to this org (nebari-dev). Note this wildcard covers | |
| # ANY single-label subdomain of the zone, not just previews — fine | |
| # as long as the zone isn't also hosting unrelated services outside | |
| # this org's control. | |
| # - Secret CLOUDFLARE_TUNNEL_ACCOUNT_ID: that account's id (the | |
| # `cfd_tunnel` API is account-scoped; can't be derived from the | |
| # token alone). | |
| # - Secret CLOUDFLARE_TUNNEL_API_TOKEN: a custom token scoped to | |
| # EXACTLY three permissions, nothing broader: | |
| # * Account -> Cloudflare Tunnel -> Edit | |
| # * Zone -> Zone -> Read (to resolve the zone id by name) | |
| # * Zone -> DNS -> Edit (to create/delete the CNAME record) | |
| # Zone Resources: Include -> Specific zone -> the PREVIEW_DOMAIN zone. | |
| # Account Resources: Include -> Specific account -> OpenTeams Account. | |
| # | |
| # Only runs when a maintainer/collaborator adds the `deploy-preview` label | |
| # (GitHub restricts who can label a PR) — arbitrary PR authors, including | |
| # from forks, cannot trigger this themselves. Even so, a fork PR still runs | |
| # attacker-authored code once labeled; the comment flags this so whoever | |
| # labels it is doing so knowingly. | |
| # | |
| # Residual risk not covered here: kind nodes share the runner's Docker daemon | |
| # rather than being hardware-isolated, and kind's default CNI (kindnet) | |
| # does not enforce NetworkPolicy, so a container escape or outbound abuse | |
| # from inside a spawned notebook pod is not blocked at the network layer. | |
| # Per-job GITHUB_TOKEN permissions are scoped to the minimum each job | |
| # needs so a compromised runner in the exposed 90-minute window can't use | |
| # an ambient token to touch other workflows or repo state. | |
| on: | |
| pull_request: | |
| types: [labeled, unlabeled, synchronize] | |
| concurrency: | |
| group: k8s-preview-${{ github.event.pull_request.number }} | |
| cancel-in-progress: true | |
| env: | |
| PREVIEW_LABEL: deploy-preview | |
| PREVIEW_DOMAIN: ${{ vars.PREVIEW_DOMAIN }} | |
| CLOUDFLARED_VERSION: "2026.7.3" | |
| # sha256 of cloudflared-linux-amd64 for the pinned version above, | |
| # computed from the official release asset at | |
| # https://github.com/cloudflare/cloudflared/releases/tag/2026.7.3 | |
| CLOUDFLARED_SHA256: "9d71c677db00134c1bd4144b7783486b654ad281b1ea62b4972098d19f770f17" | |
| jobs: | |
| deploy-preview: | |
| if: contains(github.event.pull_request.labels.*.name, 'deploy-preview') && github.event.action != 'unlabeled' | |
| name: Deploy preview | |
| runs-on: ubuntu-24.04 | |
| timeout-minutes: 90 | |
| permissions: | |
| contents: read | |
| pull-requests: write | |
| steps: | |
| - name: Checkout | |
| uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 | |
| # v0.32.0+ required: the sandbox's kind cluster uses containerd's | |
| # config v4 format, which `kind load` on older CLI releases can't | |
| # parse ("ERROR: unknown containerd config version: 4"). | |
| - name: Install kind | |
| uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # v1.14.0 | |
| with: | |
| version: v0.33.0 | |
| install_only: true | |
| - name: Provision sandbox (kind + full NIC platform stack) | |
| id: sandbox | |
| uses: nebari-dev/action-nebari-sandbox@9ac369ebf87ac2ae217504dcbf824c77f70e429a # v3.0.0 | |
| with: | |
| cluster-name: pr-preview-${{ github.event.pull_request.number }} | |
| - name: Set up Docker Buildx | |
| uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0 | |
| # Every run gets a fresh runner (empty Docker daemon), so a plain | |
| # `docker build` re-runs every layer -- including the apt/pixi | |
| # installs -- from scratch every time. GitHub Actions cache (type=gha) | |
| # persists layers across runs so an unchanged pixi.lock/pixi.toml | |
| # reuses the previous run's install instead of redoing it. | |
| - name: Build hub image from this PR | |
| uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0 | |
| with: | |
| context: images/ | |
| target: jupyterhub | |
| tags: nebari-data-science-pack-jupyterhub:preview | |
| load: true | |
| cache-from: type=gha,scope=k8s-preview-jupyterhub | |
| cache-to: type=gha,scope=k8s-preview-jupyterhub,mode=max | |
| - name: Side-load hub image into the sandbox cluster | |
| run: kind load docker-image nebari-data-science-pack-jupyterhub:preview --name ${{ steps.sandbox.outputs.cluster-name }} | |
| # charts/ is gitignored (dependency .tgz files aren't committed), so a | |
| # fresh checkout needs this before `helm upgrade --install` can find | |
| # the jupyterhub subchart. Resolves against the version/digest already | |
| # pinned in the committed Chart.lock, not a new or bumped dependency. | |
| - name: Fetch chart dependencies | |
| run: | | |
| helm repo add jupyterhub https://hub.jupyter.org/helm-chart/ | |
| helm dependency build . | |
| # nic-operator only reconciles NebariApps in namespaces opted into | |
| # Nebari management; without this label it sets condition | |
| # NamespaceNotOptedIn and never provisions the Keycloak client at | |
| # all (confirmed via `kubectl get nebariapp -o yaml` after a deploy | |
| # failure -- not a slow-provisioning race, a permanent gate that | |
| # would never resolve on its own). | |
| - name: Create + label the preview namespace for the operator | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| kubectl create namespace pr-preview --dry-run=client -o yaml | kubectl apply -f - | |
| kubectl label namespace pr-preview nebari.dev/managed=true --overwrite | |
| # Keycloak's own KC_HOSTNAME (bitnami/keycloakx) is fixed to NIC's | |
| # internal keycloak.nebari.local by default -- every self-referencing | |
| # URL Keycloak renders (login form action, issuer, redirects) is | |
| # absolute and uses that value regardless of the incoming Host | |
| # header, confirmed live: a request proxied to Keycloak with the Host | |
| # header rewritten to keycloak.nebari.local still rendered a login | |
| # form whose action="https://keycloak.nebari.local/..." -- unreachable | |
| # from a real browser, since .local is never publicly resolvable. | |
| # | |
| # Point Keycloak's own hostname AND the operator's KEYCLOAK_EXTERNAL_URL | |
| # at the same public, per-PR, single-label hostname the Cloudflare | |
| # Tunnel already routes (no DNS/cert changes, no Worker, no collision | |
| # risk) so Keycloak's self-generated URLs are actually reachable and | |
| # the operator writes a matching issuer into hub's OIDC client secret. | |
| - name: Point Keycloak's own hostname at the public tunnel route | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| echo "--- current keycloak-keycloakx env (KC_HOSTNAME*) ---" | |
| kubectl -n keycloak get statefulset keycloak-keycloakx \ | |
| -o jsonpath='{.spec.template.spec.containers[0].env}' | jq '[.[] | select(.name | test("HOSTNAME"))]' | |
| kc_public_url="https://keycloak-pr-${{ github.event.pull_request.number }}-data-science-pack.${{ env.PREVIEW_DOMAIN }}" | |
| kubectl -n keycloak set env statefulset/keycloak-keycloakx \ | |
| KC_HOSTNAME="$kc_public_url" KC_HOSTNAME_STRICT=true | |
| kubectl -n keycloak rollout status statefulset/keycloak-keycloakx --timeout=180s | |
| operator_deploy=$(kubectl get deploy -A -o json | jq -r '.items[] | select(.metadata.name | test("operator")) | "\(.metadata.namespace)/\(.metadata.name)"' | head -1) | |
| kubectl -n "${operator_deploy%%/*}" set env deployment/"${operator_deploy##*/}" \ | |
| KEYCLOAK_EXTERNAL_URL="$kc_public_url" | |
| kubectl -n "${operator_deploy%%/*}" rollout status deployment/"${operator_deploy##*/}" --timeout=120s | |
| # No --wait here: with nebariapp.auth.enabled=true, the hub pod reads | |
| # the operator-provisioned Keycloak client Secret at import time | |
| # (config/jupyterhub/00-gateway-auth.py) and crashes with | |
| # FileNotFoundError if it starts before the operator has created it -- | |
| # the Secret volume is mounted `optional: true` so the pod itself | |
| # comes up fine, only the hub container's Python process crash-loops. | |
| # That race is inherent (operator reconciliation runs concurrently | |
| # with the chart install, not before it), so we wait for the Secret | |
| # separately below instead of blocking helm on hub's rollout status. | |
| # | |
| # jupyterhub.hub.extraVolumes[1] (values.yaml) hardcodes | |
| # secretName: data-science-pack-nebari-data-science-pack-oidc-client | |
| # -- a literal string baked in for a release named "data-science-pack", | |
| # not templated off .Release.Name. Every restart/retry against the | |
| # real (correctly-populated) operator secret still 404'd on the mount | |
| # because the pod was mounting a DIFFERENT, nonexistent secret name | |
| # the whole time (release name here is "preview") -- confirmed only | |
| # after ruling out timing races and kubelet caching across 5 retries. | |
| # Override it explicitly to match this release's actual name. Using | |
| # --set with list-index syntax (extraVolumes[1].secret.secretName=...) | |
| # against a values.yaml-defined list corrupts the OTHER elements' | |
| # `name` fields too (Helm's --set doesn't deep-merge per-element into | |
| # an existing default list) -- confirmed by a real failure: | |
| # "volumes[2].name: Required value, volumes[3].name: Required value". | |
| # --set-json with the full, correct array sidesteps that. | |
| - name: Deploy chart | |
| id: deploy | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| helm upgrade --install preview . \ | |
| --namespace pr-preview --create-namespace \ | |
| --set jupyterhub.hub.image.name=nebari-data-science-pack-jupyterhub \ | |
| --set jupyterhub.hub.image.tag=preview \ | |
| --set nebariapp.enabled=true \ | |
| --set nebariapp.hostname="pr-${{ github.event.pull_request.number }}-data-science-pack.${{ env.PREVIEW_DOMAIN }}" \ | |
| --set jupyterhub.custom.external-url="pr-${{ github.event.pull_request.number }}-data-science-pack.${{ env.PREVIEW_DOMAIN }}" \ | |
| --set-json 'jupyterhub.hub.extraVolumes=[{"name":"custom-config","configMap":{"name":"nebari-data-science-pack-hub-config"}},{"name":"oauth-client","secret":{"secretName":"preview-nebari-data-science-pack-oidc-client","optional":true}},{"name":"org-ca","configMap":{"name":"nebari-trust-bundle","optional":true}},{"name":"ca-merged","emptyDir":{}}]' | |
| # Secret name convention: {Release.Name}-{Chart.Name}-oidc-client | |
| # (see values.yaml, jupyterhub.hub.extraVolumes comment). Poll by | |
| # label-free name match rather than assuming a fixed provisioning | |
| # time -- the operator's reconcile loop has no SLA. | |
| # Checking the Secret merely EXISTS isn't enough: the operator creates | |
| # it with client-id/client-secret on its first reconcile pass, then | |
| # patches in issuer-url on a later pass (GetExternalIssuerURL). A hub | |
| # restart triggered right after the Secret's first appearance still | |
| # hits FileNotFoundError on /etc/oauth/issuer-url specifically -- wait | |
| # for that key's actual (non-empty) value, not just the object. | |
| - name: Wait for operator to provision the Keycloak client secret | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| for i in $(seq 1 36); do | |
| issuer_b64=$(kubectl -n pr-preview get secret preview-nebari-data-science-pack-oidc-client -o jsonpath='{.data.issuer-url}' 2>/dev/null) | |
| if [ -n "$issuer_b64" ]; then | |
| echo "operator secret's issuer-url populated after $(( i * 5 ))s" | |
| exit 0 | |
| fi | |
| sleep 5 | |
| done | |
| echo "::error::operator never populated issuer-url on the Keycloak client secret within 3m" | |
| kubectl -n pr-preview get nebariapp -o yaml || true | |
| exit 1 | |
| # A single restart isn't reliable here even though the API server | |
| # confirms issuer-url is populated: kubelet's own Secret volume cache | |
| # (node-local, ~1min TTL) can still hand a freshly-restarted pod the | |
| # pre-population snapshot it fetched for the pod's first, crash-looped | |
| # attempt -- confirmed by the same FileNotFoundError recurring on a | |
| # restart issued 5s after the secret was already confirmed complete. | |
| # Retry the restart until a rollout actually succeeds, giving the | |
| # kubelet cache time to expire between attempts, instead of assuming | |
| # one restart is enough. | |
| - name: Restart hub until it picks up the operator secret | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| for attempt in 1 2 3 4 5; do | |
| kubectl -n pr-preview rollout restart deployment/hub | |
| if kubectl -n pr-preview rollout status deployment/hub --timeout=90s; then | |
| echo "hub ready on attempt $attempt" | |
| exit 0 | |
| fi | |
| echo "hub not ready on attempt $attempt, retrying..." | |
| done | |
| echo "::error::hub never became ready after 5 restart attempts" | |
| exit 1 | |
| - name: Wait for proxy | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: kubectl -n pr-preview rollout status deployment/proxy --timeout=180s | |
| - name: Port-forward JupyterHub proxy | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| kubectl -n pr-preview port-forward svc/proxy-public 8000:80 \ | |
| > /tmp/port-forward.log 2>&1 & | |
| echo "PORT_FORWARD_PID=$!" >> "$GITHUB_ENV" | |
| sleep 3 | |
| # Keycloak itself, so a reviewer can sign in there directly and land | |
| # on JupyterHub already authenticated (same Keycloak SSO cookie the | |
| # hub's own OAuth redirect relies on). Keycloak's own hostname was | |
| # repointed at this same public route earlier (see "Point Keycloak's | |
| # own hostname..."), so no Host-header rewrite is needed here -- the | |
| # request just flows straight through with its real Host header. | |
| - name: Port-forward Keycloak | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| kubectl -n keycloak port-forward svc/keycloak-keycloakx-http 8001:8080 \ | |
| > /tmp/port-forward-keycloak.log 2>&1 & | |
| echo "KEYCLOAK_PORT_FORWARD_PID=$!" >> "$GITHUB_ENV" | |
| sleep 3 | |
| # jhub-apps runs as a managed service subprocess inside the hub pod, | |
| # not its own Deployment, so a crash there doesn't fail `helm --wait` | |
| # or the rollout checks above -- it only shows up as a 502 on | |
| # /services/japps/* once someone hits it. Hit its root path directly | |
| # (bypassing Cloudflare, straight to CHP) so a crashed/never-bound | |
| # uvicorn process shows up here instead of only from a live login. | |
| - name: Smoke-test jhub-apps service | |
| if: always() | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| echo "--- GET /services/japps/ ---" | |
| curl -sS -o /tmp/japps-root.html -w 'HTTP %{http_code}\n' http://localhost:8000/services/japps/ || true | |
| head -c 2000 /tmp/japps-root.html || true | |
| echo | |
| echo "--- japps process in hub pod ---" | |
| kubectl -n pr-preview exec deploy/hub -- ps aux | grep -i "uvicorn\|japps" || true | |
| - name: Dump hub logs (jhub-apps startup) | |
| if: always() | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: kubectl -n pr-preview logs deployment/hub --tail=500 | |
| # Cleanup deletes the whole cluster next, so this is the only chance | |
| # to see why a pod/job didn't reach Ready if `helm --wait` timed out. | |
| - name: Debug pod/job status on deploy failure | |
| if: failure() | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| for ns in pr-preview keycloak; do | |
| echo "=== namespace: $ns ===" | |
| kubectl -n "$ns" get pods -o wide || true | |
| kubectl -n "$ns" get jobs || true | |
| kubectl -n "$ns" get secrets || true | |
| kubectl -n "$ns" get events --sort-by=.lastTimestamp || true | |
| for pod in $(kubectl -n "$ns" get pods -o name 2>/dev/null); do | |
| echo "--- describe $pod ($ns) ---" | |
| kubectl -n "$ns" describe "$pod" || true | |
| echo "--- logs $pod ($ns) ---" | |
| kubectl -n "$ns" logs "$pod" --all-containers --tail=100 || true | |
| done | |
| done | |
| kubectl -n pr-preview get nebariapp -o yaml || true | |
| echo "=== nebari-operator deployment env ===" | |
| for d in $(kubectl get deploy -A -o json | jq -r '.items[] | select(.metadata.name | test("operator")) | "\(.metadata.namespace)/\(.metadata.name)"'); do | |
| ns="${d%%/*}"; name="${d##*/}" | |
| echo "--- $d ---" | |
| kubectl -n "$ns" get deploy "$name" -o jsonpath='{.spec.template.spec.containers[0].env}' | jq . || true | |
| done | |
| echo "=== oidc-client secret: keys + issuer-url presence (no values printed) ===" | |
| kubectl -n pr-preview get secret preview-nebari-data-science-pack-oidc-client -o json 2>/dev/null | jq -r '.data | keys' || true | |
| issuer_b64=$(kubectl -n pr-preview get secret preview-nebari-data-science-pack-oidc-client -o jsonpath='{.data.issuer-url}' 2>/dev/null) | |
| echo "issuer-url key present: $([ -n "$issuer_b64" ] && echo yes || echo no); decoded byte length: $(echo -n "$issuer_b64" | base64 -d 2>/dev/null | wc -c)" | |
| echo "=== NIC config domain ===" | |
| find /tmp -maxdepth 1 -iname "nic-config*.yaml" -exec grep -H "^domain:" {} \; || true | |
| # Interactive SSH debug session into the live runner (cluster still | |
| # up, KUBECONFIG still valid) instead of guessing blind from static | |
| # logs. limit-access-to-actor restricts the SSH session to whoever | |
| # triggered this run — required on a public repo. Bounded to 20min | |
| # so a forgotten session doesn't eat the whole 90min job timeout. | |
| - name: Debug via tmate SSH on deploy failure | |
| if: failure() | |
| uses: mxschmitt/action-tmate@35b54afac29c97fb54faba5b513f8fbd1882f113 # v3.24 | |
| timeout-minutes: 20 | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| with: | |
| limit-access-to-actor: true | |
| - name: Install cloudflared | |
| run: | | |
| curl -fsSL -o /tmp/cloudflared \ | |
| "https://github.com/cloudflare/cloudflared/releases/download/${CLOUDFLARED_VERSION}/cloudflared-linux-amd64" | |
| echo "${CLOUDFLARED_SHA256} /tmp/cloudflared" | sha256sum -c - | |
| chmod +x /tmp/cloudflared | |
| # A per-run named Tunnel (not the anonymous quick-tunnel) so: | |
| # (a) it can sit behind an Access application (quick tunnels have no | |
| # account/zone attached, so no policy can be bound to them), and | |
| # (b) each PR gets its own tunnel + hostname, so two PRs previewing | |
| # at once don't share one route and cross-talk. | |
| - name: Create Cloudflare Tunnel for this PR | |
| id: cf_tunnel | |
| env: | |
| CF_API_TOKEN: ${{ secrets.CLOUDFLARE_TUNNEL_API_TOKEN }} | |
| CF_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_TUNNEL_ACCOUNT_ID }} | |
| PREVIEW_HOSTNAME: pr-${{ github.event.pull_request.number }}-data-science-pack.${{ env.PREVIEW_DOMAIN }} | |
| KEYCLOAK_HOSTNAME: keycloak-pr-${{ github.event.pull_request.number }}-data-science-pack.${{ env.PREVIEW_DOMAIN }} | |
| run: | | |
| tunnel_secret=$(openssl rand -base64 32) | |
| echo "::add-mask::${tunnel_secret}" | |
| tunnel_name="pr-${{ github.event.pull_request.number }}-${{ github.run_id }}" | |
| create_resp=$(curl -sS -X POST \ | |
| "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/cfd_tunnel" \ | |
| -H "Authorization: Bearer ${CF_API_TOKEN}" \ | |
| -H "Content-Type: application/json" \ | |
| -d "$(jq -n --arg name "$tunnel_name" --arg secret "$tunnel_secret" \ | |
| '{name: $name, config_src: "cloudflare", tunnel_secret: $secret}')") | |
| tunnel_id=$(jq -r '.result.id // empty' <<< "$create_resp") | |
| # A GitHub Actions retry reuses the same run_id (only run_attempt | |
| # changes), so a re-run after the first attempt already created | |
| # this tunnel (and didn't get to clean it up) hits a 409 name | |
| # conflict here. Reuse the existing tunnel by name instead of | |
| # failing — it doesn't need the original tunnel_secret, just a | |
| # fresh --token from the /token endpoint below. | |
| if [ -z "$tunnel_id" ]; then | |
| echo "::warning::Tunnel create failed (likely a name conflict from a retry), looking up existing tunnel named ${tunnel_name}: $create_resp" | |
| tunnel_id=$(curl -fsS "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/cfd_tunnel?name=${tunnel_name}&is_deleted=false" \ | |
| -H "Authorization: Bearer ${CF_API_TOKEN}" | jq -r '.result[0].id // empty') | |
| fi | |
| if [ -z "$tunnel_id" ]; then | |
| echo "::error::Tunnel creation failed and no existing tunnel named ${tunnel_name} found: $create_resp" | |
| exit 1 | |
| fi | |
| echo "tunnel_id=${tunnel_id}" >> "$GITHUB_OUTPUT" | |
| echo "TUNNEL_ID=${tunnel_id}" >> "$GITHUB_ENV" | |
| token_resp=$(curl -fsS \ | |
| "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/cfd_tunnel/${tunnel_id}/token" \ | |
| -H "Authorization: Bearer ${CF_API_TOKEN}") | |
| tunnel_token=$(jq -r '.result' <<< "$token_resp") | |
| echo "::add-mask::${tunnel_token}" | |
| echo "TUNNEL_TOKEN=${tunnel_token}" >> "$GITHUB_ENV" | |
| curl -fsS -X PUT \ | |
| "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/cfd_tunnel/${tunnel_id}/configurations" \ | |
| -H "Authorization: Bearer ${CF_API_TOKEN}" \ | |
| -H "Content-Type: application/json" \ | |
| -d "$(jq -n --arg host "$PREVIEW_HOSTNAME" --arg kchost "$KEYCLOAK_HOSTNAME" \ | |
| '{config: {ingress: [ | |
| {hostname: $host, service: "http://localhost:8000"}, | |
| {hostname: $kchost, service: "http://localhost:8001"}, | |
| {service: "http_status:404"} | |
| ]}}')" \ | |
| > /dev/null | |
| - name: Point DNS at the tunnel | |
| id: cf_dns | |
| env: | |
| CF_API_TOKEN: ${{ secrets.CLOUDFLARE_TUNNEL_API_TOKEN }} | |
| PREVIEW_HOSTNAME: pr-${{ github.event.pull_request.number }}-data-science-pack.${{ env.PREVIEW_DOMAIN }} | |
| KEYCLOAK_HOSTNAME: keycloak-pr-${{ github.event.pull_request.number }}-data-science-pack.${{ env.PREVIEW_DOMAIN }} | |
| run: | | |
| zone_id=$(curl -fsS "https://api.cloudflare.com/client/v4/zones?name=${PREVIEW_DOMAIN}" \ | |
| -H "Authorization: Bearer ${CF_API_TOKEN}" | jq -r '.result[0].id') | |
| if [ -z "$zone_id" ] || [ "$zone_id" = "null" ]; then | |
| echo "::error::Could not resolve zone id for ${PREVIEW_DOMAIN}" | |
| exit 1 | |
| fi | |
| echo "ZONE_ID=${zone_id}" >> "$GITHUB_ENV" | |
| record_resp=$(curl -fsS -X POST "https://api.cloudflare.com/client/v4/zones/${zone_id}/dns_records" \ | |
| -H "Authorization: Bearer ${CF_API_TOKEN}" \ | |
| -H "Content-Type: application/json" \ | |
| -d "$(jq -n --arg host "$PREVIEW_HOSTNAME" --arg target "${TUNNEL_ID}.cfargotunnel.com" \ | |
| '{type: "CNAME", name: $host, content: $target, proxied: true}')") | |
| record_id=$(jq -r '.result.id' <<< "$record_resp") | |
| if [ -z "$record_id" ] || [ "$record_id" = "null" ]; then | |
| echo "::error::DNS record creation failed: $record_resp" | |
| exit 1 | |
| fi | |
| echo "DNS_RECORD_ID=${record_id}" >> "$GITHUB_ENV" | |
| echo "url=https://${PREVIEW_HOSTNAME}" >> "$GITHUB_OUTPUT" | |
| kc_record_resp=$(curl -fsS -X POST "https://api.cloudflare.com/client/v4/zones/${zone_id}/dns_records" \ | |
| -H "Authorization: Bearer ${CF_API_TOKEN}" \ | |
| -H "Content-Type: application/json" \ | |
| -d "$(jq -n --arg host "$KEYCLOAK_HOSTNAME" --arg target "${TUNNEL_ID}.cfargotunnel.com" \ | |
| '{type: "CNAME", name: $host, content: $target, proxied: true}')") | |
| kc_record_id=$(jq -r '.result.id' <<< "$kc_record_resp") | |
| if [ -z "$kc_record_id" ] || [ "$kc_record_id" = "null" ]; then | |
| echo "::error::Keycloak DNS record creation failed: $kc_record_resp" | |
| exit 1 | |
| fi | |
| echo "KEYCLOAK_DNS_RECORD_ID=${kc_record_id}" >> "$GITHUB_ENV" | |
| echo "keycloak_url=https://${KEYCLOAK_HOSTNAME}" >> "$GITHUB_OUTPUT" | |
| # The URL itself (pr-<n>-data-science-pack.<domain>) is identical on every run, so | |
| # without a timestamp the sticky comment would post byte-identical | |
| # text each redeploy and look like it never updated. | |
| - name: Compute deployment timestamps | |
| id: timestamps | |
| run: | | |
| echo "deployed_at=$(date -u +'%Y-%m-%d %H:%M UTC')" >> "$GITHUB_OUTPUT" | |
| echo "expires_at=$(date -u -d '+90 minutes' +'%Y-%m-%d %H:%M UTC')" >> "$GITHUB_OUTPUT" | |
| - name: Comment preview link on PR | |
| uses: marocchino/sticky-pull-request-comment@5770ad5eb8f42dd2c4f34da00c94c5381e49af88 # v3.0.5 | |
| with: | |
| header: k8s-preview | |
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | |
| message: | | |
| **K8s stack preview** for `${{ github.event.pull_request.head.ref }}`: | |
| ${{ steps.cf_dns.outputs.url }} | |
| Deployed: ${{ steps.timestamps.outputs.deployed_at }} · Expires: ${{ steps.timestamps.outputs.expires_at }} | |
| ${{ github.event.pull_request.head.repo.fork && '⚠️ **This PR is from a fork** — the code running in this preview is not from a trusted maintainer branch.' || '' }} | |
| You'll be asked to sign in via Cloudflare Access (GitHub SSO) first — | |
| only accounts on the Access application's allow-list get through. | |
| **Then JupyterHub login:** goes through Keycloak (the operator-provisioned | |
| OIDC client for this preview's NebariApp). | |
| Keycloak's own login page is also reachable directly: | |
| ${{ steps.cf_dns.outputs.keycloak_url }} | |
| Signing in there first establishes the SSO session, so visiting | |
| JupyterHub afterward skips straight past the login prompt. | |
| Live until the expiry time above, or until the `deploy-preview` label | |
| is removed. Push a new commit or re-add the label to redeploy. | |
| - name: Run tunnel until the job times out | |
| run: /tmp/cloudflared tunnel --no-autoupdate run --token "${TUNNEL_TOKEN}" | |
| # Runs once the tunnel step above ends (timeout or a manual cancel), | |
| # so this captures anything logged in response to real traffic during | |
| # the tunnel's lifetime -- unlike the earlier startup-time log dump. | |
| - name: Dump hub logs after tunnel closes | |
| if: always() | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: kubectl -n pr-preview logs deployment/hub --tail=1000 || true | |
| - name: Delete DNS record | |
| if: always() | |
| env: | |
| CF_API_TOKEN: ${{ secrets.CLOUDFLARE_TUNNEL_API_TOKEN }} | |
| run: | | |
| [ -n "${ZONE_ID:-}" ] && [ -n "${DNS_RECORD_ID:-}" ] || exit 0 | |
| curl -fsS -X DELETE \ | |
| "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records/${DNS_RECORD_ID}" \ | |
| -H "Authorization: Bearer ${CF_API_TOKEN}" || true | |
| [ -n "${KEYCLOAK_DNS_RECORD_ID:-}" ] || exit 0 | |
| curl -fsS -X DELETE \ | |
| "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records/${KEYCLOAK_DNS_RECORD_ID}" \ | |
| -H "Authorization: Bearer ${CF_API_TOKEN}" || true | |
| - name: Delete Cloudflare Tunnel | |
| if: always() | |
| env: | |
| CF_API_TOKEN: ${{ secrets.CLOUDFLARE_TUNNEL_API_TOKEN }} | |
| CF_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_TUNNEL_ACCOUNT_ID }} | |
| run: | | |
| [ -n "${TUNNEL_ID:-}" ] || exit 0 | |
| curl -fsS -X DELETE \ | |
| "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/cfd_tunnel/${TUNNEL_ID}" \ | |
| -H "Authorization: Bearer ${CF_API_TOKEN}" || true | |
| cleanup-preview: | |
| if: github.event.action == 'unlabeled' && github.event.label.name == 'deploy-preview' | |
| name: Stop preview | |
| runs-on: ubuntu-latest | |
| permissions: | |
| pull-requests: write | |
| actions: write | |
| steps: | |
| - name: Cancel the in-flight preview run for this PR | |
| env: | |
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | |
| run: | | |
| run_id=$(gh api "repos/${{ github.repository }}/actions/runs?event=pull_request&status=in_progress" \ | |
| --jq '.workflow_runs[] | select(.name == "K8s Stack Preview") | select(.pull_requests[]?.number == ${{ github.event.pull_request.number }}) | .id' \ | |
| | head -1) | |
| if [ -n "$run_id" ]; then | |
| gh run cancel "$run_id" --repo "${{ github.repository }}" | |
| fi | |
| - name: Comment that the preview stopped | |
| uses: marocchino/sticky-pull-request-comment@5770ad5eb8f42dd2c4f34da00c94c5381e49af88 # v3.0.5 | |
| with: | |
| header: k8s-preview | |
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | |
| message: | | |
| **K8s stack preview** stopped — the `deploy-preview` label was removed. | |
| Add it again to redeploy. |