ci: add k8s stack PR preview via labeled deploy + Cloudflare Tunnel #63
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: K8s Stack Preview | |
| # Deploys the full Nebari platform stack (Keycloak + nic-operator + Envoy | |
| # Gateway, via nebari-dev/action-nebari-sandbox) plus this PR's chart into | |
| # an ephemeral kind cluster on the runner, then exposes | |
| # JupyterHub through a per-PR Cloudflare Tunnel behind Cloudflare Access | |
| # (GitHub SSO) so a reviewer can click a link and use it. | |
| # | |
| # This repo is public, so a plain shared secret posted in the PR comment | |
| # (a quick-tunnel URL, a basic-auth password) is readable by anyone who | |
| # opens the PR, not just intended reviewers. Access closes that gap by | |
| # authenticating the *person*, not a string in the comment: Cloudflare | |
| # challenges every request to *.<vars.PREVIEW_DOMAIN> with a GitHub SSO | |
| # login and only lets it through to cloudflared if the signed-in account | |
| # matches the Access application's policy (currently an explicit email | |
| # allow-list, not org membership -- see the Cloudflare Access application | |
| # for this domain for who's currently allowed). The PR comment only ever | |
| # contains a URL. | |
| # | |
| # The domain is a repo Variable (Settings -> Secrets and variables -> | |
| # Actions -> Variables -> PREVIEW_DOMAIN, currently openteams.app), not | |
| # hardcoded, so it can be repointed without editing this file. Hostnames | |
| # built from it stay single-level (pr-<n>-data-science-pack.<domain>, | |
| # never pr-<n>.data-science-pack.<domain>) deliberately: Cloudflare's | |
| # free Universal SSL only auto-covers the zone apex plus one wildcard | |
| # level (<domain> + *.<domain>); a second level needs the paid Advanced | |
| # Certificate Manager add-on, which this setup doesn't use. | |
| # | |
| # Scope: the tunnel points straight at the `proxy-public` service. The | |
| # chart deploys with nebariapp.enabled=true (nebariapp.auth.enabled=true | |
| # by chart default), so login goes through the real operator-provisioned | |
| # Keycloak OIDC client, exercising the same auth path production deploys | |
| # use. Getting this working required: (1) not blocking `helm upgrade | |
| # --install` with `--wait`, since the hub pod crash-loops on | |
| # FileNotFoundError until the operator's async client-provisioning | |
| # Secret exists -- a separate step now polls for that Secret and force- | |
| # restarts hub once it appears; (2) labelling the release namespace | |
| # `nebari.dev/managed=true` before install, since nic-operator otherwise | |
| # never reconciles the NebariApp at all (condition NamespaceNotOptedIn), | |
| # which is a permanent gate, not a slow-provisioning race. | |
| # | |
| # The link only lives for 20 minutes (the tunnel step's own timeout); it | |
| # is not a persistent per-PR environment. Each run creates its own | |
| # Cloudflare Tunnel + DNS record (so concurrent previews on different PRs | |
| # don't collide on the same route) and deletes both on cleanup. | |
| # | |
| # One-time setup this workflow assumes already exists (Cloudflare Zero | |
| # Trust dashboard, done by a repo admin, not scripted here). The tunnel, | |
| # the PREVIEW_DOMAIN zone, and Zero Trust/Access all live in ONE | |
| # Cloudflare account (OpenTeams Account), not the account behind | |
| # CLOUDFLARE_API_TOKEN / CLOUDFLARE_ACCOUNT_ID, which docs.yml uses for | |
| # Pages: | |
| # - Variable PREVIEW_DOMAIN (repo Settings -> Actions -> Variables): | |
| # the zone name, e.g. openteams.app. | |
| # - That zone in the Cloudflare account, with an Access self-hosted | |
| # application for `*.<PREVIEW_DOMAIN>`, GitHub as identity provider, | |
| # policy scoped to this org (nebari-dev). Note this wildcard covers | |
| # ANY single-label subdomain of the zone, not just previews, fine | |
| # as long as the zone isn't also hosting unrelated services outside | |
| # this org's control. | |
| # - Secret CLOUDFLARE_TUNNEL_ACCOUNT_ID: that account's id (the | |
| # `cfd_tunnel` API is account-scoped; can't be derived from the | |
| # token alone). | |
| # - Secret CLOUDFLARE_TUNNEL_API_TOKEN: a custom token scoped to | |
| # EXACTLY three permissions, nothing broader: | |
| # * Account -> Cloudflare Tunnel -> Edit | |
| # * Zone -> Zone -> Read (to resolve the zone id by name) | |
| # * Zone -> DNS -> Edit (to create/delete the CNAME record) | |
| # Zone Resources: Include -> Specific zone -> the PREVIEW_DOMAIN zone. | |
| # Account Resources: Include -> Specific account -> OpenTeams Account. | |
| # | |
| # Only runs when a maintainer/collaborator adds the `deploy-preview` label | |
| # (GitHub restricts who can label a PR): arbitrary PR authors, including | |
| # from forks, cannot trigger this themselves. Even so, a fork PR still runs | |
| # attacker-authored code once labeled; the comment flags this so whoever | |
| # labels it is doing so knowingly. | |
| # | |
| # Residual risk not covered here: kind nodes share the runner's Docker daemon | |
| # rather than being hardware-isolated, and kind's default CNI (kindnet) | |
| # does not enforce NetworkPolicy, so a container escape or outbound abuse | |
| # from inside a spawned notebook pod is not blocked at the network layer. | |
| # Per-job GITHUB_TOKEN permissions are scoped to the minimum each job | |
| # needs so a compromised runner in the exposed 20-minute window can't use | |
| # an ambient token to touch other workflows or repo state. | |
| on: | |
| pull_request: | |
| types: [labeled, unlabeled, synchronize] | |
| # cancel-in-progress is normally what we want (one live preview per PR, | |
| # newest commit wins), but it operates on the whole workflow RUN, not on | |
| # whether deploy-preview's job condition ends up true. Adding the | |
| # extend-preview label (see the tunnel step) fires a `labeled` event just | |
| # like any other label add, which would otherwise queue a new run here and | |
| # cancel the very run extend-preview was meant to extend -- even though | |
| # that new run's job condition skips it and does nothing. Excluding that | |
| # one case lets the new (skipped) run queue harmlessly behind the current | |
| # one instead of cancelling it. | |
| concurrency: | |
| group: k8s-preview-${{ github.event.pull_request.number }} | |
| cancel-in-progress: ${{ !(github.event.action == 'labeled' && github.event.label.name == 'extend-preview') }} | |
| env: | |
| PREVIEW_LABEL: deploy-preview | |
| PREVIEW_DOMAIN: ${{ vars.PREVIEW_DOMAIN }} | |
| CLOUDFLARED_VERSION: "2026.7.3" | |
| # sha256 of cloudflared-linux-amd64 for the pinned version above, | |
| # computed from the official release asset at | |
| # https://github.com/cloudflare/cloudflared/releases/tag/2026.7.3 | |
| CLOUDFLARED_SHA256: "9d71c677db00134c1bd4144b7783486b654ad281b1ea62b4972098d19f770f17" | |
| jobs: | |
| # A `labeled` event fires the whole workflow for ANY label add, not just | |
| # deploy-preview -- e.g. someone adding extend-preview (see the tunnel | |
| # step below) would otherwise start a second run of this job, which | |
| # `concurrency: cancel-in-progress` then uses to cancel the very run | |
| # extend-preview was meant to extend. Scope labeled/unlabeled triggers to | |
| # the deploy-preview label itself; extend-preview is read by the | |
| # already-running tunnel step polling the API, not by starting a new run. | |
| deploy-preview: | |
| if: >- | |
| contains(github.event.pull_request.labels.*.name, 'deploy-preview') && | |
| github.event.action != 'unlabeled' && | |
| (github.event.action != 'labeled' || github.event.label.name == 'deploy-preview') | |
| name: Deploy preview | |
| runs-on: ubuntu-24.04 | |
| timeout-minutes: 90 | |
| permissions: | |
| contents: read | |
| pull-requests: write | |
| issues: write | |
| deployments: write | |
| steps: | |
| - name: Checkout | |
| uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 | |
| # So it shows up as an existing, pickable label in the PR's label | |
| # picker instead of requiring someone to freehand-type a brand-new | |
| # name (which GitHub does support, but it's an easy thing to miss). | |
| # `gh label create` errors if the label already exists; that's fine, | |
| # this step is just making sure it's there. | |
| - name: Ensure the extend-preview label exists | |
| run: | | |
| python -m scripts.preview.github_api ensure-label-exists \ | |
| --repo "${{ github.repository }}" --token "${{ secrets.GITHUB_TOKEN }}" \ | |
| --name extend-preview --color BFD4F2 \ | |
| --description "Push this preview's expiry back 20 minutes" | |
| # v0.32.0+ required: the sandbox's kind cluster uses containerd's | |
| # config v4 format, which `kind load` on older CLI releases can't | |
| # parse ("ERROR: unknown containerd config version: 4"). | |
| - name: Install kind | |
| uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # v1.14.0 | |
| with: | |
| version: v0.33.0 | |
| install_only: true | |
| - name: Provision sandbox (kind + full NIC platform stack) | |
| id: sandbox | |
| uses: nebari-dev/action-nebari-sandbox@9ac369ebf87ac2ae217504dcbf824c77f70e429a # v3.0.0 | |
| with: | |
| cluster-name: pr-preview-${{ github.event.pull_request.number }} | |
| - name: Set up Docker Buildx | |
| uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0 | |
| # Every run gets a fresh runner (empty Docker daemon), so a plain | |
| # `docker build` re-runs every layer -- including the apt/pixi | |
| # installs -- from scratch every time. GitHub Actions cache (type=gha) | |
| # persists layers across runs so an unchanged pixi.lock/pixi.toml | |
| # reuses the previous run's install instead of redoing it. | |
| - name: Build hub image from this PR | |
| uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0 | |
| with: | |
| context: images/ | |
| target: jupyterhub | |
| tags: nebari-data-science-pack-jupyterhub:preview | |
| load: true | |
| cache-from: type=gha,scope=k8s-preview-jupyterhub | |
| cache-to: type=gha,scope=k8s-preview-jupyterhub,mode=max | |
| - name: Side-load hub image into the sandbox cluster | |
| run: kind load docker-image nebari-data-science-pack-jupyterhub:preview --name ${{ steps.sandbox.outputs.cluster-name }} | |
| # charts/ is gitignored (dependency .tgz files aren't committed), so a | |
| # fresh checkout needs this before `helm upgrade --install` can find | |
| # the jupyterhub subchart. Resolves against the version/digest already | |
| # pinned in the committed Chart.lock, not a new or bumped dependency. | |
| - name: Fetch chart dependencies | |
| run: | | |
| helm repo add jupyterhub https://hub.jupyter.org/helm-chart/ | |
| helm dependency build . | |
| # nic-operator only reconciles NebariApps in namespaces opted into | |
| # Nebari management; without this label it sets condition | |
| # NamespaceNotOptedIn and never provisions the Keycloak client at | |
| # all (confirmed via `kubectl get nebariapp -o yaml` after a deploy | |
| # failure -- not a slow-provisioning race, a permanent gate that | |
| # would never resolve on its own). | |
| - name: Create + label the preview namespace for the operator | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| kubectl create namespace pr-preview --dry-run=client -o yaml | kubectl apply -f - | |
| kubectl label namespace pr-preview nebari.dev/managed=true --overwrite | |
| # Why this is needed (ArgoCD selfHeal reverting a direct kubectl | |
| # patch, Keycloak's KC_HOSTNAME being otherwise unreachable from a | |
| # real browser) is explained in scripts/preview/keycloak_gitops.py's | |
| # module docstring, next to the logic itself. | |
| - name: Point Keycloak's own hostname at the public tunnel route | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| python -m scripts.preview.keycloak_gitops patch \ | |
| --gitops-dir "$HOME/.nic/gitops/${{ steps.sandbox.outputs.cluster-name }}" \ | |
| --kc-public-url "https://keycloak-pr-${{ github.event.pull_request.number }}-data-science-pack.${{ env.PREVIEW_DOMAIN }}" | |
| # No --wait here: with nebariapp.auth.enabled=true, the hub pod reads | |
| # the operator-provisioned Keycloak client Secret at import time | |
| # (config/jupyterhub/00-gateway-auth.py) and crashes with | |
| # FileNotFoundError if it starts before the operator has created it -- | |
| # the Secret volume is mounted `optional: true` so the pod itself | |
| # comes up fine, only the hub container's Python process crash-loops. | |
| # That race is inherent (operator reconciliation runs concurrently | |
| # with the chart install, not before it), so we wait for the Secret | |
| # separately below instead of blocking helm on hub's rollout status. | |
| # | |
| # jupyterhub.hub.extraVolumes[1] (values.yaml) hardcodes | |
| # secretName: data-science-pack-nebari-data-science-pack-oidc-client | |
| # -- a literal string baked in for a release named "data-science-pack", | |
| # not templated off .Release.Name. Every restart/retry against the | |
| # real (correctly-populated) operator secret still 404'd on the mount | |
| # because the pod was mounting a DIFFERENT, nonexistent secret name | |
| # the whole time (release name here is "preview") -- confirmed only | |
| # after ruling out timing races and kubelet caching across 5 retries. | |
| # Override it explicitly to match this release's actual name. Using | |
| # --set with list-index syntax (extraVolumes[1].secret.secretName=...) | |
| # against a values.yaml-defined list corrupts the OTHER elements' | |
| # `name` fields too (Helm's --set doesn't deep-merge per-element into | |
| # an existing default list) -- confirmed by a real failure: | |
| # "volumes[2].name: Required value, volumes[3].name: Required value". | |
| # --set-json with the full, correct array sidesteps that. | |
| # | |
| # keycloak.backchannelURL: Keycloak's public hostname sits behind | |
| # Cloudflare Access (same as the JupyterHub hostname), which is fine | |
| # for the browser-facing authorize/login legs -- the reviewer already | |
| # has an Access session -- but hub's OWN server-to-server token | |
| # exchange has no such session and gets blocked/redirected by | |
| # Access instead of getting a JSON response back, confirmed live: | |
| # json.decoder.JSONDecodeError: Expecting value: line 1 column 1 | |
| # (an empty/non-JSON body from the token endpoint). Point the | |
| # backchannel (token_url, userdata_url only -- authorize_url stays | |
| # on the public issuer) straight at Keycloak's in-cluster Service, | |
| # bypassing Cloudflare/Access entirely for that leg. | |
| - name: Deploy chart | |
| id: deploy | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| helm upgrade --install preview . \ | |
| --namespace pr-preview --create-namespace \ | |
| --set jupyterhub.hub.image.name=nebari-data-science-pack-jupyterhub \ | |
| --set jupyterhub.hub.image.tag=preview \ | |
| --set nebariapp.enabled=true \ | |
| --set nebariapp.hostname="pr-${{ github.event.pull_request.number }}-data-science-pack.${{ env.PREVIEW_DOMAIN }}" \ | |
| --set jupyterhub.custom.external-url="pr-${{ github.event.pull_request.number }}-data-science-pack.${{ env.PREVIEW_DOMAIN }}" \ | |
| --set keycloak.backchannelURL="http://keycloak-keycloakx-http.keycloak.svc.cluster.local:8080" \ | |
| --set-json 'jupyterhub.hub.extraVolumes=[{"name":"custom-config","configMap":{"name":"nebari-data-science-pack-hub-config"}},{"name":"oauth-client","secret":{"secretName":"preview-nebari-data-science-pack-oidc-client","optional":true}},{"name":"org-ca","configMap":{"name":"nebari-trust-bundle","optional":true}},{"name":"ca-merged","emptyDir":{}}]' | |
| # Secret name convention: {Release.Name}-{Chart.Name}-oidc-client | |
| # (see values.yaml, jupyterhub.hub.extraVolumes comment). Poll by | |
| # label-free name match rather than assuming a fixed provisioning | |
| # time -- the operator's reconcile loop has no SLA. | |
| # Checking the Secret merely EXISTS isn't enough: the operator creates | |
| # it with client-id/client-secret on its first reconcile pass, then | |
| # patches in issuer-url on a later pass (GetExternalIssuerURL). A hub | |
| # restart triggered right after the Secret's first appearance still | |
| # hits FileNotFoundError on /etc/oauth/issuer-url specifically -- wait | |
| # for that key's actual (non-empty) value, not just the object. | |
| # Why a mere Secret-exists check or a single restart isn't reliable | |
| # here (the operator populates issuer-url on a later reconcile pass; | |
| # kubelet's Secret volume cache can hand a fresh pod a stale | |
| # snapshot) is explained in scripts/preview/k8s_wait.py. | |
| - name: Wait for operator to provision the Keycloak client secret | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| python -m scripts.preview.k8s_wait wait-for-secret-key \ | |
| --namespace pr-preview --secret preview-nebari-data-science-pack-oidc-client \ | |
| --key issuer-url --timeout-s 180 --poll-interval-s 5 | |
| - name: Restart hub until it picks up the operator secret | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| python -m scripts.preview.k8s_wait restart-until-ready \ | |
| --namespace pr-preview --deployment hub --rollout-timeout-s 90 --max-attempts 5 | |
| - name: Wait for proxy | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: kubectl -n pr-preview rollout status deployment/proxy --timeout=180s | |
| - name: Port-forward JupyterHub proxy | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| kubectl -n pr-preview port-forward svc/proxy-public 8000:80 \ | |
| > /tmp/port-forward.log 2>&1 & | |
| echo "PORT_FORWARD_PID=$!" >> "$GITHUB_ENV" | |
| sleep 3 | |
| # Keycloak itself, so a reviewer can sign in there directly and land | |
| # on JupyterHub already authenticated (same Keycloak SSO cookie the | |
| # hub's own OAuth redirect relies on). Keycloak's own hostname was | |
| # repointed at this same public route earlier (see "Point Keycloak's | |
| # own hostname..."), so no Host-header rewrite is needed here -- the | |
| # request just flows straight through with its real Host header. | |
| - name: Port-forward Keycloak | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| kubectl -n keycloak port-forward svc/keycloak-keycloakx-http 8001:8080 \ | |
| > /tmp/port-forward-keycloak.log 2>&1 & | |
| echo "KEYCLOAK_PORT_FORWARD_PID=$!" >> "$GITHUB_ENV" | |
| sleep 3 | |
| # Cloudflare Access is the real security boundary for this preview | |
| # (only allow-listed accounts reach the tunnel at all) -- a simple, | |
| # known password for the Keycloak-side login is fine here, so | |
| # reviewers don't need to hunt for real credentials on a throwaway | |
| # cluster. The admin password is piped straight from the Secret into | |
| # the token request and never echoed or logged. | |
| - name: Create a test login user in Keycloak | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| admin_password=$(kubectl -n keycloak get secret keycloak-admin-credentials -o jsonpath='{.data.admin-password}' | base64 -d) | |
| python -m scripts.preview.keycloak create-reviewer-user \ | |
| --base-url "http://localhost:8001" --realm nebari --admin-password "$admin_password" | |
| # jhub-apps runs as a managed service subprocess inside the hub pod, | |
| # not its own Deployment, so a crash there doesn't fail `helm --wait` | |
| # or the rollout checks above -- it only shows up as a 502 on | |
| # /services/japps/* once someone hits it. Hit its root path directly | |
| # (bypassing Cloudflare, straight to CHP) so a crashed/never-bound | |
| # uvicorn process shows up here instead of only from a live login. | |
| - name: Smoke-test jhub-apps service | |
| if: always() | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| echo "--- GET /services/japps/ ---" | |
| curl -sS -o /tmp/japps-root.html -w 'HTTP %{http_code}\n' http://localhost:8000/services/japps/ || true | |
| head -c 2000 /tmp/japps-root.html || true | |
| echo | |
| echo "--- japps process in hub pod ---" | |
| kubectl -n pr-preview exec deploy/hub -- ps aux | grep -i "uvicorn\|japps" || true | |
| - name: Dump hub logs (jhub-apps startup) | |
| if: always() | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: kubectl -n pr-preview logs deployment/hub --tail=500 | |
| # Cleanup deletes the whole cluster next, so this is the only chance | |
| # to see why a pod/job didn't reach Ready if `helm --wait` timed out. | |
| - name: Debug pod/job status on deploy failure | |
| if: failure() | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: | | |
| for ns in pr-preview keycloak; do | |
| echo "=== namespace: $ns ===" | |
| kubectl -n "$ns" get pods -o wide || true | |
| kubectl -n "$ns" get jobs || true | |
| kubectl -n "$ns" get secrets || true | |
| kubectl -n "$ns" get events --sort-by=.lastTimestamp || true | |
| for pod in $(kubectl -n "$ns" get pods -o name 2>/dev/null); do | |
| echo "--- describe $pod ($ns) ---" | |
| kubectl -n "$ns" describe "$pod" || true | |
| echo "--- logs $pod ($ns) ---" | |
| kubectl -n "$ns" logs "$pod" --all-containers --tail=100 || true | |
| done | |
| done | |
| kubectl -n pr-preview get nebariapp -o yaml || true | |
| echo "=== nebari-operator deployment env ===" | |
| for d in $(kubectl get deploy -A -o json | jq -r '.items[] | select(.metadata.name | test("operator")) | "\(.metadata.namespace)/\(.metadata.name)"'); do | |
| ns="${d%%/*}"; name="${d##*/}" | |
| echo "--- $d ---" | |
| kubectl -n "$ns" get deploy "$name" -o jsonpath='{.spec.template.spec.containers[0].env}' | jq . || true | |
| done | |
| echo "=== oidc-client secret: keys + issuer-url presence (no values printed) ===" | |
| kubectl -n pr-preview get secret preview-nebari-data-science-pack-oidc-client -o json 2>/dev/null | jq -r '.data | keys' || true | |
| issuer_b64=$(kubectl -n pr-preview get secret preview-nebari-data-science-pack-oidc-client -o jsonpath='{.data.issuer-url}' 2>/dev/null) | |
| echo "issuer-url key present: $([ -n "$issuer_b64" ] && echo yes || echo no); decoded byte length: $(echo -n "$issuer_b64" | base64 -d 2>/dev/null | wc -c)" | |
| echo "=== NIC config domain ===" | |
| find /tmp -maxdepth 1 -iname "nic-config*.yaml" -exec grep -H "^domain:" {} \; || true | |
| # Interactive SSH debug session into the live runner (cluster still | |
| # up, KUBECONFIG still valid) instead of guessing blind from static | |
| # logs. limit-access-to-actor restricts the SSH session to whoever | |
| # triggered this run, required on a public repo. Bounded to 20min | |
| # so a forgotten session doesn't eat the whole 90min job timeout. | |
| - name: Debug via tmate SSH on deploy failure | |
| if: failure() | |
| uses: mxschmitt/action-tmate@35b54afac29c97fb54faba5b513f8fbd1882f113 # v3.24 | |
| timeout-minutes: 20 | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| with: | |
| limit-access-to-actor: true | |
| - name: Install cloudflared | |
| run: | | |
| curl -fsSL -o /tmp/cloudflared \ | |
| "https://github.com/cloudflare/cloudflared/releases/download/${CLOUDFLARED_VERSION}/cloudflared-linux-amd64" | |
| echo "${CLOUDFLARED_SHA256} /tmp/cloudflared" | sha256sum -c - | |
| chmod +x /tmp/cloudflared | |
| # A per-run named Tunnel (not the anonymous quick-tunnel) so: | |
| # (a) it can sit behind an Access application (quick tunnels have no | |
| # account/zone attached, so no policy can be bound to them), and | |
| # (b) each PR gets its own tunnel + hostname, so two PRs previewing | |
| # at once don't share one route and cross-talk. | |
| - name: Create Cloudflare Tunnel for this PR | |
| id: cf_tunnel | |
| env: | |
| CF_API_TOKEN: ${{ secrets.CLOUDFLARE_TUNNEL_API_TOKEN }} | |
| CF_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_TUNNEL_ACCOUNT_ID }} | |
| run: | | |
| python -m scripts.preview.cloudflare create-tunnel \ | |
| --account-id "$CF_ACCOUNT_ID" --api-token "$CF_API_TOKEN" \ | |
| --name "pr-${{ github.event.pull_request.number }}-${{ github.run_id }}" \ | |
| --preview-hostname "pr-${{ github.event.pull_request.number }}-data-science-pack.${{ env.PREVIEW_DOMAIN }}" \ | |
| --preview-service "http://localhost:8000" \ | |
| --keycloak-hostname "keycloak-pr-${{ github.event.pull_request.number }}-data-science-pack.${{ env.PREVIEW_DOMAIN }}" \ | |
| --keycloak-service "http://localhost:8001" | |
| - name: Point DNS at the tunnel | |
| id: cf_dns | |
| env: | |
| CF_API_TOKEN: ${{ secrets.CLOUDFLARE_TUNNEL_API_TOKEN }} | |
| run: | | |
| python -m scripts.preview.cloudflare create-dns \ | |
| --api-token "$CF_API_TOKEN" --domain "${{ env.PREVIEW_DOMAIN }}" \ | |
| --preview-hostname "pr-${{ github.event.pull_request.number }}-data-science-pack.${{ env.PREVIEW_DOMAIN }}" \ | |
| --keycloak-hostname "keycloak-pr-${{ github.event.pull_request.number }}-data-science-pack.${{ env.PREVIEW_DOMAIN }}" \ | |
| --target "${TUNNEL_ID}.cfargotunnel.com" | |
| # The URL itself (pr-<n>-data-science-pack.<domain>) is identical on every run, so | |
| # without a timestamp the sticky comment would post byte-identical | |
| # text each redeploy and look like it never updated. | |
| # ISO timestamps feed <relative-time> in the comment below -- GitHub's | |
| # own web component (used all over its UI for "3 minutes ago") that | |
| # ticks live client-side once rendered, confirmed via `gh api /markdown` | |
| # to survive comment sanitization unstripped. Beats a static UTC | |
| # string the reader has to convert to their own timezone by hand, and | |
| # unlike re-editing the comment every minute, needs no extra CI runs. | |
| # The plain UTC strings are the fallback text shown before JS | |
| # hydrates <relative-time>. | |
| - name: Compute deployment timestamps | |
| id: timestamps | |
| run: | | |
| echo "deployed_at=$(date -u +'%Y-%m-%d %H:%M UTC')" >> "$GITHUB_OUTPUT" | |
| echo "expires_at=$(date -u -d '+20 minutes' +'%Y-%m-%d %H:%M UTC')" >> "$GITHUB_OUTPUT" | |
| echo "deployed_at_iso=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> "$GITHUB_OUTPUT" | |
| echo "expires_at_iso=$(date -u -d '+20 minutes' +'%Y-%m-%dT%H:%M:%SZ')" >> "$GITHUB_OUTPUT" | |
| # A GitHub Deployment/Environment, not just the comment below: GitHub | |
| # renders this as its own "View deployment" box pinned near the top | |
| # of the PR (like Vercel's bot), so the latest link + status is | |
| # visible without scrolling into the comment thread -- unlike the | |
| # sticky comment, which is an in-place edit of a comment created on | |
| # the PR's first deploy, so it doesn't move and is easy to miss. | |
| # `required_contexts: []` skips gating this on other checks for the | |
| # same commit (e.g. lint/test workflows still running) -- this is a | |
| # preview link, not a merge gate. auto_inactive (default true on the | |
| # status call) marks any prior deployment to this same per-PR | |
| # environment inactive, so redeploys don't leave stale green boxes. | |
| - name: Create GitHub deployment | |
| run: | | |
| python -m scripts.preview.github_api create-and-activate \ | |
| --repo "${{ github.repository }}" --token "${{ secrets.GITHUB_TOKEN }}" \ | |
| --ref "${{ github.event.pull_request.head.sha }}" \ | |
| --environment "pr-${{ github.event.pull_request.number }}-preview" \ | |
| --task "deploy:preview" --description "K8s stack preview" \ | |
| --environment-url "${{ steps.cf_dns.outputs.url }}" \ | |
| --log-url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \ | |
| --status-description "Live for 20 minutes" | |
| - name: Render the ready-preview comment body | |
| id: render_ready | |
| run: | | |
| python -m scripts.preview.comment render-ready \ | |
| --url "${{ steps.cf_dns.outputs.url }}" --keycloak-url "${{ steps.cf_dns.outputs.keycloak_url }}" \ | |
| --deployed-at "${{ steps.timestamps.outputs.deployed_at }}" \ | |
| --deployed-at-iso "${{ steps.timestamps.outputs.deployed_at_iso }}" \ | |
| --expires-at "${{ steps.timestamps.outputs.expires_at }}" \ | |
| --expires-at-iso "${{ steps.timestamps.outputs.expires_at_iso }}" \ | |
| ${{ github.event.pull_request.head.repo.fork && '--fork' || '' }} | |
| - name: Comment preview link on PR | |
| id: comment_preview | |
| uses: marocchino/sticky-pull-request-comment@5770ad5eb8f42dd2c4f34da00c94c5381e49af88 # v3.0.5 | |
| with: | |
| header: k8s-preview | |
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | |
| message: ${{ steps.render_ready.outputs.body }} | |
| # Bounded to 20min by default so the live preview doesn't sit open | |
| # (and burn CI minutes) indefinitely, and so this step ends itself | |
| # well under the job's 90min timeout-minutes -- letting the job's own | |
| # timeout kill this step instead makes GitHub mark the whole run | |
| # "cancelled" with a "job has exceeded the maximum execution time" | |
| # failure annotation, which looks like a real failure even though | |
| # nothing broke -- the preview is *meant* to expire eventually. | |
| # | |
| # Runs cloudflared in the background instead of blocking on it, so | |
| # this loop can poll for the `extend-preview` label between checks | |
| # and push the deadline back 20 minutes from whenever it actually | |
| # sees the label (not just tack 20 more onto whatever's left) -- | |
| # each occurrence is a one-shot reset, consumed by removing the | |
| # label, so it can be reused any number of times before expiry. | |
| # Ultimately still bounded by the job's own 90min timeout-minutes | |
| # regardless of how many times it's extended. | |
| # | |
| # `kill -0 $pid` checks liveness without signaling; if cloudflared | |
| # exits on its own (a real crash, not us closing it), `wait` reaps | |
| # its actual exit code so a genuine failure still fails the step -- | |
| # `|| ec=$?` catches that under `bash -e` (GitHub Actions' default | |
| # for run: steps) without the step aborting mid-check. | |
| - name: Run tunnel until it times out (extend via the extend-preview label) | |
| env: | |
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | |
| run: | | |
| /tmp/cloudflared tunnel --no-autoupdate run --token "${TUNNEL_TOKEN}" & | |
| cloudflared_pid=$! | |
| deadline=$(( $(date +%s) + 1200 )) | |
| ec=0 | |
| while true; do | |
| if ! kill -0 "$cloudflared_pid" 2>/dev/null; then | |
| wait "$cloudflared_pid" || ec=$? | |
| break | |
| fi | |
| now=$(date +%s) | |
| if [ "$now" -ge "$deadline" ]; then | |
| kill "$cloudflared_pid" 2>/dev/null || true | |
| wait "$cloudflared_pid" 2>/dev/null || true | |
| break | |
| fi | |
| if gh api "repos/${{ github.repository }}/issues/${{ github.event.pull_request.number }}/labels" --jq '.[].name' 2>/dev/null \ | |
| | grep -qx "extend-preview"; then | |
| deadline=$(( now + 1200 )) | |
| gh api -X DELETE "repos/${{ github.repository }}/issues/${{ github.event.pull_request.number }}/labels/extend-preview" \ | |
| > /dev/null 2>&1 || true | |
| echo "extend-preview seen -- new deadline: $(date -u -d "@$deadline" +'%Y-%m-%d %H:%M UTC')" | |
| fi | |
| sleep 15 | |
| done | |
| [ "$ec" -eq 0 ] || exit "$ec" | |
| # Runs once the tunnel step above ends (its own timeout, cloudflared | |
| # exiting, or a manual cancel), so this captures anything logged in | |
| # response to real traffic during the tunnel's lifetime -- unlike the | |
| # earlier startup-time log dump. | |
| - name: Dump hub logs after tunnel closes | |
| if: always() | |
| env: | |
| KUBECONFIG: ${{ steps.sandbox.outputs.kubeconfig }} | |
| run: kubectl -n pr-preview logs deployment/hub --tail=1000 || true | |
| # The URL in the deployment box above is dead once the tunnel closes | |
| # -- mark it inactive so the box reflects that instead of still | |
| # showing green with a link that no longer resolves. | |
| - name: Mark GitHub deployment inactive | |
| if: always() | |
| run: | | |
| python -m scripts.preview.github_api mark-inactive \ | |
| --repo "${{ github.repository }}" --token "${{ secrets.GITHUB_TOKEN }}" \ | |
| --deployment-id "${DEPLOYMENT_ID:-}" --description "Preview expired" | |
| # The comment above still says "Ready" / "Expires in N minutes" once | |
| # the tunnel closes -- it was only ever written once, at deploy time, | |
| # and <relative-time> only keeps the clock ticking, not the | |
| # surrounding wording or status. Edit it again now that the run is | |
| # actually tearing down, so a reader doesn't see a green "Ready" row | |
| # next to a relative time that has flipped to the past. Only do this | |
| # when the first post succeeded (skip if the run failed before ever | |
| # reaching that step, since there is no live-preview comment to | |
| # correct in that case). | |
| - name: Render the expired-preview comment body | |
| id: render_expired | |
| if: always() && steps.comment_preview.outcome == 'success' | |
| run: | | |
| python -m scripts.preview.comment render-expired \ | |
| --expires-at "${{ steps.timestamps.outputs.expires_at }}" \ | |
| --expires-at-iso "${{ steps.timestamps.outputs.expires_at_iso }}" | |
| - name: Comment that the preview expired | |
| if: always() && steps.comment_preview.outcome == 'success' | |
| uses: marocchino/sticky-pull-request-comment@5770ad5eb8f42dd2c4f34da00c94c5381e49af88 # v3.0.5 | |
| with: | |
| header: k8s-preview | |
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | |
| message: ${{ steps.render_expired.outputs.body }} | |
| - name: Delete DNS record | |
| if: always() | |
| env: | |
| CF_API_TOKEN: ${{ secrets.CLOUDFLARE_TUNNEL_API_TOKEN }} | |
| run: | | |
| python -m scripts.preview.cloudflare delete-dns \ | |
| --api-token "$CF_API_TOKEN" --zone-id "${ZONE_ID:-}" \ | |
| --record-id "${DNS_RECORD_ID:-}" --record-id "${KEYCLOAK_DNS_RECORD_ID:-}" | |
| - name: Delete Cloudflare Tunnel | |
| if: always() | |
| env: | |
| CF_API_TOKEN: ${{ secrets.CLOUDFLARE_TUNNEL_API_TOKEN }} | |
| CF_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_TUNNEL_ACCOUNT_ID }} | |
| run: | | |
| python -m scripts.preview.cloudflare delete-tunnel \ | |
| --account-id "$CF_ACCOUNT_ID" --api-token "$CF_API_TOKEN" --tunnel-id "${TUNNEL_ID:-}" | |
| cleanup-preview: | |
| if: github.event.action == 'unlabeled' && github.event.label.name == 'deploy-preview' | |
| name: Stop preview | |
| runs-on: ubuntu-latest | |
| permissions: | |
| pull-requests: write | |
| actions: write | |
| deployments: write | |
| steps: | |
| - name: Checkout | |
| uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 | |
| - name: Cancel the in-flight preview run for this PR | |
| run: | | |
| python -m scripts.preview.github_api cancel-in-flight-run \ | |
| --repo "${{ github.repository }}" --token "${{ secrets.GITHUB_TOKEN }}" \ | |
| --workflow-name "K8s Stack Preview" --pr "${{ github.event.pull_request.number }}" | |
| - name: Mark GitHub deployment inactive | |
| run: | | |
| python -m scripts.preview.github_api mark-latest-inactive \ | |
| --repo "${{ github.repository }}" --token "${{ secrets.GITHUB_TOKEN }}" \ | |
| --environment "pr-${{ github.event.pull_request.number }}-preview" \ | |
| --description "Preview stopped (label removed)" | |
| - name: Render the stopped-preview comment body | |
| id: render_stopped | |
| run: python -m scripts.preview.comment render-stopped | |
| - name: Comment that the preview stopped | |
| uses: marocchino/sticky-pull-request-comment@5770ad5eb8f42dd2c4f34da00c94c5381e49af88 # v3.0.5 | |
| with: | |
| header: k8s-preview | |
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | |
| message: ${{ steps.render_stopped.outputs.body }} |