CICD - Unit | Integration | E2E

[JoanMVPopov]

Closes CAI-165. Check out the CI/CD article and the package.json scripts article for more info

CI/CD Pipeline: Unit, Integration, and E2E Testing

This PR is related to our two-stage CI/CD pipeline that gates merges to main, along with things related to the Docker Compose configuration, and Playwright E2E infrastructure.

Pipeline Overview

The pipeline is defined in .github/workflows/ci.yml and runs on every pull request targeting main. It consists of two jobs that run in parallel:

1. Unit & Integration Tests - runs on the GitHub Actions runner itself using Vitest and a local Supabase Docker subset

2. End-to-End Tests - runs Playwright against a real staging deployment on a Hetzner VPS managed by Coolify

Both jobs must pass before a PR can be merged. Merging to main triggers production deployment via Coolify

Job 1: Unit & Integration Tests (unit-integration)

Concurrency: cancel-in-progress per PR. Pushing new commits to the same PR cancels the previous run immediately.

Timeout: 20 minutes.

Steps

1. Checkout the PR code

2. Configure system limits - works around a known Supabase/Docker issue where the supavisor container requires a nofile ulimit of 100,000+. GitHub Actions runners default to a lower value, causing the container to fail with "Operation not permitted." This step sets the ulimit to 200,000 and configures the Docker daemon accordingly, then restarts Docker. See CLI on Github Actions supavisor fails on RLIMIT_NOFILE = 100000 supabase/cli#4443

3. Setup Node.js using the version from .nvmrc with npm caching

4. Install dependencies via npm ci

5. Load app environment variables from cicd/.env.app.ci into $GITHUB_ENV. These are the SvelteKit app env vars (DATABASE_URL, PUBLIC_SUPABASE_URL, etc.) and are slightly different from the Docker .env.ci used by the compose stack

6. Run unit tests via npm run test:unit (Vitest with vitest.config.unit.ts)

7. Restore/load Docker image cache -- uses actions/cache@v4 to cache Docker images between runs, keyed on the CI compose file hash. On cache hit, images are loaded from tar files

8. Run integration tests via npm run test:integration, which invokes scripts/run-integration.sh. This script spins up a Supabase subset via Docker Compose, waits for health checks, then runs Vitest with vitest.config.integration.ts

9. Save Docker images to cache on first run (when cache miss)

10. Upload coverage artifacts (unit and integration) as GitHub Actions artifacts with 7-day retention

11. Report coverage as PR comments using davelosert/vitest-coverage-report-action@v2, one comment for unit coverage and one for integration

The integration test steps run with if: always() so they execute even if unit tests fail, giving visibility into both results in a single run.

Job 2: End-to-End Tests (e2e)

Concurrency: single global group (staging-test-e2e), no cancel-in-progress. Since staging is a shared single resource, E2E runs queue rather than cancel each other.

Timeout: 30 minutes.

Steps

1. Checkout the code (merge commit)

2. Setup Node.js and install dependencies

3. Install Playwright with Chromium only (npx playwright install --with-deps chromium). Firefox and WebKit are omitted to save CI time

4. Push PR code to the staging branch via git push --force origin HEAD:refs/heads/staging. This is how the PR's code reaches the staging VPS. Force-push is safe because staging is not a working branch, the concurrency lock prevents parallel pushes, and we are pushing a specific SHA.

5. Stop both applications via scripts/cicd/stop-and-wait.sh. This calls the Coolify API to stop both the Supabase and SvelteKit applications, then polls until each reaches the exited state. This stop-first approach prevents a race condition where health checks could pass against old (still-running) containers from the previous deployment

6. Deploy and health-check Supabase via scripts/cicd/deploy-supabase-to-staging.sh. Triggers a start via the Coolify API, polls until the application status is running, then verifies three endpoints: the dashboard (expects HTTP 307), the auth service (/auth/v1/health, expects 200), and the REST API (/rest/v1/, expects 200).

7. Setup SSH using webfactory/ssh-agent@v0.9.0 with a dedicated cicd user's private key

8. Wipe the staging database by SSHing into the staging server and running a PL/pgSQL block inside the Postgres container. This truncates all tables in the public schema (with RESTART IDENTITY CASCADE) except _prisma_migrations, which tracks applied migrations and must remain intact. The container is discovered dynamically using docker ps --format '{{.Names}}' | grep '^db-' because Coolify generates random container name suffixes

9. Deploy and health-check SvelteKit via scripts/cicd/deploy-sveltekit-to-staging.sh. Same pattern as Supabase: start via API, poll until running, then check /api/health returns 200. On startup, the SvelteKit app runs init-db.sh which applies Prisma migrations, sets up SAML, seeds triggers, RLS policies, and storage buckets. This is idempotent and works correctly after a database wipe

10. Run Playwright tests with npx playwright test against the staging URL

11. Upload Playwright HTML report as an artifact with 14-day retention, even on test failure

Required GitHub Secrets for E2E

• STAGING_COOLIFY_BASE_URL -- Coolify API base URL
• STAGING_COOLIFY_API_TOKEN -- Coolify API bearer token
• STAGING_COOLIFY_SUPABASE_UUID -- Coolify application UUID for the Supabase stack
• STAGING_COOLIFY_SVELTEKIT_UUID -- Coolify application UUID for the SvelteKit app
• STAGING_SUPABASE_URL -- public Supabase URL on staging
• STAGING_ANON_KEY -- Supabase anonymous key for staging
• STAGING_DASHBOARD_USERNAME / STAGING_DASHBOARD_PASSWORD -- Supabase dashboard basic auth credentials
• STAGING_URL -- public SvelteKit URL on staging
• STAGING_SSH_PRIVATE_KEY -- ed25519 private key for the cicd user on the VPS
• STAGING_SSH_HOST -- staging server IP
• STAGING_SSH_USER -- SSH username (cicd)

CI/CD Scripts (scripts/cicd/)

All scripts use set -euo pipefail for strict error handling.

stop-and-wait.sh

Stops both Coolify applications (Supabase and SvelteKit) and polls their status via the Coolify API until each reaches the exited state. Skips the stop request if the application is already stopped. Times out after 150 seconds (30 polls at 5-second intervals). All configuration (COOLIFY_URL, TOKEN, SUPABASE_UUID, SVELTEKIT_UUID) is read from environment variables injected by GitHub secrets.

deploy-supabase-to-staging.sh

Starts the Supabase application via the Coolify API, polls until the application status is running, then runs health checks against three endpoints (dashboard, auth, REST). Uses || true on all curl commands to prevent set -e from killing the script when the server is unreachable during startup. Uses if blocks instead of && chains for boolean checks, also to avoid premature exit from set -e.

deploy-sveltekit-to-staging.sh

Same pattern as the Supabase script but checks a single endpoint (/api/health).

CI override: cicd/docker-compose.ci.yml

A minimal override file that layers on top of the base compose. It uses the !override YAML tag to replace (not merge) specific properties:

• db volumes: replaces the dev volume list to remove the persistent data volume (./volumes/db/data) and the db-config named volume. Paths use ../docker/volumes/ because Docker Compose resolves relative paths from the directory of the file they are defined in. Adds tmpfs: /var/lib/postgresql/data for ephemeral storage on the GitHub runner
• Top-level volumes: overridden to an empty object (volumes: !override {}) since the CI setup uses no named volumes

Usage in CI: docker compose -f docker/docker-compose.yml -f cicd/docker-compose.ci.yml --env-file cicd/.env.ci

Environment files

cicd/.env.ci

Docker Compose environment variables for the CI Supabase stack. Contains throwaway credentials safe to commit. Includes variables that were previously hardcoded in the old standalone CI compose file (MINIO_ROOT_USER, MINIO_ROOT_PASSWORD, SECRET_KEY_BASE, VAULT_ENC_KEY, SECRET_KEY_BASE_REALTIME)

cicd/.env.app.ci

SvelteKit application environment variables loaded into $GITHUB_ENV during CI. Contains DATABASE_URL, SERVICE_ROLE_KEY, PUBLIC_SUPABASE_URL, PUBLIC_SUPABASE_ANON_KEY, FILESYSTEM, and PUBLIC_ENVIRONMENT

Playwright Configuration

playwright.config.ts

Switches behavior based on process.env.CI (automatically set to true by GitHub Actions runners):

• CI: baseURL set from STAGING_URL env var, reporters are github (for inline CI annotations) and html (downloadable report, open: 'never')
• Local: baseURL is http://localhost:4173, reporters are list (terminal output) and html (open: 'never' to prevent the report server from blocking script exit)

Workers set to 1 (serial execution). Test directory is tests/e2e.

Testing Scripts

scripts/run-integration.sh (integration tests)

Updated to support both CI and local environments using a single script:

• Sets NODE_ENV=test
• Pre-checks that port 4173 is free
• Determines the compose command based on the CI environment variable:
  • CI: docker compose -f docker/docker-compose.yml -f cicd/docker-compose.ci.yml --env-file cicd/.env.ci with a specific service subset (db, analytics, supavisor, kong, auth, rest, storage, imgproxy, minio, minio-createbucket)
  • Local: docker compose -f docker/docker-compose.yml --env-file docker/.env with all services
• Waits for health checks, loads the appropriate env file
• On local runs: wipes the database before tests (same PL/pgSQL truncation as staging)
• Builds the app, starts a preview server on port 4173, health-checks it, then runs Vitest
• Cleanup via trap kills the server process, kills anything on port 4173 (needed because killing npm does not always kill the child vite process), and tears down Docker with -v (ephemeral test data)

scripts/run-e2e.sh (local E2E tests)

New script for running Playwright locally:

• Starts the local Docker Compose stack, waits for health checks
• Wipes the database using the same PL/pgSQL truncation approach
• Builds the app and starts a preview server on port 4173
• Health-checks the server by polling /api/health for HTTP 200
• Runs npx playwright test
• Cleanup via trap kills the server, Docker stack, and anything on port 4173

scripts/init-db.sh (database initialization)

Separated from the build step. Previously, npm run build included database operations (Prisma generate, migrate, seeding). Now:

• npm run build only runs vite build (plus dirname.sh)
• npm run start:preview and npm run start run init-db.sh before starting the server

init-db.sh runs: prisma generate, prisma migrate deploy, SAML IDP setup, trigger seeding, RLS policy seeding, and bucket seeding (if FILESYSTEM=SUPABASE)

This is idempotent and safe to run after a database wipe, since it only sets up infrastructure (schema, triggers, policies, buckets), not user data

Deployment Configuration

nixpacks.toml

Unchanged in purpose but works with the new script structure:

• Build phase: npm run build (now just vite build)
• Start command: npm start (now runs init-db.sh before node build)

[github-actions]

Coverage Report for Unit

Status	Category	Percentage	Covered / Total
🟢	Lines	38.96% (🎯 25%)	362 / 929
🟢	Statements	38.02% (🎯 25%)	381 / 1002
🟢	Functions	31.42% (🎯 25%)	88 / 280
🟢	Branches	35.88% (🎯 25%)	169 / 471

File Coverage

No changed files found.

Generated in workflow #25 for commit 41f6d06 by the Vitest Coverage Report Action

[github-actions]

Coverage Report for Integration

Status	Category	Percentage	Covered / Total
🟢	Lines	5.09% (🎯 4%)	65 / 1276
🟢	Statements	4.82% (🎯 4%)	67 / 1389
🟢	Functions	5.73% (🎯 4%)	20 / 349
🟢	Branches	4.81% (🎯 4%)	31 / 644

File Coverage

No changed files found.

Generated in workflow #25 for commit 41f6d06 by the Vitest Coverage Report Action