Ci pinned services stack (#121)

* update ci tests to use pinned local services stack

Stop test and coverage workflows from tunneling to production by downloading RaidHub-Services at a pinned commit, booting its docker dependencies locally, and running migrations/seeding before API tests.

Made-with: Cursor

* move services pin file to avoid remote path conflict

Rename the pinned RaidHub-Services version file to .github/raidhub-services-version.env and update the CI setup script to use it so branch pushes succeed against the current remote.

Made-with: Cursor

* use services make env in ci bootstrap script

Replace manual .env key injection with RaidHub-Services' make env target so CI setup follows the same environment bootstrap path as services.

Made-with: Cursor

* use real postgres fixtures in status integration tests

Create and clean up deterministic instance/pgcr rows inside status tests so they validate route behavior against real database state instead of mocking instance reads.

Made-with: Cursor

* lazy-init database clients to reduce import-time coupling

Avoid creating Postgres and ClickHouse clients at module import time by switching to lazy singletons, so one-off scripts like docs generation can load route modules without unnecessary DB setup.

Made-with: Cursor

* format

* bump pinned services commit for ci schema parity

Update the pinned RaidHub-Services SHA used by CI so migrations and seeded schemas match API test query expectations.

Made-with: Cursor

* stabilize ci tests with local-db assumptions

Remove dependency on non-portable activity_version columns and update brittle tests to use local database-backed fixtures or discovered rows so the CI stack seeded from RaidHub-Services can run reliably.

Made-with: Cursor

* format player fixture test for ci checks

Apply Prettier formatting to the updated player service fixture test so static PR checks pass.

Made-with: Cursor

* make dataset-dependent tests resilient for local ci data

Update pgcr, player search, and weapon metrics tests to validate schema and bounds without requiring production-sized datasets, so they run reliably against locally seeded CI infrastructure.

Made-with: Cursor

* format pgcr service test for static checks

Run Prettier on pgcr service test to satisfy CI formatting checks.

Made-with: Cursor

* make integration tests tolerant of seeded local datasets

Relax strict non-empty expectations in dataset-dependent tests and allow nullable leaderboard search results so CI can validate shapes/behavior against the local seeded stack instead of production-sized data.

Made-with: Cursor

* stabilize instance service tests for local seeded db

Switch instance service tests from hardcoded production instance IDs to dynamic local rows and schema checks so they pass against ephemeral CI seed data.

Made-with: Cursor

* remove remaining prod-data assumptions in service tests

Relax atlas and reporting standing assertions for seeded CI data and fix instance metadata test hash typing to avoid integer overflow against bigint hashes.

Made-with: Cursor

* stabilize player and clan tests for local ci data

Use DB-discovered membership IDs in player route tests and relax fixed-size clan leaderboard assertions so these tests pass against seeded local datasets.

Made-with: Cursor

* use discovered player ids in history and teammates route tests

Replace hardcoded membership IDs with DB-discovered public/private players and make middleware assertions robust to seeded CI datasets.

Made-with: Cursor

* make expectOk/expectErr only assert result type

Route errors now include parsed context payloads, so shared test helpers should not require null parsed values when asserting ok/err outcomes.

Made-with: Cursor

* add deterministic fixtures for remaining brittle route tests

Use explicit postgres fixtures for profile and admin standing routes and make pantheon search assertions data-aware so tests execute fully without relying on production-only records.

Made-with: Cursor

* format reporting standing fixture tests for ci

Apply Prettier formatting to new reporting standing fixture tests so static checks pass.

Made-with: Cursor

* make leaderboard search route tests data-tolerant in ci

Update leaderboard route tests to assert success for list queries while allowing search queries to return not-on-leaderboard errors when seeded CI data lacks matching memberships.

Made-with: Cursor

* fix profile and instance-standing test setup execution

Run authorized profile assertion inside a test case and split multi-statement fixture setup queries so pg prepared statements work in CI.

Made-with: Cursor

* set instance-standing fixture skull hashes to empty array

Populate skull_hashes with an empty bigint array in the fixture instance so route response schema validation matches expected array type.

Made-with: Cursor

* test: add DB fixtures and drop vacuous skips

Replace early returns and weak assertions with deterministic inserts
via a shared fixture pool, and align teammates/history/instances
tests with real instance_player rows so routes exercise real data.

Made-with: Cursor

* test: assert teammates payload and error codes

Made-with: Cursor

* test: extend review themes to leaderboards and service history

Add leaderboard pagination invariants, tighten standing/history
assertions with real fixture data, and replace vacuous length checks
across route and service leaderboard tests.

Made-with: Cursor

* fix(ci): repair leaderboard TS narrowing, PGCR gzip, instance_character

- Narrow LeaderboardData union before individual vs team membership asserts
- Use err-branch type guard for pantheon search fallback
- Store gzip-compressed minimal PGCR JSON for raw.pgcr fixtures
- Seed extended.instance_character so getInstanceExtended players[].characters is an array

Made-with: Cursor

* test: align history final-raid and feat list with CI seed

Made-with: Cursor

Author: owens1127

.github/raidhub-services-version.env

@@ -0,0 +1,2 @@
+RAIDHUB_SERVICES_REPO=Raid-Hub/RaidHub-Services
+RAIDHUB_SERVICES_COMMIT=858b76cf0314c1d5547a60011bf444eb63615e88

.github/scripts/setup-local-services-ci.sh

@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+VERSIONS_FILE="${ROOT_DIR}/.github/raidhub-services-version.env"
+SERVICES_DIR="${ROOT_DIR}/RaidHub-Services"
+
+if [[ ! -f "${VERSIONS_FILE}" ]]; then
+    echo "Missing versions file at ${VERSIONS_FILE}"
+    exit 1
+fi
+
+set -a
+source "${VERSIONS_FILE}"
+set +a
+
+if [[ -z "${RAIDHUB_SERVICES_REPO:-}" || -z "${RAIDHUB_SERVICES_COMMIT:-}" ]]; then
+    echo "RAIDHUB_SERVICES_REPO and RAIDHUB_SERVICES_COMMIT must be set in ${VERSIONS_FILE}"
+    exit 1
+fi
+
+if [[ ! "${RAIDHUB_SERVICES_COMMIT}" =~ ^[0-9a-f]{40}$ ]]; then
+    echo "RAIDHUB_SERVICES_COMMIT must be a full 40-character commit hash"
+    exit 1
+fi
+
+echo "Checking out ${RAIDHUB_SERVICES_REPO}@${RAIDHUB_SERVICES_COMMIT}"
+rm -rf "${SERVICES_DIR}"
+git init "${SERVICES_DIR}"
+git -C "${SERVICES_DIR}" remote add origin "https://github.com/${RAIDHUB_SERVICES_REPO}.git"
+git -C "${SERVICES_DIR}" fetch --depth=1 origin "${RAIDHUB_SERVICES_COMMIT}"
+git -C "${SERVICES_DIR}" checkout --detach FETCH_HEAD
+
+(
+    cd "${SERVICES_DIR}"
+    make env
+)
+
+docker compose -f "${SERVICES_DIR}/docker-compose.yml" --env-file "${SERVICES_DIR}/.env" up -d postgres rabbitmq clickhouse prometheus
+
+wait_for() {
+    local name="$1"
+    local cmd="$2"
+    local attempts="${3:-60}"
+    local sleep_seconds="${4:-2}"
+
+    for _ in $(seq 1 "${attempts}"); do
+        if eval "${cmd}" >/dev/null 2>&1; then
+            echo "${name} is ready"
+            return 0
+        fi
+        sleep "${sleep_seconds}"
+    done
+
+    echo "${name} did not become ready in time"
+    return 1
+}
+
+echo "Waiting for PostgreSQL..."
+wait_for "PostgreSQL" "docker compose -f \"${SERVICES_DIR}/docker-compose.yml\" --env-file \"${SERVICES_DIR}/.env\" exec -T postgres pg_isready -U dev -d raidhub"
+
+echo "Waiting for ClickHouse..."
+wait_for "ClickHouse" "curl -fsS \"http://localhost:8123/ping\" | grep -q '^Ok\\.$'"
+
+echo "Waiting for RabbitMQ API..."
+wait_for "RabbitMQ API" "curl -fsS -u \"dev:password\" \"http://localhost:15672/api/overview\""
+
+(
+    cd "${SERVICES_DIR}"
+    make migrate
+    make seed
+)
+
+if [[ -n "${GITHUB_ENV:-}" ]]; then
+    echo "RAIDHUB_SERVICES_DIR=${SERVICES_DIR}" >> "${GITHUB_ENV}"
+fi

.github/workflows/coverage.yml

@@ -16,39 +16,29 @@ jobs:
         steps:
             - uses: actions/checkout@v4
             - uses: oven-sh/setup-bun@v1
+            - uses: actions/setup-go@v5
+              with:
+                  go-version: "1.24"
 
             - name: Bun Install
               run: |
                   bun install --frozen-lockfile
 
-            - name: Install cloudflared
-              run: ./.github/scripts/install-cloudflared.sh
-
-            - name: Load SSH Key
-              uses: shimataro/ssh-key-action@v2
-              with:
-                  key: ${{ secrets.VPS_SSH_PRIVATE_KEY }}
-                  known_hosts: ${{ secrets.VPS_SSH_KNOWN_HOSTS }}
-
-            - name: Tunnel to VPS
-              env:
-                  TUNNEL_SERVICE_TOKEN_ID: ${{ secrets.CF_GHA_CLIENT_ID }}
-                  TUNNEL_SERVICE_TOKEN_SECRET: ${{ secrets.CF_GHA_CLIENT_SECRET }}
-              run: |
-                  ssh -N -L 5432:localhost:5432 -L 8123:localhost:8123 -L 9090:localhost:9090 -L 15672:localhost:15672 -o ProxyCommand="cloudflared access ssh --hostname %h" root@ssh.raidhub.io &
+            - name: Start pinned local services stack
+              run: bash ./.github/scripts/setup-local-services-ci.sh
 
             - name: Run Tests
               env:
                   CLIENT_SECRET: "secret-token"
                   ADMIN_CLIENT_SECRET: "another-secret-token"
                   JWT_SECRET: "jwt-secret"
-                  POSTGRES_USER: readonly
-                  POSTGRES_PASSWORD: ${{ secrets.POSTGRES_READONLY_PASSWORD }}
-                  CLICKHOUSE_USER: ${{ secrets.CLICKHOUSE_READONLY_USER }}
-                  CLICKHOUSE_PASSWORD: ${{ secrets.CLICKHOUSE_READONLY_PASSWORD }}
+                  POSTGRES_USER: dev
+                  POSTGRES_PASSWORD: password
+                  CLICKHOUSE_USER: default
+                  CLICKHOUSE_PASSWORD: ""
                   BUNGIE_API_KEY: ${{ secrets.BUNGIE_API_KEY }}
-                  RABBIT_API_USER: ${{ secrets.RABBIT_API_USER }}
-                  RABBIT_API_PASSWORD: ${{ secrets.RABBIT_API_PASSWORD }}
+                  RABBIT_API_USER: dev
+                  RABBIT_API_PASSWORD: password
                   R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
                   R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
                   R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
@@ -57,6 +47,16 @@ jobs:
               run: bun test --timeout=30000 --coverage
               continue-on-error: true
 
+            - name: Dump services logs on failure
+              if: failure()
+              run: |
+                  docker compose -f RaidHub-Services/docker-compose.yml --env-file RaidHub-Services/.env ps
+                  docker compose -f RaidHub-Services/docker-compose.yml --env-file RaidHub-Services/.env logs --tail=200 postgres rabbitmq clickhouse
+
+            - name: Tear down local services stack
+              if: always()
+              run: docker compose -f RaidHub-Services/docker-compose.yml --env-file RaidHub-Services/.env down -v
+
             - name: Upload coverage reports
               uses: actions/upload-artifact@v4
               with:

.github/workflows/test.yml

@@ -17,6 +17,9 @@ jobs:
             - name: Checkout
               uses: actions/checkout@v4
             - uses: oven-sh/setup-bun@v1
+            - uses: actions/setup-go@v5
+              with:
+                  go-version: "1.24"
 
             - name: Download code coverage report
               uses: dawidd6/action-download-artifact@v3
@@ -26,45 +29,42 @@ jobs:
                   workflow_conclusion: success
                   name: test-coverage.info
 
-            - name: Install cloudflared
-              run: ./.github/scripts/install-cloudflared.sh
-
-            - name: Load SSH Key
-              uses: shimataro/ssh-key-action@v2
-              with:
-                  key: ${{ secrets.VPS_SSH_PRIVATE_KEY }}
-                  known_hosts: ${{ secrets.VPS_SSH_KNOWN_HOSTS }}
-
             - name: Install dependencies
               run: |
                   bun install --frozen-lockfile
 
-            - name: Tunnel to VPS
-              env:
-                  TUNNEL_SERVICE_TOKEN_ID: ${{ secrets.CF_GHA_CLIENT_ID }}
-                  TUNNEL_SERVICE_TOKEN_SECRET: ${{ secrets.CF_GHA_CLIENT_SECRET }}
-              run: |
-                  ssh -N -L 5432:localhost:5432 -L 8123:localhost:8123 -L 9090:localhost:9090 -L 15672:localhost:15672 -o ProxyCommand="cloudflared access ssh --hostname %h" root@ssh.raidhub.io &
+            - name: Start pinned local services stack
+              run: bash ./.github/scripts/setup-local-services-ci.sh
 
             - name: Run Tests
               env:
                   CLIENT_SECRET: "secret-token"
                   ADMIN_CLIENT_SECRET: "another-secret-token"
                   JWT_SECRET: "jwt-secret"
-                  POSTGRES_USER: readonly
-                  POSTGRES_PASSWORD: ${{ secrets.POSTGRES_READONLY_PASSWORD }}
-                  CLICKHOUSE_USER: ${{ secrets.CLICKHOUSE_READONLY_USER }}
-                  CLICKHOUSE_PASSWORD: ${{ secrets.CLICKHOUSE_READONLY_PASSWORD }}
+                  POSTGRES_USER: dev
+                  POSTGRES_PASSWORD: password
+                  CLICKHOUSE_USER: default
+                  CLICKHOUSE_PASSWORD: ""
                   BUNGIE_API_KEY: ${{ secrets.BUNGIE_API_KEY }}
-                  RABBIT_API_USER: ${{ secrets.RABBIT_API_USER }}
-                  RABBIT_API_PASSWORD: ${{ secrets.RABBIT_API_PASSWORD }}
+                  RABBIT_API_USER: dev
+                  RABBIT_API_PASSWORD: password
                   R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
                   R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
                   R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
                   R2_BUCKET: ${{ secrets.R2_BUCKET }}
 
               run: bun test --coverage --bail=5 --timeout=20000
 
+            - name: Dump services logs on failure
+              if: failure()
+              run: |
+                  docker compose -f RaidHub-Services/docker-compose.yml --env-file RaidHub-Services/.env ps
+                  docker compose -f RaidHub-Services/docker-compose.yml --env-file RaidHub-Services/.env logs --tail=200 postgres rabbitmq clickhouse
+
+            - name: Tear down local services stack
+              if: always()
+              run: docker compose -f RaidHub-Services/docker-compose.yml --env-file RaidHub-Services/.env down -v
+
             #  Compares two code coverage files and generates report as a comment
             - name: Generate Code Coverage report
               id: code-coverage

src/integrations/clickhouse/client.ts

@@ -1,4 +1,5 @@
 import { LogFields, Logger } from "@/lib/utils/logging"
+import type { ClickHouseClient } from "@clickhouse/client"
 import {
     ClickHouseLogLevel,
     ErrorLogParams,
@@ -58,19 +59,27 @@ class ClickhouseLogger implements ClickhouseLoggingInterface {
     }
 }
 
-export const clickhouse = createClient({
-    username: process.env.CLICKHOUSE_USER,
-    password: process.env.CLICKHOUSE_PASSWORD,
-    application: process.env.PROD ? "RaidHub-API-Prod" : "RaidHub-API-Dev",
-    database: process.env.CLICKHOUSE_DATABASE ?? "default",
-    request_timeout: 5000,
-    log: {
-        LoggerClass: ClickhouseLogger,
-        level:
-            process.env.NODE_ENV === "test"
-                ? ClickHouseLogLevel.OFF
-                : process.env.PROD
-                  ? ClickHouseLogLevel.WARN
-                  : ClickHouseLogLevel.DEBUG
+let clickhouseClient: ClickHouseClient | null = null
+
+export const getClickhouseClient = () => {
+    if (!clickhouseClient) {
+        clickhouseClient = createClient({
+            username: process.env.CLICKHOUSE_USER,
+            password: process.env.CLICKHOUSE_PASSWORD,
+            application: process.env.PROD ? "RaidHub-API-Prod" : "RaidHub-API-Dev",
+            database: process.env.CLICKHOUSE_DATABASE ?? "default",
+            request_timeout: 5000,
+            log: {
+                LoggerClass: ClickhouseLogger,
+                level:
+                    process.env.NODE_ENV === "test"
+                        ? ClickHouseLogLevel.OFF
+                        : process.env.PROD
+                          ? ClickHouseLogLevel.WARN
+                          : ClickHouseLogLevel.DEBUG
+            }
+        })
     }
-})
+
+    return clickhouseClient
+}

src/integrations/postgres/index.ts

@@ -4,22 +4,50 @@ import { createTransactional } from "./transactional"
 
 configurePostgresParsers()
 
-export const pgReader = createReader({
-    user: process.env.POSTGRES_USER,
-    password: process.env.POSTGRES_PASSWORD,
-    database: "raidhub",
-    min: process.env.PROD ? 5 : 1,
-    max: process.env.PROD ? 150 : 10,
-    idleTimeoutMillis: 30000,
-    connectionTimeoutMillis: 10000
-})
-
-export const pgAdmin = createTransactional({
-    user: process.env.POSTGRES_WRITABLE_USER,
-    password: process.env.POSTGRES_WRITABLE_PASSWORD,
-    database: "raidhub",
-    min: process.env.PROD ? 2 : 1,
-    max: process.env.PROD ? 15 : 3,
-    idleTimeoutMillis: 30000,
-    connectionTimeoutMillis: 10000
-})
+let pgReaderClient: ReturnType<typeof createReader> | null = null
+let pgAdminClient: ReturnType<typeof createTransactional> | null = null
+
+const getPgReader = () => {
+    if (!pgReaderClient) {
+        pgReaderClient = createReader({
+            user: process.env.POSTGRES_USER,
+            password: process.env.POSTGRES_PASSWORD,
+            database: "raidhub",
+            min: process.env.PROD ? 5 : 1,
+            max: process.env.PROD ? 150 : 10,
+            idleTimeoutMillis: 30000,
+            connectionTimeoutMillis: 10000
+        })
+    }
+
+    return pgReaderClient
+}
+
+const getPgAdmin = () => {
+    if (!pgAdminClient) {
+        pgAdminClient = createTransactional({
+            user: process.env.POSTGRES_WRITABLE_USER,
+            password: process.env.POSTGRES_WRITABLE_PASSWORD,
+            database: "raidhub",
+            min: process.env.PROD ? 2 : 1,
+            max: process.env.PROD ? 15 : 3,
+            idleTimeoutMillis: 30000,
+            connectionTimeoutMillis: 10000
+        })
+    }
+
+    return pgAdminClient
+}
+
+export const pgReader: ReturnType<typeof createReader> = {
+    queryRow: (sql, options) => getPgReader().queryRow(sql, options),
+    queryRows: (sql, options) => getPgReader().queryRows(sql, options),
+    prepare: sql => getPgReader().prepare(sql)
+}
+
+export const pgAdmin: ReturnType<typeof createTransactional> = {
+    queryRow: (sql, options) => getPgAdmin().queryRow(sql, options),
+    queryRows: (sql, options) => getPgAdmin().queryRows(sql, options),
+    prepare: sql => getPgAdmin().prepare(sql),
+    transaction: callback => getPgAdmin().transaction(callback)
+}