Skip to content

feat: gate main on a real Grafana Cloud consistency check #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
terchris merged 1 commit into from
Jul 14, 2026
Merged

feat: gate main on a real Grafana Cloud consistency check #25

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 99 additions & 12 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,16 +45,11 @@ jobs:
- name: Build TypeScript
run: npm run build

# There is no separate unit/integration test tier: `npm run test:unit`
# and `npm run test:integration` were never implemented for any
# language (see website/docs/contributor/06-test-scenarios.md — test/e2e/company-lookup/
# is the only REQUIRED test path). The project's actual correctness gate
# is the E2E company-lookup test verified against a real OTLP backend
# (Loki/Tempo/Prometheus), which needs a live Kubernetes cluster CI
# doesn't have — see contributor/testing/uis.md. Run it locally via
# `dct-exec` and `tools/validation/uis/compare-with-master.sh`.
# - name: Run E2E tests
# run: npm run test:e2e
# The real E2E-against-a-live-backend correctness gate now runs for
# real, in the grafana-cloud-consistency job below (added 2026-07-14) —
# it doesn't need a live Kubernetes cluster the way the UIS-backed
# version does, since Grafana Cloud is reachable from any CI runner.
# See tools/validation/grafana-cloud/README.md.

- name: Verify build artifacts
run: |
Expand Down Expand Up @@ -138,10 +133,96 @@ jobs:
console.log('✅ All required package.json fields present');
"

grafana-cloud-consistency:
name: Grafana Cloud Consistency Check
runs-on: ubuntu-latest
# Real end-to-end proof, not just a build check: writes to a local log
# file (ground truth), validates the file's format, then reads back from
# Loki/Prometheus/Tempo and diffs each response against that same file,
# field by field. Uses sovdev-ci-ingest/sovdev-ci-verify — dedicated CI
# credentials, LBAC-scoped to sovdev-ci-company-lookup* only, entirely
# separate from the maintainer's personal dev keys or any customer's.
# See tools/validation/grafana-cloud/README.md and
# contributor/testing/grafana-cloud.md's "CI's own consistency check"
# section for the full setup story.
env:
OTEL_SERVICE_NAME: sovdev-ci-company-lookup
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: https://otlp-gateway-prod-eu-west-0.grafana.net/otlp/v1/logs
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: https://otlp-gateway-prod-eu-west-0.grafana.net/otlp/v1/metrics
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: https://otlp-gateway-prod-eu-west-0.grafana.net/otlp/v1/traces
OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
LOG_TO_FILE: "true"
LOG_TO_CONSOLE: "true"
GRAFANA_CLOUD_INGEST_TOKEN: ${{ secrets.SOVDEV_CI_INGEST_TOKEN }}
GRAFANA_CLOUD_VERIFY_TOKEN: ${{ secrets.SOVDEV_CI_VERIFY_TOKEN }}
# Stack-wide constants, not secrets -- already published in plain text
# in using/onboarding/index.md and the .env.example files.
GRAFANA_CLOUD_OTLP_ENDPOINT: https://otlp-gateway-prod-eu-west-0.grafana.net/otlp
GRAFANA_CLOUD_OTLP_INSTANCE_ID: "484308"
GRAFANA_CLOUD_LOKI_URL: https://logs-prod-eu-west-0.grafana.net
GRAFANA_CLOUD_LOKI_INSTANCE_ID: "333665"
GRAFANA_CLOUD_PROMETHEUS_URL: https://prometheus-prod-01-eu-west-0.grafana.net
GRAFANA_CLOUD_PROMETHEUS_INSTANCE_ID: "669389"
GRAFANA_CLOUD_TEMPO_URL: https://tempo-eu-west-0.grafana.net
GRAFANA_CLOUD_TEMPO_INSTANCE_ID: "330178"

steps:
- name: Checkout code
uses: actions/checkout@v7

- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: 22

- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: '3.11'

- name: Install Python validator dependencies
run: pip install jsonschema

- name: Install and build typescript
working-directory: ./typescript
run: |
npm ci
npm run build

- name: Compute OTLP Basic Auth header
run: |
HEADER=$(echo -n "${GRAFANA_CLOUD_OTLP_INSTANCE_ID}:${GRAFANA_CLOUD_INGEST_TOKEN}" | base64 -w0)
echo "OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic ${HEADER}" >> "$GITHUB_ENV"

# Fail fast: this is the exact tool a new customer runs to verify their
# own connection (see contributor/testing/selftest-cli.md). No point
# spending ~2 minutes on the full E2E test below if a basic write+
# read-back against this same backend doesn't even work.
- name: Quick connection check (sovdev-selftest)
working-directory: ./typescript
run: node dist/cli/selftest.js --backend grafana-cloud

- name: Install E2E test dependencies
working-directory: ./typescript/test/e2e/company-lookup
run: npm ci

- name: Install validation tool dependencies
working-directory: ./tools/validation/grafana-cloud
run: npm ci

- name: Run full consistency check
working-directory: ./tools/validation/grafana-cloud
run: |
chmod +x ./full-consistency-check.sh
# .env.ci deliberately doesn't exist in the checked-out repo --
# run-test.sh falls back to the already-exported env vars above,
# exactly matching how a real consumer's own CI would use this.
./full-consistency-check.sh --env-file .env.ci

# This job will be used later when we have multiple languages
summary:
name: CI Summary
needs: [test-typescript, code-quality]
needs: [test-typescript, code-quality, grafana-cloud-consistency]
runs-on: ubuntu-latest
if: always()

Expand All @@ -164,13 +245,19 @@ jobs:
echo "❌ **Code Quality**: Failed" >> $GITHUB_STEP_SUMMARY
fi

if [[ "${{ needs.grafana-cloud-consistency.result }}" == "success" ]]; then
echo "✅ **Grafana Cloud Consistency**: Passed" >> $GITHUB_STEP_SUMMARY
else
echo "❌ **Grafana Cloud Consistency**: Failed" >> $GITHUB_STEP_SUMMARY
fi

echo "" >> $GITHUB_STEP_SUMMARY
echo "---" >> $GITHUB_STEP_SUMMARY
echo "_Note: This workflow is designed to support multi-language testing in the future_" >> $GITHUB_STEP_SUMMARY

- name: Set final status
run: |
if [[ "${{ needs.test-typescript.result }}" == "success" && "${{ needs.code-quality.result }}" == "success" ]]; then
if [[ "${{ needs.test-typescript.result }}" == "success" && "${{ needs.code-quality.result }}" == "success" && "${{ needs.grafana-cloud-consistency.result }}" == "success" ]]; then
echo "✅ All checks passed!"
exit 0
else
Expand Down
56 changes: 56 additions & 0 deletions tools/validation/grafana-cloud/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Sovdev Logger Verification Tools — Grafana Cloud Backend

TypeScript tools that query Grafana Cloud's hosted Loki, Prometheus, and Tempo directly via HTTP Basic Auth, and compare the results against a candidate language's E2E test output.

For the local UIS equivalent (bash, `kubectl` instead of HTTP), see [`../uis/`](../uis/).

---

## Prerequisites

1. **`.env`** — copy `.env.example`, fill in real values from your Grafana Cloud portal (Security → Access Policies for tokens; each stack's Loki/Tempo/Prometheus connection page for the base URLs and Instance IDs).
2. **`npm install`** in this directory — installs `tsx`/`typescript`/`@types/node`.
3. **Python 3.7+ with `jsonschema`** — the comparison logic lives in `../validators/`, unchanged regardless of backend.

---

## Tools

| Script | Purpose | Usage |
|--------|---------|-------|
| [**query-loki.ts**](query-loki.ts) | Query Loki for logs; `--compare-with FILE` for exact entry-by-entry match against a log file | `npx tsx query-loki.ts <service-name> --compare-with logs/dev.log` |
| [**query-prometheus.ts**](query-prometheus.ts) | Same, for Prometheus metrics | `npx tsx query-prometheus.ts <service-name> --compare-with logs/dev.log` |
| [**query-tempo.ts**](query-tempo.ts) | Same, for Tempo traces | `npx tsx query-tempo.ts <service-name> --compare-with logs/dev.log` |
| [**check-connection.ts**](check-connection.ts) | Quick connectivity/credential smoke test, no comparison | `npx tsx check-connection.ts` |
| [**probe-otlp-ingest.ts**](probe-otlp-ingest.ts), [**probe-tempo-prometheus.ts**](probe-tempo-prometheus.ts) | One-off debugging probes used while first mapping this stack's endpoints — see `INVESTIGATE-grafana-cloud-validator.md` | ad hoc |
| [**generate-e2e-env.ts**](generate-e2e-env.ts) | Generates `typescript/test/e2e/company-lookup/.env.grafana-cloud` from this directory's `.env` | `npx tsx generate-e2e-env.ts` |
| [**full-consistency-check.sh**](full-consistency-check.sh) | **The single command for "does this actually work end-to-end."** Runs the E2E test, validates the log file, then runs all three `--compare-with` checks in sequence — see below. | `./full-consistency-check.sh [--env-file PATH]` |

**Common flags on all three query scripts:** `--json`, `--compare-with FILE` (the strongest check — cross-checks every entry against a log file by `trace_id`/`event_id`, real mismatches fail loudly), `--limit N`, `--time-range R`.

---

## `full-consistency-check.sh` — the real verification flow, in one command

Matches how this project's maintainer originally verified changes by hand: write to a local file first (a ground truth you can inspect directly), validate the file, then read back from each backend and diff the response against that file — not just check that a query returned "something."

```bash
./full-consistency-check.sh
```

1. Runs `typescript/test/e2e/company-lookup`'s E2E test against Grafana Cloud — generates `logs/dev.log`.
2. Validates `dev.log`'s format against the JSON Schema.
3. Queries Loki, Prometheus, and Tempo, each with `--compare-with` against that same file. Tempo's search index lags behind Loki/Prometheus (confirmed empirically) — this step polls with backoff (up to ~60s) rather than failing on the first attempt.

Exit code 0 only if every step passes. **This is the gate referenced in `PLANS.md`: before a change to `typescript/src/**.ts` is pushed to main, this script must exit 0** — it now runs for real in CI too, see below.

By default it uses your own personal dev credentials (`.env` in this directory, `.env.grafana-cloud` in the E2E test directory). The `GRAFANA_CLOUD_*` env vars must already be present in the environment before calling it — it never sources a `.env` file itself, so it can't silently override credentials you set up on purpose. Use `--env-file PATH` to point the E2E test step at different application credentials (e.g. CI's own).

## CI's own consistency check

`.github/workflows/ci.yml`'s `grafana-cloud-consistency` job runs a two-stage check on every push/PR, using dedicated CI-only credentials (`sovdev-ci-ingest`/`sovdev-ci-verify`) — entirely separate from the maintainer's personal dev keys and from any customer's, same reasoning as every other Access Policy in this project:

1. **Fail fast**: `sovdev-selftest --backend grafana-cloud` — the exact tool a real customer runs to verify their own connection. No point spending ~2 minutes on the full E2E test if a basic write+read-back against this same backend doesn't even work.
2. **Full consistency check**: this script, only if step 1 passes.

See [Testing against Grafana Cloud](https://sovdev-logger.sovereignsky.no/contributor/testing/grafana-cloud)'s "CI's own consistency check" section for the full setup story (how the keys were created, how they're stored, how to rotate them).
166 changes: 166 additions & 0 deletions tools/validation/grafana-cloud/full-consistency-check.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
#!/bin/bash
# filename: tools/validation/grafana-cloud/full-consistency-check.sh
# description: The single command for "does this actually work end-to-end against Grafana Cloud"
#
# Purpose:
# Runs the complete verification flow in one command, matching how this
# project's maintainer originally verified changes by hand: write to a
# local file first (ground truth), validate the file, then read back from
# each backend (Loki, Prometheus, Tempo) and diff the response against
# that same file, field by field. A tool's own "found it" report is not
# proof by itself -- only a field-by-field match against a known-good
# local record is.
#
# 1. Run typescript/test/e2e/company-lookup's E2E test against Grafana
# Cloud -- generates logs/dev.log (the ground truth).
# 2. Validate dev.log's format against the JSON Schema.
# 3. Query Loki, Prometheus, and Tempo, each with --compare-with against
# that same dev.log -- fails loudly on any mismatch, not just "no data
# found".
#
# Usage:
# ./full-consistency-check.sh [--env-file PATH]
#
# --env-file PATH Passed through to run-test.sh's own --env-file (path
# relative to typescript/test/e2e/company-lookup/).
# Defaults to .env.grafana-cloud (the maintainer's own
# personal dev credentials). Use this to point at a
# different backend's credentials (e.g. CI's own) --
# see terchris/sovdev-ci-grafana.env for that file's
# Group 1 half.
#
# Environment:
# The GRAFANA_CLOUD_* vars (see .env.example) must already be present in
# the environment before calling this script -- either by sourcing
# tools/validation/grafana-cloud/.env yourself first (the maintainer's own
# personal dev credentials), or by exporting a different backend's values
# (e.g. CI's own, from terchris/sovdev-ci-grafana.env's Group 2 half, or
# from real GitHub Actions secrets). This script never sources a .env file
# itself, specifically so it can't silently override credentials the
# caller already set up on purpose.
#
# Exit Codes:
# 0 - Every step passed: the library's real behavior, end-to-end against
# a real backend, matches the local log file exactly.
# 1 - Some step failed -- see output for which one, or required env vars
# were never set.
#
# This is the gate referenced in CLAUDE.md / PLANS.md: before a change to
# typescript/src/**.ts is pushed to main, this script must exit 0.

set -o pipefail

RED='\033[0;31m'
GREEN='\033[0;32m'
BLUE='\033[0;34m'
NC='\033[0m'

print_step() { echo -e "${BLUE}▶ $1${NC}"; }
print_ok() { echo -e "${GREEN}✅ $1${NC}"; }
print_fail() { echo -e "${RED}❌ $1${NC}" >&2; }

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SOVDEV_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
E2E_DIR="${SOVDEV_ROOT}/typescript/test/e2e/company-lookup"
LOG_FILE="${E2E_DIR}/logs/dev.log"

ENV_FILE_ARG=".env.grafana-cloud"
while [[ $# -gt 0 ]]; do
case "$1" in
--env-file)
ENV_FILE_ARG="$2"
shift 2
;;
*)
echo "Unknown argument: $1" >&2
exit 1
;;
esac
done

REQUIRED_VARS=(GRAFANA_CLOUD_INGEST_TOKEN GRAFANA_CLOUD_VERIFY_TOKEN GRAFANA_CLOUD_OTLP_ENDPOINT GRAFANA_CLOUD_OTLP_INSTANCE_ID GRAFANA_CLOUD_LOKI_URL GRAFANA_CLOUD_LOKI_INSTANCE_ID GRAFANA_CLOUD_PROMETHEUS_URL GRAFANA_CLOUD_PROMETHEUS_INSTANCE_ID GRAFANA_CLOUD_TEMPO_URL GRAFANA_CLOUD_TEMPO_INSTANCE_ID)
MISSING=()
for v in "${REQUIRED_VARS[@]}"; do
[ -z "${!v}" ] && MISSING+=("$v")
done
if [ ${#MISSING[@]} -ne 0 ]; then
print_fail "Missing required env vars: ${MISSING[*]}"
echo " Source tools/validation/grafana-cloud/.env (your own dev credentials) yourself" >&2
echo " first, or export a different backend's values (e.g. CI's) before calling this" >&2
echo " script -- it never sources a .env file on its own." >&2
exit 1
fi

FAILED=0

print_step "Step 1/3: running the E2E test against Grafana Cloud (writing ${LOG_FILE})"
(cd "${E2E_DIR}" && ./run-test.sh --skip-validation --env-file "${ENV_FILE_ARG}")
if [ $? -ne 0 ]; then
print_fail "E2E test execution failed"
exit 1
fi
print_ok "E2E test ran, log file written"
echo ""

print_step "Step 2/3: validating ${LOG_FILE}'s format against the schema"
python3 "${SOVDEV_ROOT}/tools/validation/validators/validate-log-format.py" "${LOG_FILE}"
if [ $? -ne 0 ]; then
print_fail "Log file format validation failed"
exit 1
fi
print_ok "Log file format valid"
echo ""

SERVICE_NAME=$(grep -h '^OTEL_SERVICE_NAME=' "${E2E_DIR}/${ENV_FILE_ARG}" 2>/dev/null | tail -1 | cut -d= -f2)
if [ -z "${SERVICE_NAME}" ]; then
# Env file didn't exist or had no OTEL_SERVICE_NAME (e.g. CI, which
# exports vars directly rather than via a checked-in file) -- fall back
# to whatever's already in the environment.
SERVICE_NAME="${OTEL_SERVICE_NAME}"
fi
if [ -z "${SERVICE_NAME}" ]; then
print_fail "Could not determine OTEL_SERVICE_NAME (checked ${E2E_DIR}/${ENV_FILE_ARG} and the environment)"
exit 1
fi

print_step "Step 3/3: reading back from Loki, Prometheus, and Tempo, diffing each against the file"
cd "${SCRIPT_DIR}"

echo ""
echo "--- Loki (logs) ---"
npx tsx query-loki.ts "${SERVICE_NAME}" --compare-with "${LOG_FILE}"
[ $? -ne 0 ] && { print_fail "Loki consistency check failed"; FAILED=1; }

echo ""
echo "--- Prometheus (metrics) ---"
npx tsx query-prometheus.ts "${SERVICE_NAME}" --compare-with "${LOG_FILE}"
[ $? -ne 0 ] && { print_fail "Prometheus consistency check failed"; FAILED=1; }

echo ""
echo "--- Tempo (traces) ---"
# Tempo's search index lags behind Loki/Prometheus (confirmed empirically,
# unlike those two, query-tempo.ts has no built-in retry) -- poll with
# backoff rather than fail on the first attempt, which reliably races a
# fresh write's indexing delay.
TEMPO_OK=1
for attempt in 1 2 3 4 5 6; do
TEMPO_OUTPUT=$(npx tsx query-tempo.ts "${SERVICE_NAME}" --compare-with "${LOG_FILE}" 2>&1)
TEMPO_EXIT=$?
if [ ${TEMPO_EXIT} -eq 0 ]; then
TEMPO_OK=0
break
fi
echo " (attempt ${attempt}/6: not all spans indexed yet, waiting 10s...)"
sleep 10
done
echo "${TEMPO_OUTPUT}"
[ ${TEMPO_OK} -ne 0 ] && { print_fail "Tempo consistency check failed after 6 attempts (~60s)"; FAILED=1; }

echo ""
if [ ${FAILED} -eq 0 ]; then
print_ok "ALL CHECKS PASSED — verified end-to-end against real Grafana Cloud data"
exit 0
else
print_fail "One or more consistency checks failed — see above"
exit 1
fi
Loading
Loading