Feat/unified configuration

.agents/skills/srql-fixtures-db-tests/SKILL.md

@@ -86,11 +86,11 @@ export SRQL_FIXTURE_SSLMODE=verify-full
 export PGSSLSERVERNAME=srql-fixture-rw.srql-fixtures.svc.cluster.local
 export SRQL_TEST_DATABASE_SERVER_NAME="$PGSSLSERVERNAME"
 
-run_entropy="${HOSTNAME:-workstation}-$(date +%s)-$$-${RANDOM:-0}"
-run_checksum="$(printf '%s' "$run_entropy" | cksum | awk '{print $1}')"
-export GITHUB_RUN_ID="$(date +%s)${run_checksum}"
-export GITHUB_RUN_ATTEMPT=1
-COMMON=(-c opt --//build:enable_integration_tests)
+# The run correlation id: minted ONCE and passed to every invocation below, because they are
+# separate bazel commands that share no process and must agree on one disposable database name
+# while not colliding with anyone else's run. It has no default -- see //build/run_id.bzl.
+RUN_ID="$(uuidgen | tr -d - | tr 'A-Z' 'a-z' | cut -c1-8)"
+COMMON=(-c opt --//build:enable_integration_tests "--//build:run_id=$RUN_ID")
 if [ -f .bazelrc.remote ]; then
   COMMON+=(--config=cache_only)
 fi
@@ -217,6 +217,45 @@ cd elixir/serviceradar_core
 MIX_ENV=test mix compile --warnings-as-errors
 ```
 
+## Probe The Endpoint Without A CI Run
+
+A CI cycle is an expensive way to learn that a hostname does not resolve. Any lifecycle target
+run with a DELIBERATELY WRONG password answers reachability in about 30 seconds, because the
+error tells you exactly how far the connection got:
+
+```bash
+SERVICERADAR_ENV=ci \
+SERVICERADAR_SECRET_DATABASE_PASSWORD=wrong-on-purpose \
+SERVICERADAR_SECRET_DATABASE_ADMIN_PASSWORD=wrong-on-purpose \
+bazel test --config=remote \
+  --//build:enable_integration_tests --//build:run_id=diag0001 \
+  --test_env=SERVICERADAR_ENV \
+  --test_env=SERVICERADAR_SECRET_DATABASE_PASSWORD \
+  --test_env=SERVICERADAR_SECRET_DATABASE_ADMIN_PASSWORD \
+  --test_output=all --nocache_test_results \
+  //rust/integration-db:sweep_stale_dbs
+```
+
+`--config=remote` is what makes this a probe rather than a local run: the test action executes
+in an OCI container on a cluster executor, which is the same kind of network namespace the
+BuildBuddy workflow runner gives a local test action. A workstation is NOT a substitute -- the
+fixture LoadBalancer is announced on the cluster's L2 and is unreachable from outside it.
+
+Read the outcome from the failure, all of which now name the endpoint, role, database, TLS mode
+and CA bundle:
+
+| Error | Meaning |
+| --- | --- |
+| `password authentication failed for user "..."` | Everything works: DNS, route, CA fetch, TLS verification, and the role exists. Only the password was wrong. |
+| `failed to lookup address information` | The name does not resolve in that namespace. A `.svc.cluster.local` host will always fail here. |
+| `Network is unreachable` / `connection refused` | The name resolves; the address does not route from this caller. |
+| `role "..." does not exist` | `database.connecting_role` / `admin_role` names a role the fixture does not have. |
+| `fetch CA bundle <url>` | The CA endpoint, not the database. |
+
+USE A THROWAWAY PASSWORD, never the real one. A remote action ships its environment to the
+executor and BuildBuddy records action metadata; the fixture password stays out of that. This is
+also why the CI lifecycle keeps `--strategy=TestRunner=local` for the runs that must succeed.
+
 ## Common Failures
 
 - `pg_hba.conf rejects ... no encryption`: require TLS and provide the fixture CA.

.bazelrc

@@ -9,8 +9,6 @@ common --noannounce_rc
 common --experimental_convenience_symlinks=clean
 common --color=yes
 common --ui_event_filters=-INFO
-# Do NOT put repository_cache / disk_cache / distdir under the source tree.
-# Use Bazel defaults (~/.cache/bazel) or machine-local .bazelrc.local instead.
 # Allow platform-specific build:* sections (linux/macos/windows).
 common --enable_platform_specific_config
 # Mirror-first URL rewrites for deps that ship a single download URL. See the file.
@@ -22,6 +20,11 @@ common --downloader_config=tools/bazel/downloader.cfg
 common --noslim_profile
 # common --experimental_profile_include_target_label
 # common --experimental_profile_include_primary_output
+# Do NOT put repository_cache / disk_cache / distdir under the source tree.
+# Use Bazel defaults (~/.cache/bazel) or machine-local .bazelrc.local instead.
+
+# This is a deliberate exception to the "no distdir under the source tree" rule above. These are immutable vendored source crates.
+common --distdir=%workspace%/third_party/crate_mirror
 
 build --keep_going
 build --jobs=auto
@@ -31,12 +34,7 @@ build --action_env=NEXT_PRIVATE_SKIP_TURBOPACK=1
 build --action_env=TURBOPACK=
 build --action_env=NEXT_SKIP_TURBO=1
 
-# NO OPENSSL_DIR / OPENSSL_LIB_DIR / OPENSSL_INCLUDE_DIR HERE, and no AARCH64_* variants.
-# NO CC_/CXX_/AR_/CARGO_TARGET_*_LINKER HERE. Compiler config is all done via the Hermetic LLVM toolchain in MODULE.bazel.
-# This is a deliberate exception to the "no distdir under the source tree" rule above.
-# These are immutable vendored source crates.
-common --distdir=%workspace%/third_party/crate_mirror
-
+# Can these be removed as we build hermetic??
 build --action_env=PKG_CONFIG_SYSROOT_DIR_aarch64_unknown_linux_gnu=/
 build --action_env=PKG_CONFIG_PATH_aarch64_unknown_linux_gnu=/usr/lib/postgresql/18/lib/pkgconfig:/usr/lib/aarch64-linux-gnu/pkgconfig:/usr/lib/pkgconfig:/usr/share/pkgconfig
 
@@ -59,77 +57,7 @@ build --@io_bazel_rules_go//go/config:pure
 
 # Standard test ergonomics: show failing logs and retry known flaky tests once.
 test --test_output=errors
-# Deliberately NOT set. --test_strategy=standalone pins every test to local, unsandboxed
-# execution, and nothing in build:ci undid it -- so under --config=ci the build went to the
-# executors while the tests stayed on the runner, and a target's exec_properties (Firecracker
-# size, init-dockerd) were inert because the test never left the runner.
-#
-# Unset, Bazel picks the strategy from the available execution platforms: remote under
-# --config=ci, local otherwise -- the `test --remote_executor=` line below clears the
-# executor for a bare `bazel test`.
-#
-# Note the second effect locally: tests now run sandboxed rather than standalone. A test that
-# needs the sandbox off must say so itself with tags = ["no-sandbox"], the way
-# //elixir/web-ng:precommit already does.
-#test --test_strategy=standalone
 test --remote_executor=
-# THE FIXTURE ENVIRONMENT IS OPT-IN. These values include live DSNs and client-key paths.
-# Forwarding them from the global `test` profile puts them in every remote unit-test action's
-# command environment and cache metadata, even when that test never opens a database. Only the
-# guarded, TestRunner-local, non-uploaded database invocations select `--config=database_env`.
-#
-# Keep SRQL_TEST_* as the base URLs. SERVICERADAR_TEST_DATABASE_URL and
-# SERVICERADAR_TEST_ADMIN_URL are intentionally absent: those are final per-test overrides and
-# suppress integration_env.exs' per-shard database-name derivation when inherited accidentally.
-test:database_env --test_env=SRQL_TEST_DATABASE_URL
-test:database_env --test_env=SRQL_TEST_ADMIN_URL
-test:database_env --test_env=SRQL_TEST_DATABASE_CA_CERT
-test:database_env --test_env=SRQL_TEST_DATABASE_CA_CERT_FILE
-test:database_env --test_env=SRQL_TEST_DATABASE_CERT
-test:database_env --test_env=SRQL_TEST_DATABASE_KEY
-test:database_env --test_env=SRQL_TEST_DATABASE_SERVER_NAME
-test:database_env --test_env=SERVICERADAR_TEST_DATABASE_CA_CERT
-test:database_env --test_env=SERVICERADAR_TEST_DATABASE_CA_CERT_FILE
-test:database_env --test_env=SERVICERADAR_TEST_DATABASE_CERT
-test:database_env --test_env=SERVICERADAR_TEST_DATABASE_KEY
-test:database_env --test_env=SERVICERADAR_TEST_DATABASE_OWNER
-test:database_env --test_env=SERVICERADAR_TEST_DATABASE_SERVER_NAME
-test:database_env --test_env=PGSSLROOTCERT
-test:database_env --test_env=PGSSLSERVERNAME
-test:database_env --test_env=PGSSLCERT
-test:database_env --test_env=PGSSLKEY
-test:database_env --test_env=CNPG_HOST
-test:database_env --test_env=CNPG_PORT
-test:database_env --test_env=CNPG_DATABASE
-test:database_env --test_env=CNPG_USERNAME
-test:database_env --test_env=CNPG_PASSWORD
-test:database_env --test_env=CNPG_ADMIN_USERNAME
-test:database_env --test_env=CNPG_ADMIN_PASSWORD
-test:database_env --test_env=CNPG_SSL_MODE
-test:database_env --test_env=CNPG_CERT_DIR
-test:database_env --test_env=CNPG_CA_FILE
-test:database_env --test_env=CNPG_CERT_FILE
-test:database_env --test_env=CNPG_KEY_FILE
-test:database_env --test_env=CNPG_TLS_SERVER_NAME
-test:database_env --test_env=TEST_CNPG_HOST
-test:database_env --test_env=TEST_CNPG_PORT
-test:database_env --test_env=TEST_CNPG_DATABASE
-test:database_env --test_env=TEST_CNPG_USERNAME
-test:database_env --test_env=TEST_CNPG_PASSWORD
-test:database_env --test_env=GITHUB_RUN_ID
-test:database_env --test_env=GITHUB_RUN_ATTEMPT
-# NATS URLs and key material are likewise opt-in. Database integration inherits this profile
-# because a subset of the core shards exercises the event pipeline; other NATS-facing callers
-# must select it explicitly rather than leaking credentials into the generic remote sweep.
-test:database_env --config=nats_env
-test:nats_env --test_env=NATS_URL
-test:nats_env --test_env=NATS_SERVER_NAME
-test:nats_env --test_env=NATS_CA_FILE
-test:nats_env --test_env=NATS_CERT_FILE
-test:nats_env --test_env=NATS_KEY_FILE
-test:nats_env --test_env=NATS_CA_B64
-test:nats_env --test_env=NATS_CERT_B64
-test:nats_env --test_env=NATS_KEY_B64
 test --@io_bazel_rules_go//go/config:pure
 test --flaky_test_attempts=2
 
@@ -155,6 +83,7 @@ build:remote_base --config=cache_only
 # Number of parallel jobs on remote executors
 build:remote_base --jobs=100
 build:remote_base --remote_download_minimal
+
 build:remote_base --remote_executor=grpcs://carverauto.buildbuddy.io
 # Remote platforms
 build:remote_base --host_platform=//build/rbe:rbe_platform
@@ -175,16 +104,32 @@ build:remote_base --remote_timeout=15m
 build:remote_base --test_output=errors
 build:remote_base --noenable_platform_specific_config
 
-# CI profile on top of the remote config
+# CI profile on top of the remote config. Client cache paths are set per profile below and
+# never in remote_base -- setting them in both made Bazel warn that --disk_cache was
+# expanded twice from --config=ci.
 build:ci --config=remote_base
+# The Bazel CLIENT's caches, on the node volume //k8s/buildbuddy bind-mounts into the workflow
+# runner at /bazel-cache, so a fresh runner does not re-download and re-extract every external
+# repo. The repo caches are the ones that keep toolchains off the network; the disk cache is
+# action outputs. CI-only -- this path exists on the executors and nowhere else, and the init
+# container in //k8s/buildbuddy/values-workflows.yaml is what makes it writable by the runner.
+build:ci --disk_cache=/bazel-cache/disk
+build:ci --repository_cache=/bazel-cache/repo
+build:ci --repo_contents_cache=/bazel-cache/repo-contents
+build:ci --experimental_disk_cache_gc_max_size=100G
 build:ci --keep_going
 build:ci --build_metadata=REPO_URL=https://github.com/carverauto/serviceradar.git
 build:ci --build_metadata=ROLE=CI
 build:ci --build_metadata=VISIBILITY=PUBLIC
 build:ci --build_metadata=USER=gh
 
-# `--config=remote` is an ALIAS for remote_base
+# `--config=remote` is remote_base plus workstation-local cache paths. `make test` uses this
+# profile, so the paths must exist on a developer machine -- /bazel-cache does not.
 build:remote --config=remote_base
+build:remote --disk_cache=~/.cache/serviceradar-bazel/disk
+build:remote --repository_cache=~/.cache/serviceradar-bazel/repo
+build:remote --repo_contents_cache=~/.cache/serviceradar-bazel/repo-contents
+build:remote --experimental_disk_cache_gc_max_size=50G
 
 build:darwin_pkg --platforms=//build/platforms:darwin_exec
 build:darwin_pkg --host_platform=//build/platforms:darwin_exec

.gitignore

@@ -299,3 +299,5 @@ __pycache__/
 # If a future pattern ever does catch an archive, re-add a narrow negation for THAT pattern
 # rather than a directory wildcard. The wildcard is what makes an un-ignore dangerous: it
 # re-admits everything the rest of this file is trying to exclude.
+.local-fixture-certs/
+.local-secrets/

AGENTS.md

@@ -142,7 +142,9 @@ This file applies repo-wide, but subdirectories may include their own `AGENTS.md
 ## Build & Test Commands
 
 - **Every unit test, the way CI runs them: `make test`** — an alias for
-  `bazel test -c opt --config=ci //... --test_tag_filters=-integration_test,-acceptance_test`.
+  `bazel test -c opt --config=remote //... --test_tag_filters=-integration_test,-acceptance_test`.
+  `--config=remote`, not `--config=ci`: the CI profile points its caches at `/bazel-cache`, the
+  node volume only the BuildBuddy executors mount, so it cannot run on a workstation.
   **Run this before opening a PR and before cutting any release.** It is the only command
   that covers the whole repo, because the Elixir unit shards exist ONLY as bazel targets
   (`//elixir/serviceradar_core:unit_tests_*`, `//elixir/web-ng:unit_tests_*`) and are
@@ -856,8 +858,10 @@ the eight-shard suite. Every test/lifecycle invocation needs
 database tests, and always invoke `teardown_db` after a red shard. Bazel has no cross-invocation
 finalizer; the stale sweep is the backstop for a killed host.
 
-Keep fixture base URLs in `SRQL_TEST_DATABASE_URL` and `SRQL_TEST_ADMIN_URL`, set one unique
-numeric `GITHUB_RUN_ID`/`GITHUB_RUN_ATTEMPT` pair for the whole sequence, and leave
+Keep fixture base URLs in `SRQL_TEST_DATABASE_URL` and `SRQL_TEST_ADMIN_URL`, mint ONE run id
+for the whole sequence and pass it to every invocation as `--//build:run_id=<id>` (8-32 chars of
+`[a-z0-9]`; it has no default, because a constant fallback let two runs share one database), and
+leave
 `SERVICERADAR_TEST_DATABASE_URL` unset so each shard derives its disposable database. When using
 a NodePort, export both `PGSSLSERVERNAME` and `SRQL_TEST_DATABASE_SERVER_NAME` with the CNPG
 certificate's DNS name so the Rust and Elixir clients verify the same certificate.