ci(integration): enforce a skip contract + executed-case floor

haybarn-unittest exits 0 whether one test skipped or every test did — a failed
require/require-env is a SKIP, not an error (and the runner's skip_error_messages
HTTP policy turns http errors into skips too) — so "All tests passed" is not
evidence anything ran. The failure mode is documented in CLAUDE.md (the lost-HTTP
-skip mode, 37048ef) but nothing enforced it; a dead shared worker or an empty
stage would still read green. This is the same detector vgi-typescript has had
from the start and vgi-go just gained.

summarize_run, run after each unittest invocation:

  * Skip allowlist: every reason the runner prints must be in
    EXPECTED_SKIP_REASONS (a base set + http-lane additions, since the bad-enum /
    dedicated-crash / launcher tests RUN on launch but gate off over http). An
    unlisted reason — a whole-file `require httpfs: 227` — fails the lane.
    Reasons captured from run 30041947972.
  * Executed-case floor: the main suite's executed count must exceed a per-lane
    MIN_EXECUTED — launch/shm 255, http 248, ~16 below the 2026-07-24 measured
    271 / 265 against Query-farm/vgi main. A collapse in this number is the
    signature of a silent skip; a floor, not an equality, so upstream growth is
    fine.
  * Empty-stage guard: a "No test cases matched" report fails instead of passing.

run_unittest also folds in the fork-child fatal-signal scan (a SIGTERM handler a
fork()ed child runs against the parent's counters prints a FAILED block the exit
code never sees), matching the rust and typescript harnesses. The gzip_fallback
step runs in a subshell so its env overrides don't leak and its single-test count
doesn't perturb the main-suite floor.

Verified against Query-farm/vgi main: launch 271, shm 271, http 265 executed —
all above floors, no unexpected skips. Negative tests confirm both guards bite:
an injected `require httpfs` skip fails as unexpected while allowlisted reasons
pass, and an executed count below the floor errors.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: rustyconover

ci/README.md

@@ -71,6 +71,33 @@ artifact). The HTTP / bearer / dynamic-code /
 `schema_reconcile` tests skip via their `require-env` gates (we don't set those
 workers), exactly as in the reference harness.
 
+## The skip contract + executed-case floor
+
+`haybarn-unittest` exits 0 whether one test skipped or every test did — a failed
+`require` / `require-env` is a **skip**, not an error (compounded here by the
+runner's `skip_error_messages HTTP` policy, which also turns http errors into
+skips). So "All tests passed" on its own is not evidence anything ran: a dead
+shared worker, an empty stage, or a mis-wired env var all read as green while the
+suite quietly tested nothing. `run-integration.sh` closes that gap with two
+guards in `summarize_run`:
+
+- **Skip allowlist.** Every skip reason the runner prints must appear in
+  `EXPECTED_SKIP_REASONS` (a base list plus, on the http lane, the bad-enum /
+  dedicated-crash / launcher reasons that gate off there but *run* on launch). An
+  unlisted reason — the signature of a whole file silently gated off, like
+  `require httpfs: 227` — fails the lane. A genuinely new, legitimate skip is
+  added to the list *with the reason why*.
+- **Executed-case floor.** The main suite's executed-case count must stay above a
+  per-lane `MIN_EXECUTED` (launch/shm 255, http 248 — floors ~16 below the
+  measured 271 / 265, leaving room for upstream growth). A collapse in this
+  number is the tell of a suite-wide silent skip. It is a floor, not an equality:
+  **do not lower it to make a run pass** — find what stopped running. Ported from
+  vgi-typescript, whose harness had this from the start.
+
+`run_unittest` also scans for the `fatal error condition` block a fork()ed child
+prints against the parent's counters — a failure invisible to the exit code by
+construction.
+
 ## Run it locally
 
 ```bash

ci/run-integration.sh

@@ -178,6 +178,60 @@ case "$TRANSPORT" in
     echo "::error::unknown TRANSPORT=$TRANSPORT (expected launch|http)"; exit 1 ;;
 esac
 
+# ---------------------------------------------------------------------------
+# Skip contract + executed-case floor.
+#
+# haybarn-unittest exits 0 whether one test skipped or every test did: a failed
+# `require` / `require-env` is a SKIP, not an error. So "All tests passed" is not
+# evidence anything ran — a dead shared worker, an empty stage, or a mis-wired
+# env var all read as green while the suite quietly tested nothing. Two guards
+# close that gap (summarize_run, below):
+#
+#   * every skip reason must be named in EXPECTED_SKIP_REASONS — an unlisted
+#     reason (e.g. a whole-suite `require httpfs`) fails the lane;
+#   * the count of executed cases must stay above MIN_EXECUTED — the signature
+#     of a silent collapse is this number falling off a cliff.
+#
+# The strings are the exact reasons the runner prints under "Skipped tests for
+# the following reasons:". These are fixtures / lanes this port deliberately does
+# not stand up; each is a legitimate, load-bearing skip.
+EXPECTED_SKIP_REASONS=(
+  'require-env VGI_BAD_PROTOCOL_WORKER'          # incompatible-protocol fixture, not wired here
+  'require-env VGI_RULES_WORKER'                 # vgi-rust multibatch-repro fixture
+  'require-env VGI_SCHEMA_RECONCILE_DB'          # schema-reconcile sqlite fixture, not set here
+  'require-env VGI_WORKER_SUPPORTS_DYNAMIC_CODE' # dynamic-code registration, not implemented
+  'require-env VGI_HTTP_TRANSPORT'               # http-identity tests; this port never sets the flag
+  'require-env VGI_HTTP_DISABLE_ZSTD'            # gzip-fallback; skips in the main run, runs in its own step
+  'require-env VGI_HTTP_NO_COMPRESSION'          # no-compression fixture server (Python-side)
+  'require-env VGI_TEST_BEARER_TOKEN'            # bearer-auth fixture server
+  'require-env VGI_TEST_COMPANION_TARGET'        # companion-catalog fixture (Python-side)
+  'require-env VGI_TEST_ICEBERG'                 # iceberg fixture, not stood up here
+  'require-env VGI_DOCKER_IMAGE'                 # containerised worker lane
+  'require-env VGI_DOCKER_TCP_IMAGE'             # containerised worker over TCP
+  'require-env VGI_GITHUB_NETWORK_TESTS'         # hits github.com; opt-in only
+)
+# Lane-specific additions — a skip expected on one lane is a red flag on another.
+# The launch lane wires the bad-enum / dedicated-crash / launcher workers, so
+# those tests RUN there; over http they gate off (shared server / no launcher).
+if [ "$TRANSPORT" = "http" ]; then
+  EXPECTED_SKIP_REASONS+=(
+    'require-env VGI_BAD_ENUM_WORKER'
+    'require-env VGI_REQUIRE_LAUNCHER_TRANSPORT'
+    'require-env VGI_TEST_DEDICATED_WORKER'
+  )
+fi
+
+# Floor on executed cases in the main suite invocation. Deliberately a floor, not
+# an equality — the upstream suite grows. Measured 2026-07-24 against
+# Query-farm/vgi main: the launch (and shm, which runs TRANSPORT=launch) lane
+# executes 271, the http lane 265. The floors sit ~16 below, leaving headroom for
+# churn while staying far above the handful a silent collapse would leave.
+if [ "$TRANSPORT" = "http" ]; then
+  MIN_EXECUTED="${MIN_EXECUTED:-248}"
+else
+  MIN_EXECUTED="${MIN_EXECUTED:-255}"
+fi
+
 cd "$STAGE"
 
 echo "Warming the extension cache (vgi from community, deps from core) ..."
@@ -203,26 +257,106 @@ EOF
 "$HAYBARN_UNITTEST" "test/_warm.test" >/dev/null 2>&1 || echo "::warning::extension warm step did not fully succeed"
 rm -f "$STAGE/test/_warm.test"
 
-# Run the whole suite in ONE unittest invocation (as `make test_launcher`
-# does), streaming the runner's native sqllogictest report: a `[i/N] (..%):
-# test/...` progress line per file and the final
-# `All tests passed (.. N assertions in M test cases)` summary (a failure
-# prints the offending query + a `M test cases | K failed` summary). This keeps
-# the CI log showing that the tests actually ran — and how many assertions —
-# rather than a rolled-up count. Out-of-scope tests were already dropped at
-# staging, so the glob never matches them; any failed assertion exits non-zero
-# and fails the job (via `set -e`).
+# summarize_run <log> — parse the runner's console report and enforce the skip
+# contract + record the executed-case count. Returns non-zero on an unexpected
+# skip reason or a runner that matched no tests at all (the empty-stage
+# signature); a suite-assertion failure is caught by the exit code (below).
+#
+#   total    = the N in the last "[i/N] (..%):" progress line (cases staged)
+#   skipped  = the sum of the "Skipped tests for the following reasons:" block
+#   executed = total - skipped
+MAIN_EXECUTED=0
+summarize_run() {
+  local log="$1" total skipped executed rc=0 reason
+  if grep -q 'No test cases matched\|No tests ran' "$log"; then
+    echo "::error::the runner matched no test cases — the glob or the staging is wrong" \
+         "(an empty stage still exits 0). transport=$TRANSPORT"
+    return 1
+  fi
+  total="$(sed -n 's/^\[[0-9]*\/\([0-9]*\)\].*/\1/p' "$log" | tail -1)"
+  [ -n "$total" ] || { echo "::error::could not parse a test-case total out of the report"; return 1; }
+  skipped="$(awk '
+    /^Skipped tests for the following reasons:/ { in_block = 1; next }
+    in_block && /^[[:space:]]*$/               { in_block = 0; next }
+    in_block && match($0, /: [0-9]+[[:space:]]*$/) { n += substr($0, RSTART + 2) }
+    END { print n + 0 }' "$log")"
+  executed=$(( total - skipped ))
+  echo "  executed: $executed / $total   skipped: $skipped   (transport=$TRANSPORT)"
+  # Every skip reason must be on the allowlist; an unlisted one fails the lane.
+  if [ "$skipped" -gt 0 ]; then
+    while IFS= read -r reason; do
+      [ -n "$reason" ] || continue
+      if printf '%s\n' "${EXPECTED_SKIP_REASONS[@]}" | grep -qxF "$reason"; then
+        continue
+      fi
+      echo "::error::unexpected skip reason '$reason'. Either a gate regressed and a" \
+           "whole file silently stopped running, or this skip is legitimate — if so add" \
+           "it to EXPECTED_SKIP_REASONS in ci/run-integration.sh with the reason why."
+      rc=1
+    done < <(awk '
+      /^Skipped tests for the following reasons:/ { in_block = 1; next }
+      in_block && /^[[:space:]]*$/               { in_block = 0; next }
+      in_block && match($0, /: [0-9]+[[:space:]]*$/) { print substr($0, 1, RSTART - 1) }' "$log")
+  fi
+  MAIN_EXECUTED="$executed"
+  return "$rc"
+}
+
+# run_unittest <glob...> — run one haybarn-unittest invocation, streaming its
+# report, then fail on anything its exit code cannot express: an unexpected skip,
+# an empty stage, or a fatal-signal report a fork()ed child printed against the
+# parent's counters (invisible to the exit code by construction).
+run_unittest() {
+  local log rc=0
+  log="$(mktemp)"
+  # `&& rc=0 || rc=${PIPESTATUS[0]}` keeps the suite's own exit code without
+  # tripping errexit and without a trailing `|| true` (which, as a new simple
+  # command, would overwrite PIPESTATUS with 0 — the accounting must still run
+  # when the suite itself failed).
+  "$HAYBARN_UNITTEST" "$@" 2>&1 | tee "$log" && rc=0 || rc="${PIPESTATUS[0]}"
+  if grep -q 'due to a fatal error condition' "$log"; then
+    echo "::error::a forked child ran the test harness's signal handler (see the" \
+         "'fatal error condition' block above). The parent exited $rc and would" \
+         "otherwise have passed. This is invisible to the exit code by construction."
+    rc=1
+  fi
+  summarize_run "$log" || rc=1
+  rm -f "$log"
+  return "$rc"
+}
+
+# Run the whole suite in ONE invocation (as `make test_launcher` does), streaming
+# the runner's native sqllogictest report: a `[i/N] (..%): test/...` progress
+# line per file and the final `All tests passed (.. N assertions in M test
+# cases)` summary. Out-of-scope tests were dropped at staging, so the glob never
+# matches them; any failed assertion, unexpected skip, or executed-case collapse
+# fails the job. run_unittest returns non-zero on failure → `set -e` aborts.
 echo "Running suite (single invocation — native sqllogictest report) ..."
-"$HAYBARN_UNITTEST" "test/sql/integration/*"
+run_unittest "test/sql/integration/*"
 
 # gzip_fallback.test needs VGI_TEST_WORKER itself to be a zstd-disabled HTTP
 # worker (it attaches it directly and asserts the gzip codec fallback). That
 # conflicts with the main suite's launch: worker, so run it as a dedicated
 # single-test invocation against its own worker. Only on the launch lane (shm is
-# the same coverage; the http lane is local-only).
+# the same coverage; the http lane is local-only). The subshell keeps the env
+# overrides from leaking (a `VAR=v func` prefix persists in bash, unlike
+# `VAR=v some_binary`) and isolates the summarize_run accounting — the floor
+# below is checked against the MAIN suite, not this single-test run.
 if [ "$TRANSPORT" = "launch" ]; then
   echo "Running http/gzip_fallback.test (zstd-disabled http worker) ..."
   gz_port="$(VGI_HTTP_DISABLE_ZSTD=1 boot_http_worker "$VGI_WORKER_BIN")"
-  VGI_TEST_WORKER="http://localhost:${gz_port}" VGI_HTTP_DISABLE_ZSTD=1 \
-    "$HAYBARN_UNITTEST" "test/sql/integration/http/gzip_fallback.test"
+  ( export VGI_TEST_WORKER="http://localhost:${gz_port}" VGI_HTTP_DISABLE_ZSTD=1
+    run_unittest "test/sql/integration/http/gzip_fallback.test" )
+fi
+
+# Executed-case floor on the main suite — the collapse-detector. Reached only if
+# the runs above passed (a failure already aborted via set -e); MAIN_EXECUTED
+# holds the main invocation's count (the gzip subshell does not touch it).
+if [ "$MAIN_EXECUTED" -lt "$MIN_EXECUTED" ]; then
+  echo "::error::only $MAIN_EXECUTED test cases executed on the $TRANSPORT lane," \
+       "floor is MIN_EXECUTED=$MIN_EXECUTED. This is the signature of a suite-wide" \
+       "silent skip (a failed require is a SKIP, not an error). Do NOT lower the floor" \
+       "to make this pass — find what stopped running."
+  exit 1
 fi
+echo "Executed $MAIN_EXECUTED main-suite test cases (floor $MIN_EXECUTED) on the $TRANSPORT lane."