A BigQuery profile estimate reserves for an escalation query only where the probe could actually issue one (#302)

* Add constants for value domain limitations in base adapter

* Refactor BigQuery adapter to enhance estimate composition with finer breakdown and reserved queries tracking

* Enhance cost gate with OverCeilingError handling and reserve composition for improved estimate transparency

* Refactor profile to reuse value domain constants from base adapter

* Add unit tests for value domain and reserve behavior in profiling estimates

* Add safety_spine test for BigQuery adapter profiling to ensure reserve narrowing aligns with billing estimates

* Update docs

* Cut release 1.6.4

Author: marcociav-exmergo

.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "dex",
   "description": "Analytics engineering for Claude Code and any agent: data warehouse exploration, dbt transformation and semantic modeling, and schema-drift maintenance on dbt.",
-  "version": "1.6.3",
+  "version": "1.6.4",
   "author": {
     "name": "Exmergo, Inc.",
     "email": "support@exmergo.com",

CHANGELOG.md

@@ -9,6 +9,8 @@ tag releases both in lockstep, so entries below are keyed by the engine version.
 
 ## [Unreleased]
 
+## [1.6.4] - 2026-08-13
+
 ### Added
 
 - **A verified join with a catastrophic orphan rate is now a finding, not
@@ -168,6 +170,45 @@ tag releases both in lockstep, so entries below are keyed by the engine version.
   profiled still needs a connection to build its sample, so it reaches the
   opener at the bottom of `cluster` and raises there.
 
+### Fixed
+
+- **A BigQuery profile estimate reserves for an escalation query only where the
+  probe could actually issue one, and says how much of itself is reserve**
+  ([#299]). Since 1.6.0 the estimate held three per-table 10 MB floors rather
+  than two: the value-domain probe added in that release
+  ([#203]) took a reserve beside the near-unique and composite ones. The reserve
+  scales with object count rather than data size, so on a warehouse of many
+  small tables the release moved a 12-object `explore map` estimate by 125.8 MB
+  in one step and started refusing a nightly refresh that had run for months.
+
+  Three of the reserves were provably unspendable rather than merely unlikely,
+  and are now dropped. A view has no row count (BigQuery reports none, and the
+  profiling aggregate's own `COUNT(*)` is read per batch and never written back
+  to the object), and all three probes return early without one, so a view held
+  three floors no run could ever spend. Nested and repeated columns get no
+  approximate distinct in the aggregate batch, and every probe's eligibility
+  starts from one, so a table of nothing else can no more escalate than a blob
+  column already excluded from the scan; the composite reserve was already
+  conditioned on having two columns, but counted columns that can never join a
+  pair. And a value domain needs at least one distinct value within a tenth of
+  the non-null rows, so no column of a table below ten rows can qualify. The
+  thresholds that imply that floor moved next to `ValueDomainSample` so the
+  estimator and the probe cannot drift apart. Nothing else was narrowed: the
+  reserve is dropped only where a probe's own guard already rules the query out,
+  because an estimate that reserves for a query that cannot run is merely loose,
+  while one that skips a query that can is the defect [#107] closed.
+
+  That leaves the common case, a warehouse of small flat tables, reserving
+  exactly as much as before, which is the second half of this. The confirm
+  handshake and the over-ceiling refusal now split the estimate into measured
+  dry-run scan and held reserve, in prose and in `reserved_bytes` /
+  `reserved_queries`. A refusal previously said only "raise the budget or narrow
+  the work" about a number that could be three quarters reserve, leaving the
+  operator to reconstruct the split from `.dex/spend.jsonl` afterwards to find
+  out whether the estimate grew because the warehouse did or because dex added a
+  probe. Over-ceiling refusals reach the connector's own description of the
+  estimate for the first time; before this only the confirmation payload did.
+
 ## [1.6.3] - 2026-08-10
 
 ### Changed

packages/dex-core/src/exmergo_dex_core/adapters/base.py

@@ -23,6 +23,7 @@
 from dataclasses import dataclass
 from datetime import date, datetime, time
 from decimal import Decimal
+from math import ceil
 from typing import Protocol, runtime_checkable
 
 from ..envelope import Paradigm
@@ -123,6 +124,24 @@ class ValueDomainSample:
     total_distinct: int
 
 
+# A column's value domain is reported only when its distinct count clears BOTH
+# bars: small absolutely (the cap) and small relative to the table (the
+# fraction), so a tiny table's near-key column does not qualify on the absolute
+# count alone. Deliberately conservative: this codebase's general posture is to
+# under-report rather than over-report, and a false negative here just costs one
+# more `explore query`.
+#
+# They live beside the sample type rather than in `explore.profile` because a
+# metered adapter has to reserve for the probe *before* profiling runs, and the
+# two sides would drift apart the moment one of them moved. VALUE_DOMAIN_MIN_ROWS
+# is what the fraction implies rather than a threshold of its own: a domain needs
+# at least one distinct value, so a table below it can never clear the fraction
+# and never produces a probe worth reserving for.
+VALUE_DOMAIN_CAP = 25
+VALUE_DOMAIN_MAX_FRACTION = 0.10
+VALUE_DOMAIN_MIN_ROWS = ceil(1 / VALUE_DOMAIN_MAX_FRACTION)
+
+
 @dataclass(frozen=True)
 class QueryResult:
     """The result of one firewall-approved agent query, columnar.

packages/dex-core/src/exmergo_dex_core/adapters/bigquery.py

@@ -17,6 +17,7 @@
 from __future__ import annotations
 
 from collections.abc import Callable
+from dataclasses import dataclass
 from typing import Any
 
 from ..config import BigQueryTarget
@@ -25,6 +26,7 @@
 from ..guards.cost_guard import CostGate, OverCeilingError
 from ..guards.sql_guard import assert_select_only
 from .base import (
+    VALUE_DOMAIN_MIN_ROWS,
     ColumnAggregate,
     ColumnMeta,
     ObjectMeta,
@@ -63,6 +65,25 @@
 _NESTED_FIELD_TYPES = {"RECORD", "STRUCT", "JSON", "GEOGRAPHY", "RANGE", "INTERVAL"}
 
 
+@dataclass(frozen=True)
+class _EstimateComposition:
+    """What a profile estimate is made of: measured dry-run scan versus
+    escalation reserve.
+
+    The two halves answer different questions and a caller deciding whether to
+    raise a budget needs them apart. The scan half is a dry-run of statements
+    that will certainly run; the reserve half is headroom held for probes that
+    may never be issued (see :meth:`BigQueryAdapter.profile_estimate`). Quoting
+    only the sum is what left a refused nightly run unable to tell a warehouse
+    that had grown from a release that had added a reserve (issue #299).
+    """
+
+    scan_bytes: float
+    scan_queries: int
+    reserved_bytes: float
+    reserved_queries: int
+
+
 def _regexp_predicate(qcol: str, pattern: str) -> str:
     # Raw string literal; REGEXP_CONTAINS matches substrings, so the shared
     # patterns' anchors make it a full match.
@@ -130,6 +151,10 @@ def __init__(
         self._tables: dict[str, Any] = {}
         self._resolved_datasets: list[str] | None = None
         self._notes: dict[str, list[str]] = {}
+        # What the last profile estimate was made of, so the handshake and the
+        # over-ceiling refusal can attribute the number they quote. Per command,
+        # like `_notes`: one command builds at most one profile estimate.
+        self._composition: _EstimateComposition | None = None
 
     # --- capabilities ---------------------------------------------------------
 
@@ -684,18 +709,18 @@ def profile_estimate(
         what BigQuery actually bills, and an unfloored estimate would send the
         agent into a ladder of budget rejections.
 
-        The total also reserves one floor per table for each of the two
-        possible escalation queries (``exact_distinct_counts`` for a
-        near-unique column, ``distinct_combination_counts`` for a composite
-        key candidate): whether either actually runs depends on the aggregate
-        batch's own approximate results, which do not exist yet at estimate
-        time, so there is no way to dry-run them here. Reserving their floor
-        unconditionally keeps this total a ceiling profiling will not exceed,
-        rather than a number a run that does escalate blows past -- the exact
-        gap this estimator used to leave open (issue #107). Skipped only when
-        a table is provably empty (``row_count == 0``); an unknown row count
-        (views, whose count is never known before the aggregate that reveals
-        it) still reserves, since escalation is not ruled out.
+        The total also reserves one floor per table for each of the three
+        escalation queries a profile may still issue after that batch
+        (:meth:`exact_distinct_counts`, :meth:`value_domain_counts`,
+        :meth:`distinct_combination_counts`). Whether any of them runs depends
+        on the batch's own approximate results, which do not exist yet at
+        estimate time, so there is no way to dry-run them here. Holding their
+        floor keeps this total a ceiling profiling will not exceed rather than
+        a number a run that does escalate blows past, which is the gap this
+        estimator used to leave open (issue #107). See
+        :meth:`_escalation_reserve` for what narrows each one, and
+        :attr:`_composition` for how the reserve is reported apart from the
+        scan it rides with.
 
         Blob-type columns are excluded from the batches the same way
         ``explore.profile.profile`` excludes them from the scan itself
@@ -704,6 +729,9 @@ def profile_estimate(
 
         blob_paths = include_blobs or set()
         per_table: dict[str, float] = {}
+        scan_bytes = 0.0
+        scan_queries = 0
+        reserved_queries = 0
         for identifier in identifiers:
             meta, columns = self.table_metadata(identifier)
             if self._unqueryable(identifier):
@@ -732,21 +760,68 @@ def profile_estimate(
                     sample_percent=sample_percent,
                 )
                 try:
-                    total += max(self._dry_run(sql), float(_MIN_BILLED_BYTES))
+                    batch = max(self._dry_run(sql), float(_MIN_BILLED_BYTES))
                 except self._api_exceptions.BadRequest:
                     self._note(
                         identifier,
                         "could not estimate an aggregate scan (dry-run failed); "
                         "the object is skipped",
                     )
-            if scan_columns and meta.row_count != 0:
-                total += float(_MIN_BILLED_BYTES)  # exact_distinct_counts
-                total += float(_MIN_BILLED_BYTES)  # value_domain_counts
-                if len(scan_columns) >= 2:
-                    total += float(_MIN_BILLED_BYTES)  # distinct_combination_counts
+                    continue
+                total += batch
+                scan_bytes += batch
+                scan_queries += 1
+            reserved = self._escalation_reserve(meta, scan_columns)
+            total += reserved * float(_MIN_BILLED_BYTES)
+            reserved_queries += reserved
             per_table[identifier] = total
+        self._composition = _EstimateComposition(
+            scan_bytes=scan_bytes,
+            scan_queries=scan_queries,
+            reserved_bytes=reserved_queries * float(_MIN_BILLED_BYTES),
+            reserved_queries=reserved_queries,
+        )
         return sum(per_table.values()), per_table
 
+    def _escalation_reserve(
+        self, meta: ObjectMeta, scan_columns: list[ColumnMeta]
+    ) -> int:
+        """How many escalation queries to hold a billing floor for on one table.
+
+        A reserve is dropped only where the probe's own guard already rules the
+        query out from metadata alone, never as a guess about what is likely: an
+        estimate that reserves for a query that cannot run is merely loose, while
+        one that skips a query that can is the defect issue #107 closed. So each
+        condition below mirrors one in ``explore.profile``, and moving one
+        without the other is the bug to watch for.
+
+        - Nothing at all without a row count. All three probes return early on a
+          falsy one, and a BigQuery view never has one: ``_object_meta`` nulls it
+          and the aggregate's own ``COUNT(*)`` is read per batch, never written
+          back to the object. So a view provably cannot escalate, and the reserve
+          it used to hold was money no run could spend.
+        - Nothing for columns BigQuery cannot count distinctly. Nested and
+          repeated fields get no approximate distinct in the aggregate batch, and
+          every probe's eligibility starts from one, so they can no more trigger
+          an escalation than a blob column already excluded from the scan.
+        - No value domain below :data:`VALUE_DOMAIN_MIN_ROWS` rows, which is what
+          the probe's row-relative fraction implies once a domain needs at least
+          one distinct value.
+        - No composite probe below two countable columns, since a combination
+          needs two members. This was already conditioned, but on the raw column
+          count, which counts columns that can never join a pair.
+        """
+
+        countable = [c for c in scan_columns if not self._is_nested(c.data_type)]
+        if not countable or not meta.row_count:
+            return 0
+        reserved = 1  # exact_distinct_counts
+        if meta.row_count >= VALUE_DOMAIN_MIN_ROWS:
+            reserved += 1  # value_domain_counts
+        if len(countable) >= 2:
+            reserved += 1  # distinct_combination_counts
+        return reserved
+
     def query_estimate(self, sql: str) -> float:
         """The dry-run byte estimate for one firewall-approved query, floored to
         what BigQuery will actually bill (the per-referenced-table minimum), so
@@ -761,33 +836,65 @@ def describe_estimate(
         """The bytes-scanned handshake payload: names the per-query billing
         floor baked into every number here, so a small-table estimate reads as
         a trustworthy ceiling instead of one the actual bill will exceed
-        (issue #107). The escalation-reserve clause only applies to a
-        multi-table call (``per_table`` set, i.e. a profile-shaped estimate);
-        a single ad-hoc query or cluster sample has no such reserve to explain.
+        (issue #107).
+
+        Deliberately a pure function of its arguments. What a *profile* estimate
+        is made of is answered by :meth:`profile_reserve` instead, because this
+        method also describes estimates that carry no reserve (an ad-hoc query,
+        a mid-command verify checkpoint) and it cannot tell which it was handed.
         """
 
-        note = (
-            f"BigQuery bills at least {_MIN_BILLED_BYTES:,} bytes (10 MB) per "
-            "query; every number here already reflects that floor"
-        )
-        if per_table:
-            note += (
-                ", including a reserve for the escalation queries a "
-                "multi-table profile may still add after its initial scan"
-            )
         data: dict[str, object] = {
             "estimated_bytes": estimate,
             "hint": (
                 "review the estimate, then re-run with --confirm --budget "
                 "<bytes> (the ceiling in bytes; 10000000000 is 10 GB, about "
                 "$0.06 on-demand)"
             ),
-            "notes": [note],
+            "notes": [
+                f"BigQuery bills at least {_MIN_BILLED_BYTES:,} bytes (10 MB) "
+                "per query; every number here already reflects that floor"
+            ],
         }
         if per_table:
             data["per_table_bytes"] = per_table
         return data
 
+    def profile_reserve(self, estimate: float) -> dict | None:
+        """How much of ``estimate`` is escalation reserve rather than measured
+        scan, as a sentence plus the two numbers behind it. ``None`` when this
+        command priced no profile, so there is nothing to attribute.
+
+        The confirm handshake and the over-ceiling refusal both reach for this,
+        which is the point of it existing: a refusal that quotes a number
+        without saying what it is made of leaves the operator to reconstruct
+        the split from the spend ledger by hand, which is what issue #299
+        reported doing. Only the command-level handshake asks, because only
+        that estimate is the one :meth:`profile_estimate` built.
+
+        The remainder is described rather than the recorded scan half quoted,
+        because a caller may have added statement estimates of its own to the
+        total (``explore query`` prices an auto-profile and its statements in
+        one handshake) and those are dry-run figures too.
+        """
+
+        composition = self._composition
+        if composition is None or not composition.reserved_queries:
+            return None
+        return {
+            "note": (
+                f"{composition.reserved_bytes:,.0f} bytes of this estimate is "
+                f"escalation reserve: {composition.reserved_queries} queries at "
+                f"BigQuery's {_MIN_BILLED_BYTES:,}-byte per-query minimum, held "
+                "for probes a profile may add after its aggregate scan and may "
+                "never issue. The remaining "
+                f"{estimate - composition.reserved_bytes:,.0f} bytes is dry-run "
+                "scan"
+            ),
+            "reserved_bytes": composition.reserved_bytes,
+            "reserved_queries": composition.reserved_queries,
+        }
+
     def _min_billed_floor(self, sql: str) -> float:
         """BigQuery bills at least ``_MIN_BILLED_BYTES`` per table a query
         references. The floor for one query is that minimum times its distinct

packages/dex-core/src/exmergo_dex_core/command_args.py

@@ -18,7 +18,7 @@
 from .adapters.base import Adapter
 from .config import CONFIG_FILE
 from .envelope import Cost
-from .guards.cost_guard import ConfirmationRequiredError, CostGate
+from .guards.cost_guard import ConfirmationRequiredError, CostGate, OverCeilingError
 from .results import ConfirmationRequest, Result
 from .storage import DEX_DIR
 
@@ -117,8 +117,24 @@ def billed_handshake(
     gate = cost_gate(adapter)
     if gate is None:
         return
+    # What a connector can say about the composition of the estimate it just
+    # priced, asked once and used by both refusals below. Only this handshake
+    # asks: it is the one whose estimate the connector's profile pricing built,
+    # and a mid-command phase (verify probes) prices something else entirely.
+    reserve = getattr(adapter, "profile_reserve", None)
+    reserve = reserve(estimate) if reserve is not None else None
     try:
         gate.preflight_command(estimate)
+    except OverCeilingError as exc:
+        # A refusal is the message an operator acts on and the one that carries
+        # the least. Without this, an estimate padded with escalation reserve
+        # reads as work the warehouse is about to do, and the only way to tell
+        # the two apart is to reconstruct the split from the spend ledger
+        # afterwards (issue #299). Re-raised rather than mutated so the
+        # exception stays a plain immutable refusal, carrying the same cost.
+        if reserve is None:
+            raise
+        raise OverCeilingError(f"{exc}. {reserve['note']}", cost=exc.cost) from exc
     except ConfirmationRequiredError as exc:
         # The payload speaks the connector's unit. An adapter that knows more
         # than the raw magnitude (Snowflake's credit translation, its
@@ -139,6 +155,14 @@ def billed_handshake(
             }
             if per_table:
                 data["per_table_bytes"] = per_table
+        if reserve is not None:
+            # Prose and flat keys both: the note is what a person reads when
+            # deciding whether raising the budget buys work or headroom, and
+            # the keys are so a host branching on the split does not have to
+            # parse the sentence to find it.
+            data["reserved_bytes"] = reserve["reserved_bytes"]
+            data["reserved_queries"] = reserve["reserved_queries"]
+            data["notes"] = [*data.get("notes", []), reserve["note"]]
         if notes:
             data.setdefault("notes", [])
             data["notes"] = [*data["notes"], *notes]