You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Splitting a multi-table statement into one statement per table hits, first, a
documented limit that is clearly worded:
13 statements in one call exceeds the query.max_statements limit of 10;
split the call, or raise the limit in .dex/config.yml
Split into 7 + 6 and confirmed at a budget derived from the returned estimate, each batch lost its final statement:
statement 7 failed: the remaining budget (9382090 bytes) is below BigQuery's
10485760-byte minimum billed per query; raise --budget or narrow the work
statement 6 failed: the remaining budget (10459437 bytes) is below BigQuery's
10485760-byte minimum billed per query; raise --budget or narrow the work
Every statement pays BigQuery's 10 MB floor no matter how little it reads, so an
N-statement call needs at least N x 10 MB of headroom. The confirmation
envelope prices the batch by its data size, so a budget taken from dex's own
estimate is guaranteed to strand the tail.
Why it matters
The failure is partial and late. Statements 1..N-1 have already billed when
it fires, so a retry re-pays for them, and the operator learns the arithmetic
only after spending.
It punishes the shape the tool asks for. Multi-statement support landed in explore query: accept more than one statement per call #265
precisely so a chain of questions is one call, and the guidance says to "ask a
whole chain of questions in one call rather than one call each".
The refusal states the right arithmetic. That same arithmetic is fully knowable
at confirmation time, which is where it belongs.
Proposal
Include statement_count x 10,485,760 in the multi-statement estimate.
Refuse up front naming the shortfall, rather than part-way through a batch that
has already spent money.
Acceptance
An N-statement call whose statements each read almost nothing returns estimated_bytes >= N x 10,485,760.
Confirming at that estimate completes all N statements.
Related: #275 is the same floor counted per probe on maintain check.
Sibling: #320 is the same floor arithmetic failing on the other side, where the
server-side cap on a single multi-table statement is set below what BigQuery
will bill.
Environment: dex 1.6.3, BigQuery connector.
Problem
Splitting a multi-table statement into one statement per table hits, first, a
documented limit that is clearly worded:
Split into 7 + 6 and confirmed at a budget derived from the returned estimate,
each batch lost its final statement:
Every statement pays BigQuery's 10 MB floor no matter how little it reads, so an
N-statement call needs at least
N x 10 MBof headroom. The confirmationenvelope prices the batch by its data size, so a budget taken from dex's own
estimate is guaranteed to strand the tail.
Why it matters
it fires, so a retry re-pays for them, and the operator learns the arithmetic
only after spending.
explore query: accept more than one statement per call #265precisely so a chain of questions is one call, and the guidance says to "ask a
whole chain of questions in one call rather than one call each".
at confirmation time, which is where it belongs.
Proposal
statement_count x 10,485,760in the multi-statement estimate.has already spent money.
Acceptance
estimated_bytes >= N x 10,485,760.Related: #275 is the same floor counted per probe on
maintain check.Sibling: #320 is the same floor arithmetic failing on the other side, where the
server-side cap on a single multi-table statement is set below what BigQuery
will bill.