Skip to content

explore query: a multi-statement call strands its last statement, because the estimate reserves one 10 MB floor for the whole batch #321

Description

@marcociav-exmergo

Environment: dex 1.6.3, BigQuery connector.

Problem

Splitting a multi-table statement into one statement per table hits, first, a
documented limit that is clearly worded:

13 statements in one call exceeds the query.max_statements limit of 10;
split the call, or raise the limit in .dex/config.yml

Split into 7 + 6 and confirmed at a budget derived from the returned estimate,
each batch lost its final statement:

statement 7 failed: the remaining budget (9382090 bytes) is below BigQuery's
10485760-byte minimum billed per query; raise --budget or narrow the work
statement 6 failed: the remaining budget (10459437 bytes) is below BigQuery's
10485760-byte minimum billed per query; raise --budget or narrow the work

Every statement pays BigQuery's 10 MB floor no matter how little it reads, so an
N-statement call needs at least N x 10 MB of headroom. The confirmation
envelope prices the batch by its data size, so a budget taken from dex's own
estimate is guaranteed to strand the tail.

Why it matters

  • The failure is partial and late. Statements 1..N-1 have already billed when
    it fires, so a retry re-pays for them, and the operator learns the arithmetic
    only after spending.
  • It punishes the shape the tool asks for. Multi-statement support landed in explore query: accept more than one statement per call #265
    precisely so a chain of questions is one call, and the guidance says to "ask a
    whole chain of questions in one call rather than one call each".
  • The refusal states the right arithmetic. That same arithmetic is fully knowable
    at confirmation time, which is where it belongs.

Proposal

  • Include statement_count x 10,485,760 in the multi-statement estimate.
  • Refuse up front naming the shortfall, rather than part-way through a batch that
    has already spent money.

Acceptance

  • An N-statement call whose statements each read almost nothing returns
    estimated_bytes >= N x 10,485,760.
  • Confirming at that estimate completes all N statements.

Related: #275 is the same floor counted per probe on maintain check.

Sibling: #320 is the same floor arithmetic failing on the other side, where the
server-side cap on a single multi-table statement is set below what BigQuery
will bill.

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions