Skip to content

SCRUM-1161-explore(query): state the exact shortfall on a mid-batch refusal - #332

Open
emnarahmouni-exmergo wants to merge 1 commit into
mainfrom
SCRUM-1161-issue-321
Open

SCRUM-1161-explore(query): state the exact shortfall on a mid-batch refusal#332
emnarahmouni-exmergo wants to merge 1 commit into
mainfrom
SCRUM-1161-issue-321

Conversation

@emnarahmouni-exmergo

Copy link
Copy Markdown
Contributor

Closes : #321
("a multi-statement call strands its last statement, because the estimate reserves one 10 MB floor for the whole batch") and fixes the real, narrower gap found. What shipped is not the issue's literal proposal, because the literal bug doesn't reproduce.

What I found

The issue's headline claim doesn't reproduce on this codebase. A 7-statement batch, each statement reading almost nothing, already gets estimated_bytes == 7 x 10,485,760 (summed per-statement, since each already floors on its own distinct tables via query_estimate), and confirming at exactly that estimate already completes all 7 statements. Both acceptance criteria hold today for that literal scenario.

Pushing further, the same execution-time billing variance #320 fixes within one statement also applies across a batch: if each statement's real bill sits just far enough above its own floor, and the dry run under-reports enough that the estimate still floors to exactly the per-query minimum (masking the gap) while the real bill stays above it, the shortfall compounds across the batch until the tail is stranded after several statements have already run and billed. I reproduced this exactly (6 of 7 statements succeed, the 7th fails just under the floor).

This is a real, narrower gap than the issue's literal framing, and it's why #320 and #321 are siblings: same underlying mechanism, one hitting a single multi-table statement's own cap, the other hitting a multi-statement batch's cumulative total.

The fix

_execute already keeps and reports every statement that completed before a cost-guard refusal, since it's already been paid for that part of the existing design is sound and unchanged. What was missing: the refusal for the failed statement said why it failed, but not what it would take to finish.

A mid-batch OverCeilingError now recomputes, from the same query_estimate the original handshake used, exactly how many more bytes complete every remaining (not-yet-run) statement, and states that in the same refusal:

"...raise --budget or narrow the work. Completing the remaining 1 statement(s) in this batch needs at least 10,892,606 more bytes_scanned; the statement(s) already run above are saved, so a wider --budget on re-run does not re-pay for them."

A caller now sets the right --budget on the first re-run instead of raising it by a guess and possibly hitting the same wall again. A no-op for a lone, non-batch statement, where the existing message already says everything there is to say.

Before the fix :
image
After the fix :
image
the error now continues: "Completing the remaining 1 statement(s) in this batch needs at least 10,892,606 more bytes_scanned; the statement(s) already run above are saved, so a wider --budget on re-run does not re-pay for them."

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

explore query: a multi-statement call strands its last statement, because the estimate reserves one 10 MB floor for the whole batch

1 participant