Environment: dex 1.6.3, BigQuery connector.
Problem
budget.session_ceiling caps cumulative spend per UTC day, and the check is
against each command's estimate. On a mid-sized project the two routine
whole-project commands price like this:
| command |
estimate |
billed |
transform build --target dev |
6.70 GB |
3.45 GB |
explore map --full |
8.69 GB |
5.12 GB |
Estimated together, 15.4 GB. Billed together, 8.6 GB.
With the ceiling at 10 GB, a figure chosen deliberately to allow one full build
plus a maintain pass, the second command is refused:
estimated cost 8685697105.0 exceeds the ceiling 6328935424.0 (bytes_scanned);
raise the budget or narrow the work
after a build that billed 3.45 GB. The ledger records billed, so 6.6 GB of the
ceiling was genuinely unspent. The refusal is against an upper bound that never
materialized.
Why it matters
- It forbids the hygiene it exists to protect. Re-baselining is required
after any change that alters relation types or columns, on the same day as the
build that made the change: explore map --full, then maintain snapshot.
The ceiling blocks that on exactly the days it is needed, and the snapshot
discipline in the maintain skill assumes it is always available.
- The pessimism compounds. Each command is priced at an upper bound and the
ceiling accumulates those bounds across a session, so a day of ordinary work
exhausts a ceiling that real spending would not have approached.
- Narrowing is not a workaround here. A scoped
explore map replaces the
cache rather than merging it, so profiles for out-of-scope datasets are
dropped. Trading a spend guard for a coverage hole is not a smaller decision.
- The remaining escape hatch is to raise a governance number that was set
deliberately, which turns a considered control into one that is routinely
overridden. That is how ceilings stop meaning anything.
Proposal
Gate the cumulative ceiling on billed bytes, which the ledger already
records, and keep the estimate for the per-command handshake where it belongs.
The per-command --budget stays the pre-spend control; the daily ceiling becomes
a real backstop rather than a running sum of worst cases.
If the check must stay on estimates, apply the ledger's own historical
estimate-to-billed ratio (the calibration #278 already proposes computing) before
comparing.
Acceptance
A day containing one full build and one full re-map does not exhaust a ceiling
set above their combined billed cost.
Related: #278 (cite the ratio in a refusal) and #283 (prompt when the ceiling is
unset).
Sibling: #323 is one large source of the over-estimation this compounds.
Environment: dex 1.6.3, BigQuery connector.
Problem
budget.session_ceilingcaps cumulative spend per UTC day, and the check isagainst each command's estimate. On a mid-sized project the two routine
whole-project commands price like this:
transform build --target devexplore map --fullEstimated together, 15.4 GB. Billed together, 8.6 GB.
With the ceiling at 10 GB, a figure chosen deliberately to allow one full build
plus a maintain pass, the second command is refused:
after a build that billed 3.45 GB. The ledger records billed, so 6.6 GB of the
ceiling was genuinely unspent. The refusal is against an upper bound that never
materialized.
Why it matters
after any change that alters relation types or columns, on the same day as the
build that made the change:
explore map --full, thenmaintain snapshot.The ceiling blocks that on exactly the days it is needed, and the snapshot
discipline in the
maintainskill assumes it is always available.ceiling accumulates those bounds across a session, so a day of ordinary work
exhausts a ceiling that real spending would not have approached.
explore mapreplaces thecache rather than merging it, so profiles for out-of-scope datasets are
dropped. Trading a spend guard for a coverage hole is not a smaller decision.
deliberately, which turns a considered control into one that is routinely
overridden. That is how ceilings stop meaning anything.
Proposal
Gate the cumulative ceiling on billed bytes, which the ledger already
records, and keep the estimate for the per-command handshake where it belongs.
The per-command
--budgetstays the pre-spend control; the daily ceiling becomesa real backstop rather than a running sum of worst cases.
If the check must stay on estimates, apply the ledger's own historical
estimate-to-billed ratio (the calibration #278 already proposes computing) before
comparing.
Acceptance
A day containing one full build and one full re-map does not exhaust a ceiling
set above their combined billed cost.
Related: #278 (cite the ratio in a refusal) and #283 (prompt when the ceiling is
unset).
Sibling: #323 is one large source of the over-estimation this compounds.