Skip to content

budget.session_ceiling gates on estimates, so a project cannot be re-baselined on any day it was built #324

Description

@marcociav-exmergo

Environment: dex 1.6.3, BigQuery connector.

Problem

budget.session_ceiling caps cumulative spend per UTC day, and the check is
against each command's estimate. On a mid-sized project the two routine
whole-project commands price like this:

command estimate billed
transform build --target dev 6.70 GB 3.45 GB
explore map --full 8.69 GB 5.12 GB

Estimated together, 15.4 GB. Billed together, 8.6 GB.

With the ceiling at 10 GB, a figure chosen deliberately to allow one full build
plus a maintain pass, the second command is refused:

estimated cost 8685697105.0 exceeds the ceiling 6328935424.0 (bytes_scanned);
raise the budget or narrow the work

after a build that billed 3.45 GB. The ledger records billed, so 6.6 GB of the
ceiling was genuinely unspent. The refusal is against an upper bound that never
materialized.

Why it matters

  • It forbids the hygiene it exists to protect. Re-baselining is required
    after any change that alters relation types or columns, on the same day as the
    build that made the change: explore map --full, then maintain snapshot.
    The ceiling blocks that on exactly the days it is needed, and the snapshot
    discipline in the maintain skill assumes it is always available.
  • The pessimism compounds. Each command is priced at an upper bound and the
    ceiling accumulates those bounds across a session, so a day of ordinary work
    exhausts a ceiling that real spending would not have approached.
  • Narrowing is not a workaround here. A scoped explore map replaces the
    cache rather than merging it, so profiles for out-of-scope datasets are
    dropped. Trading a spend guard for a coverage hole is not a smaller decision.
  • The remaining escape hatch is to raise a governance number that was set
    deliberately, which turns a considered control into one that is routinely
    overridden. That is how ceilings stop meaning anything.

Proposal

Gate the cumulative ceiling on billed bytes, which the ledger already
records, and keep the estimate for the per-command handshake where it belongs.
The per-command --budget stays the pre-spend control; the daily ceiling becomes
a real backstop rather than a running sum of worst cases.

If the check must stay on estimates, apply the ledger's own historical
estimate-to-billed ratio (the calibration #278 already proposes computing) before
comparing.

Acceptance

A day containing one full build and one full re-map does not exhaust a ceiling
set above their combined billed cost.

Related: #278 (cite the ratio in a refusal) and #283 (prompt when the ceiling is
unset).

Sibling: #323 is one large source of the over-estimation this compounds.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions