Skip to content
Open
35 changes: 30 additions & 5 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,13 +62,13 @@ project's real config instead of a wrong default.
| `transform macro [name]` | no name lists the shipped dbt macros; a name proposes scaffolding it into the project's macro directory as a plan (dbt-parse-checked, applied with `transform apply`); re-running diffs the project's copy against the shipped version |
| `transform build --target dev` | prod-looking targets refused outright; then a free dev-target preflight (refuses when `.dex/config.yml` and the rendered `profiles.yml` disagree, or when the dev database does not exist, naming the fix); then the cost preflight, priced upfront by a free `dbt compile` dry-run of each node (a partial floor when a cold dev target has not built a node's inputs yet; degrades to no estimate when dex cannot open its own connection); runs only with `--confirm` and a budget; cwd pinned to the project dir; auto-runs `dbt deps` when packages are declared but not installed |
| `transform deps` | install/refresh dbt packages (repo-confined; no warehouse spend) |
| `semantic define\|update\|plan ... --edits-file <f>` | dbt semantic model edits as diffs; validated up to and including dbt's own parser (a throwaway project copy) before the plan is stored; `plan` accepts a mix and classifies per name; degrades to a warning when dbt is absent, `--no-parse` skips; applied with `transform apply` like any other plan |
| `semantic define\|update\|plan ... --edits-file <f>\|--definitions-file <f>` | dbt semantic model edits as diffs; validated up to and including dbt's own parser (a throwaway project copy) before the plan is stored; `plan` accepts a mix and classifies each name `defined`, `updated`, or `unchanged`; `--definitions-file` names one definition at a time instead of a whole file; degrades to a warning when dbt is absent, `--no-parse` skips; applied with `transform apply` like any other plan |
| `maintain snapshot` | capture/refresh the known-good baseline in `.dex/snapshot.json` (pins the `.dex/` map + per-layer definition fingerprints) |
| `maintain check` | sweep every drift axis vs the snapshot; ranked drift report (read-only); two-phase on billed connectors (free axes now, one estimate for the scanning axes) |
| `maintain check` | sweep every drift axis vs the snapshot; ranked drift report (read-only); two-phase on billed connectors: the free axes complete and return `ok`, with one estimate for the scanning axes under `data.offer` |
| `maintain schema [<objects>]` | structural drift: columns/tables added, dropped, retyped, renamed; nullability; dangling sources (free) |
| `maintain volume [<objects>]` | freshness drift: row counts that collapsed, emptied, or spiked (free metadata) |
| `maintain grain [<objects>]` | cardinality/identity drift: lost key uniqueness, changed grain, join fanout (scans; gated on billed connectors) |
| `maintain semantic [<objects>]` | definition drift and dangling refs (free) plus categorical dimension cardinality change (scans; gated on billed connectors) |
| `maintain semantic [<objects>]` | definition drift and dangling refs (free, and returned as `ok`) plus categorical dimension cardinality change (scans; offered under `data.offer` and gated on billed connectors) |
| `maintain reconcile [<class>]` | propose the dbt edits that reconcile detected drift, as a stored plan of diffs tagged mechanical or advisory (never applied; apply with `transform apply <plan-id>`) |
| `viz preview` | emit the dbt semantic model to the Viz preview (not yet implemented) |

Expand All @@ -82,14 +82,29 @@ is not free: `schema`, `volume`, and the reference half of `semantic` are metada
scan and go through the `--confirm --budget` handshake on billed connectors. The
engine does not care which skill fronts a subcommand.

A command whose free half completed reports `ok` and puts the price of the
scanning half in `data.offer`, rather than gating the whole answer behind a
confirmation. `needs_confirmation` means dex is waiting on you for work you asked
for; an offer is work you did not ask for, and ignoring it is a valid choice.
Read `data.axes_run` for what completed and `data.offer.axes` for what the
estimate would add, since with an `ok` status those are no longer implied.

Authored content reaches the engine through `--edits-file <path>` (or `-` for
stdin): a JSON payload of `{"edits": [{"path", "kind", "op", "content"}, ...]}`
with `kind` one of `model_sql`, `schema_yml`, `semantic_yml`, `packages_yml` (the
guarded way to author the project-root `packages.yml`/`dependencies.yml`, so
declaring a dbt package is a reviewable diff too), `macro_sql`, `project_yml`
(the project-root `dbt_project.yml`), or `profiles_yml` (the project-root
`profiles.yml`, secret-guarded so a credential never enters the diff: reference
secrets via `{{ env_var('NAME') }}`). `op` is `upsert` (create or update, the
secrets via `{{ env_var('NAME') }}`). The semantic commands take a second,
narrower payload instead: `--definitions-file <path|->` with
`{"definitions": [{"kind", "path", "content"}, ...]}`, where `kind` is
`semantic_model` or `metric` and `content` is that one definition's YAML body.
The name is read from the content, and `path` may be omitted for a definition
the project already declares, in which case it is rewritten where it lives. Use
it whenever a change touches part of a shared file: the engine writes each
definition in place and leaves every other byte, including comments, untouched,
so the diff and the classification both describe only what changed. `op` is `upsert` (create or update, the
default, carrying `content`) or `delete` (remove the file, no `content`); a
delete is a reviewable diff too, guarded so the plan is refused if any surviving
file still `ref()`s a deleted model, and a rename is one plan (delete old, create
Expand All @@ -110,7 +125,17 @@ would spend requires an explicit `--confirm` and a session budget: on a
metered connector (BigQuery, Snowflake, Databricks, Redshift, and Postgres)
the first call returns `needs_confirmation` with a free estimate, and the
same command is re-issued with `--confirm --budget <magnitude>` once the user
has agreed to the spend. The magnitude is paradigm-relative: **bytes** on
has agreed to the spend.

One exception to the status, not to the rule: a command that finished free work
the caller did want, and can offer paid work they did not ask for, returns `ok`
with the estimate under `data.offer` instead. That is `maintain check` and
`maintain semantic`. The re-issue is identical (`--confirm --budget`), nothing
runs until it arrives, and `cost.estimate` stays empty so an `ok` never reads as
though it spent. Reserve `needs_confirmation` for reading "dex is waiting on
me", and an offer for "there is more available if I want it".

The magnitude is paradigm-relative: **bytes** on
BigQuery (an exact free dry-run figure), **warehouse-seconds** on Snowflake
(a heuristic labeled `estimate_quality: "heuristic"`, with a credit
translation alongside) and on Databricks (a floor labeled
Expand Down
88 changes: 88 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,94 @@ tag releases both in lockstep, so entries below are keyed by the engine version.

## [Unreleased]

### Changed

- **A free answer stops arriving shaped like a bill** ([#136]). `maintain check`
and `maintain semantic` complete their free axes on every call: schema, volume,
and the reference and definition half of semantic are metadata reads that finish
and settle. Both returned that finished work inside a `needs_confirmation`
envelope, because the axes that scan were priced and unconfirmed. So the command
a maintenance session opens with reported its entire triage, in one field report
373 findings, in a response shaped like a pending charge for work the caller had
not asked for and might never want.

Two costs came out of that. Confirming things that cost nothing is a habit, and
the handshake only works on commands where it does cost something. And the
framing taught the wrong reading: reaching for `data.findings` inside a refusal
is the natural move, and doing it is how the stale-baseline line in `warnings`
got missed.

The split is now on whether the caller asked. `needs_confirmation` means dex is
waiting on you for work you requested, and nothing you asked for has run.
Optional priced work rides on a completed answer instead: `status: ok`, findings
final, and the estimate under `data.offer` with the same breakdown and
`--confirm --budget` hint a refusal carried. `data.offer.axes` names what the
estimate would add and `data.axes_run` what already finished, which is what now
separates "grain found nothing" from "grain did not run" since the status no
longer implies it. `cost.estimate` stays unset, so an `ok` never carries a
number that reads as spend. Nothing about the spend gate moved: the confirmed
re-issue is identical and no scan runs without it. `explore relationships
--verify` and `explore map --verify` keep `needs_confirmation`, correctly, since
there the caller did ask for the probes and the budget ran out mid-command.

A host reading `data["estimated_bytes"]` on these two commands reads
`data["offer"]["estimated_bytes"]`.

- **The same two commands stopped dropping their baseline caveats on the
unconfirmed call.** The branch that returned early built its result without
`_baseline_warnings`, which the settled branch includes, so the warnings that a
baseline no longer describes the warehouse (a cache newer than the snapshot, a
snapshot pinned from an already-stale cache) were missing from precisely the
response most sessions read. Confirmed against the dogfood project, where
`maintain semantic` reported a 327-hour-old baseline and `maintain check`, same
session and same baseline, reported nothing. Both paths now carry identical
warnings, because what bounds the settled answer bounds the free one.

- **`semantic plan` reports what changed, not what was re-typed** ([#109]).
Classification compared names against the project and nothing else, so any name
already present read as `updated`. The edit unit is a whole file, so extending a
shared `semantic_models.yml` means re-stating every definition in it, and a
two-metric change reported 27 objects as updated with a `+16/-0` diff. The one
place a reviewer confirms blast radius was the place it was hidden.

There is now a third class. `updated` means the parsed definition actually
differs from the project's; a definition re-stated identically in the file that
already holds it is `unchanged`. Key order and formatting are not changes; list
order is, and identical content written to a different file is a move, so both
still read as `updated`. A plan whose every definition is unchanged warns that
it changes nothing.

### Added

- **A per-definition edit unit for the semantic layer** ([#109]).
`semantic define|update|plan` take `--definitions-file <path|->` beside
`--edits-file`: `{"definitions": [{"kind", "path", "content"}, ...]}`, where
`kind` is `semantic_model` or `metric` and `content` is that one definition's
YAML body. This is the stronger half of the fix. The whole-file unit is what
generated the `updated` noise, and re-typing twenty-seven untouched definitions
to add two is also how a stray key gets injected into a metric by hand, caught
in the field only by eye and by the parse gate.

The name is read from the content, so the two cannot disagree, and `path` may be
omitted for a definition the project already declares, defaulting to the file
that holds it. An explicit path that would relocate an existing definition is
refused, because writing it to a second file duplicates the name. Each
definition is spliced into its file as text, preserving every other byte
including the comments a semantic layer accumulates; a round trip through
`safe_dump` would reformat the file and produce a larger diff than the payload
it replaces. The result is re-parsed and compared against what was sent, and a
layout the splice cannot span safely (a flow-style sequence, anchors or aliases,
multiple documents, tab indentation) is refused with `--edits-file` named as the
way in. Classification is scoped to the definitions named, so a spliced file's
other definitions appear in no class at all.

It lowers to the whole-file `PlanEdit` the engine already stores, so the plan
format, the diffs, the conflict hashing, and `transform apply` are unchanged.
Deleting a definition remains a whole-file edit.

`AGENTS.md`, `references/command-contract.md`, and both the `transform` and
`maintain` skills document the new payload, the third class, and the offer.

## [1.6.1] - 2026-08-09

### Fixed
Expand Down
3 changes: 3 additions & 0 deletions packages/dex-core/src/exmergo_dex_core/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -261,6 +261,9 @@ def _build_parser() -> argparse.ArgumentParser:
if group == "semantic":
sp.add_argument("argument", nargs="?", default=None)
sp.add_argument("--edits-file", default=None)
# The per-definition payload: name only what changes, and the
# engine writes it into the file that holds it.
sp.add_argument("--definitions-file", default=None)
sp.add_argument("--no-parse", action="store_true", default=False)
# maintain detectors take an optional object scope (default: whole
# project); reconcile takes an optional drift class to fix.
Expand Down
11 changes: 11 additions & 0 deletions packages/dex-core/src/exmergo_dex_core/command_args.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ def billed_handshake(
*,
per_table: dict[str, float] | None = None,
notes: list[str] | None = None,
axes: list[str] | None = None,
) -> None:
"""The cost-before-spend handshake on billed connectors.

Expand Down Expand Up @@ -139,6 +140,11 @@ def billed_handshake(
}
if per_table:
data["per_table_bytes"] = per_table
if axes:
# What the estimate would add. Load-bearing for an offer, whose
# envelope reports `ok`: without it, an axis that did not run is
# indistinguishable from one that ran and found nothing.
data["axes"] = axes
if notes:
data.setdefault("notes", [])
data["notes"] = [*data["notes"], *notes]
Expand All @@ -165,6 +171,11 @@ def confirmation_request(
let this raise, because discarding them to ask about the billed half would
make the caller pay attention twice for one answer. That is ``maintain
check`` and ``maintain semantic``, whose free axes always complete.

Those two carry the returned request as ``Result.pending_offer`` rather than
``pending_confirmation``, because the caller never asked for the scanning
axes: the request is priced work on offer, not a charge dex is waiting on.
Pass ``axes`` so the offer names what the estimate would add.
"""

try:
Expand Down
Loading
Loading