feat: pair renames by field ID, split widening from additive, record enum drift

Closes #1.

The agent reported `checkNumber -> check_reference` as two findings -- a
`breaking` drop and an `additive` add -- on the one table whose headline
property is that a rename costs nothing because the field ID does not move.
The evidence was already being fetched and thrown away: schema_history()
returns field_ids, observe() read only ["columns"].

`build_rename_map` groups every historical column name by field ID, so a name
gone from the latest schema whose ID is still present is a rename, not a drop.
Transitive renames (a -> b -> c) resolve in one hop because the map keys on the
ID rather than on consecutive pairs. A field ID absent from the latest schema
stays a genuine drop and stays breaking.

Measured at full scale, no ops.monitor_results history:
  all -> agent                        0 findings          (unchanged)
  all -> drift-demo -> agent          5 -> 4 findings, 2 incidents
  all -> monitor -> drift-demo -> agent   6 -> 5 findings, 3 incidents

`renaming` is ACTIONABLE: no data moved, but the published contract now names a
column that does not exist, and that needs a recorded decision. `widening` is
not -- an int -> long and a brand-new column are both safe, but only one means
the contract is behind on a column that already existed.

Enum drift lands as `enum_watch`, a new contract key, NOT an expectation.
Expectations are fail-closed gates and `_evaluate` raises on unknown types,
which is what stops a typo'd check reading green forever; a watch that must
never block would have needed a branch that always passes -- the vacuous check
that design exists to prevent. baseType keeps its accepted_values gate and is
deliberately not watched: one column, one semantics. A watch that reads no
values reports breaking rather than passing, and one aimed at a >200-cardinality
column reports itself misconfigured.

A guard test in the first draft was vacuous. It supplied `renames` as a fixture,
so it exercised detect() and never build_rename_map, and a planted
position-matching mutant passed all 19 tests. A rename does not move a column,
so ordinal matching gets plain renames right and breaks only when a column is
deleted from the middle. Three tests now build the map from a
{a:1,b:2,c:3} -> {a:1,c_new:3} history where position and field ID disagree;
the position mutant and a name-similarity mutant both fail against them, as does
a real Iceberg table with a delete and a rename in one step. Written up in
docs/ai-sdlc/decisions/0006.

Reviewed: 246 tests pass (224 before), ruff clean, both drift sequences re-run
end to end at full scale and the README numbers replaced with what was measured.
The settlementDays widening still reports as `additive` because drift-demo adds
that column before widening it, so no narrower type was ever declared; stated in
the README rather than papered over by editing the contract, which would make
the clean run report a phantom drop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199ePw5w34FyfGAB41mkv3G

Author: rubinder

README.md

@@ -38,6 +38,12 @@ A complete data platform, built end to end, small enough to read:
   feature independently recomputed by a second implementation on every training
   row.
 - **An agentic ops layer** — LangGraph, rules-first, dry-run by default, offline.
+  Schema changes are classified `breaking` / `renaming` / `widening` / `additive`
+  / `enum_drift`, with renames paired **by Iceberg field ID** so one rename is one
+  finding rather than a phantom drop plus a phantom add.
+- **Enum drift** on 7 categorical columns — new values are recorded and reported,
+  never used to block a build. Deliberately not the same mechanism as the
+  fail-closed `accepted_values` gate on `baseType`.
 - **17 data-quality monitors + 8 arrival checks** with persisted metric history
   and robust (median/MAD) baselines. 10 of the 17 are declared in YAML; the
   other 7 are column-type checks generated per contract field.
@@ -185,12 +191,12 @@ To see the whole story in order:
 
 ```bash
 uv run make all && uv run make agent          # clean: 0 findings
-uv run make drift-demo && uv run make agent   # after drift: 5 findings, 2 incidents
+uv run make drift-demo && uv run make agent   # after drift: 4 findings, 2 incidents
 ```
 
 Both transcripts below are from that exact sequence, on a warehouse with no
 `ops.monitor_results` history. Slot a `make monitor` in before `drift-demo` and
-the post-drift run reports **6** findings and 3 incidents, not 5 and 2: the
+the post-drift run reports **5** findings and 3 incidents, not 4 and 2: the
 persisted baseline of 250,000 added records lets `bronze_txn_row_count` breach
 as well, which the agent folds in as a `monitor_breach` finding on top of the
 volume anomaly its own sensor already found. That is the monitors working, not
@@ -438,7 +444,7 @@ sense ──▶ monitor ──▶ classify ──┬──▶ act ──▶ END
 |---|---|
 | `sense` | Reads Iceberg **metadata** — schema history, snapshot log, per-snapshot added-records, newest date. No data scan where metadata suffices. |
 | `monitor` | Runs the `src/ops` monitors and arrival SLAs; converts breaches into findings. |
-| `classify` | Severity per finding: `breaking` / `additive` / `benign`. **Rules first**; the optional Claude call handles only unmatched cases. |
+| `classify` | Severity per finding: `breaking` / `renaming` / `widening` / `additive` / `enum_drift` / `benign`. **Rules first**; the optional Claude call handles only unmatched cases. |
 | `act` | Writes `docs/incidents/<slug>.md` and would file a GitHub issue. **Dry-run by default.** |
 
 **Classification is rules-first because CI must never depend on a model call**
@@ -453,28 +459,78 @@ $ uv run make agent                       # clean lakehouse
 agent: as_of=2026-06-30 findings=0
 
 $ uv run make drift-demo && uv run make agent
-agent: as_of=2026-06-30 findings=5
-  [breaking] column 'checkNumber' declared in contract but absent from table
-  [additive] column 'check_reference' present in table but not declared in contract
+agent: as_of=2026-06-30 findings=4
+  [renaming] column 'checkNumber' was renamed to 'check_reference' (field id 20); the contract still declares the old name
   [additive] column 'merchantCategoryCode' present in table but not declared in contract
   [additive] column 'settlementDays' present in table but not declared in contract
   [breaking] latest snapshot added 5 rows, below 50% of trailing median 250,000
-  -> incident: bronze-yodlee_transactions_raw-checkNumber-3955ba01.md
-  -> DRY-RUN: would run `gh issue create ...` -> [breaking] schema_drift on bronze.yodlee_transactions_raw
+  -> incident: bronze-yodlee_transactions_raw-checkNumber-7b8d24a4.md
+  -> DRY-RUN: would run `gh issue create ...` -> [renaming] schema_drift on bronze.yodlee_transactions_raw
   -> incident: bronze-yodlee_transactions_raw-volume_anomaly-f0497e07.md
   -> DRY-RUN: would run `gh issue create ...` -> [breaking] volume_anomaly on bronze.yodlee_transactions_raw
 agent: dry-run (pass --execute to file GitHub issues)
 ```
 
-Five findings, **two** incident files — breaking only. Incident slugs are
+Four findings, **two** incident files — actionable only. Incident slugs are
 sha256-derived and therefore stable across processes, so re-running rewrites the
 same two files instead of accumulating duplicates. Committed examples are in
 [`docs/incidents/`](docs/incidents/).
 
-The rename classified as `breaking` is the right answer, not noise: Silver reads
-Bronze by name and the contract declares columns by name, so a rename *is* a drop
-to every consumer that selects it, even though Iceberg lost nothing. Storage
-fine, consumers broken — exactly what a human should review.
+### The rename used to be two findings, and that was the bug
+
+An earlier version of this agent reported the rename as **five** findings, not
+four:
+
+```
+  [breaking] column 'checkNumber' declared in contract but absent from table
+  [additive] column 'check_reference' present in table but not declared in contract
+```
+
+The reasoning behind `breaking` was not wrong as far as it went — Silver reads
+Bronze by name and the contract declares columns by name, so to a by-name
+consumer a rename really does read as a disappearance. Storage fine, consumers
+broken, worth a human's time. All of that still holds, which is why `renaming`
+is in `ACTIONABLE` and still files an incident.
+
+What was wrong is that **one event was reported as two unrelated ones**, and
+neither of them named what happened. A reader was told a column had vanished and,
+separately, that an unfamiliar column had appeared — and left to guess these were
+the same column. On the exact table whose headline property is that a rename
+costs nothing because the field ID does not move, the agent could not see the
+field ID: `schema_history()` returned `field_ids` and `observe()` discarded them.
+
+`build_rename_map` now groups every historical column name by field ID, so a name
+that has vanished from the latest schema while its ID is still present is a
+rename, not a drop — and `checkNumber` and `check_reference` are both field 20.
+Transitive renames (`a → b → c`) resolve to the current name in one hop, because
+the map is keyed on the ID rather than on consecutive pairs.
+
+**Pairing by field ID is not a stylistic preference over pairing by name or
+position, and the test suite has to prove that.** A rename does not move a
+column, so ordinal matching gets a plain rename right and looks correct; it
+breaks only when a column is deleted from the middle and everything after it
+shifts up, at which point it invents a rename that never happened *and* hides the
+real drop. A first draft of these tests supplied the rename map as a fixture
+rather than building it, and a deliberately planted position-matching mutant
+passed all 19 of them. The tests now build the map from schema history for a
+`{a:1, b:2, c:3} → {a:1, c_new:3}` evolution, where position and field ID give
+different answers, and both the position mutant and a name-similarity mutant fail
+against them — as does a real Iceberg table with a column deleted and another
+renamed in the same step.
+
+`widening` likewise stopped being a synonym for `additive`. A widened `int → long`
+and a brand-new column are both safe and neither pages anyone, but only one of
+them means the contract is behind on a column that already existed.
+
+**One honest gap in the demo**: `drift-demo`'s own `settlementDays int → long`
+step shows up above as `additive`, not `widening`. That is correct — `drift-demo`
+*adds* `settlementDays` before widening it, so the shipped contract never declared
+it and the agent has no narrower type to compare against. The `widening` class
+fires when a column the contract declares gets wider, which is covered against a
+real table by `test_real_widening_classifies_as_widening`. Declaring
+`settlementDays` in the contract to make the demo prettier would make the clean
+run report a phantom dropped column, so the demo stays as it is and the gap is
+stated instead.
 
 The incident file is careful about *which* consumers, and that took a fix. The
 generated reasoning used to end "its absence will fail the next Silver build",

contracts/bronze_yodlee_transactions.yaml

@@ -39,3 +39,30 @@ expectations:
   - {type: accepted_values, column: baseType, values: [CREDIT, DEBIT]}
   - {type: row_count, min: 1000}
   - {type: freshness, column: transactionDate, max_lag_days: 3}
+
+# Categorical columns watched for NEW values. Deliberately separate from
+# `expectations`: an `accepted_values` expectation is a fail-closed gate that
+# stops the build, which is right for `baseType` (the pipeline's sign
+# convention branches on it -- an unrecognised value would silently mis-sign
+# spend) and wrong for everything below, where upstream adding a category
+# breaks nothing but must not pass unnoticed.
+#
+# `baseType` is deliberately NOT repeated here. One column, one semantics:
+# it is a gate, not a watch.
+enum_watch:
+  - column: category
+    known_values: [Restaurants/Dining, General Merchandise, Clothing/Shoes,
+                   Personal Care, Other Expenses]
+  - column: detailCategory
+    known_values: [Coffee Shop, Fast Food, Department Store, Athletic Apparel,
+                   Pizza, Cosmetics, Uncategorized]
+  - column: subType
+    known_values: [PAYMENT, CREDIT]
+  - column: categoryType
+    known_values: [EXPENSE, INCOME]
+  - column: status
+    known_values: [POSTED]
+  - column: categorySource
+    known_values: [SYSTEM]
+  - column: sourceType
+    known_values: [AGGREGATED]

docs/ai-sdlc/decisions/0006-renames-paired-by-field-id-and-a-vacuous-guard-test.md

@@ -0,0 +1,99 @@
+# 0006 — Renames paired by field ID, and the guard test that was vacuous
+
+**Type:** feature ([#1](https://github.com/rubinder/autonomous_data_platform/issues/1)),
+plus a self-inflicted vacuous test caught by mutation
+**Made:** during implementation, by planting a position-matching mutant
+**Shipped:** `sensors.build_rename_map()` groups historical column names by
+Iceberg field ID; `classify()` gains `renaming`, `widening` and `enum_drift`
+
+## What changed
+
+The agent used to report `checkNumber → check_reference` as two findings — a
+`breaking` drop and an `additive` add — on the one table whose headline property
+is that a rename is metadata-only because the field ID does not move. The
+evidence to do better was already being fetched and then discarded:
+`pyiceberg_engine.schema_history()` returns `field_ids`, and `observe()` read
+only `["columns"]`.
+
+Measured, at full scale, on a warehouse with no `ops.monitor_results` history:
+
+| Sequence | Before | After |
+|---|---|---|
+| `all → agent` | 0 findings | 0 findings |
+| `all → drift-demo → agent` | 5 findings, 2 incidents | **4 findings, 2 incidents** |
+| `all → monitor → drift-demo → agent` | 6 findings, 3 incidents | **5 findings, 3 incidents** |
+
+`renaming` is in `ACTIONABLE`: no data moved, but the published contract now
+names a column that does not exist, and that needs a recorded decision. The
+severity split only earns its keep if it changes what the agent *does*, so
+there is a test asserting exactly which severities act.
+
+## The vacuous guard test
+
+The first draft included a test called
+`test_rename_pairing_does_not_capture_an_unrelated_added_column`, written
+specifically to prove the pairing was by field ID and not by position or name.
+It constructed an `ObservedState` with `renames=` supplied as a **fixture** —
+so it exercised `detect()`'s use of the map and never `build_rename_map` itself.
+
+A planted mutant that replaced field-ID grouping with ordinal position matching
+**passed all 19 tests.**
+
+The reason it survives is worth recording, because it is not obvious: *a rename
+does not move a column*. Ordinal matching therefore gets a plain rename right,
+and gets the rename-plus-add case right too, since PyIceberg appends new columns
+at the end. It breaks only when a column is deleted from the middle and every
+later column shifts up by one — at which point it invents a rename that never
+happened and simultaneously hides a real drop.
+
+So the discriminating case had to be built deliberately:
+
+```
+history:      {a:1, b:2, c:3}  ->  {a:1, c_new:3}
+by position:  [a,b,c] vs [a,c_new]  ->  pairs b -> c_new     (wrong, twice)
+by field id:  id 2 is gone (a real drop); id 3 changed name  (the rename)
+```
+
+Three tests now cover it: a unit test on `build_rename_map` with that history, an
+end-to-end `detect()` test asserting `b` is still `breaking`, and a real-Iceberg
+test that deletes `detailCategoryId` and renames `checkNumber` in one step. The
+position mutant and a name-similarity mutant both fail against them.
+
+**The generalisable point:** a test that supplies the thing under test as a
+fixture is testing the consumer, not the producer, however precisely its name
+describes the producer's contract. This is the third instance of the same shape
+in this repository (Task 8's dedup test, Task 16's widening test), and the only
+reliable way any of the three were found was planting the mutant and watching the
+suite stay green.
+
+## Enum drift is a separate contract key, not an expectation
+
+`enum_watch` is a new top-level key in the contract YAML rather than a new
+`expectations` entry. Everything in `expectations` is a fail-closed gate:
+`assert_valid` raises on failure, and `_evaluate` raises on any type it does not
+recognise — which is precisely what stops a typo'd check from reading green
+forever. An enum watch must never block a build, so putting it in `expectations`
+would have required a branch that always passes: a check that displays green
+while measuring nothing, which is the exact defect
+`silver_txn_quarantine_rate: SELECT 0.0` already was once.
+
+`baseType` is deliberately *not* watched. It already has a fail-closed
+`accepted_values` expectation because the pipeline's sign convention branches on
+it, and an unrecognised value there would silently mis-sign spend. One column,
+one semantics: a gate or a watch, never both.
+
+A watch whose column produces no values classifies `breaking`, not silent-pass —
+a watch that cannot see its column occupies the slot where the alert would have
+been. A watch on a column with more than `ENUM_CARDINALITY_LIMIT` (200) distinct
+values reports itself misconfigured rather than emitting a finding carrying the
+whole cardinality.
+
+## Known gap, stated rather than hidden
+
+`drift-demo`'s `settlementDays int → long` step reports as `additive`, not
+`widening`, because `drift-demo` adds that column before widening it — the
+shipped contract never declared it, so there is no narrower declared type to
+compare against. `widening` fires when a *declared* column gets wider, covered
+against a real table by `test_real_widening_classifies_as_widening`. Adding
+`settlementDays` to the contract to make the demo read better would make the
+clean run report a phantom dropped column, which is a worse trade.

docs/incidents/bronze-yodlee_transactions_raw-checkNumber-3955ba01.md

@@ -0,0 +1,19 @@
+# Schema Drift — bronze.yodlee_transactions_raw
+
+**Severity:** renaming
+
+**Detected as of:** 2026-06-30
+
+## What was observed
+
+column 'checkNumber' was renamed to 'check_reference' (field id 20); the contract still declares the old name
+
+## Why this severity
+
+'checkNumber' was renamed to 'check_reference'. Both are field id 20: Iceberg resolves columns by ID, not by name, so every file written under the old name still reads back correctly and no data was rewritten or lost. What is wrong is the published contract, which still declares 'checkNumber'. That needs a recorded decision, not a rollback.
+
+## Evidence
+
+```
+{'change': 'renamed', 'column': 'checkNumber', 'renamed_to': 'check_reference', 'field_id': 20, 'declared_type': 'string', 'observed_type': 'string', 'type_compatible': True}
+```