Daily platform report with day-over-day finding diffs (#8)

* feat: daily platform report with day-over-day finding diffs

Closes #6.

The platform was accumulating a narrative and discarding it. ops.monitor_results
has carried run_at since Task 18 and nothing ever read it back. Worse, the one
artifact that looked like a history was destroying one: incident slugs are
deliberately stable so a recurring finding rewrites one file rather than
spawning many, which means write_incident OVERWRITES and yesterday is gone.
Nothing could answer "when did this start?" or "did yesterday's problem clear?"

Two append-only Iceberg tables:
  ops.finding_log  every finding on every run, keyed on sensors.finding_key
  ops.agent_runs   one row per run, findings or not

The second exists because a clean run writes no finding rows, so without it
"ran and found nothing" and "never ran" are the same empty table -- and a report
would render the second as a clean bill of health. Same empty-delta blind spot
this codebase has now closed four times (quarantine_rate SELECT 0.0, range on an
all-NULL column, freshness on no values, an enum watch on an empty column).

src/ops/report.py assembles seven sections. build_report() takes a dataclass and
returns a string: it is handed no engine, so it CANNOT compute a metric even by
accident, which is what makes "the report recomputes nothing" a property rather
than a comment. A test asserts the module never references monitors.evaluate,
sensors.detect or validator.validate. A report that recomputes can disagree with
the monitor that raised the alert, and the report is the one people believe.

Absences are reported as absences: no run recorded says so instead of rendering
clean; no monitor results is "an absence of evidence, not a pass"; no previous
run says "nothing to diff against" instead of calling every standing finding new;
a monitor with no prior value prints an em dash, never 0. An unknown finding kind
routes to Errors rather than vanishing.

drift-demo --day N applies one scheduled change instead of all four, so the agent
can run between them and the log accumulates a real timeline. Bare drift-demo is
unchanged. reports/ holds a four-day sequence from one continuous history: clean,
+column, rename, then the rest plus a volume collapse -- 0, 1, 2, 8 findings.

Fixed a latent bug found on the way in: incident_slug() hashed
table|kind|column|change with no monitor name, so EVERY monitor_breach on a table
hashed identically. With one breach firing that was invisible; with two the
second incident file silently overwrote the first, and this feature's diff would
have reported a finding as "cleared" while it was still breaching. Extracted
sensors.finding_key() as the single definition of finding identity, shared by the
incident filename and the log row -- they must agree or "first seen" describes
something other than the file on disk. Incident filenames also stopped rendering
"None" for breaches carrying no column. The committed incident slugs change
accordingly; this is a one-time migration.

Also fixed: the new tests reached graph.run()'s act node and wrote into the
repo's real docs/incidents/. test_agent_graph and test_bronze already guard
against this; test_report now uses the same autouse fixture. The suite no longer
dirties the committed artifact set.

Reviewed: 275 tests pass (250 before), ruff clean, four-day sequence re-run end
to end at full scale and every committed report and incident regenerated from it.
Day four runs one day past the end of the data on purpose, which is why three
arrival SLAs breach; stated in the README rather than left to look accidental.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199ePw5w34FyfGAB41mkv3G

* fix: review round 1 -- re-running the agent doubled findings; report render was unstable

Two defects found by probing the shipped code rather than re-reading the diff.

1. `make agent` twice on one date -- an ordinary thing to do -- appended the
   same logical finding to ops.finding_log twice, and the report counted rows
   rather than findings. One open finding rendered as "2 still open". A diff
   whose counts are wrong is worse than no diff, because the counts are read as
   a measurement. `_latest_per_key` collapses to one row per finding_key per
   run; `make monitor` gets the same treatment.

2. The diff sections were rendered in set-iteration order, so regenerating a
   report reshuffled its lines with no content change. Reports are committed
   artifacts: an unstable render puts noise in every git diff, and a diff that
   is usually noise stops being read. Sorted by finding key, with a test that
   renders twice and compares.

Also added a test that an unescaped pipe in detail text cannot split a Markdown
table cell -- it was already handled, but nothing pinned it.

Reviewed: 279 tests pass (275 before), ruff clean. reports/daily-2026-07-01.md
re-rendered under the stable ordering; counts are unchanged (6 new, 0 cleared,
2 still open), only line order. Verified byte-stable across two regenerations.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199ePw5w34FyfGAB41mkv3G

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: rubinder

Makefile

@@ -1,4 +1,4 @@
-.PHONY: all bronze silver gold forecast agent monitor arrival test lint clean timetravel drift-demo schema-history cross-version maintenance
+.PHONY: all bronze silver gold forecast agent monitor arrival report test lint clean timetravel drift-demo schema-history cross-version maintenance
 UV := uv run
 
 all: bronze silver gold forecast
@@ -10,6 +10,7 @@ forecast:    ; $(UV) python -m src.forecast.report
 agent:       ; $(UV) python -m src.agent.graph
 monitor:     ; $(UV) python -m src.ops.runner
 arrival:     ; $(UV) python -m src.ops.arrival
+report:      ; $(UV) python -m src.ops.report
 timetravel:  ; $(UV) python -m src.lakehouse.maintenance timetravel
 drift-demo:  ; $(UV) python -m src.lakehouse.maintenance drift-demo
 schema-history: ; $(UV) python -m src.lakehouse.maintenance schema-history

README.md

@@ -47,6 +47,10 @@ A complete data platform, built end to end, small enough to read:
 - **17 data-quality monitors + 8 arrival checks** with persisted metric history
   and robust (median/MAD) baselines. 10 of the 17 are declared in YAML; the
   other 7 are column-type checks generated per contract field.
+- **A daily report that tells the story** — `make report` assembles verdict,
+  schema evolution, data quality, anomalies, errors and a **day-over-day diff**
+  from `ops.*`. It reads; it never recomputes. A three-day sample sequence is in
+  [`reports/`](reports/).
 - **The AI-SDLC record** — the spec, the plan, every task brief, every review
   diff, and a ledger of what was found and changed. See
   [`docs/ai-sdlc/`](docs/ai-sdlc/workflow.md).
@@ -170,12 +174,13 @@ uv run make lint           # ruff
 |---|---|
 | `make all` | `bronze silver gold forecast` end to end, offline, deterministic |
 | `make monitor` | 17 data-quality checks; persists metrics to `ops.monitor_results` and routes alerts to `ops.alert_log` (**dry-run** unless `--execute`) |
+| `make report` | the daily platform report for `AS_OF_DATE`, assembled from `ops.*` and written to `reports/daily-<date>.md` |
 | `make arrival` | 8 checks across 3 arrival SLAs, on the NYSE trading calendar |
 | `make agent` | the LangGraph ops agent (**dry-run** unless `--execute`) |
 | `make schema-history` | every schema version with field ids |
 | `make cross-version` | one query spanning snapshots written under different schemas |
 | `make timetravel` | read at snapshot N−1 vs N |
-| `make drift-demo` | evolve the schema and inject a volume collapse |
+| `make drift-demo` | evolve the schema and inject a volume collapse (`--day N` applies one scheduled change at a time) |
 | `make maintenance` | expire old snapshots |
 | `make smoke` | `make all` at `N_TRANSACTIONS=5000` |
 
@@ -436,15 +441,16 @@ live behaviour on the strength of its unit tests. It runs now.
 A LangGraph state machine (`src/agent/graph.py`):
 
 ```
-sense ──▶ monitor ──▶ classify ──┬──▶ act ──▶ END
-                                 └──▶ END (no action)
+sense ──▶ monitor ──▶ classify ──▶ persist ──┬──▶ act ──▶ END
+                                             └──▶ END (no action)
 ```
 
 | Node | What it does |
 |---|---|
 | `sense` | Reads Iceberg **metadata** — schema history, snapshot log, per-snapshot added-records, newest date. No data scan where metadata suffices. |
 | `monitor` | Runs the `src/ops` monitors and arrival SLAs; converts breaches into findings. |
 | `classify` | Severity per finding: `breaking` / `renaming` / `widening` / `additive` / `enum_drift` / `benign`. **Rules first**; the optional Claude call handles only unmatched cases. |
+| `persist` | Appends this run to `ops.finding_log` and `ops.agent_runs`. Runs on **every** path, including the clean one. |
 | `act` | Writes `docs/incidents/<slug>.md` and would file a GitHub issue. **Dry-run by default.** |
 
 **Classification is rules-first because CI must never depend on a model call**
@@ -548,6 +554,104 @@ PyIceberg will not narrow a column type and faking that state would misrepresent
 what the storage layer does
 ([decisions/0005](docs/ai-sdlc/decisions/0005-drift-scenario-coverage-split.md)).
 
+## The daily report
+
+Everything above measures. `make report` is the thing that *reads it back*.
+
+The platform was already accumulating a narrative and then throwing it away.
+`ops.monitor_results` has carried `run_at` since Task 18 and nothing ever queried
+it. `docs/incidents/*.md` records the latest state of each open finding — and
+because the incident slug is deliberately stable so a recurring finding rewrites
+one file instead of spawning many, **every write destroyed the previous state**.
+Nothing in the repository could answer "when did this start?" or "did yesterday's
+problem go away?"
+
+Two append-only tables fix that. `ops.finding_log` records every finding on every
+run, keyed on `sensors.finding_key`. `ops.agent_runs` records the run itself —
+because a clean run writes no finding rows, so without it "the agent ran and
+found nothing" and "the agent never ran" are the same empty table, and only one
+of those is good news.
+
+```bash
+uv run make report                       # today
+AS_OF_DATE=2026-06-29 uv run make report # any recorded day
+```
+
+### A four-day sequence
+
+`drift-demo --day N` applies one scheduled schema change instead of all four, so
+the agent can run between them and the log accumulates a real timeline. The four
+files in [`reports/`](reports/) are this sequence, verbatim, from one continuous
+warehouse history:
+
+```bash
+AS_OF_DATE=2026-06-28 make monitor agent report   # clean:  0 findings
+uv run python -m src.lakehouse.maintenance drift-demo --day 1   # +merchantCategoryCode
+AS_OF_DATE=2026-06-29 make monitor agent report   #         1 finding
+uv run python -m src.lakehouse.maintenance drift-demo --day 4   # checkNumber -> check_reference
+AS_OF_DATE=2026-06-30 make monitor agent report   #         2 findings
+uv run python -m src.lakehouse.maintenance drift-demo           # the rest + a volume collapse
+AS_OF_DATE=2026-07-01 make monitor agent report   #         8 findings
+```
+
+Day three's diff section — the sentence the platform previously could not
+produce at all:
+
+```markdown
+Compared with `2026-06-29`: **1 new**, **0 cleared**, **1 still open**.
+
+### New
+- `[renaming]` column 'checkNumber' was renamed to 'check_reference' (field id 20)
+
+### Still open
+- `[additive]` column 'merchantCategoryCode' present in table but not declared
+  in contract (open 1d (since 2026-06-29))
+```
+
+Not "there is a finding", but *this one is new, this one has been open since
+Monday, and nothing was resolved.*
+
+Day four is where every section fills in at once: two schema changes still open,
+a volume collapse, the `bronze_txn_row_count` monitor breaching against its now
+250,000-row baseline, and three arrival-SLA gaps.
+
+**Why day four fires arrival SLAs, stated plainly:** the dataset ends
+`2026-06-30`, so running the fourth day at `2026-07-01` is deliberately one day
+past the end of the data. That is what makes `silver.transactions`,
+`silver.stock_prices` and `gold.forecast_training_set` report a missing period —
+the same mechanism the `AS_OF_DATE=2026-07-31 make arrival` example uses. It is a
+real breach of a real SLA against a logical clock, not a contrivance, but the
+sequence that produced it is stated with it.
+
+### The report reads; it does not measure
+
+`build_report()` takes a `ReportSnapshot` dataclass and returns a string. It is
+handed no engine, so it **cannot** compute a metric even by accident — which is
+what makes the claim testable rather than a comment. `load_snapshot()` is the
+only function in the module that touches a warehouse.
+
+The reason is not tidiness. A report that recomputes a number can disagree with
+the monitor that raised the alert, and when those two disagree, the report is the
+one people believe.
+
+### Absences are reported as absences
+
+Three of them, each a place where a naive report would print something reassuring:
+
+| Situation | What a naive report shows | What this one shows |
+|---|---|---|
+| No agent run for the date | empty findings table → looks clean | `Verdict — NO RUN RECORDED`, "this is **not** a clean bill of health" |
+| No monitor results | empty quality table → looks clean | "an absence of evidence, not a pass — run `make monitor`" |
+| No previous run to diff | every finding listed as "new" | "nothing to diff against… not because it is new" |
+
+A monitor metric with no prior value prints `—`, never `0`: unchanged and
+never-measured are different facts, and printing zero for the second is a small
+lie that reads as a measurement.
+
+A finding kind the report has no section for is routed to Errors rather than
+dropped. A new check family that is measured, recorded, and invisible is the
+exact failure this file exists to prevent.
+
 ## Results
 
 **Read the caveat at the top of this file first.** Prices are synthetic with a

docs/ai-sdlc/decisions/0007-finding-history-in-iceberg-not-in-the-incident-files.md

@@ -0,0 +1,119 @@
+# 0007 — Finding history lives in Iceberg, not in the incident files
+
+**Type:** design decision for [#6](https://github.com/rubinder/autonomous_data_platform/issues/6)
+**Alternative considered:** an append-only event log inside each `docs/incidents/*.md`
+**Shipped:** two append-only Iceberg tables, `ops.finding_log` and `ops.agent_runs`
+
+## The problem
+
+The platform was accumulating a narrative and discarding it.
+
+`ops.monitor_results` has carried `run_at` (partitioned by month) since Task 18,
+and **nothing ever read it back**. `make monitor` wrote; no code queried.
+
+Worse, the one artifact that looked like a history was actively destroying one.
+`actions.incident_slug()` is deliberately stable so a recurring finding rewrites
+a single file rather than spawning one per run — the right call for bounded tree
+churn, made deliberately by a human, and the reason `docs/incidents/` is
+committable at all. But it means `write_incident` **overwrites**. Yesterday's
+state is gone. Nothing in the repository could answer "when did this start?" or
+"did yesterday's problem clear?"
+
+## Why the table, not the file
+
+The rejected alternative was to append a dated event line inside each incident
+file, keeping everything readable in the repo with no warehouse.
+
+Against it:
+
+- **Diffing means parsing prose.** "What changed since the previous run" becomes
+  a Markdown-scraping exercise over files whose format exists for humans. Any
+  format change silently breaks the diff.
+- **It reintroduces unbounded growth** — the exact thing the stable slug was
+  chosen to avoid. A finding open for six months is a file with 180 event lines,
+  and every agent run dirties the git tree.
+- **Findings that never file an incident would have nowhere to live.** Only
+  `ACTIONABLE` severities write incident files. `additive`, `widening` and
+  `enum_drift` findings would be absent from a file-based history entirely —
+  and those are precisely the ones whose *accumulation over time* is the story
+  the report is supposed to tell.
+- The ops tables already exist, already carry `run_at`, and are already
+  partitioned by month. The table answer is consistent with how this codebase
+  already records operational facts.
+
+The incident files keep their existing job: the current state of each open
+actionable finding, readable in the repo, one file per finding.
+
+## Why `ops.agent_runs` is a second table and not a flag
+
+A clean run produces no findings, so it writes no rows to `ops.finding_log`. A
+report reading only that table cannot distinguish **"the agent ran and found
+nothing"** from **"the agent never ran"** — and it would render the second as a
+clean bill of health.
+
+That is the same empty-delta blind spot this codebase has now closed four times:
+`silver_txn_quarantine_rate: SELECT 0.0`, the `range` check passing on an
+all-NULL column, `freshness` on no values, and (in
+[0006](0006-renames-paired-by-field-id-and-a-vacuous-guard-test.md)) an enum
+watch on a column holding nothing. One row per run, findings or not, closes it
+structurally rather than by remembering to check.
+
+## The report takes no engine
+
+`build_report(snapshot: ReportSnapshot) -> str` is handed a dataclass, not a
+warehouse. It therefore **cannot** compute a metric even by accident, which is
+what makes "the report recomputes nothing" a property rather than a comment.
+`load_snapshot()` is the only function in the module that touches an engine, and
+a test asserts the module never references `monitors.evaluate`, `sensors.detect`
+or `validator.validate`.
+
+The reason is not tidiness. A report that recomputes a number can disagree with
+the monitor that raised the alert — and when those two disagree, the report is
+the one people believe.
+
+## A bug found on the way in
+
+`incident_slug()` hashed `table | kind | column | change`, with no monitor name.
+Every `monitor_breach` on a given table therefore hashed **identically**:
+
+```
+bronze.yodlee_transactions_raw|monitor_breach|None|
+```
+
+With one monitor breaching, invisible. With two, the second incident file
+silently overwrote the first — and the day-over-day diff would have reported a
+finding as "cleared" while it was still breaching, which is the worst possible
+lie for this feature to tell.
+
+Fixed by extracting `sensors.finding_key()` as the single definition of finding
+identity, used by both the incident filename and the `ops.finding_log` row. They
+have to agree, or "first seen" describes something different from the file on
+disk. The filename also stopped rendering `None` for breaches that carry no
+column.
+
+## Review round: two defects the tests did not reach
+
+**1. Running `make agent` twice in one day doubled the findings.**
+`ops.finding_log` is append-only, and re-running the agent on the same date is
+an entirely ordinary thing for a human to do. The report counted *rows*, not
+findings, so a second run turned one open finding into "2 still open". A
+day-over-day diff whose counts are wrong is worse than no diff at all, because
+the counts are read as a measurement. `_latest_per_key` now collapses to one row
+per `finding_key` per run; the same applies to `make monitor`.
+
+**2. The report was not byte-stable across regenerations.**
+The diff sections were built from set differences and rendered in set-iteration
+order, so simply regenerating a report reshuffled its lines. These are committed
+artifacts: an unstable render produces a git diff on every regeneration that has
+nothing to do with what changed, and a diff that is usually noise is a diff
+nobody reads. Now sorted by finding key, with a test that renders twice and
+compares.
+
+Both were found by probing the shipped code, not by re-reading the diff. Neither
+would have failed a test that only asked "does the report render?"
+
+## Scope note
+
+`drift-demo --day N` was added because the report needed something to narrate.
+`evolve_all()` applies four schema changes in one shot, which produces one before
+and one after and no story in between. Bare `drift-demo` is unchanged.

docs/incidents/bronze-yodlee_transactions_raw-bronze_txn_row_count-68d8938c.md

@@ -0,0 +1,19 @@
+# Monitor Breach — bronze.yodlee_transactions_raw
+
+**Severity:** breaking
+
+**Detected as of:** 2026-07-01
+
+## What was observed
+
+[bronze_txn_row_count] 5 is 0% of trailing median 250000
+
+## Why this severity
+
+Monitor 'bronze_txn_row_count' breached on bronze.yodlee_transactions_raw: [bronze_txn_row_count] 5 is 0% of trailing median 250000 A monitor breach is, by definition, already a judged verdict -- there is no additive reading of a check that has already failed.
+
+## Evidence
+
+```
+{'monitor': 'bronze_txn_row_count', 'kind': 'row_count', 'column': None, 'metric': 5.0, 'baseline': 250000.0, 'status': 'breach'}
+```

…transactions_raw-checkNumber-7b8d24a4.md …transactions_raw-checkNumber-13673376.mddocs/incidents/bronze-yodlee_transactions_raw-checkNumber-7b8d24a4.md renamed to docs/incidents/bronze-yodlee_transactions_raw-checkNumber-13673376.md docs/incidents/bronze-yodlee_transactions_raw-checkNumber-7b8d24a4.md renamed to docs/incidents/bronze-yodlee_transactions_raw-checkNumber-13673376.md

@@ -2,7 +2,7 @@
 
 **Severity:** renaming
 
-**Detected as of:** 2026-06-30
+**Detected as of:** 2026-07-01
 
 ## What was observed
 

…nsactions_raw-volume_anomaly-f0497e07.md …nsactions_raw-volume_anomaly-9fdcb62b.mddocs/incidents/bronze-yodlee_transactions_raw-volume_anomaly-f0497e07.md renamed to docs/incidents/bronze-yodlee_transactions_raw-volume_anomaly-9fdcb62b.md docs/incidents/bronze-yodlee_transactions_raw-volume_anomaly-f0497e07.md renamed to docs/incidents/bronze-yodlee_transactions_raw-volume_anomaly-9fdcb62b.md

@@ -2,7 +2,7 @@
 
 **Severity:** breaking
 
-**Detected as of:** 2026-06-30
+**Detected as of:** 2026-07-01
 
 ## What was observed
 

docs/incidents/gold-forecast_training_set-trade_date-d2acdf09.md

@@ -0,0 +1,19 @@
+# Monitor Breach — gold.forecast_training_set
+
+**Severity:** breaking
+
+**Detected as of:** 2026-07-01
+
+## What was observed
+
+[gold.forecast_training_set_arrival_gap] 1 missing period(s): 2026-07-01
+
+## Why this severity
+
+Monitor 'gold.forecast_training_set_arrival_gap' breached on gold.forecast_training_set: [gold.forecast_training_set_arrival_gap] 1 missing period(s): 2026-07-01 A monitor breach is, by definition, already a judged verdict -- there is no additive reading of a check that has already failed.
+
+## Evidence
+
+```
+{'monitor': 'gold.forecast_training_set_arrival_gap', 'kind': 'arrival_gap', 'column': 'trade_date', 'metric': 1.0, 'baseline': None, 'status': 'breach'}
+```

docs/incidents/silver-stock_prices-trade_date-5f2232da.md

@@ -0,0 +1,19 @@
+# Monitor Breach — silver.stock_prices
+
+**Severity:** breaking
+
+**Detected as of:** 2026-07-01
+
+## What was observed
+
+[silver.stock_prices_arrival_gap] 1 missing period(s): 2026-07-01
+
+## Why this severity
+
+Monitor 'silver.stock_prices_arrival_gap' breached on silver.stock_prices: [silver.stock_prices_arrival_gap] 1 missing period(s): 2026-07-01 A monitor breach is, by definition, already a judged verdict -- there is no additive reading of a check that has already failed.
+
+## Evidence
+
+```
+{'monitor': 'silver.stock_prices_arrival_gap', 'kind': 'arrival_gap', 'column': 'trade_date', 'metric': 1.0, 'baseline': None, 'status': 'breach'}
+```

docs/incidents/silver-transactions-txn_date-d5f55e4e.md

@@ -0,0 +1,19 @@
+# Monitor Breach — silver.transactions
+
+**Severity:** breaking
+
+**Detected as of:** 2026-07-01
+
+## What was observed
+
+[silver.transactions_arrival_gap] 1 missing period(s): 2026-07-01
+
+## Why this severity
+
+Monitor 'silver.transactions_arrival_gap' breached on silver.transactions: [silver.transactions_arrival_gap] 1 missing period(s): 2026-07-01 A monitor breach is, by definition, already a judged verdict -- there is no additive reading of a check that has already failed.
+
+## Evidence
+
+```
+{'monitor': 'silver.transactions_arrival_gap', 'kind': 'arrival_gap', 'column': 'txn_date', 'metric': 1.0, 'baseline': None, 'status': 'breach'}
+```