lakehouse-reconciler

Prove a Delta table matches its source of truth, name which kind of corruption it doesn't, and bisect the table's own commit history to find the commit that caused it.

Not "these two tables differ". lakerec reports double load, partition wipe, timezone partition shift, stale rows or one of 11 corruption modes, with the evidence that justified the claim and the action a human should take. Then it finds the version that introduced it:

$ lakerec reconcile extract.parquet lake/orders --contract contracts/orders.yaml
verdict: diverges   (level rows)
rows:    source 5,400  target 6,300  partitions 6 (1 disagreeing)
hash:    source 4f7da72cac5bd031  target 8b1e0c94aa7f2265
[x] double_load: 900 duplicate row(s) across 900 key(s): the target holds copies of rows the
    source has once

$ lakerec bisect lake/orders --predicate unique-keys --keys order_id
first bad version: 6
culprit:           version 6 · WRITE · 2026-08-17T19:47:12Z · {"mode": "Append"} · +1 file(s)/900 row(s)
last good:         5
examined:          7 version(s) (exhaustive)

No JVM, no cluster, no network. Everything below runs in one Python process, against real Delta tables, in about twenty seconds.

Watch the 57-second demo (MP4). A clean Delta table is built, broken twice, diagnosed, and bisected. Every line of terminal text is the real stdout of the command above it, captured by tools/record_demo.py while it breaks the table; the reveal speed is paced by each command's measured wall time. It is a replay of real output, not a live screen recording, and docs/video/manifest.json lists every command with its exit code and timing.

More proof of work: the double-load report · the bisect report

---

Why this exists

Every data platform has a monitor for the pipeline and none for the table. The job succeeded, the dashboard is green, and the numbers have been wrong since Tuesday because a retried batch committed twice, or an overwrite's predicate matched one partition more than it should have, or somebody derived a partition date from a local timestamp instead of a UTC one.

Those three failures have nothing in common except that they are invisible to a row count. And a row count is what most reconciliation checks are, because the alternative sounds like a nightly full scan of two systems that nobody will pay for.

This is the alternative that is affordable: five levels, of which the two cheapest read only the Delta transaction log, and a taxonomy that says which incident you have rather than that you have one.

What it does

1. A checksum that survives being a lakehouse

A table is an unordered pile of files, so the checksum has to be order-independent. The obvious combiner, XOR of per-row digests, is fatally wrong here: x ⊕ x = 0, so a table where every row appears twice checksums identically to an empty table, and double-loading is the most common lakehouse corruption there is. Measured, on the fixture:

XOR combiner
  empty table              00000000000000000000000000000000
  every row exactly twice  00000000000000000000000000000000
  -> a perfect double load is INVISIBLE to XOR

lakerec sums the digests modulo 2^128 instead. Order-independent, sees duplicates, and composable: the six partition checksums add up to the table checksum, which is what makes the drill-down free. ADR-001.

2. Five levels, two of which never open a Parquet file

level	question	target rows read
schema	do both sides have the same columns?	0
counts	does every partition hold the right number of rows?	0
checksum	does every partition hold the right rows?	5,400
keys	which keys are missing, extra, duplicated, changed?	5,400
rows	which columns disagree, on which rows?	5,400

Delta's add actions carry each file's row count and per-column min/max/null-count, so the top two levels are a few kilobytes of JSON. The bottom three read the same bytes, because the key diff and the example rows come from digests the checksum pass already produced. The deepest answer is free, which is why it is the default. ADR-005.

A cold run reads 5,400 rows. Give it the baseline from the previous run and a partition whose file set is unchanged has its checksum inherited (a guarantee of a log-structured format, not an assumption), so a run where one of six partitions changed comes down to 1,800 rows: 67% of the target-side read avoided. The source side has no log and is re-read in full; the tool prints both numbers rather than the flattering one.

3. A corruption taxonomy, injected for real

16 scenarios, each starting from the same clean table and then doing something a real pipeline does: appending a batch twice, DELETE with a predicate that catches a neighbour, overwriting with a range that resolves too wide, deriving dt in the wrong zone. All through the ordinary Delta write path, so the transaction log afterwards is a genuine log.

make matrix runs all 16 at all five levels and publishes what each level caught, including what it missed:

corruption	first level that notices	first level that names it	data read to name it
double load	counts	keys	6,300 rows
missing rows	counts	counts	0 rows
extra rows	counts	counts	0 rows
partition wipe	counts	counts	0 rows
overwrite wider than declared	counts	counts	0 rows
timezone partition shift	counts	keys	5,400 rows
stale rows	checksum	keys	5,400 rows
content divergence	checksum	keys	5,400 rows
schema drift	schema	schema	0 rows
one recomputed double	checksum	checksum	5,400 rows
Unicode normalisation	checksum	keys	5,400 rows
null keys	checksum	keys	5,400 rows
float drift (must stay silent)	never (correct)	nothing to name	n/a
control, no corruption	never (correct)	nothing to name	n/a

All 14 corruptions are named at some level, and 5 are diagnosed from the transaction log alone. Two scenarios exist to make sure the tool stays quiet, and both do. Full matrix: docs/experiments/detection_matrix.json.

4. Bisect, and the reason the obvious version of it is wrong

The first design searched the history for the newest version that matched the source. It runs, and it answers version 0 every time. The source of truth is now and version 4 is then, so version 4 legitimately lacks everything loaded since. A moving reference makes the predicate non-monotone in the variable being searched over, and the search fails confidently rather than loudly.

So lakerec bisect searches only over predicates the table can answer alone:

lakerec bisect lake/orders --predicate unique-keys  --keys order_id       # a double load
lakerec bisect lake/orders --predicate no-null-keys --keys order_id       # log only, no data read
lakerec bisect lake/orders --predicate row-count    --expect 1200000 --partition dt=2026-08-11
lakerec bisect lake/orders --predicate matches-source --source extract.parquet \
                           --partition dt=2026-08-10                      # a *closed* partition only

matches-source refuses to run without --partition, and says why. Restricted to a partition that is no longer written, the source stops moving and the search is sound again. ADR-004.

Three ways it could still mislead, all handled:

• Broken and then repaired, every post-incident investigation. Both ends are clean, so a binary search has nothing to find. Under 48 versions the range is walked instead: healed: true, v6 broke, v7 recovered.
• Too big to walk, then it is a binary search, monotone is None rather than True, and the note says the assumption was made rather than checked.
• History vacuumed away, versions no checkpoint covers are unreadable, neither clean nor broken. The answer becomes "it already fails at the oldest version this log still holds, and the true first bad version is not recoverable from this table."

5. The row count that lies

The last step of make demo is worth reading closely. The table has had one day's batch applied twice and a different day deleted, so:

verdict: diverges   (level counts)
rows:    source 5,400  target 5,400  partitions 6 (2 disagreeing)
[x] partition_wipe: 1 partition(s) are empty in the target while the source has rows: dt=2026-08-10

The total row count is exactly right. A monitor comparing table totals, which is what most reconciliation is, reports green while a day's data is gone and another day is duplicated. Going one level down, to counts per partition, catches both, still without opening a Parquet file.

6. Forensics on the log

$ lakerec forensics lake/orders --partition-by dt
 ver  when                  operation         +files  -files     +rows  partitions
   5  2026-08-17T19:47:12Z  WRITE                  1       0       900  dt=2026-08-11
   6  2026-08-17T19:47:13Z  WRITE                  1       2       900  dt=2026-08-07

worth a second look:
  v6 partition_emptied: WRITE removed every file it touched in dt=2026-08-08 and added none back
  v6 overwrite_shrank_the_table: overwrite removed files from 2 partition(s) and wrote 1;
     the difference is data that is now gone

add and remove actions are read straight out of _delta_log/*.json, which is what turns "the table is wrong" into "version 6 removed every file in dt=2026-08-08 while its parameters name a range it should not have covered".

---

Three results that contradicted my own intuition

Published as they came out, because a repository where every experiment confirmed the author is a repository where the experiments were decorative.

A bit-exact hash over a float column flags 28.3% of a partition where nothing is wrong. The target's amount_float was recomputed as (v*0.7 + v*0.2) + v*0.1, the same money, grouped differently, which is what a changed join order does. 255 of 900 rows differ bitwise, the largest gap being 5.7e-14. That is the false-positive rate a "just hash everything" design ships with, and it is why a float column is refused until the contract says what equal means. ADR-003.

One column, 200 shuffles, 27 different answers. Summing the same 900 values in different orders with sum() produces 27 different answers; math.fsum produces one. Which is why the aggregate that stands in for an excluded float column uses fsum, working around order-dependent arithmetic with an order-dependent check would be silly.

And the exact totals agreed anyway. The fsum totals of the source and target float columns are identical, bit for bit, while 255 rows differ. The errors cancel, so even with the tolerance set to zero the aggregate check cannot see this. That generalises: a sum is a lossy summary of a column and what it loses is exactly the small stuff. An aggregate is a guard against gross error, not a proof of agreement, and it is in the detection matrix as a miss rather than glossed over.

And the whole test suite passed while the installed command was broken. delta-rs 1.6.2 aborts the process during interpreter shutdown after a data read, terminate called without an active exception, which a shell reports as exit 134, on roughly half of runs. Every test called main() as a function inside pytest's own process, where that shutdown never happens, so the whole suite was green on top of a command that intermittently returned exit 134 instead of 0. For a tool whose documented interface is its exit codes, that is the interface broken.

The fix is in the console-script entry point: flush both streams and leave via os._exit with the code the tool decided on, skipping the finalisers that crash. The bug is upstream's; the consequence was mine. tests/test_exit_codes.py now runs the real installed command six times per case and asserts the code is always the documented one, the only shape of test that could have caught it.

One more, from a different direction: the null_keys corruption could not be injected at all on a table that declares order_id NOT NULL. Delta enforces nullability at commit time, the write raises, and the bad data never lands. There is a test asserting exactly that, because it is a stronger result than any detection: the cheapest reconciliation is the one the table format makes unnecessary.

---

Getting started

pip install -e ".[dev]"

make demo          # build a corrupted table and reconcile it, end to end
make matrix        # the detection matrix: 16 scenarios x 5 levels
make verify        # lint, 258 tests, and every published number re-measured

As a CI gate

- run: |
    lakerec reconcile extract.parquet lake/orders \
      --contract contracts/orders.yaml \
      --baseline .lakerec/baseline.json \
      --write-baseline .lakerec/baseline.json \
      --markdown "$GITHUB_STEP_SUMMARY" \
      --html reports/reconcile.html

Exit codes are the interface, so they are fixed: 0 nothing blocking, 1 a blocking finding, 2 the invocation was wrong, 3 a side could not be read. A gate that cannot tell "the data is broken" from "the tool is broken" is a gate that gets ignored for a month.

Late-arriving data is reported as expected and exits 0, which is the difference between a gate people keep and a gate people mute.

The contract

Everything a comparison needs that the data cannot tell you lives in one reviewed file (contracts/orders.yaml), the key, the partitioning, what to ignore, what "equal" means for a float, whether a naive timestamp may be assumed to be UTC, how far ahead the source is allowed to be, and which column orders two versions of a row. There are no flags for those: they are claims about the pipeline, and a claim about the pipeline belongs next to the pipeline. ADR-002.

---

What it does not do

• It cannot see a change that conserves every row. Delete a row and reinsert an identical one and nothing observable happened. Correct, rather than a gap.
• It cannot see float drift that cancels in the aggregate, as above, unless the column is hashed, which costs the 28.3% false-positive rate.
• It does not validate. It compares two systems. If the extract is wrong in the same way as the lake, both agree and both are wrong; run an expectation suite as well.
• It is single-process. The row digest is a pure function of a row and the per-file checksums compose by addition, so pushing it into Spark or DuckDB is a UDF rather than a redesign, but the reference implementation reads a partition into memory, and this line is here instead of an implication that it scales.
• Deletion vectors are untested. delta-rs exposes them; this reader does not account for them, so a table using them needs testing before I would claim support.

---

The repository

src/lakerec/
  canonical.py    what a value's bytes are, and what is refused rather than guessed
  checksum.py     the order-independent, composable multiset hash
  contract.py     the reviewed file that says what a comparison means
  sides.py        the source and the Delta target, with their reads counted
  reconcile.py    the five levels
  classify.py     evidence -> a named corruption, only where the evidence names one
  findings.py     the 11 corruption modes: what each means and what to do about it
  bisect.py       the version search, and the predicates it is allowed to search over
  history.py      _delta_log read as an incident record
  report.py       markdown, self-contained HTML, JSON
corruptions/      16 injected scenarios against real Delta tables
experiments/      detection matrix · checksum algebra · float drift · cost · bisect histories
tools/            collect_metrics.py re-measures; check_numbers.py fails the build on a stale claim
docs/adr/         the five decisions worth arguing about
docs/defense-guide.md   the interview conversation, including the questions with no good answer

266 tests, 95.1% line coverage, 86.0% branch. make verify runs the suite, re-measures every number in this README, and fails if a document quotes one that is no longer true, including this sentence.

The receipts checker does more than match digits. A metric can declare an anchor phrase: for 255 of 900 rows to pass, the document has to contain that phrase, not merely the digits 255 somewhere on the page. An earlier project of mine passed its number check while three claims were false, because the digits were present in a sentence that had come to mean something else.

Licence

MIT.