Catalog-authoritative snapshots (Unity Catalog) - tracking

[cdelmonte-zg]

Today catalog-managed Delta tables (catalogManaged / catalogOwned-preview) are deliberately detected and refused: a filesystem-only read of _delta_log is not authoritative for them, because the catalog can hold commits not yet published to storage and defines the maximum authoritative version.

The planned next step moves that boundary deliberately instead of deleting it: obtain the snapshot through delta-kernel-rs's catalog-managed path (catalog-provided log tail, catalog-authoritative max version, vended storage credentials) and make the snapshot's provenance part of the report - which authority established the version, and what came from the catalog vs. what was observed on storage. Unity Catalog is the first concrete adapter, isolated from the core analysis path; the pruning pipeline itself is unchanged once the snapshot exists.

Blocked upstream: the Unity Catalog crates (delta-kernel-unity-catalog, unity-catalog-delta-client-api, unity-catalog-delta-rest-client) exist on delta-kernel-rs main at 0.1.0 but are not yet published to crates.io - crate metadata and READMEs landed 2026-08-07 (delta-io/delta-kernel-rs#3061, delta-io/delta-kernel-rs#3062), and the remaining piece is the per-crate release tooling in delta-io/delta-kernel-rs#3066 (draft). A published crate is a hard requirement for our own crates.io distribution (git dependencies cannot be published). The core snapshot-builder APIs (with_log_tail, with_max_catalog_version) are already in the kernel version we pin.

Plan once unblocked:

1. bump to a kernel version compatible with the published UC crates (Bump delta_kernel 0.26 -> 0.27.1 #137 keeps us close in the meantime);
2. ADR (proposed before implementing, per docs/adr/README.md) settling the open design questions - CLI surface for table references and endpoint configuration, provenance schema, feature gating and distribution matrix;
3. break this tracker into sub-issues from the accepted ADR and schedule them in a release milestone.

Until then this issue tracks the upstream dependency; watch delta-io/delta-kernel-rs#3066.