Metadata-only delete when a DELETE removes every live row of a data file

[paultmathew]

What / why

I hit this building a delete + insert incremental event-stream pipeline (repeatedly deleting a partition and re-inserting it). Every DELETE writes merge-on-read delete files/DVs even when the predicate removes every live row of a data file, so the dead files pile up and every later scan / MERGE / DELETE still has to read them plus their delete files. When a delete fully empties a file, I'd rather drop it via metadata.

This is Phase 1 and I'd like a sanity check on direction before going further.

Phase 1 (this PR) — drop fully-deleted files at flush time

We already scan every matching row, so at flush we know the exact deleted positions per file. When existing + new deleted positions == record_count (positions are 0-based ordinals, so this holds exactly when no live row survives), invalidate the data file instead of writing a delete file; otherwise fall back to the current path.

• Exact check (no stats/null/NaN edge cases), and naturally covers whole-partition deletes.
• Reuses the existing IcebergManifestDeletes -> RewriteManifestFile commit path (the one iceberg_rewrite_data_files uses).
• Adds an invalidate-only AddSnapshot path for when a delete only drops whole files (no new manifest entries).
• Also invalidates delete files provably scoped to the dropped data file (v3 deletion vectors, and DuckDB's file-scoped positional deletes via referenced_data_file). Multi-file positional delete files written by other engines still apply to surviving files, so they're left for that engine's maintenance to reclaim once dangling.

Phase 2 (follow-up) — skip the scan via strict metrics

On top of Phase 1, add a strict metrics evaluator (the strict dual of the inclusive IcebergPredicate::MatchBounds) to check per-file stats against the predicate before scanning, so provably-fully-covered files skip the scan entirely; partial files keep the MOR path. Best-effort with a safe fallback.

Tests

Fixture tests under .../catalog_agnostic/delete/:

• test_metadata_only_full_file_delete.test — partitioned v2 (positional) + v3 (DV): whole-partition delete writes no delete file; partial delete does.
• test_metadata_only_unpartitioned.test — multiple/single data files, whole vs partial, and a file emptied across two deletes (existing deletes + new == record_count) which also drops the dangling file-scoped delete file.
• test_metadata_only_with_equality_deletes.test (gated on EQUALITY_DELETE_WRITES_ENABLED) — safety: with equality deletes present, deleting the remaining live rows is not mis-detected as a whole-file drop; it falls back to MOR and reads stay correct.

Known limitations (deliberate for this phase)

• Snapshot summary counts (deleted-data-files / deleted-records) are not decremented for the metadata drop (pre-existing accounting gap); reads/cardinality are correct.
• No write-isolation conflict detection (same as today's deletes).

Questions

1. Does this direction fit what you had in mind for Feature Request: Support for INSERT OVERWRITE / Atomic Table Overwrite #620 / Feature request, copy to iceberg add support for append and overwrite :) #826?
2. Prefer keeping it transparent inside the DELETE plan (Phase 1 at flush, Phase 2 at plan time), or opt-in?
3. Anything on summary correctness or write isolation you'd want handled here vs. deferred?