Feature Request: Support for INSERT OVERWRITE / Atomic Table Overwrite

[carc-prathyush-shankar]

Summary

Add INSERT OVERWRITE support to enable atomic single-snapshot table overwrites, matching the behavior available in Spark, Trino, and PyIceberg. There currently does not seem to be a way to atomically replace all rows in a table with new data in a single commit.

Use Case

Batch ETL pipelines commonly need to fully replace table contents on each run. For example:

INSERT OVERWRITE my_catalog.my_schema.target_table
SELECT * FROM source_table WHERE ...

This pattern is standard in workflows where a batch job reads from source tables, applies transformations, and overwrites a target table.

Additional Context

The Iceberg spec already supports the overwrite snapshot operation type, and I see it's defined in the codebase (IcebergSnapshotOperationType::OVERWRITE). The AddUpdateSnapshot function already creates overwrite snapshots for UPDATE operations, so the underlying infrastructure may partially exist.

[djouallah]

currently i use truncate and insert but sometimes, insert does not works and i endup with an empty table.

[paultmathew]

@Tishj @Tmonster when you get a chance, could you take a look at the direction in #1178? Mainly want to check the approach before I invest further.

The core idea: a DELETE that removes every live row of a data file drops the file via a metadata-only manifest edit instead of writing a positional delete / DV, reusing the IcebergManifestDeletes → RewriteManifestFile path that iceberg_rewrite_data_files already uses.

One thing I want to be upfront about on the design. This PR detects the whole-file delete at flush time — it still scans the matching rows and only then notices they cover the entire file. So it fixes downstream read amplification (no delete files left for later scans/merges to apply), but it doesn't avoid the read cost of the delete itself. The more efficient version is a plan-time path that identifies fully-covered files/partitions straight from manifest metadata and skips the scan entirely (Iceberg/Spark's strict-projection deleteWhere model). I'm treating that as a future phase — it needs strict-projection onto the partition spec and a no-scan plan shape, whereas the flush-time approach here is exact and simple. It seems like a decent, if less efficient, first step, and the commit-layer plumbing (invalidate-only snapshot, file-scoped delete-file cleanup) is shared with the plan-time path either way.

Three things I'd value your read on:

1. Does this fit what you had in mind for Feature Request: Support for INSERT OVERWRITE / Atomic Table Overwrite #620 / Feature request, copy to iceberg add support for append and overwrite :) #826?
2. Is the flush-time approach a reasonable first step, with the plan-time (no-scan) path as a follow-up — or would you rather go straight to plan-time?

No rush, and happy to rework the shape if you'd approach it differently.