Skip to content

[DO NOT MERGE (yet)] Avoid Iceberg CTAS for table materialization#725

Draft
dataders wants to merge 1 commit into
duckdb:masterfrom
dataders:codex/iceberg-rest-ctas-workaround
Draft

[DO NOT MERGE (yet)] Avoid Iceberg CTAS for table materialization#725
dataders wants to merge 1 commit into
duckdb:masterfrom
dataders:codex/iceberg-rest-ctas-workaround

Conversation

@dataders

@dataders dataders commented Apr 24, 2026

Copy link
Copy Markdown
Contributor

Warning

this is hopefully a soon to be obviated workaround.
the ideal fix is upstream in the duckdb-iceberg extension itself: duckdb/duckdb-iceberg#962

What changed

  • Detect DuckDB attachments declared with type: iceberg.
  • Avoid Iceberg CREATE TABLE AS for table materializations.
  • Use CREATE TABLE (...) followed by INSERT INTO ... SELECT for Iceberg targets.
  • Keep the existing temp+rename path for normal DuckDB and DuckLake tables.

Why

DuckDB Iceberg REST can execute CTAS during prepare through ADBC/dbt-fusion, then fail during execute because the table already exists. This mirrors the Fusion fix in https://github.com/dbt-labs/fs/pull/9658 and tracks dbt-labs/dbt-core#14547.

Verification

uv run --with pytest --with-editable . pytest tests/unit/test_duckdb_adapter.py -k iceberg
uv run --with pytest --with-editable . pytest tests/functional/adapter/test_ducklake_partitioned_by.py

Draft PR for early review and adapter-maintainer heads-up.

@mironovich

mironovich commented May 20, 2026

Copy link
Copy Markdown

@dataders
hi! big thanks for your work!
I'm using duckdb with iceberg and thus watch closely this issue.
Had problem with CTAS
Now duckdb 1.5.3 came out, CTAS work, but i get error on next step

sources:
  - name: streams
    database: sq
    schema: streams
    tables:
      - name: google_114

Where sq - attached iceberg warehouse with rest catalog

-- model: stg_google with data from
{{ config(materialized='table') }}
select * from {{ source('streams', 'google_114')}}

-- command: dbt run -s stg_google

-- sql in logs:
create  table
      "sq"."analytics"."stg_google__dbt_tmp"
    as (
select * from "sq"."streams"."google_114";

alter table "sq"."analytics"."stg_google__dbt_tmp" rename to "stg_google"
DuckDB adapter: duckdb error: Catalog Error: This table (stg_google__dbt_tmp) was modified already, can't be renamed!

here is PR for alter table .. rename to..
duckdb/duckdb-iceberg#924
where its said
We don't allow creation of the table or other ALTER statements to happen in the same transaction as a RENAME of the table.
Is this the reason?
Can you please give some advice, where to look?

Should i just rewrite materialization?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants