Add Iceberg View support for REST catalogs by LucaSoato · Pull Request #887 · duckdb/duckdb-iceberg

LucaSoato · 2026-04-03T15:56:45Z

Summary

Implements CREATE VIEW, DROP VIEW, and view loading/querying for Iceberg REST catalogs, following the Iceberg View Spec.

Views are stored with dialect "duckdb" per the spec's multi-dialect representation model
When loading, prefers "duckdb" dialect, falls back to any available SQL representation
Views from other engines (Spark, Trino) that can't be parsed are still visible in the catalog and droppable — querying them gives a clear error instead of crashing
Works with any Iceberg REST catalog (tested against apache/iceberg-rest-fixture)

Closes #314

Known workaround

CreateViewInfo::Copy() in DuckDB core does not copy the names field. We patch it after copying. Fix submitted upstream: duckdb/duckdb#21817

Test plan

CREATE VIEW, SELECT from view, DROP VIEW
CREATE VIEW IF NOT EXISTS (idempotent)
CREATE OR REPLACE VIEW (drop + create in same transaction)
DROP VIEW IF EXISTS on non-existent view
View with column aliases
Views from other engines give clear error on SELECT, still droppable
Full irc_any_catalog test suite passes (169 cases, 83,661 assertions, 0 regressions)

LucaSoato · 2026-04-03T18:38:51Z

Updated: fixed clang-format violations and force-pushed.

LucaSoato · 2026-04-04T11:32:14Z

Hi, I've tested a lot this yesterday and I'm adding a small fix for a deadlock in ScanViews. When views exist in the REST catalog but aren't cached locally ScanViews held entry_lock and called GetViewEntry which tried to acquire the same lock. I've split it into a locking wrapper + internal method. Includes a regression test that hangs without the fix and passes with it.

Tmonster · 2026-04-08T21:18:17Z

Hi @LucaSoato,

Thanks for the PR! This is a pretty big feature, but we will take a look soon. Unfortunately, this is a bit too big of a feature to merge for the v1.5.2 release. We will try to keep main compatible with v1.5.2 so you can install a nightly binary to use this feature if/when we merge

LucaSoato · 2026-04-16T11:59:28Z

Hi @Tmonster,
It's totally understandable!
I'm using it as unsigned extension right now and testing it thoroughly for several use cases.
One thing I'm noticing is that sometimes it gets stuck in the duckdb ui, I think it's related to the fact that the UI uses threads in a different way.
Let me know what I can do to help you to release this feature in 1.5.3, in the meantime I'm happy that my other PR in the duckdb main project was accepted, so we don't need a workaround in CreateViewInfo::Copy().

Tmonster

@LucaSoato

Thanks! A couple of mostly minor comments.

Could you also retarget this to v1.5-variegata? That way we can try and get it in for v1.5.3

Tmonster · 2026-04-29T10:46:56Z

+	auto entry_name = info.name;
+
+	// Handle VIEW_ENTRY drops
+	if (info.type == CatalogType::VIEW_ENTRY) {


nit: can we get a switch statement here for different CatalogTypes? The default case can just throw a NotImplementedException

Switch statement added!

https://github.com/duckdb/duckdb-iceberg/blob/28bd7f45/src/catalog/rest/catalog_entry/iceberg_schema_entry.cpp#L121-L178

Tmonster · 2026-04-29T10:47:35Z

nit: Can we check cascade before the View type? I think cascade is not even an Iceberg compatible keywork

Yes, fixed!
https://github.com/LucaSoato/duckdb-iceberg/blob/59733d610d662ddba4f317417db25abcfd831c53/src/catalog/rest/catalog_entry/iceberg_schema_entry.cpp#L113-L125

Tmonster · 2026-04-29T10:50:12Z

+	auto view_key = IcebergTableInformation::GetTableKey(namespace_items, info.view_name);
+
+	// Workaround: CreateViewInfo::Copy() does not copy `names`, so we patch it manually.
+	// See: https://github.com/duckdb/duckdb/pull/21817


Looks like duckdb/duckdb#21819 was merged, so the workaround can be removed?

Done!
https://github.com/LucaSoato/duckdb-iceberg/blob/db61a521/src/catalog/rest/catalog_entry/iceberg_schema_entry.cpp#L252-L257

Tmonster · 2026-04-29T10:54:21Z

+		auto repr_obj = yyjson_mut_arr_add_obj(doc, repr_arr);
+		yyjson_mut_obj_add_strcpy(doc, repr_obj, "type", "sql");
+		yyjson_mut_obj_add_strcpy(doc, repr_obj, "sql", view_sql.c_str());
+		yyjson_mut_obj_add_strcpy(doc, repr_obj, "dialect", "duckdb");


duckdb or DuckDB or does it not matter?

I've used the lowercase and saved it to the constant IcebergConstants::ViewDuckDBDialect
https://github.com/LucaSoato/duckdb-iceberg/blob/db61a52199e69e2f2a900693752b374007342db6/src/include/common/iceberg_constants.hpp#L17-L19

In the Iceberg View examples, I saw them always using lowercase.

Tmonster · 2026-04-29T10:56:10Z

 	// commit updates to multiple tables in an atomic transaction
 	supported_urls.insert("POST /v1/{prefix}/transactions/commit");
+	// list all view identifiers
+	supported_urls.insert("GET /v1/{prefix}/namespaces/{namespace}/views");


Views are technically not in the list of default supported endpoints, so I think this should be removed

https://github.com/apache/iceberg/blob/8ac703067a2adfae1928748712dc1d47dbc3c22b/open-api/rest-catalog-open-api.yaml#L106

Removed the rows, I've left a comment so if anyone looks into this knows why the endpoints aren't included.

LucaSoato@27f4aeb

Tmonster · 2026-04-29T10:59:36Z

+	}
+
+	// Check if the view endpoint is supported
+	if (ic_catalog.supported_urls.find("GET /v1/{prefix}/namespaces/{namespace}/views/{view}") ==


Should we do this check first? Any view that we create that we later call CreateViewEntryInternal will have to pass this check right?

So right now we check:

if it's in transaction.deleted_views

if it's in view_catalog_entries cache

if we created it in transaction.created_views

and only then if views are in supported urls

Technically it's right because the first three operations are in memory and do not require URLs. The thing is that if in the same transaction we try to create a view, referencing the same view later, it will fail anyway but only during the commit of such operation for catalogs that don't support views.

If you want we can add a POST /v1/{prefix}/namespaces/{namespace}/views endpoint check at the top of CreateView so it fails immediately on no-views catalogs?

Tmonster · 2026-04-29T11:00:27Z

+	try {
+		load_result = IRCAPI::GetView(context, ic_catalog, schema, view_name);
+	} catch (...) {
+		return nullptr;


nit: Can we not have a naked catch here? Not a fan of those when it comes to debugging

I've improved this part by refactoring IRCAPI::GetView to return APIResult<unique_ptr<const LoadViewResult>>, similarly to what happens in GetTable

https://github.com/LucaSoato/duckdb-iceberg/blob/db61a52199e69e2f2a900693752b374007342db6/src/catalog/rest/api/catalog_api.cpp#L529

Tmonster · 2026-04-29T11:02:17Z

+			if (!repr.has_sqlview_representation) {
+				continue;
+			}
+			if (repr.sqlview_representation.dialect == "duckdb") {


I think we should hardcode the "duckdb" string somewhere. Maybe in include/common/IcebergConstants.hpp?

Yes, done that as well
https://github.com/LucaSoato/duckdb-iceberg/blob/db61a52199e69e2f2a900693752b374007342db6/src/catalog/rest/iceberg_table_set.cpp#L454

Tmonster · 2026-04-29T11:02:58Z

+		if (view_sql.empty()) {
+			for (auto &repr : version.representations) {
+				if (repr.has_sqlview_representation) {
+					view_sql = repr.sqlview_representation.sql;


Can we also log a warning that no DuckDB dialect has been found?
a search for DUCKDB_LOG_WARNING should reveal some examples

Yup, logged:
https://github.com/LucaSoato/duckdb-iceberg/blob/db61a52199e69e2f2a900693752b374007342db6/src/catalog/rest/iceberg_table_set.cpp#L463-L467

LucaSoato · 2026-05-05T21:10:32Z

Thanks so much for your feedback, I'll get back to this asap 🚀

i-m-nk · 2026-05-19T09:32:16Z

@LucaSoato can we have a support for user defined dialect value (may be from a variable/global), some Kond of override functionality. That way user can try to read multi dialects.

LucaSoato · 2026-05-20T13:33:55Z

@LucaSoato can we have a support for user defined dialect value (may be from a variable/global), some Kond of override functionality. That way user can try to read multi dialects.

@Tmonster what do you think about it?

Tmonster

Thanks for the updates.
With a more thorough review, I think we are still missing some testing/functionality that I would appreciate having in this PR

There are no tests on the unhappy path. Can we add some?

It would be nice to see

Creating a view that selects from table columns that do not exist. (I would expect this to error on view creation similar to how it does when operating on duckdb tables)
Selecting a column from a view where the column does not exist in the view.

Other tests I would like to see

A view that does select *
A view that does select * exclude
A view that joins tables

Also can we test how iceberg views are handled when we call

from duckdb_views();
from duckdb_tables();
show all tables;

I don't expect full Iceberg view functionality integrated here just yet, I just want to make sure DuckDB doesn't crash in this scenario.

One more piece of functionality I would like to see, is a table function called iceberg_view_metadata which returns the result of a call to the LoadView API. That can make view discovery on a Rest catalog a bit easier. We have a similar function for tables now, iceberg_load_table_response, you can use that as a template, we put a lot of the JSON into a variant.

I also like the idea of a preferred SQL dialect. Maybe that is something we can add to the view properties? I see in the LoadViewResult we can add properties to a view. I think that might be out of scope for this PR though, and can be added in a follow up. A config option would also work. DuckDB v2 will use the PEG parser, so we should be able to extend our parser support, and then we can have a statement like SELECT * FROM iceberg_view WITH ('sql'='duckdb), but that will obvs come much later.

Tmonster · 2026-05-21T07:00:30Z

+			drop_info.name = info.view_name;
+			drop_info.cascade = false;
+			drop_info.if_not_found = OnEntryNotFound::RETURN_NULL;
+			DropEntry(context, drop_info, false);


This will drop the entry regardless of if he transaction succeeds right?
Currently DuckDB-Iceberg doesn't support CREATE OR REPLACE for tables because of this scenario. I would like to stay consistent with that if possible

Tmonster · 2026-05-21T07:03:33Z

+statement error
+select * from test_view_v1;
+----
+does not exist


nit: <REGEX>:.*does not exist.* is what we normally use

Tmonster · 2026-05-21T07:11:07Z

+		}
+	}
+
+	// Generate default column names if the caller gave us types but no names.


Can we do some more work here and potentially bind the sql query to make sure it is valid?

Hi @Tmonster,
I think that this already happens here:
https://github.com/duckdb/duckdb/blob/c028854da4348b4d32731e460b75973364f79b74/src/planner/binder/statement/bind_create.cpp#L204

Tmonster · 2026-06-15T15:32:17Z

Hi @LucaSoato is this still something you still want merged, otherwise we can close to reduce clutter?

LucaSoato · 2026-06-19T13:13:40Z

Hi everyone,
Thanks for your patience, I should be able to push the requested changes this weekend.

- DropEntry: use switch on CatalogType, default throws NotImplementedException - DropEntry: move cascade check above type switch (cascade is not in the Iceberg REST spec) - CreateView: remove CreateViewInfo::Copy() names workaround (fixed by duckdb#21819) - Hoist "duckdb" dialect and "sql" representation type into IcebergConstants - Log a warning when falling back from DuckDB dialect to another dialect on view load Also set drop_info.type = TABLE_ENTRY in the internal DoTableRename and DoTableDeletes callers so the new switch default doesn't reject them.

- add iceberg_view_metadata table function (mirrors iceberg_load_table_response) - reject CREATE OR REPLACE for views, consistent with CreateTable / HandleCreateConflict - add tests: unhappy paths (bad column/table at create), select *, select * exclude, joins - test that duckdb_views() / duckdb_tables() / show all tables do not crash with iceberg views - use <REGEX> error matchers and fix the stale test name header

LucaSoato · 2026-06-25T10:42:48Z

Hi @Tmonster, thanks for the patience!

Just a small recap regarding the changes:

unhappy paths are covered, you can check here and below;
CREATE OR REPLACE now throws NotImplementedException, since it's non-transactional;
new iceberg_view_metadata table function, returns the raw LoadView response;

Related to the preferred SQL dialect, it's a nice follow-up, let's leave it out of this PR and we can add it later.
I'm replying also to the other comment of yours related to the binding of the sql query, it's something that we have already.

Thanks again and, as always, let me know if there's anything else left for this PR.

LucaSoato · 2026-06-25T10:44:39Z

p.s.
I've rebased on top of v1.5-variegata, the tests go on fine except for two that fail even on a clean v1.5-variegata build.

LucaSoato · 2026-06-25T12:52:58Z

(a small format fix is coming)

Tmonster · 2026-06-25T15:38:34Z

Thanks @LucaSoato, will get to reviewing, It hurts me to say this, but we are going to limit new features to v1.5. Would you mind rebasing and retargeting main for this?

LucaSoato · 2026-06-25T17:00:39Z

It's alright, will rebase to main.

Tishj · 2026-07-08T10:24:57Z

+		throw BinderException("Cannot create view in Iceberg without a query");
+	}
+	auto &context = transaction.GetContext();
+


We should check that binding_mode is set to "bind on create":

//! Whether or not to bind the view on create CreateViewBindingMode binding_mode = CreateViewBindingMode::BIND_ON_CREATE;

Since we don't want to submit to the Catalog without verifying correctness

LucaSoato force-pushed the feature/iceberg-view-support branch from 1a92243 to 3c0c883 Compare April 3, 2026 18:15

Tmonster reviewed Apr 29, 2026

View reviewed changes

LucaSoato changed the base branch from main to v1.5-variegata May 13, 2026 13:40

LucaSoato force-pushed the feature/iceberg-view-support branch 2 times, most recently from f4a95bd to 267f371 Compare May 13, 2026 13:51

Tmonster reviewed May 21, 2026

View reviewed changes

LucaSoato added 8 commits June 24, 2026 16:27

add iceberg view support for REST catalogs (CREATE/DROP/SELECT)

9e504b4

fix deadlock in ScanViews when views are not cached

ab73caa

use switch on CatalogType in DropEntry cascade check

ac84a11

replace naked catch in GetView path with APIResult error handling

ad626ee

remove redundant types/names re-binding in CreateView

488adc5

remove view endpoints from default supported set

27a9b89

LucaSoato force-pushed the feature/iceberg-view-support branch from 27f4aeb to 686885f Compare June 25, 2026 10:29

fix view test name header and account for view-listing http requests

10eec60

Tishj reviewed Jul 8, 2026

View reviewed changes

Uh oh!

Conversation

LucaSoato commented Apr 3, 2026

Summary

Known workaround

Test plan

Uh oh!

LucaSoato commented Apr 3, 2026

Uh oh!

LucaSoato commented Apr 4, 2026

Uh oh!

Tmonster commented Apr 8, 2026

Uh oh!

LucaSoato commented Apr 16, 2026

Uh oh!

Tmonster left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LucaSoato May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LucaSoato commented May 5, 2026

Uh oh!

i-m-nk commented May 19, 2026

Uh oh!

LucaSoato commented May 20, 2026

Uh oh!

Tmonster left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tmonster commented Jun 15, 2026

Uh oh!

LucaSoato commented Jun 19, 2026

Uh oh!

LucaSoato commented Jun 25, 2026

Uh oh!

LucaSoato commented Jun 25, 2026

LucaSoato May 18, 2026 •

edited

Loading