feat(s3tables): support for AWS Lake Formation filters for S3 Tables by The-Alchemist · Pull Request #1108 · duckdb/duckdb-iceberg

The-Alchemist · 2026-06-23T19:44:32Z

Summary

Adds Lake Formation (LF from now on ) data filter support for AWS Glue Iceberg catalogs. When enabled, DuckDB participates in Lake Formation’s application integration / external data filtering flow: it discovers the caller’s effective row filter via Glue, vends scoped S3 credentials via Lake Formation, and enforces the filter at scan time so user predicates cannot widen the grant.

Use Case

AWS admins want to filter S3 Tables by, say, account_id, for simple row-level security. This is support OOTB in AWS, but requires a separate credential path and APIs that duckdb-iceberg does not use.

Out of Scope

Column filters (will be a separate PR). This is just for row filters.

Motivation

Glue Iceberg REST catalogs expose table metadata, but row-level access control for Lake Formation–registered tables is enforced through a separate integration path:

Glue returns the caller’s effective filter policy (GetUnfilteredTableMetadata, and GetUnfilteredPartitionsMetadata for partitioned tables).
Lake Formation returns temporary, scope-down S3 credentials (GetTemporaryGlueTableCredentials / GetTemporaryGluePartitionCredentials).

Without this, a client attached to Glue with normal credentials could read more data than Lake Formation grants allow, resulting in runtime errors.

How it works

sequenceDiagram
    participant User
    participant DuckDB
    participant GlueIRC as Glue Iceberg REST
    participant GlueAPI as Glue Data Catalog API
    participant LF as Lake Formation
    participant S3

    User->>DuckDB: ATTACH ... LAKE_FORMATION_DATA_FILTERS true
    User->>DuckDB: SELECT ... FROM catalog.schema.table
    DuckDB->>GlueIRC: GetTable (metadata, no vended-credentials header)
    DuckDB->>GlueAPI: GetUnfilteredTableMetadata (CELL_FILTER_PERMISSION)
    GlueAPI-->>DuckDB: row_filter SQL, queryAuthorizationId, ...
    DuckDB->>LF: GetTemporaryGlueTableCredentials (scoped S3 keys)
    LF-->>DuckDB: temporary credentials
    DuckDB->>DuckDB: Apply mandatory row filter (bind + optimizer)
    DuckDB->>S3: Read Parquet with LF-scoped credentials

Credential path: IRC access_delegation_mode stays none (no X-Iceberg-Access-Delegation: vended-credentials). S3 access uses LF temporary credentials instead of catalog-vended credentials.
Policy discovery: Per-table LF policy is fetched from Glue and cached on IcebergTableInformation. For partitioned tables, partition-scoped credentials are refreshed lazily during manifest enumeration.
Enforcement: LF row-filter SQL is parsed and bound as mandatory scan filters at bind time, with a LakeFormationRowFilterOptimizer safety net so predicates like WHERE id = 999 cannot bypass the grant.
Auth: Requires SIGV4 catalog authentication. If the catalog secret uses assume_role_arn, the extension assumes that role with an STS session tag LakeFormationAuthorizedCaller=<LF_SESSION_TAG> (see application integration settings).

Usage

CREATE SECRET glue_lf_secret (
    TYPE S3,
    REGION 'us-east-1',
    PROVIDER credential_chain,
    CHAIN 'sts',
    ASSUME_ROLE_ARN 'arn:aws:iam::<account>:role/<lf-role>'
);

ATTACH '<catalog-id>' AS lf_datalake (
    TYPE ICEBERG,
    ENDPOINT_TYPE 'GLUE',
    SECRET glue_lf_secret,
    LAKE_FORMATION_DATA_FILTERS true,
    LF_SESSION_TAG 'duckdb'
);

SELECT * FROM lf_datalake.test_inserts.basic_insert_test;

Attach options

Option	Required	Description
`LAKE_FORMATION_DATA_FILTERS`	Yes	Enables Lake Formation mode
`LF_SESSION_TAG`	Yes	Must match a value in the account’s authorized session tag list
`ENDPOINT_TYPE 'GLUE'`	Yes	LF integration is Glue-only
`access_delegation_mode`	No	Must be `none` (default) or omitted; not `vended_credentials`
`ENDPOINT_TYPE 'GLUE'`	Yes	LF integration requires the Glue Iceberg REST endpoint (even when the catalog is an S3 Tables catalog)
`ENDPOINT_TYPE 'S3TABLES'`	—	Not supported with `LAKE_FORMATION_DATA_FILTERS`
`access_delegation_mode 'lf_filtered'` is rejected; use `LAKE_FORMATION_DATA_FILTERS true` instead.

v1 limitations

Fails closed on grant shapes we do not enforce yet:

Row-level filters only — rejects cell-level / data cell filters
No column-level permissions — tables must grant all columns (or use row filters only)
Not available in WASM builds (EMSCRIPTEN)

Testing

Cloud tests: test/sql/cloud/lakeformation/* — row filters (partitioned/unpartitioned), bypass attempt, no grant, unsupported column filter
- requires set up on the AWS side; not sure if the AWS permissions are set up correctly to allow creation of necessary resources
CI setup: scripts/setup_lf_data_filters.py provisions data cell filters and grants; enables external data filtering on the test account
Table fixtures: scripts/create_s3_insert_table.py extended for LF test tables on Glue
- again, not sure if the AWS account the CI environment uses can make these resources

AWS documentation

Glue vs S3 Tables

This integration uses the Glue Iceberg REST catalog (ENDPOINT_TYPE 'GLUE'), which is the entry point Lake Formation’s application integration expects. It does not support attaching with ENDPOINT_TYPE 'S3TABLES'.

Cloud tests use this pattern: Iceberg metadata comes from Glue REST; physical table data lives in S3 Tables; LF row filters and temporary credentials are applied on top.

Enforce LF row filters at scan time with temporary Glue credentials, cloud test suite, and CI setup scripts. Preserve DuckDB Labs defaults for S3 Tables/Glue regions and bucket names so existing cloud tests pass. Document Glue/LF API boundaries where policy discovery, credential vending, and row-filter enforcement interact.

Remove duplicate jobs and duckdb_iceberg_ref passthrough so scheduled and manual cloud tests validate and run against the triggering branch.

…ckDB API compile errors. Reject access_delegation_mode lf_filtered in favor of the boolean flag, rename the internal mode to LAKE_FORMATION, and update Lake Formation code for Identifier-based secret/bind APIs.

Reintroduce REGION, TABLE_BUCKET, and DATABASE_NAME as canonical defaults for S3 Tables config and drop unused DATABASE variables.

Tmonster · 2026-06-24T08:10:09Z

Hi @The-Alchemist, can you provide a PR description with some more documentation as to what is being added here? Will make reviewing a lot easier

The-Alchemist · 2026-06-24T15:32:17Z

@Tmonster

Hi @The-Alchemist, can you provide a PR description with some more documentation as to what is being added here? Will make reviewing a lot easier

Not sure how i missed it. Thanks!

Resolve CloudTestingReusable.yml conflict by keeping duckdb_iceberg_ref checkout and Lake Formation test steps. Signed-off-by: Karl Pietrzak <karl@medplum.com>

Restore format-check compliance after CI reported formatting drift in LF integration code. Signed-off-by: Karl Pietrzak <karl@medplum.com>

Tishj · 2026-06-25T08:31:23Z

+	return schema_component.encoded;
+}
+
+static string ExtractAccountId(const string &warehouse) {


Looks a lot like logic we have in IcebergCatalog::GetConfig

Tishj · 2026-06-25T08:37:26Z

 				throw ConversionException("Could not get interval information from %s", interval_option.ToString());
 			}
 			attach_options.max_table_staleness_micros = interval_in_micros;
+		} else if (lower_name == "lake_formation_data_filters") {


SIGV4Authorization::FromAttachOptions ?

Tishj · 2026-06-25T09:16:56Z

I went over the PR, and as far as I can tell, the core of what is added is:

Re-invent vended-credentials, with the added granularity of making the credentials per partition. Something that could be proposed as an addition to the vended-credentials spec.
Push additional filters on top of the scan, in an unsafe way, as optimizers can be disabled

As it stands, this PR is much too invasive, and much too large in scope to review

The-Alchemist · 2026-06-25T13:51:02Z

I went over the PR, and as far as I can tell, the core of what is added is:

Re-invent vended-credentials, with the added granularity of making the credentials per partition. Something that could be proposed as an addition to the vended-credentials spec.

Push additional filters on top of the scan, in an unsafe way, as optimizers can be disabled

As it stands, this PR is much too invasive, and much too large in scope to review

Thanks for the feedback, @Tishj . I totally agree the PR is too big.

We can certainly break this up into separate PRs:

lake formation grants as addition to vended-credentials
AWS Glue-specific code
lake formation specific code

WDYT?

Tmonster · 2026-06-25T14:24:56Z

@The-Alchemist a bigger issue here is that this PR starts to implement features that deviate significantly from what is in the Iceberg spec and in the IRC. We would like to keep this as close to pure Iceberg features as possible. In addition, we would like to get rid of the aws dependency in this repo since we only have some legacy code that relies on it, which we will deprecate. So also for that reason, adding LakeFormation filters is not idea.

We are discussing how to go about adding this kind of functionality in the DuckDB-Iceberg ecosystem. When we have a better idea, we can update here.

The-Alchemist · 2026-06-25T16:39:03Z

@Tmonster : understood. does a Community Extension make more sense for this, then?

Tmonster · 2026-06-26T07:14:16Z

For now yes, a community extension is the way to go 👍

The-Alchemist · 2026-06-27T17:18:33Z

@Tmonster : thanks for the quick replies! WRT to removing the aws dependency: does this mean that S3 Tables support is planned to be removed? Unfortunately, AWS S3 tables auth depends on the aws library.

Anyway, wrote a separate PR to allow external auth in this repo to allow community extensions to "plug in" their own auth mechanism. That would allow this PR to basically become an external repo and community extension.

#1118

Tmonster · 2026-07-06T08:26:10Z

S3Tables will still be supported, since we have hand rolled our own Sigv4 signing in this repo. In it's current state the AWS extension is mostly used for finding aws credentials on a machine.

It is on our roadmap to expand it to include more aws api functionality.

The-Alchemist added 4 commits June 22, 2026 19:27

Fix cloud testing workflows to match upstream v1.5-variegata.

4ffb4e6

Remove duplicate jobs and duckdb_iceberg_ref passthrough so scheduled and manual cloud tests validate and run against the triggering branch.

Make LAKE_FORMATION_DATA_FILTERS the sole LF attach option and fix Du…

cd34563

…ckDB API compile errors. Reject access_delegation_mode lf_filtered in favor of the boolean flag, rename the internal mode to LAKE_FORMATION, and update Lake Formation code for Identifier-based secret/bind APIs.

Restore main-branch defaults in create_s3_insert_table.py.

8408e7a

Reintroduce REGION, TABLE_BUCKET, and DATABASE_NAME as canonical defaults for S3 Tables config and drop unused DATABASE variables.

The-Alchemist marked this pull request as ready for review June 24, 2026 00:47

The-Alchemist changed the title ~~WIP: feat(s3tables): support for AWS Lake Formation filters for S3 Tables~~ feat(s3tables): support for AWS Lake Formation filters for S3 Tables Jun 24, 2026

The-Alchemist added 2 commits June 24, 2026 11:34

Merge upstream main into feature/lake-formation-data-filters.

62c35eb

Resolve CloudTestingReusable.yml conflict by keeping duckdb_iceberg_ref checkout and Lake Formation test steps. Signed-off-by: Karl Pietrzak <karl@medplum.com>

Apply clang-format fixes for Lake Formation files.

38e4256

Restore format-check compliance after CI reported formatting drift in LF integration code. Signed-off-by: Karl Pietrzak <karl@medplum.com>

Tishj reviewed Jun 25, 2026

View reviewed changes

Tishj marked this pull request as draft June 25, 2026 12:26

Tmonster closed this Jun 25, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(s3tables): support for AWS Lake Formation filters for S3 Tables#1108

feat(s3tables): support for AWS Lake Formation filters for S3 Tables#1108
The-Alchemist wants to merge 6 commits into
duckdb:mainfrom
The-Alchemist:feature/lake-formation-data-filters

The-Alchemist commented Jun 23, 2026 •

edited

Loading

Uh oh!

Tmonster commented Jun 24, 2026

Uh oh!

The-Alchemist commented Jun 24, 2026

Uh oh!

Tishj Jun 25, 2026

Uh oh!

Tishj Jun 25, 2026

Uh oh!

Tishj commented Jun 25, 2026

Uh oh!

The-Alchemist commented Jun 25, 2026

Uh oh!

Tmonster commented Jun 25, 2026

Uh oh!

The-Alchemist commented Jun 25, 2026

Uh oh!

Tmonster commented Jun 26, 2026

Uh oh!

The-Alchemist commented Jun 27, 2026

Uh oh!

Tmonster commented Jul 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

The-Alchemist commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Use Case

Out of Scope

Motivation

How it works

Usage

Attach options

v1 limitations

Testing

AWS documentation

Glue vs S3 Tables

Uh oh!

Tmonster commented Jun 24, 2026

Uh oh!

The-Alchemist commented Jun 24, 2026

Uh oh!

Tishj Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

Tishj Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

Tishj commented Jun 25, 2026

Uh oh!

The-Alchemist commented Jun 25, 2026

Uh oh!

Tmonster commented Jun 25, 2026

Uh oh!

The-Alchemist commented Jun 25, 2026

Uh oh!

Tmonster commented Jun 26, 2026

Uh oh!

The-Alchemist commented Jun 27, 2026

Uh oh!

Tmonster commented Jul 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

The-Alchemist commented Jun 23, 2026 •

edited

Loading