Skip to content

ATTACH 'arn:aws:s3tables:...' and infra for expanding to other resources#165

Open
carlopi wants to merge 2 commits into
duckdb:mainfrom
carlopi:attach-arn
Open

ATTACH 'arn:aws:s3tables:...' and infra for expanding to other resources#165
carlopi wants to merge 2 commits into
duckdb:mainfrom
carlopi:attach-arn

Conversation

@carlopi

@carlopi carlopi commented Jul 11, 2026

Copy link
Copy Markdown
Member

Registers a pass-through:

class ArnStorageExtension : public StorageExtension {
public:
	ArnStorageExtension() {
		attach = ArnAttach;
		create_transaction_manager = ArnCreateTransactionManager;
	}
};

that will recognize paths-like strings like arn:aws:<service>:<region>:<AMI>:<type/<name>, and then either refuse them as not supported or redirect to the relevant attach.

Redirect is implemented only for arn:aws:s3tables:..., where it remaps to the iceberg-type attach on an ARN, but it's somewhat generalizable to other resources.

This PR will allow for:

LOAD aws;
--- possibly create credentials
ATTACH 'arn:aws:s3tables:us-east-1:259911478022:bucket/iceberg-on-the-browser' AS lake;
FROM lake.tpch10.customers;

and allow a path for other Catalog-shaped resources that have an ARN to be attached referencing the ARN. Once #162 lands, that would also be a prime addition to this surface.

A follow up duckdb/ducdkb side would be registering arn->aws remapping, allowing autoloading to happen, that makes so that if credentials are already on the user machine, the single statement version to also work:

ATTACH 'arn:aws:s3tables:us-east-1:259911478022:bucket/iceberg-on-the-browser' AS lake;

This is on top of #164, bumped to include the required duckdb/duckdb#23759

@carlopi carlopi changed the title Attach arn Attach 'arn:aws:s3tables:...' and infra for expanding to other resources Jul 11, 2026
@carlopi carlopi changed the title Attach 'arn:aws:s3tables:...' and infra for expanding to other resources ATTACH 'arn:aws:s3tables:...' and infra for expanding to other resources Jul 11, 2026
@carlopi
carlopi force-pushed the attach-arn branch 2 times, most recently from 6effd43 to c473277 Compare July 11, 2026 12:07
@carlopi

carlopi commented Jul 14, 2026

Copy link
Copy Markdown
Member Author

Question from @Tmonster: what happens if I do:

ATTACH 'arn:aws:s3tables:us-east-1:259911478022:bucket/iceberg-on-the-browser' AS lake (OPT1, OPT2);

? How are those treaded / conflicts & co.

Comment thread src/arn_storage.cpp
carlopi added 2 commits July 17, 2026 09:43
Parse an AWS ARN from the verbatim ATTACH path (AttachedDatabase::
GetOriginalPath()) and dispatch by service to the backing storage backend
(s3tables -> iceberg). Lives in its own arn_storage.cpp; aws_extension.cpp
just calls ArnStorage::Register().

Registered under two storage type names, both backed by the same extension
instance:
  - 'aws', for the explicit `ATTACH '<arn>' (TYPE aws)` form.
  - 'arn', because core derives the db type from the 'arn:' path prefix, so a
    bare `ATTACH '<arn>'` looks for a storage type of that name. Registering it
    here serves that form without needing a core arn->aws extension alias.

Also include duckdb/logging/logger.hpp explicitly in aws_secret.cpp; it is no
longer pulled in transitively via duckdb.hpp.

Requires a duckdb with AttachedDatabase::GetOriginalPath().

Adapted from arn_attach/.github/patches/extensions/aws/0002-arn-storage-dispatch.patch.
@carlopi

carlopi commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

Options are passed down correctly, with a throw on conflicts.

One question I had is whether there was a better than hand parsing the arn, and there is none in the AWS's SDK and non API to do so.
I have not researched third party libraries, but we might be better off doing this in a customized way.

@carlopi
carlopi requested a review from Tmonster July 17, 2026 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants