feat(iceberg): add AWS Glue Data Catalog + S3 Tables catalog modes#1451
Open
aaj3f wants to merge 2 commits into
Open
feat(iceberg): add AWS Glue Data Catalog + S3 Tables catalog modes#1451aaj3f wants to merge 2 commits into
aaj3f wants to merge 2 commits into
Conversation
Two native-AWS-SDK catalog modes for the Iceberg/R2RML graph-source path, at parity with REST (Polaris/Snowflake) and Direct: fluree iceberg map gs --mode glue --table db.tbl --region us-east-1 --r2rml m.ttl fluree iceberg map gs --mode s3tables --table ns.tbl --table-bucket-arn arn:... --r2rml m.ttl Both resolve the table's metadata_location via the native AWS SDK (aws-sdk-glue GetTable / aws-sdk-s3tables GetTableMetadataLocation) and read metadata/manifests /data from S3 with the ambient credential chain, reusing the existing direct.rs reader + from_default_chain. No aws-sigv4, no request signing, no REST prefix, no vended-credential parsing. Rationale: for a normally S3-authorized principal, neither Glue DC (IAM mode) nor S3 Tables vends credentials, so ambient reads suffice; and staying on the AWS SDK keeps consistent signing/transport paths rather than a hand-rolled REST+SigV4 client. Validated against a live Glue + S3 Tables harness (offline config tests + exact-count SPARQL e2e incl. multi-table joins and a dangling-FK check). - fluree-db-iceberg: GlueSdkCatalogClient + S3TablesSdkCatalogClient (catalog/aws_sdk.rs); CatalogConfig::Glue/S3Tables (serde glue/s3tables) — variant ungated, SDK impl aws-gated. read_glue example; offline config tests. - fluree-db-api: CatalogMode::Glue/S3Tables + to_iceberg_gs_config + two r2rml scan-path arms mirroring Direct. - fluree-db-cli / fluree-db-server: --mode glue|s3tables (+ --region, --catalog-id, --table-bucket-arn) on `iceberg map`, local and remote. - docs: graph-sources + CLI reference. Follow-ups: Lake-Formation-governed (vended-cred) Glue, and catalog browse for Glue/S3Tables.
7c83c7c to
d777a13
Compare
…query snapshot pin Before this, only map/query/list/info/drop worked for glue/s3tables; the read-only onboarding/introspection surface hard-rejected them, and the query path lacked the REST arm's per-query snapshot pin + catalog caches. Onboarding/introspection (catalog browse, table preview, R2RML auto-generate, mapping validate): - API: `rest_catalog_client` -> `catalog_client`, returning a boxed `SendCatalogClient` per mode (RestCatalogClient / GlueSdkCatalogClient / S3TablesSdkCatalogClient; Direct still rejected — no catalog to browse). Preview fetches + parses table metadata from S3 via `metadata_location` when the catalog returns no inline metadata (the SDK loadTable does), reusing the scan-path reader. generate/validate call preview per table, so they unblock transitively — no edits needed there. - Server: `IcebergConnectionRequest` gains region/catalog_id/table_bucket_arn; `build_iceberg_connection` gains glue/s3tables arms. Query-time (correctness + perf): - glue/s3tables scans now resolve loadTable through the same per-query snapshot pin + cross-query loadTable cache as REST (`sdk_resolve_load_table`), so every scan in a query reads one pinned Iceberg snapshot (was: independent per-scan SDK loads could observe different snapshots if the table committed mid-query), and the SDK client is built only on a genuine cache miss (was: rebuilt + a fresh GetTable/GetTableMetadataLocation per scan). Validated live: browse + preview against Glue (`examples/glue_onboarding.rs`); SPARQL e2e counts unchanged (300/60000/6/20000). Offline: iceberg_catalog + server route tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds two native-AWS-SDK catalog modes to the Iceberg/R2RML graph-source path — AWS Glue Data Catalog (
--mode glue) and AWS S3 Tables (--mode s3tables) — at full parity with the existing REST (Polaris / Snowflake Horizon) and Direct modes.Approach
Both modes resolve the table's
metadata_locationthrough the native AWS SDK (aws-sdk-glueGetTable/aws-sdk-s3tablesGetTableMetadataLocation), then read metadata/manifests/data from S3 with the ambient credential chain — reusing the existingdirect.rsreader andfrom_default_chain. Noaws-sigv4, no request signing, no REST prefix, no vended-credential parsing.Why SDK-native rather than REST + SigV4: empirically, for a normally S3-authorized principal, neither Glue DC (IAM mode) nor S3 Tables vends credentials, so ambient reads suffice — and staying on the AWS SDK keeps consistent signing/transport paths (prefer the SDK / override its HTTP client via
aws-smithy-httprather than roll alternatives) instead of a hand-rolled REST+SigV4 client.Changes
Commit 1 — core:
GlueSdkCatalogClient+S3TablesSdkCatalogClient(catalog/aws_sdk.rs);CatalogConfig::Glue/S3Tables(serde"glue"/"s3tables", variant ungated / SDK implaws-gated); twor2rml.rsscan arms;CatalogMode+ CLI + servermap;aws-sdk-glue/aws-sdk-s3tablesdeps;read_glueexample + offline tests.Commit 2 — full parity:
rest_catalog_client→catalog_clientreturns a boxedSendCatalogClientper mode; preview fetches+parses table metadata from S3 viametadata_locationwhen the catalog returns no inline metadata (the SDK loadTable does); serverIcebergConnectionRequest/build_iceberg_connectionwidened.generate/validateunblock transitively (they call preview per table).loadTablecache as REST (sdk_resolve_load_table), so every scan in a query reads one pinned Iceberg snapshot, and the SDK client is built only on a cache miss.Validation
iceberg_catalog+ server-route + api-lib (689) + iceberg (192) suites pass;fmt/clippyclean.read_glueexample reads exact row counts (Glue 500 / 60,000, S3 Tables 200). SPARQL e2e over Glue returns exact Athena-oracle values — 300 stores, 60,000 order-line⋈product, 6 categories, 20,000 order⋈customer — incl. multi-table joins, and a dangling-FK row confirms the R2RML inner join drops it (60,001 total / 60,000 joined).browse+previewvalidated live against Glue (fluree-db-api/examples/glue_onboarding.rs).Follow-ups (out of scope here)
configmap, not the standardizedstorage-credentialsarray that S3 Tables / LF-vended Glue return (would silently fall to ambient and 403). Not hit by this PR (ambient by design).