Skip to content

feat(iceberg): add AWS Glue Data Catalog + S3 Tables catalog modes#1451

Open
aaj3f wants to merge 2 commits into
mainfrom
feat/iceberg-glue-sigv4
Open

feat(iceberg): add AWS Glue Data Catalog + S3 Tables catalog modes#1451
aaj3f wants to merge 2 commits into
mainfrom
feat/iceberg-glue-sigv4

Conversation

@aaj3f

@aaj3f aaj3f commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds two native-AWS-SDK catalog modes to the Iceberg/R2RML graph-source path — AWS Glue Data Catalog (--mode glue) and AWS S3 Tables (--mode s3tables) — at full parity with the existing REST (Polaris / Snowflake Horizon) and Direct modes.

fluree iceberg map gs --mode glue     --table enterprise_dw.dim_store --region us-east-1 --r2rml m.ttl
fluree iceberg map gs --mode s3tables --table ns.tbl --table-bucket-arn arn:aws:s3tables:... --region us-east-1 --r2rml m.ttl

Approach

Both modes resolve the table's metadata_location through the native AWS SDK (aws-sdk-glue GetTable / aws-sdk-s3tables GetTableMetadataLocation), then read metadata/manifests/data from S3 with the ambient credential chain — reusing the existing direct.rs reader and from_default_chain. No aws-sigv4, no request signing, no REST prefix, no vended-credential parsing.

Why SDK-native rather than REST + SigV4: empirically, for a normally S3-authorized principal, neither Glue DC (IAM mode) nor S3 Tables vends credentials, so ambient reads suffice — and staying on the AWS SDK keeps consistent signing/transport paths (prefer the SDK / override its HTTP client via aws-smithy-http rather than roll alternatives) instead of a hand-rolled REST+SigV4 client.

Changes

Commit 1 — core: GlueSdkCatalogClient + S3TablesSdkCatalogClient (catalog/aws_sdk.rs); CatalogConfig::Glue/S3Tables (serde "glue"/"s3tables", variant ungated / SDK impl aws-gated); two r2rml.rs scan arms; CatalogMode + CLI + server map; aws-sdk-glue/aws-sdk-s3tables deps; read_glue example + offline tests.

Commit 2 — full parity:

  • Onboarding/introspection (catalog browse, table preview, R2RML auto-generate, mapping validate) — were hard-rejected for glue/s3tables, now work. rest_catalog_clientcatalog_client returns a boxed SendCatalogClient per mode; preview fetches+parses table metadata from S3 via metadata_location when the catalog returns no inline metadata (the SDK loadTable does); server IcebergConnectionRequest/build_iceberg_connection widened. generate/validate unblock transitively (they call preview per table).
  • Query-time (correctness + perf): glue/s3tables scans now use the same per-query snapshot pin + cross-query loadTable cache as REST (sdk_resolve_load_table), so every scan in a query reads one pinned Iceberg snapshot, and the SDK client is built only on a cache miss.

Validation

  • Offline: config + iceberg_catalog + server-route + api-lib (689) + iceberg (192) suites pass; fmt/clippy clean.
  • Live (Glue + S3 Tables harness in a sandbox): read_glue example reads exact row counts (Glue 500 / 60,000, S3 Tables 200). SPARQL e2e over Glue returns exact Athena-oracle values — 300 stores, 60,000 order-line⋈product, 6 categories, 20,000 order⋈customer — incl. multi-table joins, and a dangling-FK row confirms the R2RML inner join drops it (60,001 total / 60,000 joined). browse + preview validated live against Glue (fluree-db-api/examples/glue_onboarding.rs).

Follow-ups (out of scope here)

  • Lake-Formation-governed Glue where a restricted principal needs vended credentials (the existing Snowflake vended-cred code has the mechanics).
  • Latent bug: vended-cred parsing reads only the legacy top-level config map, not the standardized storage-credentials array that S3 Tables / LF-vended Glue return (would silently fall to ambient and 403). Not hit by this PR (ambient by design).

Two native-AWS-SDK catalog modes for the Iceberg/R2RML graph-source path, at
parity with REST (Polaris/Snowflake) and Direct:

  fluree iceberg map gs --mode glue     --table db.tbl --region us-east-1 --r2rml m.ttl
  fluree iceberg map gs --mode s3tables --table ns.tbl --table-bucket-arn arn:... --r2rml m.ttl

Both resolve the table's metadata_location via the native AWS SDK (aws-sdk-glue
GetTable / aws-sdk-s3tables GetTableMetadataLocation) and read metadata/manifests
/data from S3 with the ambient credential chain, reusing the existing direct.rs
reader + from_default_chain. No aws-sigv4, no request signing, no REST prefix, no
vended-credential parsing.

Rationale: for a normally S3-authorized principal, neither Glue DC (IAM mode) nor
S3 Tables vends credentials, so ambient reads suffice; and staying on the AWS SDK
keeps consistent signing/transport paths rather than a hand-rolled REST+SigV4
client. Validated against a live Glue + S3 Tables harness (offline config tests +
exact-count SPARQL e2e incl. multi-table joins and a dangling-FK check).

- fluree-db-iceberg: GlueSdkCatalogClient + S3TablesSdkCatalogClient
  (catalog/aws_sdk.rs); CatalogConfig::Glue/S3Tables (serde glue/s3tables) —
  variant ungated, SDK impl aws-gated. read_glue example; offline config tests.
- fluree-db-api: CatalogMode::Glue/S3Tables + to_iceberg_gs_config + two r2rml
  scan-path arms mirroring Direct.
- fluree-db-cli / fluree-db-server: --mode glue|s3tables (+ --region,
  --catalog-id, --table-bucket-arn) on `iceberg map`, local and remote.
- docs: graph-sources + CLI reference.

Follow-ups: Lake-Formation-governed (vended-cred) Glue, and catalog browse for
Glue/S3Tables.
@aaj3f aaj3f force-pushed the feat/iceberg-glue-sigv4 branch from 7c83c7c to d777a13 Compare July 7, 2026 16:24
…query snapshot pin

Before this, only map/query/list/info/drop worked for glue/s3tables; the
read-only onboarding/introspection surface hard-rejected them, and the query
path lacked the REST arm's per-query snapshot pin + catalog caches.

Onboarding/introspection (catalog browse, table preview, R2RML auto-generate,
mapping validate):
- API: `rest_catalog_client` -> `catalog_client`, returning a boxed
  `SendCatalogClient` per mode (RestCatalogClient / GlueSdkCatalogClient /
  S3TablesSdkCatalogClient; Direct still rejected — no catalog to browse).
  Preview fetches + parses table metadata from S3 via `metadata_location` when
  the catalog returns no inline metadata (the SDK loadTable does), reusing the
  scan-path reader. generate/validate call preview per table, so they unblock
  transitively — no edits needed there.
- Server: `IcebergConnectionRequest` gains region/catalog_id/table_bucket_arn;
  `build_iceberg_connection` gains glue/s3tables arms.

Query-time (correctness + perf):
- glue/s3tables scans now resolve loadTable through the same per-query snapshot
  pin + cross-query loadTable cache as REST (`sdk_resolve_load_table`), so every
  scan in a query reads one pinned Iceberg snapshot (was: independent per-scan
  SDK loads could observe different snapshots if the table committed mid-query),
  and the SDK client is built only on a genuine cache miss (was: rebuilt + a
  fresh GetTable/GetTableMetadataLocation per scan).

Validated live: browse + preview against Glue (`examples/glue_onboarding.rs`);
SPARQL e2e counts unchanged (300/60000/6/20000). Offline: iceberg_catalog +
server route tests pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant