Skip to content

feat: Delta Lake + AWS Glue catalog registration #261

Description

@malon64

Context

Floe already uses aws-sdk-glue for Iceberg catalog integration (io/write/iceberg/glue.rs). AWS Glue can also register Delta Lake tables as external tables, enabling Athena and other AWS services to query Floe-written Delta tables via the Glue Data Catalog.

This is the lowest-effort catalog addition for Delta — same SDK, different table parameters.

Architecture: post-write registration

Same pattern as Delta + Unity Catalog: Floe writes Delta files normally, then registers/updates the Glue table entry. The _catalogs parameter already present in DeltaAcceptedAdapter::write_accepted() (delta.rs:175) is wired up.

Delta vs Iceberg table parameters in Glue

Parameter Iceberg Delta
table_type EXTERNAL_TABLE EXTERNAL_TABLE
parameters.table_type ICEBERG (absent)
parameters.metadata_location S3 path to metadata.json (absent)
parameters.spark.sql.sources.provider (absent) delta
location S3 warehouse root S3 Delta table root

Implementation

1. New io/write/delta/glue.rs module

Mirrors io/write/iceberg/glue.rs but with Delta-specific Glue table parameters:

pub async fn upsert_glue_delta_table(
    cfg: &GlueDeltaCatalogConfig,
    table_path: &str,
) -> FloeResult<()> {
    let client = build_glue_client(&cfg.region).await;
    let table_input = build_glue_delta_table_input(cfg, table_path);
    // Same create-then-update-on-conflict pattern as iceberg/glue.rs
    match client.create_table()...send().await {
        Ok(_) => Ok(()),
        Err(already_exists) => client.update_table()...send().await,
    }
}

fn build_glue_delta_table_input(...) -> GlueTableInput {
    GlueTableInput::builder()
        .name(&cfg.table)
        .table_type("EXTERNAL_TABLE")
        .storage_descriptor(
            StorageDescriptor::builder()
                .location(table_path)
                .build()
        )
        .set_parameters(Some(HashMap::from([
            ("spark.sql.sources.provider".into(), "delta".into()),
            ("EXTERNAL".into(), "TRUE".into()),
        ])))
        .build()
}

2. User config

catalogs:
  definitions:
    - name: glue_delta
      type: glue           # same type as Iceberg Glue — dispatch by format
      region: us-east-1
      database: lakehouse
      warehouse_storage: s3_out
      warehouse_prefix: warehouse

entities:
  - name: orders
    sink:
      accepted:
        format: delta
        storage: s3_out
        path: warehouse/orders
        delta:
          catalog: glue_delta   # new optional field

No new dependencies

aws-sdk-glue is already a dependency. This issue adds ~100 lines of new code.

Estimated effort

~3 days. Smallest catalog addition possible — reuses existing SDK, existing Glue client builder, and existing create/update pattern from iceberg/glue.rs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    Status
    Strategic bet

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions