Context
Floe already uses aws-sdk-glue for Iceberg catalog integration (io/write/iceberg/glue.rs). AWS Glue can also register Delta Lake tables as external tables, enabling Athena and other AWS services to query Floe-written Delta tables via the Glue Data Catalog.
This is the lowest-effort catalog addition for Delta — same SDK, different table parameters.
Architecture: post-write registration
Same pattern as Delta + Unity Catalog: Floe writes Delta files normally, then registers/updates the Glue table entry. The _catalogs parameter already present in DeltaAcceptedAdapter::write_accepted() (delta.rs:175) is wired up.
Delta vs Iceberg table parameters in Glue
| Parameter |
Iceberg |
Delta |
table_type |
EXTERNAL_TABLE |
EXTERNAL_TABLE |
parameters.table_type |
ICEBERG |
(absent) |
parameters.metadata_location |
S3 path to metadata.json |
(absent) |
parameters.spark.sql.sources.provider |
(absent) |
delta |
location |
S3 warehouse root |
S3 Delta table root |
Implementation
1. New io/write/delta/glue.rs module
Mirrors io/write/iceberg/glue.rs but with Delta-specific Glue table parameters:
pub async fn upsert_glue_delta_table(
cfg: &GlueDeltaCatalogConfig,
table_path: &str,
) -> FloeResult<()> {
let client = build_glue_client(&cfg.region).await;
let table_input = build_glue_delta_table_input(cfg, table_path);
// Same create-then-update-on-conflict pattern as iceberg/glue.rs
match client.create_table()...send().await {
Ok(_) => Ok(()),
Err(already_exists) => client.update_table()...send().await,
}
}
fn build_glue_delta_table_input(...) -> GlueTableInput {
GlueTableInput::builder()
.name(&cfg.table)
.table_type("EXTERNAL_TABLE")
.storage_descriptor(
StorageDescriptor::builder()
.location(table_path)
.build()
)
.set_parameters(Some(HashMap::from([
("spark.sql.sources.provider".into(), "delta".into()),
("EXTERNAL".into(), "TRUE".into()),
])))
.build()
}
2. User config
catalogs:
definitions:
- name: glue_delta
type: glue # same type as Iceberg Glue — dispatch by format
region: us-east-1
database: lakehouse
warehouse_storage: s3_out
warehouse_prefix: warehouse
entities:
- name: orders
sink:
accepted:
format: delta
storage: s3_out
path: warehouse/orders
delta:
catalog: glue_delta # new optional field
No new dependencies
aws-sdk-glue is already a dependency. This issue adds ~100 lines of new code.
Estimated effort
~3 days. Smallest catalog addition possible — reuses existing SDK, existing Glue client builder, and existing create/update pattern from iceberg/glue.rs.
Context
Floe already uses
aws-sdk-gluefor Iceberg catalog integration (io/write/iceberg/glue.rs). AWS Glue can also register Delta Lake tables as external tables, enabling Athena and other AWS services to query Floe-written Delta tables via the Glue Data Catalog.This is the lowest-effort catalog addition for Delta — same SDK, different table parameters.
Architecture: post-write registration
Same pattern as Delta + Unity Catalog: Floe writes Delta files normally, then registers/updates the Glue table entry. The
_catalogsparameter already present inDeltaAcceptedAdapter::write_accepted()(delta.rs:175) is wired up.Delta vs Iceberg table parameters in Glue
table_typeEXTERNAL_TABLEEXTERNAL_TABLEparameters.table_typeICEBERGparameters.metadata_locationmetadata.jsonparameters.spark.sql.sources.providerdeltalocationImplementation
1. New
io/write/delta/glue.rsmoduleMirrors
io/write/iceberg/glue.rsbut with Delta-specific Glue table parameters:2. User config
No new dependencies
aws-sdk-glueis already a dependency. This issue adds ~100 lines of new code.Estimated effort
~3 days. Smallest catalog addition possible — reuses existing SDK, existing Glue client builder, and existing create/update pattern from
iceberg/glue.rs.