This is a collecton of Amazon CDK projects to show how to directly ingest streaming data from Amazon Mananged Service for Apache Kafka (MSK) and MSK Serverless into Apache Iceberg table in S3 with AWS Glue Streaming.
Stream CDC into an Amazon S3 data lake in Apache Iceberg format with AWS Glue Streaming using Amazon MSK Serverless and MSK Connect (Debezium)
Data Pipeline for CDC data from MySQL DB to Amazon S3 through Amazon MSK Serverless using Amazon MSK Connect (Debezium).
Streaming data pipeline to continuously load data from an Amazon MSK or MSK Serverless cluster to Amazon S3 using Amazon Kinesis Data Firehose.
Real-time Change Data Capture pipeline with SCD Type 2 history tracking for gaming analytics. PostgreSQL RDS → Debezium → AWS MSK Kafka → S3 Data Lake → Redshift Serverless with dimensional modeling and medallion architecture.
Kafka S3 Connector kaf-s3: A Python library that offloads Kafka messages to S3, with a producer and consumer using lightweight references and IAM-based access 🐙
Add a description, image, and links to the aws-msk-serverless topic page so that developers can more easily learn about it.
To associate your repository with the aws-msk-serverless topic, visit your repo's landing page and select "manage topics."