snowpipe

Learn how to auto-ingest streaming data into Snowflake using Snowpipe.

SQL Scripts related to my learning on the Snowflake data cloud provider

SQL, Databases, warehouses, Data lake, cloud storage, MYSQL, Data Pipeline

Generating Streaming Data for Snowpipe Streaming

SQL, Databases, warehouses, Data lake, cloud storage, MYSQL, Data Pipeline

CLI script for interacting with Snowpipe REST APIs

Realtime data pipeline using Kafka + Spark + AWS S3 (Terraform) + Snowflake

The goal is to build an example of a simple data collection pipeline which collects data from multiple customers and uploads the data to Snowflake could look like. The OpenMeteo Api acts as the "customer system" in this case.

Data Engineering project implementing a fully automated Medallion Architecture (Bronze, Silver, Gold) using Snowpipe, Streams, Tasks, and SQL to build scalable cloud-native data pipelines for analytics.

AWS S3, Snowflake (Snowpipe, Snowpark) and PowerBi.

Snowflake DataScience Realtime HandsOn Projects

❄️ Stream real-time data into Snowflake with Amazon Kinesis Firehose

Automatically move data from AWS S3 to Snowflake datawarehouse continuously with snowpipe

End-to-end Change Data Capture (CDC) pipeline from PostgreSQL to Snowflake using AWS DMS, S3 (Parquet), Snowpipe, Streams, and Tasks. Demonstrates real-time data ingestion and incremental warehouse processing.

Real-time ETL (Extract, Transform, Load) data pipeline to process insurance claims data with Snowflake, Apache Airflow, AWS S3, EC2, python pandas and creating a real time data visualization dashboard using Tableau.

This project showcases a scalable ETL pipeline that automates the extraction, transformation, and storage of Redfin housing market data using AWS, Apache Airflow, and Snowflake, with Power BI for data visualization. The pipeline is configured to run monthly, ensuring your data remains up-to-date.