data-ingestion-and-infrastructure

DataHut-DuckHouse is a modern, modular, and multi-tenant analytics platform that combines DuckDB, Apache Iceberg, Arrow Flight, dbt, and Trino to build a hybrid, lightweight, and scalable data stack ready for SaaS.

This repository contains example of a RAG chat bot with a basic architecture (designed for scenarios without network isolation), and a standard Zero-Trust Architecture deployment. It is intended for quick demos and should be adjusted as needed to fit specific use cases or requirements.

Data Pipeline creation of france courses enrollments. Every month the providers report the enrollments in their programs. The idea is to get the courses listed as well as the enrollments every month and look at the trend of enrolments and the inter comparison of the trainings s providers for different courses.

Framework Python para pipelines de ingestão de dados com arquitetura modular: conectores, validadores Pydantic, transformers e loaders (JSON, PostgreSQL) plugáveis.

A prototype energy-sector IoT + Data Storage platform that unifies everything in one place for the user.

Multi-Tool AI Agent for Bangladesh using HuggingFace datasets and LangChain

Data pipeline to study trends and rising prices in NYC taxi since post-Covid times

Building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics, as part of my journey on learning data engineering.

A fully Dockerized financial data pipeline that decouples external API fetching from data serving. Built with FastAPI, Celery, Redis, PostgreSQL, and Apache Airflow.

A scalable data ingestion pipeline that continuously scrapes, normalizes, and enriches tech ecosystem data (jobs, events, news) from ATS platforms and external APIs like Adzuna and Jooble.