I'm a Data Engineer and Data Scientist specializing in large-scale data processing, distributed computing, and Python-based data pipelines.
I currently work at LIneA, where I design and develop scalable workflows for massive scientific datasets. My work involves data ingestion, transformation, validation, quality assurance, spatial processing, crossmatching, deduplication, and distributed execution across interactive Jupyter environments and multi-node HPC clusters.
I work primarily with Python, Dask, SLURM, and HPC systems, with a strong focus on performance, memory efficiency, data quality, reproducibility, testing, and maintainable software.
I currently apply this expertise to large-scale astronomy, including projects connected to the Rubin Observatory / LSST ecosystem and international scientific collaborations.
- Designing scalable Python data pipelines
- Processing datasets ranging from millions to hundreds of millions of records
- Building distributed workflows with Dask and SLURM
- Integrating and standardizing heterogeneous data sources
- Developing data validation, quality assurance, and deduplication workflows
- Building Python packages, command-line tools, automated tests, and CI workflows
- Contributing to open-source scientific software
A distributed Python pipeline for generating large-scale HiPS catalogs using Dask, LSDB, and HPC infrastructure.
The project supports configurable selection strategies, YAML-based configuration, command-line execution, parallel processing, and reproducible large-scale catalog generation. It is also published on PyPI.
A distributed pipeline for integrating dozens of heterogeneous datasets into consolidated data products.
The workflow includes schema harmonization, quality metadata standardization, scalable spatial crossmatching, deterministic deduplication, distributed validation, and memory-efficient processing with Python, Dask, LSDB, HATS, and HPC infrastructure.
A distributed workflow for interactive analysis and visualization of datasets containing up to 691 million records.
The architecture integrates JupyterLab with a multi-node HPC cluster using Dask and SLURMCluster, combining distributed computation with HoloViews, Bokeh, and Datashader for responsive large-scale data exploration.
I also taught a short course on this workflow and published the training materials openly.
Designed and implemented the .concat API in the open-source LSDB library, enabling concatenation of distributed catalogs stored in the HATS format.
The contribution included feature design, implementation, integration with the existing catalog architecture, edge-case handling, and development of the complete unit-test suite to validate correctness and reliability.
Data Engineering & Programming
Python · SQL · Pandas · Parquet · Data Pipelines · Data Quality
Distributed Computing & HPC
Dask · SLURM · Distributed Computing · Parallel Computing · HPC
Scientific Data & Analysis
Jupyter · LSDB · HATS · HoloViews · Bokeh · Datashader
Software Engineering
Git · GitHub · Testing · CI · CLI Development · Configuration-Driven Workflows
I'm currently expanding my data engineering background through formal studies in:
- Database systems and SQL
- Data modeling
- Data warehousing
- Cloud data engineering
- AWS
This complements my professional experience with large-scale and distributed data processing.