ASF Mission Data

Data pipelines for fetching, processing and storing core ASF mission datasets to S3.

How pipelines work

Each pipeline has two or three stages: bronze (fetch and store raw data), silver (clean and transform into Parquet), and optionally gold (aggregate into dashboard-ready outputs).

Stages are implemented as Hamilton dataflows, where each function defines a transformation and its arguments declare its dependencies. Hamilton then automatically resolves the execution order of functions into a Directed Acyclic Graph (DAG). This gives us a consistent structure for writing pipelines, makes transforms easy to test in isolation, makes data dependencies explicit, and lets Hamilton render the pipeline as a visual graph. It also has built-in decorators for data quality checks.

In production, pipelines run as Docker containers on AWS ECS, triggered via GitHub Actions. Data is stored in S3 with separate dev and prod buckets.

Project structure

asf_mission_data/           # Python package
├── pipeline/
│   └── <pipeline-name>/    # one directory per pipeline, each with its own README
├── alerting.py             # Slack alerting
├── logging_utils.py        # Logging utilities
├── run.py                  # CLI entrypoint
├── storage.py              # Local and S3 read/write utilities
└── utils.py                # Shared utility functions
.github/workflows/          # CI/CD workflows
docs/                       # Guides and runbooks
infrastructure/             # AWS CDK infrastructure
scripts/                    # Utility scripts
tests/                      # Test suite
Dockerfile                  # Container image for running pipelines in AWS
pipelines.yaml              # Pipeline registry
pyproject.toml              # Project config and dependencies

Docs

Guide	Covers
Running pipelines	Local, Docker, GitHub Actions, ad hoc AWS
Development guide	Editor setup, testing, pre-commit
Adding a pipeline	Creating a new ETL pipeline
Contributing	PR process, code standards
Infrastructure	AWS CDK resources and deployment

Each pipeline has its own README at asf_mission_data/pipeline/<name>/README.md. For the full list of existing pipelines, see pipelines.yaml.

Setup

Prerequisites

• Python 3.12+ (uv will install this automatically if needed)

• uv for dependency management

• Graphviz, for pipelines that render Hamilton DAG images. The Python graphviz package is installed by uv sync, but the system dot executable must also be available on your PATH.

  macOS:

  brew install graphviz
  dot -V

  Windows:

  winget install Graphviz.Graphviz
  dot -V

  If dot -V is not found on Windows, restart your terminal and check that the Graphviz bin directory is on PATH. Chocolatey users can install it with choco install graphviz.

Installation

# Clone the repo
git clone https://github.com/nestauk/asf_mission_data.git
cd asf_mission_data

# Install dependencies (creates .venv automatically)
uv sync

# Activate virtual environment
source .venv/bin/activate

# Or run commands directly without activating, e.g.
uv run python -m asf_mission_data.run example --stage all

Infrastructure

Infrastructure is managed with AWS CDK (Python).

How this works, in three steps:

1. Build: our code (all pipelines) gets packaged into a single Docker container image and pushed to ECR (a private store for container images).

2. Run: to run a specific pipeline, someone manually triggers a task on ECS Fargate (AWS's "run a container without managing a server" service), telling it which pipeline to run from that shared image.

3. Land: the pipeline writes its output data to an S3 bucket where it's picked up by downstream tools for analysis.

Note: there's no automatic scheduling yet (as of 31 July 2026), every run is triggered manually at the moment.

See infrastructure/README.md for full CDK documentation, or docs/running-pipelines.md for how to actually trigger a run.

---

Last updated: 31 July 2026 by Elysia Lucas