common-data-pipeline

A public-facing data pipeline feeding three eventually-merged domains: U.S. election trends (Colorado-focused), home affordability, and economic indicators. Automated ingestion pipelines land clean, historicized data in object storage; a transform layer and static dashboards with county/city drill-down are built on top once enough sources are in place.

Architecture

Source API/file  →  Python fetch script  →  dated Parquet  →  Cloudflare R2
                                                                     ↑
                                                    GitHub Actions runs this on cron

Later layers (not built yet): DuckDB querying Parquet directly from R2 → dbt-core transformations → Evidence.dev or Observable Framework dashboard → Cloudflare Pages or GitHub Pages.

Each source gets its own fetch script under scripts/, built on the shared scripts/pipeline_common.py library (env checks, HTTP retries, R2 upload, validation) and a thin GitHub Actions workflow that calls the reusable .github/workflows/_pipeline.yml.

Setup

1. Clone the repo and create a project-local venv:

   python -m venv venv
   venv\Scripts\Activate.ps1
   pip install -r requirements.txt
   

   (On a fresh machine, Activate.ps1 may need Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned run once first.)
2. Copy .env.example (if present) or create a .env with the Cloudflare R2 credentials: R2_ACCOUNT_ID, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, R2_BUCKET_NAME.
3. Run a fetch script locally, e.g. python scripts/fetch_votehub.py — confirm the dated Parquet lands in data/ and uploads to R2.
4. For CI, add the same four secrets under the repo's Settings → Secrets and variables → Actions, using identical names.

Status

Currently proving out ingestion mechanics with one source (VoteHub election polling) as the template for the rest. See NEXT_STEPS.md and CLAUDE.md (both local-only, gitignored) for the backlog and conventions.