Skip to content

Finalise V1.0 of docs - #83

Merged
eglucas merged 14 commits into
devfrom
pipeline-documentation
Jul 31, 2026
Merged

Finalise V1.0 of docs#83
eglucas merged 14 commits into
devfrom
pipeline-documentation

Conversation

@eglucas

@eglucas eglucas commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Description

The following existing docs have been substantially updated:

  • repo-level README.md - rewritten as a short entry point that links out to the guides below, instead of holding everything itself
  • docs/DEVELOPMENT.md - expanded with dependency management, git workflow, notebooks, testing, and pre-commit sections
  • docs/running-pipelines.md - reworked into a quick-reference table covering local, Docker/ECR, GitHub Actions, and prod runs
  • .github/PULL_REQUEST_TEMPLATE/prod_template.md - added image-freshness and post-merge checklist items

The following documentation files have been added:

  • docs/CONTRIBUTING.md - PR process, branching conventions, and issue labelling (area:*/type:*)
  • docs/adding-pipelines.md - end-to-end guide for building a new pipeline (Hamilton dataflow pattern, bronze/silver/gold stages, registering in pipelines.yaml)
  • docs/templates/pipelines/README.md - template to copy when writing a new pipeline's README
  • Per-pipeline READMEs:
    • asf_mission_data/pipeline/example/README.md
    • asf_mission_data/pipeline/energy_price_cap_levels_annex_9/README.md
    • asf_mission_data/pipeline/heat_pump_deployment_statistics/README.md
  • .github/pull_request_template.md - single dev-merge PR template, replacing .github/PULL_REQUEST_TEMPLATE/dev_template.md

Removed:

  • docs/test.md (leftover test file)
  • docs/dag-images/.gitkeep (unused placeholder directory as we are now saving dag images in S3)

Other minor changes:

  • Standardises the docstrings in each pipeline's config.py for consistency (no behaviour change).
  • .gitignore also gained a *.ipynb/.ipynb_checkpoints/ entry to gitignore all Jupyter notebooks for development

Fixes #76

Type of change

  • New pipeline
  • Change to an existing pipeline
  • Package / infrastructure change
  • Docs only

Checklist

  • uv run pytest passes locally
  • uv run pre-commit run --all-files passes (ruff, gitleaks, etc.)
  • If this adds a new pipeline: it's registered in pipelines.yaml and has its own README.md
  • I've merged the latest dev into this branch

Testing notes

N/A. No changes made to pipeline code.

Instructions for reviewer

Please read through the docs and check if the content makes sense, anything is missing or needs to be clearer. Feel free to commit edits directly.

@AlexRossKnox In addition to double-checking the content I've written in the other docs related to running pipelines, AWS, workflows etc., please could you update infrastructure/README.md?

@eglucas

eglucas commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Note (I've already included these in the docs but just to be explicit) - I've implemented the following prod protections

  • Require 1 approval from me, Dan or Alex to be able to merge PR (no self-approvals allowed though, even if it is one of us. can get around this with the Bypass list) - this was done by adding the CODEOWNERS file and enabling specific settings in the "Protect prod" Ruleset
  • Contributors are instructed to use the prod_template PR template when merging dev to prod

Additionally, Alex has put more protection in the Run pipeline in prod workflow where a pipeline in prod can't be triggered without approval from me, Dan, Alex or Jack.

@eglucas

eglucas commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

@AlexRossKnox
On line 121 in docs/adding-pipelines.md, I've added a TODO note

<!-- TODO: once infrastructure is finalised on how stack_nameandschedule are used, update these instructions -->

I wasn't sure how stack_name and schedule will be used so please update when you know more :))

AlexRossKnox and others added 3 commits July 27, 2026 12:55
The previous version described a Lambda based architecture that doesn't match the ECS Fargate stack in the CDK app. This skeleton lays out the sections for a rewrite
Remove stale core stack read me
@eglucas
eglucas merged commit efd8b4e into dev Jul 31, 2026
2 checks passed
@eglucas
eglucas deleted the pipeline-documentation branch July 31, 2026 12:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Documentation

2 participants