Skip to content
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 86 additions & 0 deletions docs/Engineering/TechStack.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
---
sidebar_label: "Technology stack"
---

import MaintainerTag from "@site/src/components/MaintainerTag";
import ControlledDocBanner from "@site/src/components/ControlledDocBanner";


<MaintainerTag maintainerEmails={["siddharth.krishna@openenergytransition.org","johannes.hampp@openenergytransition.org"]} />

# Technology stack that we use

Recommended, not enforced.
Projects may deviate, especially in case of projects based on soft forks.


* Version control:
* `git`
* Host repositories on: `github`
* Repositories should belong to [Open Energy Transition GitHub organization](https://github.com/open-energy-transition)
* No force pushes
* Use pull requests for all changes
Comment on lines +21 to +22

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* No force pushes
* Use pull requests for all changes
* Add [main branch protection](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/managing-a-branch-protection-rule) to include:
* no force pushes;
* at least one `approve` review before merge;
* override rights for project admins only;
* no pushes directly to `main` allowed (i.e. use pull requests for all changes).

* Merge branches instead of rebasing
* Squash and merge pull requests to keep the commit history clean

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is good for python library projects, but in other project types (e.g. snakemake workflows)the full commit history can be quite instructive / can be used to resurrect superseded attempts at solving an issue when it later turns out that it was the correct approach.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The full PR branch history is preserved by GitHub, and I think this is the better place to archive this information, because it also has the context of why each change was necessary (review comments, CI failures, etc).

I would be okay with merge-merging PRs if everyone had iron discipline and kept their PR branches full of atomic, clean commits with descriptive messages, but too often we have (and it's much easier to have) commits like "fix, fix fix, typo, revert fix fix, ..."

It's been on my list for a while to make a handbook page/section on why I recommend squash-merging all PRs, but some other reasons include: easier to roll-back/revert a breaking change in production, easier to git bisect and find the source of a bug, and every commit on the main branch is a "valid" state of the project that works (or at least, CI passes).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The full PR branch history is preserved by GitHub

Is that the case even once a branch is deleted? I tend to keep repos clean by deleting branches on merging PRs but I assume at some point you can't roll back that deletion and the branch history is lost forever?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep!
Here's one of the first PRs on the benchmark repo: open-energy-transition/solver-benchmark#9
You can go through and view all the commits still, e.g.: open-energy-transition/solver-benchmark@f56d9ee

We should also add a recommendation to auto-delete PR branches on merge. :)

* Continuous integration:
* For public repositories / open source projects: `github actions`
* For private repositories / internal projects: ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If they are lightweight, github actions could still be possible (what's our total private repo allowance?)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We pay for GitHub actions per minute on private repos. It's still probably the recommended way, although I'd add that for forks such as pypsa-x it is a good idea to disable win/macOS actions since they cost 4-8x as much.

* Use a pinned environment for the CI to ensure continuity and reproducibility

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brynpickering or @SermishaNarayana or whoever knows the latest/best way to do this could you please add a section below with instructions and link to it here? Huge thanks

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a pinned environment? If we recommend using uv or pixi that would come as standard without the user really having to do anything.

* Programming languages:
* `python`
* whereever possible, including for static analysis, plotting, etc.
* `pandas`, `numpy`, `xarray` with `matplotlib`/`seaborn`/`hvplot`
* Package management:
* `uv` for new projects
* always use virtual environments (better reproducibility, isolation, and dependency management)
* `uv init` to create a new project with default structure
* `uv add <package>` to add a package to the project
* `uv pip install <package>` to install a package in the virtual environment without adding it as a dependency
* `uv run <command>` to run a command in the virtual environment
Comment on lines +34 to +39

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pixi...? :P

* `source .venv/bin/activate` to activate the virtual environment (default location)
* `uv sync` to install all dependencies from `pyproject.toml` or `uv.lock`
* Share your environment with others using `pyproject.toml` and `uv.lock`
* some projects use `conda` because of specific dependencies, where installing the packages via `pip` requires additional steps to install the required binaries separately.
Installations with `conda` using `conda-forge` on the other hand proide the binaries automatically.
Can be especially important for projects that work with geospatial packages or on windows.
* Documentation:
* doc strings follow [numpydoc style](https://numpydoc.readthedocs.io/en/latest/format.html)
* Framework: `mkdocs-material`
* Hosting: `readthedocs`

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Github pages can be a better option when there's the possibility of name clashes with existing RTD projects or when in a private repo. Not sure how best to recommend one or the other.

* Repositories should use pre-commit framework: `pre-commit`

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would consider this to be part of continuous integration

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Link to your favourite implementation of .pre-commit-config.yaml so others can copy that across to their project if they like

* To ensure code styles and checks are run consistently on PRs / commits
* Public repositories: use `pre-commit.ci` for automatic checks (instead of integrating `pre-commit` into CI)
* Private repositories: use `pre-commit` in CI
* Code style:
* `black`

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe don't even mention black since ruff handles everything?

* `ruff` for implementation
* recommended to use inside `pre-commit`
* Code linting:
* `ruff` for implementation
* recommended to use inside `pre-commit`
* Static code analysis:
* `mypy`
* Use type annotations in code please, more infomation [here](https://docs.python.org/3/library/typing.html) and [example here](https://github.com/PyPSA/technology-data/blob/40ca61b9aaf8c3019fa264a35ca999e6c41ef562/scripts/compile_cost_assumptions_usa.py#L198)
* recommended to use inside `pre-commit`
* Code testing:
* `pytest`
* unit and integration tests
* recommended to use inside your CI
* use `pytest-cov` to measure code coverage
* for faster, parallel tests, use `pytest-xdist` with `pytest -n auto`
* Workflow management:
* `snakemake` for modeling and data processing workflows
*
* Data sharing:
* `Zenodo` for sharing data publicly with a DOI
* `Google Drive` for sharing data internally or temporarily with externals
* Licensing and copyright:
* Technical: Follow the [reuse](https://reuse.software/) specification
* Use `reuse annotate` to add license and copyright information and `reuse lint` to check for compliance of a repository
* recommended to use inside `pre-commit`
* Licenses:
* `MIT` for code
* `CC-BY-4.0` for documentation
* `CC-BY-4.0` for data
* `CC0-1.0` for metadata and general repository configuration files
* For details, see handbook entry on [Licensing and copyright](../Handbook/CopyrightAndAttribution.mdx)
Loading