Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
154 changes: 154 additions & 0 deletions docs/how-to/pre-commit-hooks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
# Pre-commit hooks

This page explains the pyramids [pre-commit](https://pre-commit.com/) setup: what the hooks are, how to install them,
how to trigger them (all, a subset, or one), and how to skip the slow ones. The hook definitions live in
[`.pre-commit-config.yaml`](../../.pre-commit-config.yaml); the fast checks run in isolated tool environments while a few
heavier hooks (`mypy`, `pytest-check`, `doctest`, `notebook-check`, `pixi-lock-check`) shell out to the pixi **dev**
environment.

The same config is the local mirror of CI: the `pre-commit` job in
[`.github/workflows/lint.yml`](../../.github/workflows/lint.yml) runs `pre-commit run --all-files` with the slow test
hooks skipped, so getting a clean local run is the fastest way to keep that job green.

## Install (one-time)

Register the git hook so it fires on every `git commit`:

```bash
pixi run --frozen -e dev pre-commit install
```

`--frozen` makes pixi use `pixi.lock` as-is instead of re-solving every platform on each run (which is slow and can hang
on Windows). The first hook run also builds an isolated toolchain per hook repo (a few minutes, needs network) — that is
expected, not a failure.

## Trigger the hooks

### Automatically, on commit

Once installed, the hooks run against the **staged files** every time you commit:

```bash
git commit -m "feat(dataset): ..."
```

`fail_fast: true` is set, so the first hook that modifies a file (e.g. `ruff-format`) stops the run — re-stage and commit
again until it passes.

### Manually, on the whole tree

Run every hook against all files without committing (this is what CI does):

```bash
pixi run --frozen -e dev pre-commit run --all-files
```

### On specific files only

```bash
pixi run --frozen -e dev pre-commit run --files src/pyramids/dataset/collection.py tests/dataset/collection/test_meta.py
```

To scope the run to just what your branch changed:

```bash
pixi run --frozen -e dev pre-commit run --files $(git diff --name-only origin/main...HEAD)
```

### A single specific hook

Pass the hook **id** (the left column in the table below). Combine with `--all-files` or `--files` to choose the scope:

```bash
pixi run --frozen -e dev pre-commit run ruff-check --all-files # lint only
pixi run --frozen -e dev pre-commit run ruff-format --all-files # format only
pixi run --frozen -e dev pre-commit run mypy --all-files # type-check only
pixi run --frozen -e dev pre-commit run pytest-check --all-files # the full test suite
```

## The hooks

| id | What it does | Speed |
| --- | --- | --- |
| `check-toml` / `check-json` / `check-yaml` | Validate config file syntax | fast |
| `end-of-file-fixer` / `trailing-whitespace` / `mixed-line-ending` | Whitespace / newline normalisation | fast |
| `pretty-format-json` / `requirements-txt-fixer` | Canonicalise JSON and `requirements.txt` | fast |
| `check-added-large-files` | Block files over 2 MB | fast |
| `check-merge-conflict` / `detect-private-key` / `debug-statements` | Catch conflict markers, keys, stray `breakpoint()` | fast |
| `no-commit-to-branch` | Refuse a commit made directly on `main` | fast |
| `ruff-check` | Lint Python + notebooks (auto-fix) | fast |
| `ruff-format` | Format Python + notebooks | fast |
| `nbstripout` | Strip notebook outputs / execution counts | fast |
| `beautysh` / `shellcheck` | Format and lint shell scripts | fast |
| `check-summary-*` / `check-description-*` / `check-second-line-empty` | Commit-message conventional-commit checks | fast |
| `bandit` | Python security linter | medium |
| `gitleaks` | Scan staged changes for secrets | medium |
| `checkov` | Infrastructure-as-code security scan | medium |
| `mypy` | Static type-check (in the pixi `dev` env) | **slow** |
| `pytest-check` | Full test suite with coverage (`-m "not plot"`) | **slow** |
| `doctest` | `--doctest-modules src` under the Agg backend | **slow** |
| `notebook-check` | `nbval` execution of the example notebooks | **slow** |
| `pixi-lock-check` | Fail if `pixi.lock` is stale vs `pyproject.toml` | medium |

## Skip hooks

### Skip specific hooks with `SKIP`

pre-commit reads the `SKIP` environment variable (a comma-separated list of hook ids) at commit time and at
`pre-commit run` time. Everything else still runs. To skip the slow test trio on a commit:

```bash
# Git Bash — inline, scoped to the single command:
SKIP=pytest-check,doctest,notebook-check git commit -m "docs: ..."
```

```powershell
# PowerShell — persists for the rest of the session (clear with: Remove-Item Env:SKIP):
$env:SKIP = 'pytest-check,doctest,notebook-check'
git commit -m "docs: ..."
```

To make the skip permanent (also applies to commits from the PyCharm/JetBrains commit dialog, which inherits the
environment it was launched with):

```powershell
[Environment]::SetEnvironmentVariable('SKIP', 'pytest-check,doctest,notebook-check', 'User')
```

Restart your shells / IDE afterward. Note that a permanent `SKIP` also excludes those hooks from
`pre-commit run --all-files`, so they never run locally until you clear it — fine if you rely on CI as the backstop.

### The CI skip set

The `lint.yml` job runs everything **except** the hooks it can't or shouldn't run there. To reproduce that job exactly:

```bash
SKIP=no-commit-to-branch,gitleaks,pytest-check,doctest,notebook-check,pixi-lock-check \
pixi run --frozen -e dev pre-commit run --all-files
```

### Skip every hook

`--no-verify` bypasses **all** hooks for a single commit — use it sparingly, since it also skips the fast formatting and
lint that keep the diff clean:

```bash
git commit --no-verify -m "wip"
```

## Gotchas

- **pixi startup cost.** `pixi run` has a noticeable cold-start on Windows (~15–25 s). That is env warm-up, not a hung
hook.
- **`--frozen` always.** Without it, pixi re-solves every platform on each hook invocation, which is slow and can hang.
- **`-p no:cacheprovider`.** The pytest-based hooks disable the cache because writing `.pytest_cache` fails with
`WinError 183` on the Google-Drive-synced tree.
- **Agg backend for doctests.** The `doctest` hook forces `matplotlib.use("Agg")` so no plotting doctest pops a GUI
window that would block the commit.
- **`no-commit-to-branch` blocks `main`.** Commits must be made on a feature branch; this hook refuses a direct commit on
`main`.

## See also

- [Testing & CI](testing.md) — how the test suite is sliced across CI jobs and reproduced locally.
- [`.pre-commit-config.yaml`](../../.pre-commit-config.yaml) — the source of truth for every hook.
6 changes: 3 additions & 3 deletions docs/pyramids.drawio
Original file line number Diff line number Diff line change
Expand Up @@ -283,7 +283,7 @@
<mxRectangle x="-431" y="555" width="150" height="162" as="alternateBounds" />
</mxGeometry>
</mxCell>
<mxCell id="R0dbPGBYgG4crdKDNVuF-72" value="create_cube" style="text;align=left;verticalAlign=top;spacingLeft=4;spacingRight=4;overflow=hidden;rotatable=0;points=[[0,0.5],[1,0.5]];portConstraint=eastwest;" parent="R0dbPGBYgG4crdKDNVuF-71" vertex="1">
<mxCell id="R0dbPGBYgG4crdKDNVuF-72" value="from_dataset" style="text;align=left;verticalAlign=top;spacingLeft=4;spacingRight=4;overflow=hidden;rotatable=0;points=[[0,0.5],[1,0.5]];portConstraint=eastwest;" parent="R0dbPGBYgG4crdKDNVuF-71" vertex="1">
<mxGeometry y="26" width="150" height="34" as="geometry" />
</mxCell>
<mxCell id="R0dbPGBYgG4crdKDNVuF-73" value="update_cube" style="text;align=left;verticalAlign=top;spacingLeft=4;spacingRight=4;overflow=hidden;rotatable=0;points=[[0,0.5],[1,0.5]];portConstraint=eastwest;" parent="R0dbPGBYgG4crdKDNVuF-71" vertex="1">
Expand Down Expand Up @@ -811,7 +811,7 @@
<mxRectangle x="340" y="380" width="170" height="26" as="alternateBounds" />
</mxGeometry>
</mxCell>
<mxCell id="KzRKmL6C243oyRz3Jxsw-78" value="create_cube" style="text;align=left;verticalAlign=top;spacingLeft=4;spacingRight=4;overflow=hidden;rotatable=0;points=[[0,0.5],[1,0.5]];portConstraint=eastwest;" parent="KzRKmL6C243oyRz3Jxsw-77" vertex="1">
<mxCell id="KzRKmL6C243oyRz3Jxsw-78" value="from_dataset" style="text;align=left;verticalAlign=top;spacingLeft=4;spacingRight=4;overflow=hidden;rotatable=0;points=[[0,0.5],[1,0.5]];portConstraint=eastwest;" parent="KzRKmL6C243oyRz3Jxsw-77" vertex="1">
<mxGeometry y="26" width="150" height="34" as="geometry" />
</mxCell>
<mxCell id="KzRKmL6C243oyRz3Jxsw-79" value="update_cube" style="text;align=left;verticalAlign=top;spacingLeft=4;spacingRight=4;overflow=hidden;rotatable=0;points=[[0,0.5],[1,0.5]];portConstraint=eastwest;" parent="KzRKmL6C243oyRz3Jxsw-77" vertex="1">
Expand Down Expand Up @@ -3982,7 +3982,7 @@
<mxRectangle x="340" y="380" width="170" height="26" as="alternateBounds" />
</mxGeometry>
</mxCell>
<mxCell id="FAiVZwX0JY47LPwTAF5i-79" value="create_cube" style="text;align=left;verticalAlign=top;spacingLeft=4;spacingRight=4;overflow=hidden;rotatable=0;points=[[0,0.5],[1,0.5]];portConstraint=eastwest;" parent="FAiVZwX0JY47LPwTAF5i-78" vertex="1">
<mxCell id="FAiVZwX0JY47LPwTAF5i-79" value="from_dataset" style="text;align=left;verticalAlign=top;spacingLeft=4;spacingRight=4;overflow=hidden;rotatable=0;points=[[0,0.5],[1,0.5]];portConstraint=eastwest;" parent="FAiVZwX0JY47LPwTAF5i-78" vertex="1">
<mxGeometry y="26" width="150" height="34" as="geometry" />
</mxCell>
<mxCell id="FAiVZwX0JY47LPwTAF5i-81" value="read_multiple_files" style="text;align=left;verticalAlign=top;spacingLeft=4;spacingRight=4;overflow=hidden;rotatable=0;points=[[0,0.5],[1,0.5]];portConstraint=eastwest;" parent="FAiVZwX0JY47LPwTAF5i-78" vertex="1">
Expand Down
51 changes: 51 additions & 0 deletions docs/reference/dataset/collection.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,57 @@ flowchart LR
DC --> WR["<b>write</b><br/>to_file · to_cog_stack<br/>to_zarr · to_netcdf · to_kerchunk"]
```

## API at a glance

Two `classDiagram` views of the public surface: the **constructors** (every `@classmethod`, all returning a
`DatasetCollection`) and the **properties**. Exact parameter types and defaults live in the auto-generated
reference below; the diagrams show parameters by name to stay readable.

### Constructors (classmethods)

```mermaid
classDiagram
class DatasetCollection {
+from_dataset(dataset, time_length) DatasetCollection
+from_files(files, glob, date_format, date_regex, start, end, meta, gdal_env, validate) DatasetCollection
+from_stac(items, asset, patch_url, bbox, max_items, signer, align, skip_missing, groupby, grid) DatasetCollection
+from_point(lat, lon, collection, bands, start_date, end_date, edge_size, resolution, units, stac, query, signer, align) DatasetCollection
+from_zarr(store, storage_options) DatasetCollection
+from_archive(url_or_path, kind, member_glob, meta) DatasetCollection
+read_multiple_files(path, with_order, regex_string, date, file_name_data_fmt, start, end, fmt, glob) DatasetCollection
}
```

`from_dataset` builds an in-memory scaffold from a template `Dataset`; the other constructors read from a data
source — a folder or explicit list (`from_files`), a STAC search (`from_stac`), a single STAC point query
(`from_point`), a Zarr store (`from_zarr`), or a zip/tar archive (`from_archive`). Every constructor after
`from_dataset` takes its non-`self` extras as keyword-only arguments (see the reference for the exact `*`
boundary). `read_multiple_files` is **deprecated** — use `from_files`.

### Properties

```mermaid
classDiagram
class DatasetCollection {
+list~Dataset~ datasets
+Dataset base
+list~str~ files
+int time_length
+list time
+int rows
+int columns
+tuple shape
+Array data
+RasterMeta meta
+NDArray values
}
```

`time` and `values` are settable; every other property is read-only. `files` is `None` for an in-memory
collection and `time` is `None` until a time axis is assigned. `data` is the Dask array over
`(time, bands, rows, cols)` (Path B) and requires a file-backed collection; `values` materialises the cube
eagerly through Path A.

## The two paths

The class operates through **two distinct backing paths**, each serving a
Expand Down
2 changes: 1 addition & 1 deletion docs/tutorials/lazy/lazy-collection.md
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,7 @@ execution; useful when the store-write is one step of a larger
graph.

`collection.to_zarr` raises `RuntimeError` on a collection without a
`files` list (e.g. the legacy `create_cube(src, n)` path) — Zarr
`files` list (e.g. the in-memory `from_dataset(src, n)` path) — Zarr
writes need a source file per timestep. `ImportError` raised when
the `[lazy]` extra is missing.

Expand Down
15 changes: 9 additions & 6 deletions docs/tutorials/stac.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,18 +87,21 @@ into one timestep, mosaicked first-valid:
daily = DatasetCollection.from_stac(items, asset="visual", groupby="solar_day")
```

### Match a target grid — `like=` / `crs`+`resolution`+`bounds`
### Match a target grid — `grid=Grid(...)`

To guarantee pixel co-registration, resample every timestep onto an explicit
grid — either an existing `Dataset` (`like=`) or a CRS + resolution + bounds
(snapped to the resolution so independently-built grids align):
grid with the `grid` parameter and a `Grid` — either an existing `Dataset`
(`Grid(like=...)`) or a CRS + resolution + bounds (snapped to the resolution so
independently-built grids align):

```python
cube = DatasetCollection.from_stac(items, asset="B04", like=reference_dataset)
from pyramids.dataset import Grid

cube = DatasetCollection.from_stac(items, asset="B04", grid=Grid(like=reference_dataset))

cube = DatasetCollection.from_stac(
items, asset="B04", crs=32633, resolution=10,
bounds=(600000, 5300000, 610000, 5310000),
items, asset="B04",
grid=Grid(crs=32633, resolution=10, bounds=(600000, 5300000, 610000, 5310000)),
)
```

Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,7 @@ nav:
- Dev Container: how-to/devcontainer.md
- Docker & GHCR: how-to/docker.md
- Parallel windowed reads: how-to/parallel-reads.md
- Pre-commit hooks: how-to/pre-commit-hooks.md
- Recipes: how-to/recipes.md
- Testing & CI:
- Overview: how-to/testing.md
Expand Down
2 changes: 2 additions & 0 deletions src/pyramids/dataset/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
from pyramids.dataset.abstract_dataset import DEFAULT_NO_DATA_VALUE
from pyramids.dataset.collection import DatasetCollection
from pyramids.dataset.dataset import Dataset, NoDataSentinelWarning
from pyramids.dataset.grid import Grid
from pyramids.dataset.transform import GeoTransform
from pyramids.dataset.window import Window

Expand All @@ -13,6 +14,7 @@
"DatasetCollection",
"DEFAULT_NO_DATA_VALUE",
"GeoTransform",
"Grid",
"GroundControlPoint",
"NoDataSentinelWarning",
"RasterMeta",
Expand Down
Loading