s1iw_catalogue

s1iw_catalogue – Exhaustive catalogue of Sentinel‑1 IW SAFE products for Ifremer.

Build a single Parquet file with availability, dataset membership, category, meteorological enrichment, and acquisition geometry. Enables fast dashboards, product lookup, spatial heatmaps, and catalogue merging.

Overview

This tool creates and maintains a master Parquet file (sentinel-1_exhaustive_IW_SAFE_working_material.parquet) that answers:

• Which SAFE products (SLC, GRD, OCN) are available on Ifremer storage?
• Do they belong to internal datasets (ciaran2023, jolina26, etc.)?
• What is their dataset category (train, val, test, case-study)?
• Are they colocalised with reference listings?
• What are the associated wave (WW3) and wind (ECMWF) parameters?
• What is the acquisition start date, geometry, orbit, and polarisation?

The file is updated daily via a workflow that uses:

• cdse_match_product_type for SLC↔GRD matching
• s1ifr to check local presence at Ifremer
• familyprod to derive A21, B17, OCN and dataset membership
• Optional ECMWF/WW3 enrichment

Features

Core functionalities

• Create – Build a catalogue from listing files with dataset metadata.
• Update – Incrementally update an existing catalogue with new listings.
• Merge – Combine multiple catalogues into one (parallel processing on HPC).
• Stats – Generate statistics and completeness reports.
• Serve – Launch a web interface for visual exploration (FastAPI + Plotly).
• Query – Look up SAFE products by name.
• Backup – Create timestamped backups of the catalogue.
• Dataset categories – Assign undefined, train, val, test, case-study with priority hierarchy.
• Conflict reporting – Track dataset category conflicts in a separate file.

Column schema (key columns)

Column	Description
SAFE SLC	SAFE name for SLC product
SAFE GRD	SAFE name for GRD product
SAFE OCN	SAFE name for OCN product (or "NOT_FOUND")
PATH SLC	Full path to SLC on Ifremer storage
PATH GRD	Full path to GRD on Ifremer storage
PATH OCN	Full path to OCN on Ifremer storage
PATH L1B XSP A21	Full path to L1B XSP A21 product
PATH L1C XSP B17	Full path to L1C XSP B17 product
datasets	List of dataset names (e.g., ["ciaran2023"])
category	Dataset category: undefined, train, val, test, case-study
unit	Satellite identifier: S1A, S1B, S1C, S1D
start date SAFE	Acquisition start date
horodating	Last catalogue update timestamp
polygon SLC / polygon GRD	Footprint polygon (WKT)
S3path SLC / S3path GRD	S3 path on CDSE

Configuration

The tool uses a YAML configuration file. Example:

paths:
  reference_listings:
    ciaran2023:
      path: "/path/to/listing.txt"
      type: "slc"
      description: "Storm Ciaran over Europe"
      category: "case-study"
    jolina26:
      path: "/path/to/grd_listing.txt"
      type: "grd"
      description: "Medicane Jolina"
      category: "case-study"
  output:
    catalogue: "/path/to/catalogue.parquet"
s1ifr-config-file: "/path/to/s1ifr/config.yml"
cdse_cache_dir: "/path/to/cache"
product_versions:
  l1b: ["A21", "A23"]
  l1c: ["B17", "B21"]

Command-Line Interface

Create a catalogue

catalog-iw --config config.yml create --output catalogue.parquet

Create a catalogue for a single listing (useful for parallel HPC processing):

catalog-iw --config config.yml create --output cat_ciaran.parquet --listing ciaran2023

Update an existing catalogue

catalog-iw --config config.yml update --catalogue catalogue.parquet

Merge multiple catalogues

catalog-iw --config config.yml merge cat1.parquet cat2.parquet --output merged.parquet

Generate statistics

catalog-iw --config config.yml stats --catalogue catalogue.parquet

Filter by dataset:

catalog-iw --config config.yml stats --catalogue catalogue.parquet --dataset ciaran2023

Export to JSON:

catalog-iw --config config.yml stats --catalogue catalogue.parquet --output stats.json

Launch web interface

catalog-iw --config config.yml serve --catalogue catalogue.parquet

Options:

• --host (default: 127.0.0.1)
• --port (default: 8649)
• --reload (development mode)

Backup the catalogue

catalog-iw --config config.yml backup --catalogue catalogue.parquet

Query a SAFE product

catalog-iw --config config.yml query --catalogue catalogue.parquet --safe-name S1A_IW_SLC_...

Very first steps

Initialize your code

1. Clone the repository and go inside:

git clone https://github.com/agrouaze/s1iw_catalogue.git
cd s1iw_catalogue

2. Create a virtual environment (optional but recommended):

python -m venv venv
source venv/bin/activate  # on Windows: venv\Scripts\activate

3. Install the package with development dependencies:

make install

This runs pip install -e .[dev] and also installs mypy stub types.

4. Install pre-commit hooks:

make pre-commit-install

5. Run codestyle formatting:

make codestyle

6. Upload initial code to GitHub (if you haven't already):

git add .
git commit -m ":tada: Initial commit"
git branch -M main
git remote add origin https://github.com/agrouaze/s1iw_catalogue.git
git push -u origin main

Set up bots

• Dependabot for keeping dependencies up to date.
• Stale bot for automatic issue closing.

Installation

You can install the package directly from GitHub or PyPI (once published).

pip install git+https://github.com/agrouaze/s1iw_catalogue.git

Or after publishing:

pip install s1iw_catalogue

Note: The package requires Python 3.11 or higher. It depends on s1ifr which is fetched from Ifremer's private PyPI index (configured automatically in pyproject.toml).

Python API Usage

Build the exhaustive Parquet catalogue

from s1iw_catalogue.catalogue import S1IWCatalogue

cat = S1IWCatalogue("catalogue.parquet", config="config.yml")
cat.create()

Query the catalogue

import polars as pl

df = pl.read_parquet("catalogue.parquet")

# Find a specific SAFE
safe_row = df.filter(pl.col("SAFE SLC") == "S1A_IW_SLC__...")

# Get all SAFE belonging to a dataset
dataset_safes = df.filter(pl.col("datasets").list.contains("ciaran2023"))

# Get statistics
stats = cat.stats()
print(stats["total_count"])

Makefile usage

The Makefile provides convenient commands.

1. Install dependencies

make install

Installs the package in editable mode with all dev dependencies.

2. Pre-commit hooks

make pre-commit-install

3. Codestyle

make codestyle          # formats code with pyupgrade, isort, black
make check-codestyle    # checks only, no changes

4. Type checks

make mypy

5. Tests with coverage

make test

Generates an HTML coverage report and a badge.

6. Security checks

make check-safety

Runs pip check, safety, and bandit.

7. All linters in one

make lint

Equivalent to make test && make check-codestyle && make mypy && make check-safety.

8. Docker

make docker-build       # builds Docker image (default tag latest)
make docker-remove      # removes the image

See docker/README.md for details.

9. Cleanup

make cleanup            # removes pycache, .DS_Store, .mypy_cache, .pytest_cache, build/

Releases

We follow Semantic Versions. See GitHub Releases for changelog.

License

This project is licensed under the MIT License – see the LICENSE file for details.

Citation

@misc{s1iw_catalogue,
  author = {lops-wave},
  title = {s1iw_catalogue – exhaustive Sentinel-1 IW SAFE catalogue for Ifremer},
  year = {2026},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/agrouaze/s1iw_catalogue}}
}

Credits

This project was generated with python-package-template and adapted for s1iw_catalogue.