PZ Server Combine Redshift Catalogs

Combines multiple reference redshift catalogs into a single sample with homogenized data formats and a unique system of quality flags translated from the survey's original files.

Acknowledgements

Software developed and delivered as part of the in-kind contribution program BRA-LIN, from LIneA to the Rubin Observatory's LSST. An overview of this and other contributions is available here. The pipelines take advantage of the software support layer developed by LINCC, available as Python libraries: hats, hats-import and lsdb.

Tests

Development environment

Create the development environment with Conda or Mamba:

conda env create -f environment.dev.yaml
conda activate pipe_crd_dev

To update an existing environment after dependency changes:

conda env update -f environment.dev.yaml --prune

The development environment contains the pipeline runtime dependencies plus pytest, ruff, and pre-commit. The default suite contains only lightweight unit tests:

pytest

Pre-commit checks

With pipe_crd_dev activated, install the repository hook once:

pre-commit install

Every commit will then run essential Ruff checks and the lightweight unit-test suite. The Ruff hook checks Python errors, undefined or unused symbols, unsafe function defaults, problematic closures, and exception chaining. Notebooks, formatting, and opinionated style rules are not enforced.

Run the same checks manually across the repository with:

pre-commit run --all-files

Pull requests targeting main and commits merged into main run the same Ruff and lightweight unit-test checks automatically through GitHub Actions. The workflow also validates its own GitHub Actions configuration. New commits cancel older validation runs still in progress for the same pull request or branch.

Test data

The repository contains three small sample catalogs used by config.template.yaml for local smoke runs. Unit tests create their own temporary fixtures and do not depend on these files.

Install

The only requirement is to have micromamba available in PATH:

git clone https://github.com/linea-it/pzserver_combine_redshift_dedup && cd pzserver_combine_redshift_dedup
./install.sh

The installation script creates the pipe_crd environment with micromamba.

By default the scripts use MAMBA_ROOT_PREFIX="$HOME/.micromamba". On a Slurm cluster, point this variable to a persistent location visible to the jobs if needed:

export MAMBA_ROOT_PREFIX=/path/to/shared/or/persistent/micromamba

Run a pipeline

Copy and edit the example configuration, then execute the pipeline:

cp config.template.yaml config.yaml
./run.sh config.yaml process001