Strengthen distributed deduplication, validation, and observability in the CRC pipeline

[luigilcsilva]

Strengthen distributed deduplication, validation, and observability in the CRC pipeline

Context

The CRC pipeline performs crossmatching and connected-component deduplication over HATS catalogs, processing partitions independently with margins.

Real executions exposed risks related to:

• duplicated CRD_ID values across partitions;
• inconsistent tie_result and group_id values;
• transitive components crossing partitions and margins;
• dangling references in compared_to;
• graph truncation caused by crossmatch_n_neighbors;
• missing-redshift handling;
• unnecessary materializations in the HATS path;
• limited visibility into angular separation and graph chaining;
• repeated Dask computations;
• lack of an automated lightweight test suite.

Objective

Strengthen correctness, observability, performance, and testability without replacing the existing connected-component deduplication semantics.

Scope

Identifiers and groups

• Ensure global uniqueness of CRD_ID.
• Stabilize input partitions before calculating ID offsets.
• Generate canonical group_id values consistently between main and margin records.
• Detect components incorrectly split across groups.
• Detect dangling non-stellar references in compared_to.

Deduplication semantics

Validate the allowed tie_result patterns for every non-stellar group:

• one winner marked as 1, with all other objects marked as 0;
• two or more hard ties marked as 2, with all other objects marked as 0;
• stars marked as 3 and excluded from winner selection.

Reject invalid patterns such as:

• multiple winners;
• mixing tie_result=1 and tie_result=2;
• groups containing only losers;
• a single hard tie.

Additional rules:

• Apply non-transitive resolution based on delta_z.
• Ensure z=NaN loses against a defined redshift.
• Preserve hard ties when all redshifts are missing.
• Make the remove_all, keep_all, and draw_one policies explicit and testable.

Spatial and graph safety

• Validate that the crossmatch radius is smaller than the HATS margin.
• Monitor components approaching the margin extent.
• Detect edges connecting incompatible groups.
• Detect dangling references in compared_to.
• Monitor saturation at the configured n_neighbors limit.
• Allow warning and failure thresholds for saturation.
• Increase the default crossmatch_n_neighbors to 160.
• Preserve the default crossmatch radius of 0.5".

Production validations

Keep the following protections enabled by default:

validate_global_tie_invariants: true
validate_crd_id_uniqueness: true
crossmatch_saturation_enabled: true

Keep the more expensive distributed edge validation configurable and disabled by default:

validate_global_graph_edges: false

Optional geometric diagnostics

When enabled, record:

• angular-separation quantiles;
• fraction of edges close to the radius limit;
• neighbor-count distribution;
• fraction of objects at the n_neighbors limit;
• component-size distribution;
• large, tree-like, or low-density components;
• maximum distance from members to the group representative;
• components exceeding the radius, twice the radius, or the margin.

Representative-distance diagnostics must be aggregated on the driver so they appear in the central log independently of worker log handlers.

These secondary diagnostics must be configurable and disabled by default in production:

crossmatch_geometry_diagnostics_enabled: false
representative_radius_diagnostics_enabled: false
dedup_edge_diagnostics_enabled: false

When disabled, they must not add diagnostic columns, reductions, or unnecessary geometric processing.

HATS and performance

• Keep the final DataFrame lazy in the HATS output path.
• Avoid materializing the complete final catalog on the driver.
• Remove unnecessary head() calls and diagnostic materializations.
• Validate empty HATS outputs only after Parquet staging.
• Read only the dataproduct_type=object catalog, without mixing main and margin data.
• Restrict the no-margin fallback to the known legacy hats-import empty-margin error.
• Make costly repartitioning and global validations configurable.
• Persist compact deduplication labels once.
• Validate global tie invariants directly from the compact labels.
• Reuse persisted labels during the final merge and HATS output.
• Avoid recomputing the complete deduplication graph during consolidation.
• Group numeric-conversion validations into a shared Dask execution.
• Record driver and worker peak RSS in the central log.

Quality and CI

• Create a development Conda environment.
• Add a lightweight unit test suite.
• Exclude end-to-end tests from the default test run.
• Configure pre-commit with essential Ruff checks and lightweight pytest tests.
• Run the same validations automatically on pull requests through GitHub Actions.

Default production configuration

crossmatch_radius_arcsec: 0.5
crossmatch_n_neighbors: 160

validate_global_graph_edges: false
validate_global_tie_invariants: true
validate_crd_id_uniqueness: true
crossmatch_saturation_enabled: true

crossmatch_geometry_diagnostics_enabled: false
representative_radius_diagnostics_enabled: false
dedup_edge_diagnostics_enabled: false

Acceptance criteria

• CRD_ID generation is globally unique across partitions.
• ID generation is stabilized before partition offsets are calculated.
• Final groups respect the tie_result invariants.
• Stars do not participate in winner selection.
• Main and margin components receive consistent canonical identifiers.
• Missing redshifts lose against defined redshifts.
• Hard ties are preserved when all redshifts are missing.
• Delta-redshift resolution is non-transitive.
• remove_all, keep_all, and draw_one are explicitly tested.
• Crossmatch radius and HATS margin compatibility is validated.
• Neighbor saturation is logged and can stop the pipeline.
• Dangling references can be detected by global validation.
• The HATS path does not materialize the full catalog on the driver.
• Spatial diagnostics appear in the central log when enabled.
• Optional geometric diagnostics are disabled by default.
• Disabled diagnostics do not add the representative-radius column.
• Compact deduplication labels are persisted and reused.
• Global tie invariants are validated directly from compact labels.
• Numeric-conversion validations share one Dask execution.
• Driver and worker peak RSS are recorded centrally.
• HATS input discovery excludes margin catalogs.
• Legacy no-margin fallback is restricted to the known empty-margin failure.
• Ruff and lightweight tests pass through pre-commit.
• Ruff and lightweight tests run automatically on pull requests.
• Connected-component deduplication semantics remain unchanged.
• Run a full production-scale benchmark with the optimized labels path.
• Confirm that deduplication and HATS consolidation no longer recompute labels.
• Compare runtime and peak memory against the previous production execution.