fix: constraint- and length-aware data masking (issue #18)

CHANGELOG.md

@@ -2,6 +2,31 @@
 
 All notable changes to pgclone are documented in this file.
 
+## [4.4.2]
+
+### Fixed
+- **Constraint- and length-aware data masking** (issue #18). v4.4.1 made masking skip a strategy whose *type* a column could not store; issue #18 reported three further ways a mask could still abort or corrupt a clone. All are now handled — the offending mask is skipped with a `WARNING` (or clamped), and the clone succeeds:
+  - **Length overflow** — a `constant` (default `'REDACTED'`), `partial`, `hash`, or even `random_int` value could exceed a `varchar(N)`/`char(N)` column and fail with `value too long for type character varying(N)`. Mask output for a length-limited string column is now clamped with `left(…, N)` so it always fits.
+  - **`constant` on a numeric column** — the default `'REDACTED'` (or any non-numeric text) cannot be stored in an `integer`/`numeric` column (`invalid input syntax`). A `constant` mask on a non-string column is now applied only when the literal is a valid number for that column, otherwise skipped.
+  - **`null` on a NOT NULL column** — a `null` mask that would violate a `NOT NULL` constraint is skipped.
+  - **UNIQUE / PRIMARY KEY columns** — a value-collapsing (`name`, `constant`, `null`) or collision-prone (`random_int`) mask would break uniqueness (`duplicate key`). Only the injective `hash` strategy is applied to a UNIQUE/PK column; the rest are skipped.
+  - **FOREIGN KEY columns** — masking a FK column would break referential integrity, so it is skipped.
+- **`pgclone.discover_sensitive()` and `pgclone.masking_report()` only suggest strategies the engine will actually apply.** Foreign-key columns are omitted; a UNIQUE/PK sensitive column, or a NOT NULL column whose default strategy is `null` (e.g. `ssn`), is steered to `hash` (which preserves distinctness and non-nullness) when the type permits. A generated mask file is therefore safe to apply verbatim.
+
+### Changed
+- **`pgclone.mask_in_place()`** applies the same length clamp and constraint/type skips; when every rule is skipped it returns `OK: masked 0 rows … (0 columns — all mask rules were skipped as unsafe for their columns)`.
+- **`pgclone.create_masking_policy()`** applies the type/`constant` checks (a masked view enforces no table constraints, so uniqueness/FK/NOT NULL skips do not apply there).
+
+### Added
+- Internal C helpers in `src/pgclone.c`: `ColMaskMeta` plus `pgclone_column_maskmeta()` / `pgclone_maskmeta_from_row()` and the shared `PGCLONE_MASKMETA_COLS` catalog fragment (per-column typcategory, declared length, NOT NULL, UNIQUE/PK, FK); `pgclone_mask_skip_reason()` (single decision point for every skip rule); `pgclone_append_mask_expr_clamped()` (length clamp); `pgclone_discover_strategy()` (align suggestions with what the engine applies); and `pgclone_looks_numeric()`.
+- **pgTAP test group 27** (12 tests, plan 95 → 107) plus `test_schema.mask18_parent` / `mask18_child` fixtures (a UNIQUE NOT NULL `email`, a `varchar(4)` `code`, an `integer` `salary`, a NOT NULL `ssn`, and a FK `parent_id`): verifies length clamping, constant-on-numeric skip, uniqueness preservation (`name` skipped, `hash` applied and distinct), NOT NULL and FK skips, and that `discover_sensitive` steers the unique email to `hash`.
+
+### Compatibility
+- No SQL signature changes — the fix is entirely in the C masking engine. `sql/pgclone--4.4.1--4.4.2.sql` only bumps the installed version.
+- Behavior change is limited to masks that previously **crashed** the clone or violated a constraint: those columns are now clamped (length) or passed through unmasked with a `WARNING`. Masks that already worked are unaffected.
+- Synchronous paths only, as with the rest of the masking feature — `*_async` jobs ignore `"mask"`/`"masks"`.
+- Pure C-string/libpq implementation; identical behavior on PG 14–18.
+
 ## [4.4.1]
 
 ### Fixed

META.json

@@ -2,14 +2,14 @@
    "name": "pgclone",
    "abstract": "Clone PostgreSQL databases, schemas, tables between staging, test, dev and prod environments",
    "description": "PostgreSQL extension for easily cloning your DB, Schemas, Tables and more between environments",
-   "version": "4.4.1",
+   "version": "4.4.2",
    "maintainer": "Valeh Agayev <valeh.agayev@gmail.com>",
    "license": "postgresql",
    "provides": {
       "pgclone": {
          "abstract": "Clone PostgreSQL databases, schemas, and tables across environments",
-         "file": "sql/pgclone--4.4.1.sql",
-         "version": "4.4.1"
+         "file": "sql/pgclone--4.4.2.sql",
+         "version": "4.4.2"
       }
    },
    "prereqs": {

README.md

@@ -3,7 +3,7 @@
 [![CI](https://github.com/valehdba/pgclone/actions/workflows/ci.yml/badge.svg)](https://github.com/valehdba/pgclone/actions/workflows/ci.yml)
 [![Postgres 14–18](https://img.shields.io/badge/Postgres-14%E2%80%9318-336791?logo=postgresql&logoColor=white)](https://github.com/valehdba/pgclone)
 [![License](https://img.shields.io/badge/License-PostgreSQL-blue.svg)](https://github.com/valehdba/pgclone/blob/main/LICENSE)
-[![Version](https://img.shields.io/badge/version-4.4.1-orange)](https://github.com/valehdba/pgclone/releases/tag/v4.4.1)
+[![Version](https://img.shields.io/badge/version-4.4.2-orange)](https://github.com/valehdba/pgclone/releases/tag/v4.4.2)
 
 A PostgreSQL extension that clones databases, schemas, tables, and functions between PostgreSQL instances — directly from SQL. No `pg_dump`, no `pg_restore`, no shell scripts.
 
@@ -147,6 +147,7 @@ pgclone uses Unix domain sockets for local loopback connections, so the default
 - [x] v4.3.0: Consistent-snapshot clones — every clone reads the source under one shared `REPEATABLE READ READ ONLY` snapshot (sync, async, and parallel pool); cross-table FK consistency on a live source is now guaranteed (same model as `pg_dump -j`)
 - [x] v4.4.0: Schema/database-level in-line masking (`"masks"`) and regex table subset filters (`"tables"` / `"exclude_tables"`) — discussion #16
 - [x] v4.4.1: Type-aware masking — a mask that a column's type cannot store (e.g. `random_int` on a `boolean`) is skipped with a warning instead of aborting the clone; `discover_sensitive` no longer suggests incompatible masks — issue #17
+- [x] v4.4.2: Constraint- and length-aware masking — masks are clamped to a column's length and skipped when they would overflow, feed a bad literal to a numeric column, or violate a NOT NULL / UNIQUE / FOREIGN KEY constraint; UNIQUE/PK and NOT NULL sensitive columns are steered to `hash` — issue #18
 - [ ] v4.5.0: FK-aware referential sampling — sample N rows and follow foreign keys to keep child rows consistent (`pgclone.table_sample`)
 
 ## License

docs/USAGE.md

@@ -522,6 +522,20 @@ Each strategy emits a value of a fixed kind, and that value has to be storable i
 
 If a mask is **incompatible** with a column's type — e.g. `random_int` on a `boolean` flag, or `hash` on an `integer` — pgclone now **skips that mask and copies the column unchanged**, emitting a `WARNING` instead of aborting the clone with `invalid input syntax for type …` (issue #17). `discover_sensitive` (below) never suggests an incompatible mask in the first place, so a generated mask file is safe to apply verbatim.
 
+### Length and constraint safety (v4.4.2)
+
+Masking never breaks a clone by overflowing a column or violating a constraint (issue #18):
+
+- **Length** — output for a `varchar(N)`/`char(N)` column is clamped with `left(…, N)` so a `constant` (`'REDACTED'`), `partial`, `hash`, or `random_int` value always fits instead of failing with `value too long for type character varying(N)`.
+- **`constant` on a numeric column** — applied only when the literal is a valid number for that column; otherwise skipped (so the default `'REDACTED'` on an `integer` column no longer errors).
+- **`NOT NULL`** — a `null` mask on a `NOT NULL` column is skipped.
+- **`UNIQUE` / `PRIMARY KEY`** — only the injective `hash` strategy is applied (it keeps rows distinct); value-collapsing or collision-prone strategies (`name`, `constant`, `null`, `random_int`) are skipped. To mask a unique sensitive column (e.g. a unique `email`), use `hash`.
+- **`FOREIGN KEY`** — masking a FK column is skipped to preserve referential integrity.
+
+Skipped masks emit a `WARNING` naming the column and reason and leave the column unmasked. `discover_sensitive` / `masking_report` account for all of this: they omit FK columns and steer UNIQUE/PK and NOT NULL sensitive columns to `hash`.
+
+> **Masking keys (PK/UQ/FK).** Integer primary/unique/foreign keys generally should not be masked — no built-in strategy both fits an integer and preserves the constraint, so they are left unmasked (integer PK/UQ) or skipped (FK). If you need referentially-consistent key masking, apply the deterministic `hash` to the same column across parent and child yourself (matching values hash identically), keeping the columns a text type.
+
 ### Notes
 
 - Masking is applied on the **source** side inside `COPY (SELECT ...) TO STDOUT`, so masked data never enters the local database unmasked.

pgclone.control

@@ -1,6 +1,6 @@
 # pgclone extension
 comment = 'Clone PostgreSQL databases, schemas, tables, roles and permissions with selective columns, data filtering, data masking/anonymization, async, parallel cloning, materialized views, resume, and conflict resolution'
-default_version = '4.4.1'
+default_version = '4.4.2'
 module_pathname = '$libdir/pgclone'
 relocatable = false
 superuser = true

sql/pgclone--4.4.1--4.4.2.sql

@@ -0,0 +1,39 @@
+/* pgclone--4.4.1--4.4.2.sql */
+\echo Use "ALTER EXTENSION pgclone UPDATE" to load this file. \quit
+
+-- v4.4.2: Constraint- and length-aware data masking (GitHub issue #18).
+--
+-- No SQL signature changes — the fix lives entirely in the C masking
+-- engine (src/pgclone.c), so this upgrade script only bumps the
+-- installed version.
+--
+-- Background
+-- ----------
+-- v4.4.1 (issue #17) made masking skip a strategy whose *type* a column
+-- could not store. Issue #18 reported three further ways a mask could
+-- still break a clone:
+--   1. length  — a text/constant/partial value longer than a
+--                 varchar(N)/char(N) column -> "value too long".
+--   2. constant — the default 'REDACTED' (or any text) fed into a numeric
+--                 column -> "invalid input syntax".
+--   3. constraints — collapsing a UNIQUE/PRIMARY KEY column to one value,
+--                 nulling a NOT NULL column, or masking a FOREIGN KEY
+--                 column -> duplicate-key / not-null / FK violation.
+--
+-- Fix
+-- ---
+--   * Length: text (and any) mask output for a length-limited string
+--     column is clamped with left(..., N) so it always fits.
+--   * Constant: a "constant" mask is skipped on a non-string column
+--     unless the literal is a valid number for a numeric column.
+--   * NOT NULL: a "null" mask on a NOT NULL column is skipped.
+--   * UNIQUE/PK: only the injective "hash" strategy is applied; every
+--     value-collapsing or collision-prone strategy is skipped.
+--   * FOREIGN KEY: masking a FK column is skipped (referential integrity).
+--   * discover_sensitive() / masking_report() only suggest strategies the
+--     engine will actually apply: FK columns are omitted, and UNIQUE/PK or
+--     NOT NULL sensitive columns are steered to "hash".
+--   A masked view (create_masking_policy) enforces no constraints, so only
+--   the type/constant checks apply there.
+
+-- (No catalog changes.)