fix: type-aware masking — skip masks incompatible with a column's typ…

CHANGELOG.md

@@ -2,6 +2,27 @@
 
 All notable changes to pgclone are documented in this file.
 
+## [4.4.1]
+
+### Fixed
+- **Type-aware data masking — masks incompatible with a column's type no longer abort the clone** (issue #17). A mask strategy emits a value of a fixed kind — a text string (`email`/`name`/`phone`/`partial`/`hash`), an integer (`random_int`), SQL `NULL` (`null`), or a caller-supplied literal (`constant`). When that value was fed into a column whose type could not parse it, the clone failed deep inside the streaming `COPY`. The reported trigger: `pgclone.discover_sensitive()` matched a **boolean** column `base_income_replacement_allow` against the `%income%` financial pattern and suggested the numeric `random_int` strategy; applying it produced `ERROR: pgclone: COPY completed with error: ERROR: invalid input syntax for type boolean: "60629"`. Every mask-application site now checks the column's `pg_type.typcategory` first and **skips an incompatible mask with a `WARNING`** (leaving the column unchanged) instead of corrupting the `COPY` stream. Text masks fit string columns; `random_int` fits numeric or string columns; `null` and `constant` are left to the server to validate.
+- **`pgclone.discover_sensitive()` no longer suggests type-incompatible masks.** The scan now joins `pg_type` and omits a suggestion when the recommended strategy cannot be stored in the column's type (e.g. a numeric strategy for a boolean flag, or a text strategy for a numeric/date column), so a re-generated mask file is safe to apply verbatim.
+- **`pgclone.masking_report()`** flags a name-matched but type-incompatible column with a `Review manually: strategy X does not fit <type> column` recommendation rather than advising a mask that would fail.
+
+### Changed
+- **`pgclone.mask_in_place()`** skips incompatible column masks (with a `WARNING`) and, when *every* rule is incompatible, returns a clear `OK: masked 0 rows … (0 columns — all mask rules were incompatible with their column types)` instead of failing with an empty `UPDATE … SET`.
+- **`pgclone.create_masking_policy()`** leaves an incompatible column unmasked in the generated view, preserving the column's original type instead of silently changing it (e.g. `boolean` → `integer`).
+
+### Added
+- Internal C helpers in `src/pgclone.c`: `pgclone_mask_out_kind()` / `pgclone_strategy_fits()` / `pgclone_mask_kind_fits()` (mask-output-kind vs. `typcategory` compatibility), `pgclone_column_typcategory()` (per-column type-category lookup over a libpq connection), and `pgclone_masktype_name()` / `pgclone_typcat_desc()` (human-readable names for `WARNING` messages).
+- **pgTAP test group 26** (8 tests, plan 87 → 95) plus a `test_schema.flags` fixture (a boolean `income_verified` column whose name matches `%income%`, alongside a numeric `monthly_income` and string `contact_email`): verifies that a `random_int` mask on the boolean column is skipped and the clone succeeds, that the compatible email mask still applies, that `discover_sensitive` omits the boolean but keeps the numeric column, and that `mask_in_place` skips the incompatible mask.
+
+### Compatibility
+- No SQL signature changes — the fix is entirely in the C masking engine. `sql/pgclone--4.4.0--4.4.1.sql` only bumps the installed version.
+- Behavior change is limited to masks that previously **crashed** the clone (or, for `create_masking_policy`, silently changed a column's type): those columns are now passed through unmasked with a `WARNING`. Masks that already worked are unaffected.
+- Synchronous paths only, as with the rest of the masking feature — `*_async` jobs do not carry JSON options through shared memory and ignore `"mask"`/`"masks"`.
+- Pure C-string/libpq implementation; identical behavior on PG 14–18.
+
 ## [4.4.0]
 
 ### Added

META.json

@@ -2,14 +2,14 @@
    "name": "pgclone",
    "abstract": "Clone PostgreSQL databases, schemas, tables between staging, test, dev and prod environments",
    "description": "PostgreSQL extension for easily cloning your DB, Schemas, Tables and more between environments",
-   "version": "4.4.0",
+   "version": "4.4.1",
    "maintainer": "Valeh Agayev <valeh.agayev@gmail.com>",
    "license": "postgresql",
    "provides": {
       "pgclone": {
          "abstract": "Clone PostgreSQL databases, schemas, and tables across environments",
-         "file": "sql/pgclone--4.4.0.sql",
-         "version": "4.4.0"
+         "file": "sql/pgclone--4.4.1.sql",
+         "version": "4.4.1"
       }
    },
    "prereqs": {

README.md

@@ -3,7 +3,7 @@
 [![CI](https://github.com/valehdba/pgclone/actions/workflows/ci.yml/badge.svg)](https://github.com/valehdba/pgclone/actions/workflows/ci.yml)
 [![Postgres 14–18](https://img.shields.io/badge/Postgres-14%E2%80%9318-336791?logo=postgresql&logoColor=white)](https://github.com/valehdba/pgclone)
 [![License](https://img.shields.io/badge/License-PostgreSQL-blue.svg)](https://github.com/valehdba/pgclone/blob/main/LICENSE)
-[![Version](https://img.shields.io/badge/version-4.4.0-orange)](https://github.com/valehdba/pgclone/releases/tag/v4.4.0)
+[![Version](https://img.shields.io/badge/version-4.4.1-orange)](https://github.com/valehdba/pgclone/releases/tag/v4.4.1)
 
 A PostgreSQL extension that clones databases, schemas, tables, and functions between PostgreSQL instances — directly from SQL. No `pg_dump`, no `pg_restore`, no shell scripts.
 
@@ -146,6 +146,7 @@ pgclone uses Unix domain sockets for local loopback connections, so the default
 - [x] v4.2.0: Pre-flight validator — connection, space, permissions, version, name-conflict checks before clone (`pgclone.preflight`)
 - [x] v4.3.0: Consistent-snapshot clones — every clone reads the source under one shared `REPEATABLE READ READ ONLY` snapshot (sync, async, and parallel pool); cross-table FK consistency on a live source is now guaranteed (same model as `pg_dump -j`)
 - [x] v4.4.0: Schema/database-level in-line masking (`"masks"`) and regex table subset filters (`"tables"` / `"exclude_tables"`) — discussion #16
+- [x] v4.4.1: Type-aware masking — a mask that a column's type cannot store (e.g. `random_int` on a `boolean`) is skipped with a warning instead of aborting the clone; `discover_sensitive` no longer suggests incompatible masks — issue #17
 - [ ] v4.5.0: FK-aware referential sampling — sample N rows and follow foreign keys to keep child rows consistent (`pgclone.table_sample`)
 
 ## License

docs/USAGE.md

@@ -509,6 +509,19 @@ SELECT pgclone.table(conn, 'hr', 'employees', true, 'employees_dev',
 | `random_int` | Random in `[min, max]` | Ignores NULL (always produces value) |
 | `constant` | Fixed value | Ignores NULL (always produces value) |
 
+### Type compatibility (v4.4.1)
+
+Each strategy emits a value of a fixed kind, and that value has to be storable in the target column:
+
+| Strategy | Emits | Fits column types |
+|----------|-------|-------------------|
+| `email`, `name`, `phone`, `partial`, `hash` | text | string columns (`text`, `varchar`, `char`, …) |
+| `random_int` | integer | numeric or string columns |
+| `null` | SQL `NULL` | any nullable column |
+| `constant` | your literal | any type your value is valid for |
+
+If a mask is **incompatible** with a column's type — e.g. `random_int` on a `boolean` flag, or `hash` on an `integer` — pgclone now **skips that mask and copies the column unchanged**, emitting a `WARNING` instead of aborting the clone with `invalid input syntax for type …` (issue #17). `discover_sensitive` (below) never suggests an incompatible mask in the first place, so a generated mask file is safe to apply verbatim.
+
 ### Notes
 
 - Masking is applied on the **source** side inside `COPY (SELECT ...) TO STDOUT`, so masked data never enters the local database unmasked.
@@ -537,6 +550,27 @@ Returns JSON grouped by table with suggested mask strategies:
 
 The output can be used directly as the `"mask"` option value in a clone call. Detected patterns include: email, name (first/last/full), phone/mobile, SSN/national ID, salary/income, password/token/api_key, address/street, date of birth, credit card, and IP address.
 
+Type-incompatible suggestions are omitted automatically (v4.4.1): a `boolean` column that happens to match a financial pattern, for example, is left out rather than being paired with a numeric `random_int` strategy that the clone could not apply.
+
+### Driving masking from a file
+
+Because the options argument is just a JSON string, you can keep the (potentially large) mask definition in a file and let `psql` assemble the call — handy for automation and for tweaking the generated masks. Save `discover_sensitive`'s output, edit it, then reference it:
+
+```sh
+# 1. Generate a starting point (strip psql decoration with -tA)
+psql -tA -d mydb -c \
+  "SELECT pgclone.discover_sensitive('host=src dbname=app user=postgres','public')" \
+  > masks.json
+# 2. Edit masks.json as needed, then run the clone with the file's contents.
+#    psql's :'var' quotes and escapes the value into a SQL string literal.
+psql -d target_db \
+  -v masks="$(cat masks.json)" \
+  -c "SELECT pgclone.schema('host=src dbname=app user=postgres','public',true,
+        json_build_object('masks', :'masks'::json)::text)"
+```
+
+Any mechanism that produces the JSON string works — a heredoc, `\set masks \`cat masks.json\``, an application-side templating step, etc. pgclone only sees the final string.
+
 ---
 
 ## Static Data Masking (v3.2.0)

pgclone.control

@@ -1,6 +1,6 @@
 # pgclone extension
 comment = 'Clone PostgreSQL databases, schemas, tables, roles and permissions with selective columns, data filtering, data masking/anonymization, async, parallel cloning, materialized views, resume, and conflict resolution'
-default_version = '4.4.0'
+default_version = '4.4.1'
 module_pathname = '$libdir/pgclone'
 relocatable = false
 superuser = true

sql/pgclone--4.4.0--4.4.1.sql

@@ -0,0 +1,39 @@
+/* pgclone--4.4.0--4.4.1.sql */
+\echo Use "ALTER EXTENSION pgclone UPDATE" to load this file. \quit
+
+-- v4.4.1: Type-aware data masking (GitHub issue #17).
+--
+-- No SQL signature changes — the fix lives entirely in the C masking
+-- engine (src/pgclone.c), so this upgrade script only bumps the
+-- installed version.
+--
+-- Background
+-- ----------
+-- Every mask strategy emits a value of a fixed kind: a text string
+-- (email/name/phone/partial/hash), an integer (random_int), SQL NULL
+-- (null), or a caller-supplied literal (constant). When that value is
+-- fed into a column whose type cannot parse it, the clone aborted deep
+-- inside the streaming COPY, e.g. a numeric random_int mask applied to a
+-- BOOLEAN column named to match the %income% financial pattern:
+--
+--   ERROR:  pgclone: COPY completed with error:
+--           ERROR:  invalid input syntax for type boolean: "60629"
+--   CONTEXT:  COPY acquired_right, line 1,
+--             column base_income_replacement_allow: "60629"
+--
+-- Fix
+-- ---
+--   * Every mask-application site (schema/table/database clones,
+--     mask_in_place, create_masking_policy) now checks the column's
+--     pg_type.typcategory before applying a mask. An incompatible mask
+--     is skipped with a WARNING and the column is emitted unchanged,
+--     instead of corrupting the COPY stream. Text masks fit string
+--     columns; random_int fits numeric or string columns; null and
+--     constant are left to the server.
+--   * pgclone.discover_sensitive() no longer suggests a mask that is
+--     incompatible with a column's type (so a re-generated mask file is
+--     safe to apply verbatim), and pgclone.masking_report() flags such
+--     columns for manual review rather than recommending a mask that
+--     would fail.
+
+-- (No catalog changes.)