From 87c2732f081967fcfee270b42c82cc0e3ec80f5c Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 1 Jul 2026 12:34:10 +0000 Subject: [PATCH] =?UTF-8?q?fix:=20type-aware=20masking=20=E2=80=94=20skip?= =?UTF-8?q?=20masks=20incompatible=20with=20a=20column's=20type=20(issue?= =?UTF-8?q?=20#17)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A schema clone with masking aborted mid-COPY when a mask produced a value the target column could not parse: ERROR: pgclone: COPY completed with error: ERROR: invalid input syntax for type boolean: "60629" CONTEXT: COPY acquired_right, line 1, column base_income_replacement_allow: "60629" The trigger: discover_sensitive() matched a boolean flag column against the %income% financial pattern and suggested the numeric random_int strategy. Every mask strategy emits a value of a fixed kind (text, integer, NULL, or a caller literal); when that value was fed into an incompatible column type the streaming COPY failed. Fix: every mask-application site now checks the column's pg_type.typcategory before applying a mask and skips an incompatible one with a WARNING, emitting the column unchanged instead of corrupting the COPY stream. Text masks fit string columns; random_int fits numeric or string columns; null and constant are left to the server. - pgclone.table / schema / database clones (both the catalog-driven and the explicit-columns COPY paths): skip + WARNING, column passes through. - discover_sensitive(): joins pg_type and omits a suggestion whose strategy cannot be stored in the column's type, so a generated mask file is safe to apply verbatim. - masking_report(): flags a name-matched but type-incompatible column for manual review instead of recommending a mask that would fail. - mask_in_place(): skips incompatible rules; when all rules are incompatible it returns a clear no-op message instead of an empty UPDATE ... SET. - create_masking_policy(): leaves an incompatible column unmasked in the view, preserving its original type (was silently changing boolean -> integer). New internal helpers: pgclone_mask_out_kind / pgclone_strategy_fits / pgclone_mask_kind_fits, pgclone_column_typcategory, pgclone_masktype_name, pgclone_typcat_desc. Tests: pgTAP group 26 (8 tests, plan 87 -> 95) plus a test_schema.flags fixture with a boolean %income% column. Docs: USAGE type-compatibility table and a "driving masking from a file" workflow. Version bumped to 4.4.1 with sql/pgclone--4.4.0--4.4.1.sql (no SQL signature changes). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01S219VD9ZDg211aSEZdE3jD --- CHANGELOG.md | 21 +++ META.json | 6 +- README.md | 3 +- docs/USAGE.md | 34 ++++ pgclone.control | 2 +- sql/pgclone--4.4.0--4.4.1.sql | 39 +++++ sql/pgclone--4.4.1.sql | 190 +++++++++++++++++++++ src/pgclone.c | 302 +++++++++++++++++++++++++++++++++- test/fixtures/seed.sql | 19 +++ test/pgclone_test.sql | 66 +++++++- 10 files changed, 669 insertions(+), 13 deletions(-) create mode 100644 sql/pgclone--4.4.0--4.4.1.sql create mode 100644 sql/pgclone--4.4.1.sql diff --git a/CHANGELOG.md b/CHANGELOG.md index ad1544a..e4121b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,27 @@ All notable changes to pgclone are documented in this file. +## [4.4.1] + +### Fixed +- **Type-aware data masking — masks incompatible with a column's type no longer abort the clone** (issue #17). A mask strategy emits a value of a fixed kind — a text string (`email`/`name`/`phone`/`partial`/`hash`), an integer (`random_int`), SQL `NULL` (`null`), or a caller-supplied literal (`constant`). When that value was fed into a column whose type could not parse it, the clone failed deep inside the streaming `COPY`. The reported trigger: `pgclone.discover_sensitive()` matched a **boolean** column `base_income_replacement_allow` against the `%income%` financial pattern and suggested the numeric `random_int` strategy; applying it produced `ERROR: pgclone: COPY completed with error: ERROR: invalid input syntax for type boolean: "60629"`. Every mask-application site now checks the column's `pg_type.typcategory` first and **skips an incompatible mask with a `WARNING`** (leaving the column unchanged) instead of corrupting the `COPY` stream. Text masks fit string columns; `random_int` fits numeric or string columns; `null` and `constant` are left to the server to validate. +- **`pgclone.discover_sensitive()` no longer suggests type-incompatible masks.** The scan now joins `pg_type` and omits a suggestion when the recommended strategy cannot be stored in the column's type (e.g. a numeric strategy for a boolean flag, or a text strategy for a numeric/date column), so a re-generated mask file is safe to apply verbatim. +- **`pgclone.masking_report()`** flags a name-matched but type-incompatible column with a `Review manually: strategy X does not fit column` recommendation rather than advising a mask that would fail. + +### Changed +- **`pgclone.mask_in_place()`** skips incompatible column masks (with a `WARNING`) and, when *every* rule is incompatible, returns a clear `OK: masked 0 rows … (0 columns — all mask rules were incompatible with their column types)` instead of failing with an empty `UPDATE … SET`. +- **`pgclone.create_masking_policy()`** leaves an incompatible column unmasked in the generated view, preserving the column's original type instead of silently changing it (e.g. `boolean` → `integer`). + +### Added +- Internal C helpers in `src/pgclone.c`: `pgclone_mask_out_kind()` / `pgclone_strategy_fits()` / `pgclone_mask_kind_fits()` (mask-output-kind vs. `typcategory` compatibility), `pgclone_column_typcategory()` (per-column type-category lookup over a libpq connection), and `pgclone_masktype_name()` / `pgclone_typcat_desc()` (human-readable names for `WARNING` messages). +- **pgTAP test group 26** (8 tests, plan 87 → 95) plus a `test_schema.flags` fixture (a boolean `income_verified` column whose name matches `%income%`, alongside a numeric `monthly_income` and string `contact_email`): verifies that a `random_int` mask on the boolean column is skipped and the clone succeeds, that the compatible email mask still applies, that `discover_sensitive` omits the boolean but keeps the numeric column, and that `mask_in_place` skips the incompatible mask. + +### Compatibility +- No SQL signature changes — the fix is entirely in the C masking engine. `sql/pgclone--4.4.0--4.4.1.sql` only bumps the installed version. +- Behavior change is limited to masks that previously **crashed** the clone (or, for `create_masking_policy`, silently changed a column's type): those columns are now passed through unmasked with a `WARNING`. Masks that already worked are unaffected. +- Synchronous paths only, as with the rest of the masking feature — `*_async` jobs do not carry JSON options through shared memory and ignore `"mask"`/`"masks"`. +- Pure C-string/libpq implementation; identical behavior on PG 14–18. + ## [4.4.0] ### Added diff --git a/META.json b/META.json index a41927d..5785acb 100644 --- a/META.json +++ b/META.json @@ -2,14 +2,14 @@ "name": "pgclone", "abstract": "Clone PostgreSQL databases, schemas, tables between staging, test, dev and prod environments", "description": "PostgreSQL extension for easily cloning your DB, Schemas, Tables and more between environments", - "version": "4.4.0", + "version": "4.4.1", "maintainer": "Valeh Agayev ", "license": "postgresql", "provides": { "pgclone": { "abstract": "Clone PostgreSQL databases, schemas, and tables across environments", - "file": "sql/pgclone--4.4.0.sql", - "version": "4.4.0" + "file": "sql/pgclone--4.4.1.sql", + "version": "4.4.1" } }, "prereqs": { diff --git a/README.md b/README.md index 8e01799..20a7e01 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ [![CI](https://github.com/valehdba/pgclone/actions/workflows/ci.yml/badge.svg)](https://github.com/valehdba/pgclone/actions/workflows/ci.yml) [![Postgres 14–18](https://img.shields.io/badge/Postgres-14%E2%80%9318-336791?logo=postgresql&logoColor=white)](https://github.com/valehdba/pgclone) [![License](https://img.shields.io/badge/License-PostgreSQL-blue.svg)](https://github.com/valehdba/pgclone/blob/main/LICENSE) -[![Version](https://img.shields.io/badge/version-4.4.0-orange)](https://github.com/valehdba/pgclone/releases/tag/v4.4.0) +[![Version](https://img.shields.io/badge/version-4.4.1-orange)](https://github.com/valehdba/pgclone/releases/tag/v4.4.1) A PostgreSQL extension that clones databases, schemas, tables, and functions between PostgreSQL instances — directly from SQL. No `pg_dump`, no `pg_restore`, no shell scripts. @@ -146,6 +146,7 @@ pgclone uses Unix domain sockets for local loopback connections, so the default - [x] v4.2.0: Pre-flight validator — connection, space, permissions, version, name-conflict checks before clone (`pgclone.preflight`) - [x] v4.3.0: Consistent-snapshot clones — every clone reads the source under one shared `REPEATABLE READ READ ONLY` snapshot (sync, async, and parallel pool); cross-table FK consistency on a live source is now guaranteed (same model as `pg_dump -j`) - [x] v4.4.0: Schema/database-level in-line masking (`"masks"`) and regex table subset filters (`"tables"` / `"exclude_tables"`) — discussion #16 +- [x] v4.4.1: Type-aware masking — a mask that a column's type cannot store (e.g. `random_int` on a `boolean`) is skipped with a warning instead of aborting the clone; `discover_sensitive` no longer suggests incompatible masks — issue #17 - [ ] v4.5.0: FK-aware referential sampling — sample N rows and follow foreign keys to keep child rows consistent (`pgclone.table_sample`) ## License diff --git a/docs/USAGE.md b/docs/USAGE.md index 0ff9042..3b8b9c4 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -509,6 +509,19 @@ SELECT pgclone.table(conn, 'hr', 'employees', true, 'employees_dev', | `random_int` | Random in `[min, max]` | Ignores NULL (always produces value) | | `constant` | Fixed value | Ignores NULL (always produces value) | +### Type compatibility (v4.4.1) + +Each strategy emits a value of a fixed kind, and that value has to be storable in the target column: + +| Strategy | Emits | Fits column types | +|----------|-------|-------------------| +| `email`, `name`, `phone`, `partial`, `hash` | text | string columns (`text`, `varchar`, `char`, …) | +| `random_int` | integer | numeric or string columns | +| `null` | SQL `NULL` | any nullable column | +| `constant` | your literal | any type your value is valid for | + +If a mask is **incompatible** with a column's type — e.g. `random_int` on a `boolean` flag, or `hash` on an `integer` — pgclone now **skips that mask and copies the column unchanged**, emitting a `WARNING` instead of aborting the clone with `invalid input syntax for type …` (issue #17). `discover_sensitive` (below) never suggests an incompatible mask in the first place, so a generated mask file is safe to apply verbatim. + ### Notes - Masking is applied on the **source** side inside `COPY (SELECT ...) TO STDOUT`, so masked data never enters the local database unmasked. @@ -537,6 +550,27 @@ Returns JSON grouped by table with suggested mask strategies: The output can be used directly as the `"mask"` option value in a clone call. Detected patterns include: email, name (first/last/full), phone/mobile, SSN/national ID, salary/income, password/token/api_key, address/street, date of birth, credit card, and IP address. +Type-incompatible suggestions are omitted automatically (v4.4.1): a `boolean` column that happens to match a financial pattern, for example, is left out rather than being paired with a numeric `random_int` strategy that the clone could not apply. + +### Driving masking from a file + +Because the options argument is just a JSON string, you can keep the (potentially large) mask definition in a file and let `psql` assemble the call — handy for automation and for tweaking the generated masks. Save `discover_sensitive`'s output, edit it, then reference it: + +```sh +# 1. Generate a starting point (strip psql decoration with -tA) +psql -tA -d mydb -c \ + "SELECT pgclone.discover_sensitive('host=src dbname=app user=postgres','public')" \ + > masks.json +# 2. Edit masks.json as needed, then run the clone with the file's contents. +# psql's :'var' quotes and escapes the value into a SQL string literal. +psql -d target_db \ + -v masks="$(cat masks.json)" \ + -c "SELECT pgclone.schema('host=src dbname=app user=postgres','public',true, + json_build_object('masks', :'masks'::json)::text)" +``` + +Any mechanism that produces the JSON string works — a heredoc, `\set masks \`cat masks.json\``, an application-side templating step, etc. pgclone only sees the final string. + --- ## Static Data Masking (v3.2.0) diff --git a/pgclone.control b/pgclone.control index 132d91d..7e07768 100644 --- a/pgclone.control +++ b/pgclone.control @@ -1,6 +1,6 @@ # pgclone extension comment = 'Clone PostgreSQL databases, schemas, tables, roles and permissions with selective columns, data filtering, data masking/anonymization, async, parallel cloning, materialized views, resume, and conflict resolution' -default_version = '4.4.0' +default_version = '4.4.1' module_pathname = '$libdir/pgclone' relocatable = false superuser = true diff --git a/sql/pgclone--4.4.0--4.4.1.sql b/sql/pgclone--4.4.0--4.4.1.sql new file mode 100644 index 0000000..d00745c --- /dev/null +++ b/sql/pgclone--4.4.0--4.4.1.sql @@ -0,0 +1,39 @@ +/* pgclone--4.4.0--4.4.1.sql */ +\echo Use "ALTER EXTENSION pgclone UPDATE" to load this file. \quit + +-- v4.4.1: Type-aware data masking (GitHub issue #17). +-- +-- No SQL signature changes — the fix lives entirely in the C masking +-- engine (src/pgclone.c), so this upgrade script only bumps the +-- installed version. +-- +-- Background +-- ---------- +-- Every mask strategy emits a value of a fixed kind: a text string +-- (email/name/phone/partial/hash), an integer (random_int), SQL NULL +-- (null), or a caller-supplied literal (constant). When that value is +-- fed into a column whose type cannot parse it, the clone aborted deep +-- inside the streaming COPY, e.g. a numeric random_int mask applied to a +-- BOOLEAN column named to match the %income% financial pattern: +-- +-- ERROR: pgclone: COPY completed with error: +-- ERROR: invalid input syntax for type boolean: "60629" +-- CONTEXT: COPY acquired_right, line 1, +-- column base_income_replacement_allow: "60629" +-- +-- Fix +-- --- +-- * Every mask-application site (schema/table/database clones, +-- mask_in_place, create_masking_policy) now checks the column's +-- pg_type.typcategory before applying a mask. An incompatible mask +-- is skipped with a WARNING and the column is emitted unchanged, +-- instead of corrupting the COPY stream. Text masks fit string +-- columns; random_int fits numeric or string columns; null and +-- constant are left to the server. +-- * pgclone.discover_sensitive() no longer suggests a mask that is +-- incompatible with a column's type (so a re-generated mask file is +-- safe to apply verbatim), and pgclone.masking_report() flags such +-- columns for manual review rather than recommending a mask that +-- would fail. + +-- (No catalog changes.) diff --git a/sql/pgclone--4.4.1.sql b/sql/pgclone--4.4.1.sql new file mode 100644 index 0000000..e1ddcb4 --- /dev/null +++ b/sql/pgclone--4.4.1.sql @@ -0,0 +1,190 @@ +/* pgclone--4.4.1.sql */ +\echo Use "CREATE EXTENSION pgclone" to load this file. \quit + +-- v4.4.1: Type-aware data masking (GitHub issue #17). +-- No new SQL surface — the fix is entirely in the C masking engine. +-- A mask strategy whose output value the target column cannot store +-- (e.g. the numeric "random_int" strategy on a BOOLEAN column matched +-- by the %income% pattern) is now skipped with a WARNING instead of +-- aborting the clone with "invalid input syntax for type boolean". +-- pgclone.discover_sensitive() likewise no longer suggests a mask that +-- is incompatible with a column's type. + +-- v4.4.0: Schema/database-level in-line masking and table subset filters +-- No new SQL surface — both features are new keys in the existing JSON +-- options argument of pgclone.schema() / pgclone.database() / +-- pgclone.database_create(): +-- '{"tables": ["order_.*"], "exclude_tables": [".*_old"], +-- "masks": {"users": {"email": "email", "ssn": "null"}}}' + +-- v4.3.0: Consistent-snapshot clones (REPEATABLE READ READ ONLY) +-- No new SQL surface — the behaviour is enabled by default for every +-- table/schema/database clone (sync and async, including parallel +-- pool mode) and can be disabled per-call with the options JSON: +-- SELECT pgclone.schema('...', 'public', true, '{"consistent": false}'); +-- See CHANGELOG.md / docs/USAGE.md for details and tradeoffs. + +-- v4.0.0: All functions now live under the 'pgclone' schema. +-- Usage: SELECT pgclone.table(...), pgclone.schema(...), etc. +CREATE SCHEMA IF NOT EXISTS pgclone; + +-- SYNCHRONOUS +CREATE FUNCTION pgclone.table(source_conninfo TEXT, schema_name TEXT, table_name TEXT, include_data BOOLEAN DEFAULT true) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_table' LANGUAGE C VOLATILE STRICT; +CREATE FUNCTION pgclone.table(source_conninfo TEXT, schema_name TEXT, table_name TEXT, include_data BOOLEAN, target_table_name TEXT) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_table' LANGUAGE C VOLATILE; +CREATE FUNCTION pgclone.table(source_conninfo TEXT, schema_name TEXT, table_name TEXT, include_data BOOLEAN, target_table_name TEXT, options TEXT) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_table' LANGUAGE C VOLATILE; +COMMENT ON FUNCTION pgclone.table(TEXT, TEXT, TEXT, BOOLEAN, TEXT, TEXT) IS 'Clone table with JSON options: {"columns":["col1","col2"], "where":"status=''active''", "indexes":false, "constraints":false, "triggers":false, "mask":{"email":"email","name":"name","phone":"phone","col":{"type":"partial","prefix":2,"suffix":3},"col2":"hash","col3":"null","col4":{"type":"random_int","min":0,"max":100},"col5":{"type":"constant","value":"REDACTED"}}}'; +CREATE FUNCTION pgclone.table_ex(source_conninfo TEXT, schema_name TEXT, table_name TEXT, include_data BOOLEAN, target_table_name TEXT, include_indexes BOOLEAN DEFAULT true, include_constraints BOOLEAN DEFAULT true, include_triggers BOOLEAN DEFAULT true) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_table_ex' LANGUAGE C VOLATILE; +CREATE FUNCTION pgclone.schema(source_conninfo TEXT, schema_name TEXT, include_data BOOLEAN DEFAULT true) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_schema' LANGUAGE C VOLATILE STRICT; +CREATE FUNCTION pgclone.schema(source_conninfo TEXT, schema_name TEXT, include_data BOOLEAN, options TEXT) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_schema' LANGUAGE C VOLATILE; +COMMENT ON FUNCTION pgclone.schema(TEXT, TEXT, BOOLEAN, TEXT) IS 'Clone schema with JSON options: {"indexes":false, "constraints":false, "triggers":false, "tables":["regex1","regex2"], "exclude_tables":["regex"], "masks":{"table_name":{"col":"email"},"schema.table":{"col":"hash"}}} — tables/exclude_tables are anchored POSIX regexes selecting a subset of tables; masks applies per-table in-line masking during the COPY stream (v4.4.0).'; +CREATE FUNCTION pgclone.schema_ex(source_conninfo TEXT, schema_name TEXT, include_data BOOLEAN, include_indexes BOOLEAN DEFAULT true, include_constraints BOOLEAN DEFAULT true, include_triggers BOOLEAN DEFAULT true) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_schema_ex' LANGUAGE C VOLATILE; +CREATE FUNCTION pgclone.functions(source_conninfo TEXT, schema_name TEXT) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_functions' LANGUAGE C VOLATILE STRICT; +CREATE FUNCTION pgclone.database(source_conninfo TEXT, include_data BOOLEAN DEFAULT true) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_database' LANGUAGE C VOLATILE STRICT; +CREATE FUNCTION pgclone.database(source_conninfo TEXT, include_data BOOLEAN, options TEXT) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_database' LANGUAGE C VOLATILE; +COMMENT ON FUNCTION pgclone.database(TEXT, BOOLEAN, TEXT) IS 'Clone database with JSON options — supports the same "tables", "exclude_tables", and "masks" options as pgclone.schema (v4.4.0); masks keys may be schema-qualified ("schema.table") to disambiguate.'; + +-- v2.0.1: Create target database and clone into it +CREATE FUNCTION pgclone.database_create(source_conninfo TEXT, target_dbname TEXT, include_data BOOLEAN DEFAULT true) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_database_create' LANGUAGE C VOLATILE; +CREATE FUNCTION pgclone.database_create(source_conninfo TEXT, target_dbname TEXT, include_data BOOLEAN, options TEXT) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_database_create' LANGUAGE C VOLATILE; +COMMENT ON FUNCTION pgclone.database_create(TEXT, TEXT, BOOLEAN) IS 'Create target database if not exists, then clone all schemas/tables/functions from source. Run from postgres DB.'; + +-- ASYNC (require shared_preload_libraries = 'pgclone') +CREATE FUNCTION pgclone.table_async(source_conninfo TEXT, schema_name TEXT, table_name TEXT, include_data BOOLEAN DEFAULT true, target_table_name TEXT DEFAULT NULL, options TEXT DEFAULT NULL) RETURNS INTEGER AS 'MODULE_PATHNAME', 'pgclone_table_async' LANGUAGE C VOLATILE; +CREATE FUNCTION pgclone.schema_async(source_conninfo TEXT, schema_name TEXT, include_data BOOLEAN DEFAULT true, options TEXT DEFAULT NULL) RETURNS INTEGER AS 'MODULE_PATHNAME', 'pgclone_schema_async' LANGUAGE C VOLATILE; + +-- PROGRESS & JOB MANAGEMENT +CREATE FUNCTION pgclone.progress(job_id INTEGER) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_progress' LANGUAGE C VOLATILE STRICT; +CREATE FUNCTION pgclone.cancel(job_id INTEGER) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_cancel' LANGUAGE C VOLATILE STRICT; +CREATE FUNCTION pgclone.resume(job_id INTEGER) RETURNS INTEGER AS 'MODULE_PATHNAME', 'pgclone_resume' LANGUAGE C VOLATILE STRICT; +CREATE FUNCTION pgclone.jobs() RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_jobs' LANGUAGE C VOLATILE STRICT; +CREATE FUNCTION pgclone.clear_jobs() RETURNS INTEGER AS 'MODULE_PATHNAME', 'pgclone_clear_jobs' LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.clear_jobs() IS 'Clear completed/failed/cancelled job slots from shared memory'; + +-- v2.1.0+v2.1.1+v2.1.2: Progress Tracking View with progress bar, elapsed time, ETA +CREATE FUNCTION pgclone.progress_detail() +RETURNS TABLE ( + job_id INTEGER, + status TEXT, + op_type TEXT, + schema_name TEXT, + table_name TEXT, + current_phase TEXT, + current_table TEXT, + tables_total BIGINT, + tables_completed BIGINT, + rows_copied BIGINT, + bytes_copied BIGINT, + elapsed_ms BIGINT, + start_time TIMESTAMPTZ, + end_time TIMESTAMPTZ, + error_message TEXT, + pct_complete DOUBLE PRECISION, + progress_bar TEXT, + elapsed_time TEXT +) AS 'MODULE_PATHNAME', 'pgclone_progress_view' +LANGUAGE C VOLATILE STRICT; + +COMMENT ON FUNCTION pgclone.progress_detail() IS 'Returns tabular progress with visual progress bar and elapsed time for all clone jobs'; + +-- VIEW: convenient wrapper +CREATE VIEW pgclone.jobs_view AS + SELECT * FROM pgclone.progress_detail(); + +COMMENT ON VIEW pgclone.jobs_view IS 'Live progress tracking view with progress bar and elapsed time for all pgclone async clone jobs'; + +-- v3.1.0: Auto-discovery of sensitive data +CREATE FUNCTION pgclone.discover_sensitive(source_conninfo TEXT, schema_name TEXT) +RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_discover_sensitive' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.discover_sensitive(TEXT, TEXT) IS 'Scan source schema for columns matching sensitive data patterns (email, name, phone, ssn, salary, etc.) and return suggested mask rules as JSON'; + +-- v3.2.0: Static data masking on local tables +CREATE FUNCTION pgclone.mask_in_place(schema_name TEXT, table_name TEXT, mask_json TEXT) +RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_mask_in_place' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.mask_in_place(TEXT, TEXT, TEXT) IS 'Apply data masking to an existing local table via UPDATE. mask_json uses same format as clone mask option: {"email": "email", "name": "name", "ssn": "null"}'; + +-- v3.3.0: Dynamic data masking via views and role-based access +CREATE FUNCTION pgclone.create_masking_policy(schema_name TEXT, table_name TEXT, mask_json TEXT, privileged_role TEXT) +RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_create_masking_policy' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.create_masking_policy(TEXT, TEXT, TEXT, TEXT) IS 'Create a dynamic masking policy: creates a masked view, revokes base table access from PUBLIC, grants view to PUBLIC, grants base table to privileged role'; + +CREATE FUNCTION pgclone.drop_masking_policy(schema_name TEXT, table_name TEXT) +RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_drop_masking_policy' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.drop_masking_policy(TEXT, TEXT) IS 'Remove a dynamic masking policy: drops the masked view and restores base table access to PUBLIC'; + +-- v3.4.0: Clone roles with permissions and passwords +CREATE FUNCTION pgclone.clone_roles(source_conninfo TEXT) +RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_clone_roles' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.clone_roles(TEXT) IS 'Clone all non-system roles from source with encrypted passwords, attributes, memberships, and all permissions. Requires superuser on both source and target.'; + +CREATE FUNCTION pgclone.clone_roles(source_conninfo TEXT, role_names TEXT) +RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_clone_roles' +LANGUAGE C VOLATILE; +COMMENT ON FUNCTION pgclone.clone_roles(TEXT, TEXT) IS 'Clone specific roles (comma-separated) from source with encrypted passwords, attributes, memberships, and permissions. If role exists on target, syncs password and attributes without dropping.'; + +-- v3.5.0: Clone verification — compare row counts +CREATE FUNCTION pgclone.verify(source_conninfo TEXT, schema_name TEXT) +RETURNS TABLE ( + schema_name TEXT, + table_name TEXT, + source_rows BIGINT, + target_rows BIGINT, + match TEXT +) AS 'MODULE_PATHNAME', 'pgclone_verify' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.verify(TEXT, TEXT) IS 'Compare row counts between source and local target for all tables in a schema. Returns side-by-side comparison with match status.'; + +CREATE FUNCTION pgclone.verify(source_conninfo TEXT) +RETURNS TABLE ( + schema_name TEXT, + table_name TEXT, + source_rows BIGINT, + target_rows BIGINT, + match TEXT +) AS 'MODULE_PATHNAME', 'pgclone_verify' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.verify(TEXT) IS 'Compare row counts between source and local target for all user tables across all schemas. Returns side-by-side comparison with match status.'; + +-- v3.6.0: GDPR/Compliance masking report +CREATE FUNCTION pgclone.masking_report(schema_name TEXT) +RETURNS TABLE ( + schema_name TEXT, + table_name TEXT, + column_name TEXT, + sensitivity TEXT, + mask_status TEXT, + recommendation TEXT +) AS 'MODULE_PATHNAME', 'pgclone_masking_report' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.masking_report(TEXT) IS 'Generate GDPR/compliance audit report: lists sensitive columns, their masking status, and recommendations. Checks for masked views.'; + +-- v4.1.0: Schema diff — DDL drift detection between source and local target +CREATE FUNCTION pgclone.diff(source_conninfo TEXT, schema_name TEXT) +RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_diff' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.diff(TEXT, TEXT) IS + 'Compare DDL of a schema between source and the local target. ' + 'Returns JSON drift report listing objects only_in_source / only_in_target / modified ' + 'across tables (with per-column type/nullability/default drift), indexes, ' + 'constraints, triggers, views, and sequences. Read-only on both sides.'; + +-- v4.2.0: Pre-flight validator — connection, permissions, version, capacity, +-- name-conflict, role and tablespace checks before a clone. +CREATE FUNCTION pgclone.preflight(source_conninfo TEXT, schema_name TEXT) +RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_preflight' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.preflight(TEXT, TEXT) IS + 'Validate that a clone of the given schema from source into the local ' + 'target is likely to succeed. Returns JSON with errors / warnings / info ' + 'arrays plus a per-check breakdown covering: source/target connection, ' + 'PostgreSQL versions, schema existence, USAGE/SELECT/CREATE permissions, ' + 'estimated source size, target database size, object counts, name ' + 'conflicts on the target schema, missing extensions, missing roles, and ' + 'missing non-default tablespaces. Read-only on both sides.'; + +-- VERSION +CREATE FUNCTION pgclone.version() RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_version' LANGUAGE C IMMUTABLE STRICT; diff --git a/src/pgclone.c b/src/pgclone.c index df5516b..1ce77cd 100644 --- a/src/pgclone.c +++ b/src/pgclone.c @@ -885,6 +885,184 @@ pgclone_find_mask_rule(const CloneOptions *opts, const char *col_name) return NULL; } +/* --------------------------------------------------------------- + * v4.4.1 (issue #17): type-aware masking. + * + * A mask strategy emits a value of a fixed "kind" — a text string, + * an integer, SQL NULL, or a caller-supplied literal. When that value + * is fed into a column whose type cannot parse it (e.g. a numeric + * mask like random_int applied to a boolean flag column named + * "..._income_..."), the clone fails deep inside the streaming COPY + * with a cryptic + * ERROR: invalid input syntax for type boolean: "60629" + * These helpers let every mask-application site check up front whether + * the strategy fits the column's type and skip the mask when it does + * not, instead of corrupting the COPY stream. + * --------------------------------------------------------------- */ +typedef enum PgcloneMaskOutKind +{ + MASK_OUT_ASIS = 0, /* column emitted unchanged (no mask) */ + MASK_OUT_TEXT, /* text value: email/name/phone/partial/hash */ + MASK_OUT_NUMERIC, /* integer value: random_int */ + MASK_OUT_NULLONLY, /* SQL NULL: null */ + MASK_OUT_ANY /* caller-supplied literal: constant (trusted) */ +} PgcloneMaskOutKind; + +/* What kind of value does this mask strategy emit? */ +static PgcloneMaskOutKind +pgclone_mask_out_kind(PgcloneMaskType t) +{ + switch (t) + { + case PGCLONE_MASK_EMAIL: + case PGCLONE_MASK_NAME: + case PGCLONE_MASK_PHONE: + case PGCLONE_MASK_PARTIAL: + case PGCLONE_MASK_HASH: + return MASK_OUT_TEXT; + case PGCLONE_MASK_RANDOM_INT: + return MASK_OUT_NUMERIC; + case PGCLONE_MASK_NULL: + return MASK_OUT_NULLONLY; + case PGCLONE_MASK_CONSTANT: + return MASK_OUT_ANY; + case PGCLONE_MASK_NONE: + default: + return MASK_OUT_ASIS; + } +} + +/* + * Can a value of kind `kind` be stored in a column whose pg_type.typcategory + * is `typcat`? A '\0' typcat means "unknown" — we stay permissive and let the + * server enforce types rather than skip a mask we cannot classify. + * + * The allow-list is deliberately conservative: text masks only fit string + * columns ('S'); a numeric mask fits numeric ('N') or string ('S') columns + * (an integer renders cleanly into both). NULL and caller-supplied constants + * are left to the server (NULL casts to any type; a constant is the user's + * explicit, trusted literal). + */ +static bool +pgclone_mask_kind_fits(PgcloneMaskOutKind kind, char typcat) +{ + if (typcat == '\0') + return true; + + switch (kind) + { + case MASK_OUT_TEXT: + return typcat == 'S'; + case MASK_OUT_NUMERIC: + return typcat == 'N' || typcat == 'S'; + case MASK_OUT_NULLONLY: + case MASK_OUT_ANY: + case MASK_OUT_ASIS: + default: + return true; + } +} + +/* Same test, keyed by the strategy string used by discover_sensitive / + * masking_report. A discover-generated "constant" carries the built-in + * "REDACTED" text value, so it is classified as text here (unlike a + * user-supplied constant, which pgclone_mask_out_kind trusts as ANY). */ +static bool +pgclone_strategy_fits(const char *strategy, char typcat) +{ + PgcloneMaskOutKind kind; + + if (strategy == NULL) + return true; + else if (strcmp(strategy, "random_int") == 0) + kind = MASK_OUT_NUMERIC; + else if (strcmp(strategy, "null") == 0) + kind = MASK_OUT_NULLONLY; + else + kind = MASK_OUT_TEXT; /* email/name/phone/partial/hash/constant */ + + return pgclone_mask_kind_fits(kind, typcat); +} + +/* Human-readable mask strategy name, for WARNING messages. */ +static const char * +pgclone_masktype_name(PgcloneMaskType t) +{ + switch (t) + { + case PGCLONE_MASK_EMAIL: return "email"; + case PGCLONE_MASK_NAME: return "name"; + case PGCLONE_MASK_PHONE: return "phone"; + case PGCLONE_MASK_PARTIAL: return "partial"; + case PGCLONE_MASK_HASH: return "hash"; + case PGCLONE_MASK_NULL: return "null"; + case PGCLONE_MASK_RANDOM_INT: return "random_int"; + case PGCLONE_MASK_CONSTANT: return "constant"; + case PGCLONE_MASK_NONE: + default: return "none"; + } +} + +/* Human-readable pg_type.typcategory, for WARNING messages. */ +static const char * +pgclone_typcat_desc(char typcat) +{ + switch (typcat) + { + case 'B': return "boolean"; + case 'D': return "date/time"; + case 'T': return "timespan"; + case 'N': return "numeric"; + case 'S': return "string"; + case 'E': return "enum"; + case 'G': return "geometric"; + case 'I': return "network address"; + case 'R': return "range"; + case 'A': return "array"; + case 'U': return "binary/user-defined"; + default: return "this"; + } +} + +/* + * Look up pg_type.typcategory for schema.table.column over `conn`. + * Returns the single category char, or '\0' when it cannot be determined + * (callers then treat the mask as compatible and let the server decide). + */ +static char +pgclone_column_typcategory(PGconn *conn, const char *schema_name, + const char *table_name, const char *col_name) +{ + StringInfoData q; + PGresult *r; + char cat = '\0'; + + initStringInfo(&q); + appendStringInfo(&q, + "SELECT t.typcategory " + "FROM pg_catalog.pg_attribute a " + "JOIN pg_catalog.pg_class c ON c.oid = a.attrelid " + "JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace " + "JOIN pg_catalog.pg_type t ON t.oid = a.atttypid " + "WHERE n.nspname = %s AND c.relname = %s AND a.attname = %s " + "AND a.attnum > 0 AND NOT a.attisdropped", + quote_literal_cstr(schema_name), + quote_literal_cstr(table_name), + quote_literal_cstr(col_name)); + + r = PQexec(conn, q.data); + pfree(q.data); + + if (PQresultStatus(r) == PGRES_TUPLES_OK && PQntuples(r) > 0) + { + char *v = PQgetvalue(r, 0, 0); + if (v != NULL && v[0] != '\0') + cat = v[0]; + } + PQclear(r); + return cat; +} + /* --------------------------------------------------------------- * v4.4.0: Find the raw mask JSON for a table inside a schema clone, * or NULL if none. Schema-qualified keys ("schema.table") win over @@ -1656,10 +1834,11 @@ pgclone_copy_data(PGconn *source_conn, PGconn *local_conn, initStringInfo(&col_query); appendStringInfo(&col_query, - "SELECT a.attname " + "SELECT a.attname, t.typcategory " "FROM pg_catalog.pg_attribute a " "JOIN pg_catalog.pg_class c ON c.oid = a.attrelid " "JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace " + "JOIN pg_catalog.pg_type t ON t.oid = a.atttypid " "WHERE n.nspname = %s AND c.relname = %s " "AND a.attnum > 0 AND NOT a.attisdropped " "ORDER BY a.attnum", @@ -1683,12 +1862,29 @@ pgclone_copy_data(PGconn *source_conn, PGconn *local_conn, for (ci = 0; ci < ncols; ci++) { const char *col_name = PQgetvalue(col_res, ci, 0); + const char *typcat_s = PQgetvalue(col_res, ci, 1); + char typcat = (typcat_s != NULL) ? typcat_s[0] : '\0'; const char *col_ident = quote_identifier(col_name); const MaskRule *rule = pgclone_find_mask_rule(opts, col_name); if (ci > 0) appendStringInfoString(&select_cmd, ", "); + /* issue #17: skip a mask whose output type the column + * cannot store (e.g. random_int on a boolean); emit the + * column unchanged rather than corrupt the COPY stream. */ + if (rule != NULL && + !pgclone_mask_kind_fits(pgclone_mask_out_kind(rule->type), + typcat)) + { + ereport(WARNING, + (errmsg("pgclone: mask \"%s\" is not compatible with %s column \"%s\"; leaving it unmasked", + pgclone_masktype_name(rule->type), + pgclone_typcat_desc(typcat), + col_name))); + rule = NULL; + } + if (rule != NULL) { pgclone_build_mask_expr(&select_cmd, col_ident, rule); @@ -1714,6 +1910,26 @@ pgclone_copy_data(PGconn *source_conn, PGconn *local_conn, if (ci > 0) appendStringInfoString(&select_cmd, ", "); + /* issue #17: skip a mask that the column's type cannot + * store (looked up per masked column on the source). */ + if (rule != NULL) + { + char typcat = pgclone_column_typcategory(source_conn, + schema_name, + source_table, + opts->columns[ci]); + if (!pgclone_mask_kind_fits(pgclone_mask_out_kind(rule->type), + typcat)) + { + ereport(WARNING, + (errmsg("pgclone: mask \"%s\" is not compatible with %s column \"%s\"; leaving it unmasked", + pgclone_masktype_name(rule->type), + pgclone_typcat_desc(typcat), + opts->columns[ci]))); + rule = NULL; + } + } + if (rule != NULL) { pgclone_build_mask_expr(&select_cmd, col_ident, rule); @@ -3914,10 +4130,12 @@ pgclone_discover_sensitive(PG_FUNCTION_ARGS) initStringInfo(&query); appendStringInfo(&query, "SELECT c.relname AS table_name, " - " a.attname AS column_name " + " a.attname AS column_name, " + " t.typcategory AS typcategory " "FROM pg_catalog.pg_class c " "JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace " "JOIN pg_catalog.pg_attribute a ON a.attrelid = c.oid " + "JOIN pg_catalog.pg_type t ON t.oid = a.atttypid " "WHERE n.nspname = %s " "AND c.relkind IN ('r', 'p') " /* regular tables + partitioned */ "AND a.attnum > 0 AND NOT a.attisdropped " @@ -3950,6 +4168,8 @@ pgclone_discover_sensitive(PG_FUNCTION_ARGS) { const char *tbl = PQgetvalue(res, i, 0); const char *col = PQgetvalue(res, i, 1); + const char *typcat_s = PQgetvalue(res, i, 2); + char typcat = (typcat_s != NULL) ? typcat_s[0] : '\0'; const SensitivityRule *rule; rule = pgclone_match_sensitivity(col); @@ -3957,6 +4177,13 @@ pgclone_discover_sensitive(PG_FUNCTION_ARGS) if (rule == NULL) continue; + /* issue #17: don't suggest a mask the column type cannot + * store (e.g. random_int for a boolean "..._income_..." + * flag) — such a suggestion, applied verbatim, would abort + * the clone with "invalid input syntax for type ...". */ + if (!pgclone_strategy_fits(rule->strategy, typcat)) + continue; + /* New table? Close previous table's object and open new one. */ if (strcmp(tbl, current_table) != 0) { @@ -4027,6 +4254,7 @@ pgclone_mask_in_place(PG_FUNCTION_ARGS) StringInfoData wrapped; CloneOptions opts; int i; + int applied = 0; bool first_set = true; int64 rows_affected; @@ -4042,6 +4270,8 @@ pgclone_mask_in_place(PG_FUNCTION_ARGS) errmsg("pgclone: no valid mask rules in JSON"), errhint("Provide mask rules like: {\"email\": \"email\", \"name\": \"name\"}"))); + local_conn = pgclone_connect_local(); + /* Build UPDATE statement */ initStringInfo(&update_cmd); appendStringInfo(&update_cmd, "UPDATE %s.%s SET ", @@ -4053,6 +4283,21 @@ pgclone_mask_in_place(PG_FUNCTION_ARGS) const MaskRule *rule = &opts.masks[i]; const char *col_ident = quote_identifier(rule->column); StringInfoData expr; + char typcat; + + /* issue #17: skip a mask whose output type the column cannot + * store — otherwise the UPDATE fails with a type-mismatch error. */ + typcat = pgclone_column_typcategory(local_conn, schema_name, + table_name, rule->column); + if (!pgclone_mask_kind_fits(pgclone_mask_out_kind(rule->type), typcat)) + { + ereport(WARNING, + (errmsg("pgclone: mask \"%s\" is not compatible with %s column \"%s\"; leaving it unmasked", + pgclone_masktype_name(rule->type), + pgclone_typcat_desc(typcat), + rule->column))); + continue; + } if (!first_set) appendStringInfoString(&update_cmd, ", "); @@ -4063,9 +4308,21 @@ pgclone_mask_in_place(PG_FUNCTION_ARGS) appendStringInfo(&update_cmd, "%s = %s", col_ident, expr.data); pfree(expr.data); first_set = false; + applied++; } - local_conn = pgclone_connect_local(); + /* All masks skipped as type-incompatible — nothing to do. */ + if (applied == 0) + { + PQfinish(local_conn); + pfree(update_cmd.data); + + initStringInfo(&result); + appendStringInfo(&result, + "OK: masked 0 rows in %s.%s (0 columns — all mask rules were incompatible with their column types)", + schema_name, table_name); + PG_RETURN_TEXT_P(cstring_to_text(result.data)); + } res = PQexec(local_conn, update_cmd.data); @@ -4089,7 +4346,7 @@ pgclone_mask_in_place(PG_FUNCTION_ARGS) "OK: masked %ld rows in %s.%s (%d columns)", (long) rows_affected, schema_name, table_name, - opts.num_masks); + applied); pfree(update_cmd.data); @@ -4155,10 +4412,11 @@ pgclone_create_masking_policy(PG_FUNCTION_ARGS) /* Query local catalog for column names */ initStringInfo(&col_query); appendStringInfo(&col_query, - "SELECT a.attname " + "SELECT a.attname, t.typcategory " "FROM pg_catalog.pg_attribute a " "JOIN pg_catalog.pg_class c ON c.oid = a.attrelid " "JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace " + "JOIN pg_catalog.pg_type t ON t.oid = a.atttypid " "WHERE n.nspname = %s AND c.relname = %s " "AND a.attnum > 0 AND NOT a.attisdropped " "ORDER BY a.attnum", @@ -4200,12 +4458,28 @@ pgclone_create_masking_policy(PG_FUNCTION_ARGS) for (ci = 0; ci < ncols; ci++) { const char *col_name = PQgetvalue(col_res, ci, 0); + const char *typcat_s = PQgetvalue(col_res, ci, 1); + char typcat = (typcat_s != NULL) ? typcat_s[0] : '\0'; const char *col_ident = quote_identifier(col_name); const MaskRule *rule = pgclone_find_mask_rule(&opts, col_name); if (ci > 0) appendStringInfoString(&view_cmd, ", "); + /* issue #17: skip a mask whose output type the column cannot + * store — otherwise the masked view would silently change the + * column's type (e.g. boolean -> integer for random_int). */ + if (rule != NULL && + !pgclone_mask_kind_fits(pgclone_mask_out_kind(rule->type), typcat)) + { + ereport(WARNING, + (errmsg("pgclone: mask \"%s\" is not compatible with %s column \"%s\"; leaving it unmasked", + pgclone_masktype_name(rule->type), + pgclone_typcat_desc(typcat), + col_name))); + rule = NULL; + } + if (rule != NULL) { pgclone_build_mask_expr(&view_cmd, col_ident, rule); @@ -5094,10 +5368,11 @@ pgclone_masking_report(PG_FUNCTION_ARGS) /* Get all columns in the schema */ initStringInfo(&query); appendStringInfo(&query, - "SELECT c.relname, a.attname " + "SELECT c.relname, a.attname, t.typcategory " "FROM pg_catalog.pg_class c " "JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace " "JOIN pg_catalog.pg_attribute a ON a.attrelid = c.oid " + "JOIN pg_catalog.pg_type t ON t.oid = a.atttypid " "WHERE n.nspname = %s " "AND c.relkind IN ('r', 'p') " "AND a.attnum > 0 AND NOT a.attisdropped " @@ -5141,6 +5416,8 @@ pgclone_masking_report(PG_FUNCTION_ARGS) { const char *tbl = PQgetvalue(col_res, i, 0); const char *col = PQgetvalue(col_res, i, 1); + const char *typcat_s = PQgetvalue(col_res, i, 2); + char typcat = (typcat_s != NULL) ? typcat_s[0] : '\0'; const SensitivityRule *rule; ReportRow *row; bool is_sensitive; @@ -5183,6 +5460,17 @@ pgclone_masking_report(PG_FUNCTION_ARGS) snprintf(row->recommendation, sizeof(row->recommendation), "OK - masked via %s_masked view", tbl); } + else if (!pgclone_strategy_fits(rule->strategy, typcat)) + { + /* issue #17: the pattern matched by name but the + * suggested strategy cannot be stored in this column's + * type — flag it for manual review instead of + * recommending a mask that would break the clone. */ + strlcpy(row->mask_status, "UNMASKED", sizeof(row->mask_status)); + snprintf(row->recommendation, sizeof(row->recommendation), + "Review manually: strategy %s does not fit %s column", + rule->strategy, pgclone_typcat_desc(typcat)); + } else { strlcpy(row->mask_status, "UNMASKED", sizeof(row->mask_status)); @@ -5242,7 +5530,7 @@ PG_FUNCTION_INFO_V1(pgclone_version); Datum pgclone_version(PG_FUNCTION_ARGS) { - PG_RETURN_TEXT_P(cstring_to_text("pgclone 4.4.0")); + PG_RETURN_TEXT_P(cstring_to_text("pgclone 4.4.1")); } /* =============================================================== diff --git a/test/fixtures/seed.sql b/test/fixtures/seed.sql index 332af43..b95eaab 100644 --- a/test/fixtures/seed.sql +++ b/test/fixtures/seed.sql @@ -181,6 +181,25 @@ INSERT INTO test_schema.employees (full_name, email, phone, salary, ssn, notes) ('Diana Prince', 'diana@wonder.net', '+1-555-333-4444', 120000, '456-78-9012', 'Director'), ('Eve Wilson', 'eve@example.com', NULL, 55000, '567-89-0123', 'Intern'); +-- ---- Table for issue #17: type-aware masking ---- +-- income_verified is a BOOLEAN whose NAME matches the %income% financial +-- pattern (which recommends the numeric "random_int" strategy). Applying +-- random_int to a boolean used to abort the clone with +-- ERROR: invalid input syntax for type boolean: "60629" +-- monthly_income is a numeric column that random_int fits; contact_email +-- is a string column that the "email" strategy fits. +CREATE TABLE test_schema.flags ( + id SERIAL PRIMARY KEY, + income_verified BOOLEAN NOT NULL DEFAULT false, + monthly_income INTEGER NOT NULL, + contact_email VARCHAR(255) +); + +INSERT INTO test_schema.flags (income_verified, monthly_income, contact_email) VALUES + (true, 5000, 'a@example.com'), + (false, 6000, 'b@example.com'), + (true, 7000, 'c@example.com'); + -- ---- Roles and permissions for clone_roles tests ---- DO $$ BEGIN diff --git a/test/pgclone_test.sql b/test/pgclone_test.sql index 9908570..c3010d1 100644 --- a/test/pgclone_test.sql +++ b/test/pgclone_test.sql @@ -5,7 +5,7 @@ BEGIN; -SELECT plan(87); +SELECT plan(95); -- ============================================================ -- TEST GROUP 1: Extension loads correctly @@ -617,5 +617,69 @@ SELECT results_eq( ARRAY[2], 'keep_orders emails untouched — masks scoped per table'); +-- ============================================================ +-- TEST GROUP 26: issue #17 — type-aware masking +-- +-- A mask whose output type the target column cannot store — e.g. the +-- numeric "random_int" strategy on a BOOLEAN column named to match the +-- %income% pattern — used to abort the streaming COPY with +-- ERROR: invalid input syntax for type boolean: "60629" +-- The mask must now be skipped (leaving the column intact) while +-- compatible masks on other columns still apply. +-- ============================================================ +-- 8 tests below — keep in sync with plan() above. + +-- Clone flags applying random_int to the boolean income_verified column +-- (must be skipped) and email to the string contact_email column (must +-- apply). The clone as a whole must succeed. +SELECT lives_ok( + format('SELECT pgclone.table(%L, %L, %L, true, %L, %L)', + current_setting('app.source_conninfo'), + 'test_schema', 'flags', 'flags_typed', + '{"mask": {"income_verified": "random_int", "contact_email": "email"}}'), + 'clone with random_int mask on a boolean column succeeds (mask skipped)' +); + +SELECT has_table('test_schema', 'flags_typed', 'flags_typed table created'); + +-- Boolean values pass through unchanged (source has 2 true rows). +SELECT results_eq( + 'SELECT count(*)::integer FROM test_schema.flags_typed WHERE income_verified', + ARRAY[2], + 'boolean column left intact when its mask is type-incompatible'); + +-- The compatible email mask still applied — no original address survives. +SELECT results_eq( + $$SELECT count(*)::integer FROM test_schema.flags_typed + WHERE contact_email = 'a@example.com'$$, + ARRAY[0], + 'compatible email mask still applied to string column'); + +-- discover_sensitive must NOT suggest a mask for the boolean income column. +SELECT ok( + (SELECT pgclone.discover_sensitive( + current_setting('app.source_conninfo'), + 'test_schema')::text NOT LIKE '%income_verified%'), + 'discover skips boolean column income_verified'); + +-- ...but still suggests one for the numeric income column. +SELECT ok( + (SELECT pgclone.discover_sensitive( + current_setting('app.source_conninfo'), + 'test_schema')::text LIKE '%monthly_income%'), + 'discover still suggests a mask for numeric monthly_income'); + +-- mask_in_place must skip the incompatible boolean mask and still succeed. +SELECT lives_ok( + $$SELECT pgclone.mask_in_place( + 'test_schema', 'flags_typed', + '{"income_verified": "random_int", "monthly_income": "random_int"}')$$, + 'mask_in_place skips incompatible boolean mask and succeeds'); + +SELECT results_eq( + 'SELECT count(*)::integer FROM test_schema.flags_typed WHERE income_verified', + ARRAY[2], + 'boolean column unchanged after mask_in_place skips its mask'); + SELECT * FROM finish(); ROLLBACK;