From e6e3d6ebf193ed327457996a7cf3a65d32cf05e4 Mon Sep 17 00:00:00 2001 From: valehdba Date: Wed, 1 Jul 2026 19:15:49 +0000 Subject: [PATCH] fix: constraint- and length-aware data masking (issue #18) Builds on the type-aware masking from #17. A mask could still abort or corrupt a clone in ways type-checking alone did not catch; all are now handled by skipping the offending mask (with a WARNING) or clamping it, so the clone always succeeds: - length: a constant/partial/hash/random_int value longer than a varchar(N)/char(N) column failed with "value too long". Output for a length-limited string column is now clamped with left(..., N). - constant on a numeric column: the default 'REDACTED' (or any non-numeric text) failed to parse. A constant is applied to a non-string column only when the literal is a valid number, otherwise skipped. - null on a NOT NULL column: skipped. - UNIQUE/PRIMARY KEY column: only the injective "hash" strategy is applied; value-collapsing (name/constant/null) or collision-prone (random_int) strategies are skipped. - FOREIGN KEY column: masking is skipped to preserve referential integrity. discover_sensitive() and masking_report() now only suggest strategies the engine will apply: FK columns are omitted, and UNIQUE/PK or NOT NULL sensitive columns are steered to "hash". A masked view (create_masking_policy) enforces no constraints, so only the type/constant checks apply there. New helpers: ColMaskMeta + pgclone_column_maskmeta / pgclone_maskmeta_from_row (shared PGCLONE_MASKMETA_COLS catalog fragment), pgclone_mask_skip_reason (single decision point), pgclone_append_mask_expr_clamped, pgclone_looks_numeric, pgclone_discover_strategy. Tests: pgTAP group 27 (12 tests, plan 95 -> 107) plus mask18_parent/mask18_child fixtures covering length, constant-on-numeric, uniqueness, NOT NULL and FK. Docs: USAGE length/constraint safety section. Version bumped to 4.4.2 with sql/pgclone--4.4.1--4.4.2.sql (no SQL signature changes). --- CHANGELOG.md | 25 +++ META.json | 6 +- README.md | 3 +- docs/USAGE.md | 14 ++ pgclone.control | 2 +- sql/pgclone--4.4.1--4.4.2.sql | 39 ++++ sql/pgclone--4.4.2.sql | 201 ++++++++++++++++++ src/pgclone.c | 383 ++++++++++++++++++++++++---------- test/fixtures/seed.sql | 32 +++ test/pgclone_test.sql | 85 +++++++- 10 files changed, 679 insertions(+), 111 deletions(-) create mode 100644 sql/pgclone--4.4.1--4.4.2.sql create mode 100644 sql/pgclone--4.4.2.sql diff --git a/CHANGELOG.md b/CHANGELOG.md index e4121b7..063c56a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,31 @@ All notable changes to pgclone are documented in this file. +## [4.4.2] + +### Fixed +- **Constraint- and length-aware data masking** (issue #18). v4.4.1 made masking skip a strategy whose *type* a column could not store; issue #18 reported three further ways a mask could still abort or corrupt a clone. All are now handled — the offending mask is skipped with a `WARNING` (or clamped), and the clone succeeds: + - **Length overflow** — a `constant` (default `'REDACTED'`), `partial`, `hash`, or even `random_int` value could exceed a `varchar(N)`/`char(N)` column and fail with `value too long for type character varying(N)`. Mask output for a length-limited string column is now clamped with `left(…, N)` so it always fits. + - **`constant` on a numeric column** — the default `'REDACTED'` (or any non-numeric text) cannot be stored in an `integer`/`numeric` column (`invalid input syntax`). A `constant` mask on a non-string column is now applied only when the literal is a valid number for that column, otherwise skipped. + - **`null` on a NOT NULL column** — a `null` mask that would violate a `NOT NULL` constraint is skipped. + - **UNIQUE / PRIMARY KEY columns** — a value-collapsing (`name`, `constant`, `null`) or collision-prone (`random_int`) mask would break uniqueness (`duplicate key`). Only the injective `hash` strategy is applied to a UNIQUE/PK column; the rest are skipped. + - **FOREIGN KEY columns** — masking a FK column would break referential integrity, so it is skipped. +- **`pgclone.discover_sensitive()` and `pgclone.masking_report()` only suggest strategies the engine will actually apply.** Foreign-key columns are omitted; a UNIQUE/PK sensitive column, or a NOT NULL column whose default strategy is `null` (e.g. `ssn`), is steered to `hash` (which preserves distinctness and non-nullness) when the type permits. A generated mask file is therefore safe to apply verbatim. + +### Changed +- **`pgclone.mask_in_place()`** applies the same length clamp and constraint/type skips; when every rule is skipped it returns `OK: masked 0 rows … (0 columns — all mask rules were skipped as unsafe for their columns)`. +- **`pgclone.create_masking_policy()`** applies the type/`constant` checks (a masked view enforces no table constraints, so uniqueness/FK/NOT NULL skips do not apply there). + +### Added +- Internal C helpers in `src/pgclone.c`: `ColMaskMeta` plus `pgclone_column_maskmeta()` / `pgclone_maskmeta_from_row()` and the shared `PGCLONE_MASKMETA_COLS` catalog fragment (per-column typcategory, declared length, NOT NULL, UNIQUE/PK, FK); `pgclone_mask_skip_reason()` (single decision point for every skip rule); `pgclone_append_mask_expr_clamped()` (length clamp); `pgclone_discover_strategy()` (align suggestions with what the engine applies); and `pgclone_looks_numeric()`. +- **pgTAP test group 27** (12 tests, plan 95 → 107) plus `test_schema.mask18_parent` / `mask18_child` fixtures (a UNIQUE NOT NULL `email`, a `varchar(4)` `code`, an `integer` `salary`, a NOT NULL `ssn`, and a FK `parent_id`): verifies length clamping, constant-on-numeric skip, uniqueness preservation (`name` skipped, `hash` applied and distinct), NOT NULL and FK skips, and that `discover_sensitive` steers the unique email to `hash`. + +### Compatibility +- No SQL signature changes — the fix is entirely in the C masking engine. `sql/pgclone--4.4.1--4.4.2.sql` only bumps the installed version. +- Behavior change is limited to masks that previously **crashed** the clone or violated a constraint: those columns are now clamped (length) or passed through unmasked with a `WARNING`. Masks that already worked are unaffected. +- Synchronous paths only, as with the rest of the masking feature — `*_async` jobs ignore `"mask"`/`"masks"`. +- Pure C-string/libpq implementation; identical behavior on PG 14–18. + ## [4.4.1] ### Fixed diff --git a/META.json b/META.json index 5785acb..14de956 100644 --- a/META.json +++ b/META.json @@ -2,14 +2,14 @@ "name": "pgclone", "abstract": "Clone PostgreSQL databases, schemas, tables between staging, test, dev and prod environments", "description": "PostgreSQL extension for easily cloning your DB, Schemas, Tables and more between environments", - "version": "4.4.1", + "version": "4.4.2", "maintainer": "Valeh Agayev ", "license": "postgresql", "provides": { "pgclone": { "abstract": "Clone PostgreSQL databases, schemas, and tables across environments", - "file": "sql/pgclone--4.4.1.sql", - "version": "4.4.1" + "file": "sql/pgclone--4.4.2.sql", + "version": "4.4.2" } }, "prereqs": { diff --git a/README.md b/README.md index 20a7e01..2c4b119 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ [![CI](https://github.com/valehdba/pgclone/actions/workflows/ci.yml/badge.svg)](https://github.com/valehdba/pgclone/actions/workflows/ci.yml) [![Postgres 14–18](https://img.shields.io/badge/Postgres-14%E2%80%9318-336791?logo=postgresql&logoColor=white)](https://github.com/valehdba/pgclone) [![License](https://img.shields.io/badge/License-PostgreSQL-blue.svg)](https://github.com/valehdba/pgclone/blob/main/LICENSE) -[![Version](https://img.shields.io/badge/version-4.4.1-orange)](https://github.com/valehdba/pgclone/releases/tag/v4.4.1) +[![Version](https://img.shields.io/badge/version-4.4.2-orange)](https://github.com/valehdba/pgclone/releases/tag/v4.4.2) A PostgreSQL extension that clones databases, schemas, tables, and functions between PostgreSQL instances — directly from SQL. No `pg_dump`, no `pg_restore`, no shell scripts. @@ -147,6 +147,7 @@ pgclone uses Unix domain sockets for local loopback connections, so the default - [x] v4.3.0: Consistent-snapshot clones — every clone reads the source under one shared `REPEATABLE READ READ ONLY` snapshot (sync, async, and parallel pool); cross-table FK consistency on a live source is now guaranteed (same model as `pg_dump -j`) - [x] v4.4.0: Schema/database-level in-line masking (`"masks"`) and regex table subset filters (`"tables"` / `"exclude_tables"`) — discussion #16 - [x] v4.4.1: Type-aware masking — a mask that a column's type cannot store (e.g. `random_int` on a `boolean`) is skipped with a warning instead of aborting the clone; `discover_sensitive` no longer suggests incompatible masks — issue #17 +- [x] v4.4.2: Constraint- and length-aware masking — masks are clamped to a column's length and skipped when they would overflow, feed a bad literal to a numeric column, or violate a NOT NULL / UNIQUE / FOREIGN KEY constraint; UNIQUE/PK and NOT NULL sensitive columns are steered to `hash` — issue #18 - [ ] v4.5.0: FK-aware referential sampling — sample N rows and follow foreign keys to keep child rows consistent (`pgclone.table_sample`) ## License diff --git a/docs/USAGE.md b/docs/USAGE.md index 3b8b9c4..9fffa52 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -522,6 +522,20 @@ Each strategy emits a value of a fixed kind, and that value has to be storable i If a mask is **incompatible** with a column's type — e.g. `random_int` on a `boolean` flag, or `hash` on an `integer` — pgclone now **skips that mask and copies the column unchanged**, emitting a `WARNING` instead of aborting the clone with `invalid input syntax for type …` (issue #17). `discover_sensitive` (below) never suggests an incompatible mask in the first place, so a generated mask file is safe to apply verbatim. +### Length and constraint safety (v4.4.2) + +Masking never breaks a clone by overflowing a column or violating a constraint (issue #18): + +- **Length** — output for a `varchar(N)`/`char(N)` column is clamped with `left(…, N)` so a `constant` (`'REDACTED'`), `partial`, `hash`, or `random_int` value always fits instead of failing with `value too long for type character varying(N)`. +- **`constant` on a numeric column** — applied only when the literal is a valid number for that column; otherwise skipped (so the default `'REDACTED'` on an `integer` column no longer errors). +- **`NOT NULL`** — a `null` mask on a `NOT NULL` column is skipped. +- **`UNIQUE` / `PRIMARY KEY`** — only the injective `hash` strategy is applied (it keeps rows distinct); value-collapsing or collision-prone strategies (`name`, `constant`, `null`, `random_int`) are skipped. To mask a unique sensitive column (e.g. a unique `email`), use `hash`. +- **`FOREIGN KEY`** — masking a FK column is skipped to preserve referential integrity. + +Skipped masks emit a `WARNING` naming the column and reason and leave the column unmasked. `discover_sensitive` / `masking_report` account for all of this: they omit FK columns and steer UNIQUE/PK and NOT NULL sensitive columns to `hash`. + +> **Masking keys (PK/UQ/FK).** Integer primary/unique/foreign keys generally should not be masked — no built-in strategy both fits an integer and preserves the constraint, so they are left unmasked (integer PK/UQ) or skipped (FK). If you need referentially-consistent key masking, apply the deterministic `hash` to the same column across parent and child yourself (matching values hash identically), keeping the columns a text type. + ### Notes - Masking is applied on the **source** side inside `COPY (SELECT ...) TO STDOUT`, so masked data never enters the local database unmasked. diff --git a/pgclone.control b/pgclone.control index 7e07768..df6a0c1 100644 --- a/pgclone.control +++ b/pgclone.control @@ -1,6 +1,6 @@ # pgclone extension comment = 'Clone PostgreSQL databases, schemas, tables, roles and permissions with selective columns, data filtering, data masking/anonymization, async, parallel cloning, materialized views, resume, and conflict resolution' -default_version = '4.4.1' +default_version = '4.4.2' module_pathname = '$libdir/pgclone' relocatable = false superuser = true diff --git a/sql/pgclone--4.4.1--4.4.2.sql b/sql/pgclone--4.4.1--4.4.2.sql new file mode 100644 index 0000000..5dd124b --- /dev/null +++ b/sql/pgclone--4.4.1--4.4.2.sql @@ -0,0 +1,39 @@ +/* pgclone--4.4.1--4.4.2.sql */ +\echo Use "ALTER EXTENSION pgclone UPDATE" to load this file. \quit + +-- v4.4.2: Constraint- and length-aware data masking (GitHub issue #18). +-- +-- No SQL signature changes — the fix lives entirely in the C masking +-- engine (src/pgclone.c), so this upgrade script only bumps the +-- installed version. +-- +-- Background +-- ---------- +-- v4.4.1 (issue #17) made masking skip a strategy whose *type* a column +-- could not store. Issue #18 reported three further ways a mask could +-- still break a clone: +-- 1. length — a text/constant/partial value longer than a +-- varchar(N)/char(N) column -> "value too long". +-- 2. constant — the default 'REDACTED' (or any text) fed into a numeric +-- column -> "invalid input syntax". +-- 3. constraints — collapsing a UNIQUE/PRIMARY KEY column to one value, +-- nulling a NOT NULL column, or masking a FOREIGN KEY +-- column -> duplicate-key / not-null / FK violation. +-- +-- Fix +-- --- +-- * Length: text (and any) mask output for a length-limited string +-- column is clamped with left(..., N) so it always fits. +-- * Constant: a "constant" mask is skipped on a non-string column +-- unless the literal is a valid number for a numeric column. +-- * NOT NULL: a "null" mask on a NOT NULL column is skipped. +-- * UNIQUE/PK: only the injective "hash" strategy is applied; every +-- value-collapsing or collision-prone strategy is skipped. +-- * FOREIGN KEY: masking a FK column is skipped (referential integrity). +-- * discover_sensitive() / masking_report() only suggest strategies the +-- engine will actually apply: FK columns are omitted, and UNIQUE/PK or +-- NOT NULL sensitive columns are steered to "hash". +-- A masked view (create_masking_policy) enforces no constraints, so only +-- the type/constant checks apply there. + +-- (No catalog changes.) diff --git a/sql/pgclone--4.4.2.sql b/sql/pgclone--4.4.2.sql new file mode 100644 index 0000000..2f441c1 --- /dev/null +++ b/sql/pgclone--4.4.2.sql @@ -0,0 +1,201 @@ +/* pgclone--4.4.2.sql */ +\echo Use "CREATE EXTENSION pgclone" to load this file. \quit + +-- v4.4.2: Constraint- and length-aware data masking (GitHub issue #18). +-- No new SQL surface — the fix is entirely in the C masking engine. +-- A mask is now skipped (with a WARNING) when applying it would break +-- the clone: text/constant/partial values that overflow a varchar(N) +-- column are clamped to fit; a "constant" that is not valid for a +-- numeric column, a "null" on a NOT NULL column, a value-collapsing +-- mask on a UNIQUE/PRIMARY KEY column (only "hash" is kept), and any +-- mask on a FOREIGN KEY column are all skipped. discover_sensitive() +-- and masking_report() only suggest strategies the engine will apply +-- (UNIQUE/PK and NOT NULL sensitive columns are steered to "hash"). + +-- v4.4.1: Type-aware data masking (GitHub issue #17). +-- No new SQL surface — the fix is entirely in the C masking engine. +-- A mask strategy whose output value the target column cannot store +-- (e.g. the numeric "random_int" strategy on a BOOLEAN column matched +-- by the %income% pattern) is now skipped with a WARNING instead of +-- aborting the clone with "invalid input syntax for type boolean". +-- pgclone.discover_sensitive() likewise no longer suggests a mask that +-- is incompatible with a column's type. + +-- v4.4.0: Schema/database-level in-line masking and table subset filters +-- No new SQL surface — both features are new keys in the existing JSON +-- options argument of pgclone.schema() / pgclone.database() / +-- pgclone.database_create(): +-- '{"tables": ["order_.*"], "exclude_tables": [".*_old"], +-- "masks": {"users": {"email": "email", "ssn": "null"}}}' + +-- v4.3.0: Consistent-snapshot clones (REPEATABLE READ READ ONLY) +-- No new SQL surface — the behaviour is enabled by default for every +-- table/schema/database clone (sync and async, including parallel +-- pool mode) and can be disabled per-call with the options JSON: +-- SELECT pgclone.schema('...', 'public', true, '{"consistent": false}'); +-- See CHANGELOG.md / docs/USAGE.md for details and tradeoffs. + +-- v4.0.0: All functions now live under the 'pgclone' schema. +-- Usage: SELECT pgclone.table(...), pgclone.schema(...), etc. +CREATE SCHEMA IF NOT EXISTS pgclone; + +-- SYNCHRONOUS +CREATE FUNCTION pgclone.table(source_conninfo TEXT, schema_name TEXT, table_name TEXT, include_data BOOLEAN DEFAULT true) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_table' LANGUAGE C VOLATILE STRICT; +CREATE FUNCTION pgclone.table(source_conninfo TEXT, schema_name TEXT, table_name TEXT, include_data BOOLEAN, target_table_name TEXT) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_table' LANGUAGE C VOLATILE; +CREATE FUNCTION pgclone.table(source_conninfo TEXT, schema_name TEXT, table_name TEXT, include_data BOOLEAN, target_table_name TEXT, options TEXT) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_table' LANGUAGE C VOLATILE; +COMMENT ON FUNCTION pgclone.table(TEXT, TEXT, TEXT, BOOLEAN, TEXT, TEXT) IS 'Clone table with JSON options: {"columns":["col1","col2"], "where":"status=''active''", "indexes":false, "constraints":false, "triggers":false, "mask":{"email":"email","name":"name","phone":"phone","col":{"type":"partial","prefix":2,"suffix":3},"col2":"hash","col3":"null","col4":{"type":"random_int","min":0,"max":100},"col5":{"type":"constant","value":"REDACTED"}}}'; +CREATE FUNCTION pgclone.table_ex(source_conninfo TEXT, schema_name TEXT, table_name TEXT, include_data BOOLEAN, target_table_name TEXT, include_indexes BOOLEAN DEFAULT true, include_constraints BOOLEAN DEFAULT true, include_triggers BOOLEAN DEFAULT true) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_table_ex' LANGUAGE C VOLATILE; +CREATE FUNCTION pgclone.schema(source_conninfo TEXT, schema_name TEXT, include_data BOOLEAN DEFAULT true) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_schema' LANGUAGE C VOLATILE STRICT; +CREATE FUNCTION pgclone.schema(source_conninfo TEXT, schema_name TEXT, include_data BOOLEAN, options TEXT) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_schema' LANGUAGE C VOLATILE; +COMMENT ON FUNCTION pgclone.schema(TEXT, TEXT, BOOLEAN, TEXT) IS 'Clone schema with JSON options: {"indexes":false, "constraints":false, "triggers":false, "tables":["regex1","regex2"], "exclude_tables":["regex"], "masks":{"table_name":{"col":"email"},"schema.table":{"col":"hash"}}} — tables/exclude_tables are anchored POSIX regexes selecting a subset of tables; masks applies per-table in-line masking during the COPY stream (v4.4.0).'; +CREATE FUNCTION pgclone.schema_ex(source_conninfo TEXT, schema_name TEXT, include_data BOOLEAN, include_indexes BOOLEAN DEFAULT true, include_constraints BOOLEAN DEFAULT true, include_triggers BOOLEAN DEFAULT true) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_schema_ex' LANGUAGE C VOLATILE; +CREATE FUNCTION pgclone.functions(source_conninfo TEXT, schema_name TEXT) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_functions' LANGUAGE C VOLATILE STRICT; +CREATE FUNCTION pgclone.database(source_conninfo TEXT, include_data BOOLEAN DEFAULT true) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_database' LANGUAGE C VOLATILE STRICT; +CREATE FUNCTION pgclone.database(source_conninfo TEXT, include_data BOOLEAN, options TEXT) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_database' LANGUAGE C VOLATILE; +COMMENT ON FUNCTION pgclone.database(TEXT, BOOLEAN, TEXT) IS 'Clone database with JSON options — supports the same "tables", "exclude_tables", and "masks" options as pgclone.schema (v4.4.0); masks keys may be schema-qualified ("schema.table") to disambiguate.'; + +-- v2.0.1: Create target database and clone into it +CREATE FUNCTION pgclone.database_create(source_conninfo TEXT, target_dbname TEXT, include_data BOOLEAN DEFAULT true) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_database_create' LANGUAGE C VOLATILE; +CREATE FUNCTION pgclone.database_create(source_conninfo TEXT, target_dbname TEXT, include_data BOOLEAN, options TEXT) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_database_create' LANGUAGE C VOLATILE; +COMMENT ON FUNCTION pgclone.database_create(TEXT, TEXT, BOOLEAN) IS 'Create target database if not exists, then clone all schemas/tables/functions from source. Run from postgres DB.'; + +-- ASYNC (require shared_preload_libraries = 'pgclone') +CREATE FUNCTION pgclone.table_async(source_conninfo TEXT, schema_name TEXT, table_name TEXT, include_data BOOLEAN DEFAULT true, target_table_name TEXT DEFAULT NULL, options TEXT DEFAULT NULL) RETURNS INTEGER AS 'MODULE_PATHNAME', 'pgclone_table_async' LANGUAGE C VOLATILE; +CREATE FUNCTION pgclone.schema_async(source_conninfo TEXT, schema_name TEXT, include_data BOOLEAN DEFAULT true, options TEXT DEFAULT NULL) RETURNS INTEGER AS 'MODULE_PATHNAME', 'pgclone_schema_async' LANGUAGE C VOLATILE; + +-- PROGRESS & JOB MANAGEMENT +CREATE FUNCTION pgclone.progress(job_id INTEGER) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_progress' LANGUAGE C VOLATILE STRICT; +CREATE FUNCTION pgclone.cancel(job_id INTEGER) RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_cancel' LANGUAGE C VOLATILE STRICT; +CREATE FUNCTION pgclone.resume(job_id INTEGER) RETURNS INTEGER AS 'MODULE_PATHNAME', 'pgclone_resume' LANGUAGE C VOLATILE STRICT; +CREATE FUNCTION pgclone.jobs() RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_jobs' LANGUAGE C VOLATILE STRICT; +CREATE FUNCTION pgclone.clear_jobs() RETURNS INTEGER AS 'MODULE_PATHNAME', 'pgclone_clear_jobs' LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.clear_jobs() IS 'Clear completed/failed/cancelled job slots from shared memory'; + +-- v2.1.0+v2.1.1+v2.1.2: Progress Tracking View with progress bar, elapsed time, ETA +CREATE FUNCTION pgclone.progress_detail() +RETURNS TABLE ( + job_id INTEGER, + status TEXT, + op_type TEXT, + schema_name TEXT, + table_name TEXT, + current_phase TEXT, + current_table TEXT, + tables_total BIGINT, + tables_completed BIGINT, + rows_copied BIGINT, + bytes_copied BIGINT, + elapsed_ms BIGINT, + start_time TIMESTAMPTZ, + end_time TIMESTAMPTZ, + error_message TEXT, + pct_complete DOUBLE PRECISION, + progress_bar TEXT, + elapsed_time TEXT +) AS 'MODULE_PATHNAME', 'pgclone_progress_view' +LANGUAGE C VOLATILE STRICT; + +COMMENT ON FUNCTION pgclone.progress_detail() IS 'Returns tabular progress with visual progress bar and elapsed time for all clone jobs'; + +-- VIEW: convenient wrapper +CREATE VIEW pgclone.jobs_view AS + SELECT * FROM pgclone.progress_detail(); + +COMMENT ON VIEW pgclone.jobs_view IS 'Live progress tracking view with progress bar and elapsed time for all pgclone async clone jobs'; + +-- v3.1.0: Auto-discovery of sensitive data +CREATE FUNCTION pgclone.discover_sensitive(source_conninfo TEXT, schema_name TEXT) +RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_discover_sensitive' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.discover_sensitive(TEXT, TEXT) IS 'Scan source schema for columns matching sensitive data patterns (email, name, phone, ssn, salary, etc.) and return suggested mask rules as JSON'; + +-- v3.2.0: Static data masking on local tables +CREATE FUNCTION pgclone.mask_in_place(schema_name TEXT, table_name TEXT, mask_json TEXT) +RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_mask_in_place' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.mask_in_place(TEXT, TEXT, TEXT) IS 'Apply data masking to an existing local table via UPDATE. mask_json uses same format as clone mask option: {"email": "email", "name": "name", "ssn": "null"}'; + +-- v3.3.0: Dynamic data masking via views and role-based access +CREATE FUNCTION pgclone.create_masking_policy(schema_name TEXT, table_name TEXT, mask_json TEXT, privileged_role TEXT) +RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_create_masking_policy' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.create_masking_policy(TEXT, TEXT, TEXT, TEXT) IS 'Create a dynamic masking policy: creates a masked view, revokes base table access from PUBLIC, grants view to PUBLIC, grants base table to privileged role'; + +CREATE FUNCTION pgclone.drop_masking_policy(schema_name TEXT, table_name TEXT) +RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_drop_masking_policy' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.drop_masking_policy(TEXT, TEXT) IS 'Remove a dynamic masking policy: drops the masked view and restores base table access to PUBLIC'; + +-- v3.4.0: Clone roles with permissions and passwords +CREATE FUNCTION pgclone.clone_roles(source_conninfo TEXT) +RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_clone_roles' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.clone_roles(TEXT) IS 'Clone all non-system roles from source with encrypted passwords, attributes, memberships, and all permissions. Requires superuser on both source and target.'; + +CREATE FUNCTION pgclone.clone_roles(source_conninfo TEXT, role_names TEXT) +RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_clone_roles' +LANGUAGE C VOLATILE; +COMMENT ON FUNCTION pgclone.clone_roles(TEXT, TEXT) IS 'Clone specific roles (comma-separated) from source with encrypted passwords, attributes, memberships, and permissions. If role exists on target, syncs password and attributes without dropping.'; + +-- v3.5.0: Clone verification — compare row counts +CREATE FUNCTION pgclone.verify(source_conninfo TEXT, schema_name TEXT) +RETURNS TABLE ( + schema_name TEXT, + table_name TEXT, + source_rows BIGINT, + target_rows BIGINT, + match TEXT +) AS 'MODULE_PATHNAME', 'pgclone_verify' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.verify(TEXT, TEXT) IS 'Compare row counts between source and local target for all tables in a schema. Returns side-by-side comparison with match status.'; + +CREATE FUNCTION pgclone.verify(source_conninfo TEXT) +RETURNS TABLE ( + schema_name TEXT, + table_name TEXT, + source_rows BIGINT, + target_rows BIGINT, + match TEXT +) AS 'MODULE_PATHNAME', 'pgclone_verify' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.verify(TEXT) IS 'Compare row counts between source and local target for all user tables across all schemas. Returns side-by-side comparison with match status.'; + +-- v3.6.0: GDPR/Compliance masking report +CREATE FUNCTION pgclone.masking_report(schema_name TEXT) +RETURNS TABLE ( + schema_name TEXT, + table_name TEXT, + column_name TEXT, + sensitivity TEXT, + mask_status TEXT, + recommendation TEXT +) AS 'MODULE_PATHNAME', 'pgclone_masking_report' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.masking_report(TEXT) IS 'Generate GDPR/compliance audit report: lists sensitive columns, their masking status, and recommendations. Checks for masked views.'; + +-- v4.1.0: Schema diff — DDL drift detection between source and local target +CREATE FUNCTION pgclone.diff(source_conninfo TEXT, schema_name TEXT) +RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_diff' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.diff(TEXT, TEXT) IS + 'Compare DDL of a schema between source and the local target. ' + 'Returns JSON drift report listing objects only_in_source / only_in_target / modified ' + 'across tables (with per-column type/nullability/default drift), indexes, ' + 'constraints, triggers, views, and sequences. Read-only on both sides.'; + +-- v4.2.0: Pre-flight validator — connection, permissions, version, capacity, +-- name-conflict, role and tablespace checks before a clone. +CREATE FUNCTION pgclone.preflight(source_conninfo TEXT, schema_name TEXT) +RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_preflight' +LANGUAGE C VOLATILE STRICT; +COMMENT ON FUNCTION pgclone.preflight(TEXT, TEXT) IS + 'Validate that a clone of the given schema from source into the local ' + 'target is likely to succeed. Returns JSON with errors / warnings / info ' + 'arrays plus a per-check breakdown covering: source/target connection, ' + 'PostgreSQL versions, schema existence, USAGE/SELECT/CREATE permissions, ' + 'estimated source size, target database size, object counts, name ' + 'conflicts on the target schema, missing extensions, missing roles, and ' + 'missing non-default tablespaces. Read-only on both sides.'; + +-- VERSION +CREATE FUNCTION pgclone.version() RETURNS TEXT AS 'MODULE_PATHNAME', 'pgclone_version' LANGUAGE C IMMUTABLE STRICT; diff --git a/src/pgclone.c b/src/pgclone.c index 1ce77cd..2de20e2 100644 --- a/src/pgclone.c +++ b/src/pgclone.c @@ -1024,22 +1024,104 @@ pgclone_typcat_desc(char typcat) } } +/* --------------------------------------------------------------- + * v4.4.2 (issue #18): constraint- and length-aware masking. + * + * Type compatibility alone (issue #17) is not enough — a mask can still + * break a clone in three further ways, all reported in issue #18: + * 1. length: a text/constant/partial mask can produce a value longer + * than a varchar(N)/char(N) column -> "value too long". + * 2. constant: the default 'REDACTED' (or any text) does not parse + * into a numeric column -> "invalid input syntax". + * 3. constraints: collapsing a UNIQUE/PK column to one value, nulling + * a NOT NULL column, or masking a FOREIGN KEY column breaks the + * constraint -> duplicate/null/FK error. + * + * ColMaskMeta carries the per-column facts needed to decide, and the + * helpers below either skip the mask (with a reason) or apply it with a + * length clamp so the value always fits. + * --------------------------------------------------------------- */ +typedef struct ColMaskMeta +{ + char typcat; /* pg_type.typcategory, or '\0' if unknown */ + int char_maxlen; /* declared varchar/char length, 0 = unlimited */ + bool notnull; /* column is NOT NULL */ + bool is_unique; /* participates in a PK / UNIQUE index */ + bool is_fk; /* participates in a FOREIGN KEY constraint */ +} ColMaskMeta; + +/* True when the whole string parses as a number (constant-on-numeric). */ +static bool +pgclone_looks_numeric(const char *s) +{ + char *end; + + if (s == NULL || *s == '\0') + return false; + errno = 0; + (void) strtod(s, &end); + if (errno != 0 || end == s) + return false; + while (*end == ' ' || *end == '\t') + end++; + return *end == '\0'; +} + +/* + * The SQL that computes ColMaskMeta's columns for a table, in a fixed + * order (typcategory, char_maxlen, notnull, is_unique, is_fk). Shared by + * the single-column lookup and the bulk per-table queries so the column + * order stays in sync. `a`, `c`, `t` must be bound to pg_attribute, + * pg_class and pg_type in the surrounding query. + */ +#define PGCLONE_MASKMETA_COLS \ + "t.typcategory, " \ + "CASE WHEN a.atttypmod > 4 AND t.typcategory = 'S' " \ + " THEN a.atttypmod - 4 ELSE 0 END, " \ + "a.attnotnull, " \ + "EXISTS (SELECT 1 FROM pg_catalog.pg_index i " \ + " WHERE i.indrelid = c.oid AND i.indisunique " \ + " AND a.attnum = ANY(i.indkey)), " \ + "EXISTS (SELECT 1 FROM pg_catalog.pg_constraint con " \ + " WHERE con.conrelid = c.oid AND con.contype = 'f' " \ + " AND a.attnum = ANY(con.conkey))" + +/* Fill a ColMaskMeta from a result row starting at column `base`. */ +static ColMaskMeta +pgclone_maskmeta_from_row(PGresult *r, int row, int base) +{ + ColMaskMeta m; + char *v; + + memset(&m, 0, sizeof(m)); + v = PQgetvalue(r, row, base + 0); + if (v != NULL && v[0] != '\0') + m.typcat = v[0]; + m.char_maxlen = atoi(PQgetvalue(r, row, base + 1)); + m.notnull = (PQgetvalue(r, row, base + 2)[0] == 't'); + m.is_unique = (PQgetvalue(r, row, base + 3)[0] == 't'); + m.is_fk = (PQgetvalue(r, row, base + 4)[0] == 't'); + return m; +} + /* - * Look up pg_type.typcategory for schema.table.column over `conn`. - * Returns the single category char, or '\0' when it cannot be determined - * (callers then treat the mask as compatible and let the server decide). + * Look up masking metadata for one schema.table.column over `conn`. + * On any error an all-zero struct is returned (typcat '\0'), which the + * decision logic treats permissively — the server then enforces types. */ -static char -pgclone_column_typcategory(PGconn *conn, const char *schema_name, - const char *table_name, const char *col_name) +static ColMaskMeta +pgclone_column_maskmeta(PGconn *conn, const char *schema_name, + const char *table_name, const char *col_name) { + ColMaskMeta meta; StringInfoData q; PGresult *r; - char cat = '\0'; + + memset(&meta, 0, sizeof(meta)); initStringInfo(&q); appendStringInfo(&q, - "SELECT t.typcategory " + "SELECT " PGCLONE_MASKMETA_COLS " " "FROM pg_catalog.pg_attribute a " "JOIN pg_catalog.pg_class c ON c.oid = a.attrelid " "JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace " @@ -1054,13 +1136,96 @@ pgclone_column_typcategory(PGconn *conn, const char *schema_name, pfree(q.data); if (PQresultStatus(r) == PGRES_TUPLES_OK && PQntuples(r) > 0) + meta = pgclone_maskmeta_from_row(r, 0, 0); + PQclear(r); + return meta; +} + +/* + * Decide whether a mask may be applied to a column. Returns NULL to apply + * it, or a short human-readable reason to skip it (leaving the column + * unmasked). Consolidates every safety rule so all call sites behave the + * same: + * - issue #17: output type incompatible with the column type + * - issue #18: a constant literal that is not valid for a non-string type + * - issue #18: a null mask on a NOT NULL column + * - issue #18: masking a foreign-key column (breaks referential integrity) + * - issue #18: masking a UNIQUE/PK column with a value-collapsing or + * collision-prone strategy — only the injective `hash` keeps rows distinct + */ +static const char * +pgclone_mask_skip_reason(const MaskRule *rule, const ColMaskMeta *m) +{ + PgcloneMaskOutKind kind = pgclone_mask_out_kind(rule->type); + + if (!pgclone_mask_kind_fits(kind, m->typcat)) + return "output type is incompatible with the column type"; + + if (rule->type == PGCLONE_MASK_CONSTANT && + m->typcat != 'S' && m->typcat != '\0') { - char *v = PQgetvalue(r, 0, 0); - if (v != NULL && v[0] != '\0') - cat = v[0]; + if (!(m->typcat == 'N' && pgclone_looks_numeric(rule->constant_val))) + return "constant value is not valid for the column type"; } - PQclear(r); - return cat; + + if (rule->type == PGCLONE_MASK_NULL && m->notnull) + return "would put NULL in a NOT NULL column"; + + if (m->is_fk) + return "column is a foreign key (masking would break referential integrity)"; + + if (m->is_unique && rule->type != PGCLONE_MASK_HASH) + return "column is UNIQUE/PRIMARY KEY (only the \"hash\" strategy preserves uniqueness)"; + + return NULL; +} + +/* + * Adjust a discover_sensitive strategy suggestion so the masking engine + * will actually apply it to a column with metadata `m` (issue #18): + * - foreign-key columns are never suggested (masking breaks FK); + * - a UNIQUE/PK column, or a NOT NULL column whose base strategy is + * "null", falls back to the injective "hash" (the one strategy the + * engine keeps on such columns) when the type permits, else nothing; + * - a suggestion whose output type does not fit the column is dropped. + * Returns the strategy string to emit, or NULL to omit the column. + */ +static const char * +pgclone_discover_strategy(const char *base, const ColMaskMeta *m) +{ + const char *strat = base; + + if (m->is_fk) + return NULL; + + if ((m->is_unique && strcmp(strat, "hash") != 0) || + (strcmp(strat, "null") == 0 && m->notnull)) + strat = "hash"; + + if (!pgclone_strategy_fits(strat, m->typcat)) + return NULL; + + return strat; +} + +/* + * Append a mask expression for a column, clamped so it always fits a + * length-limited string column (issue #18). char_maxlen == 0 means the + * column has no character-length limit, so the expression is emitted as-is. + * The ::text cast lets non-text outputs (e.g. random_int) be clamped too. + */ +static void +pgclone_append_mask_expr_clamped(StringInfo out, const char *col_ident, + const MaskRule *rule, int char_maxlen) +{ + if (char_maxlen > 0) + { + appendStringInfoString(out, "left(("); + pgclone_build_mask_expr(out, col_ident, rule); + appendStringInfo(out, ")::text, %d)", char_maxlen); + } + else + pgclone_build_mask_expr(out, col_ident, rule); } /* --------------------------------------------------------------- @@ -1834,7 +1999,7 @@ pgclone_copy_data(PGconn *source_conn, PGconn *local_conn, initStringInfo(&col_query); appendStringInfo(&col_query, - "SELECT a.attname, t.typcategory " + "SELECT a.attname, " PGCLONE_MASKMETA_COLS " " "FROM pg_catalog.pg_attribute a " "JOIN pg_catalog.pg_class c ON c.oid = a.attrelid " "JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace " @@ -1862,33 +2027,35 @@ pgclone_copy_data(PGconn *source_conn, PGconn *local_conn, for (ci = 0; ci < ncols; ci++) { const char *col_name = PQgetvalue(col_res, ci, 0); - const char *typcat_s = PQgetvalue(col_res, ci, 1); - char typcat = (typcat_s != NULL) ? typcat_s[0] : '\0'; const char *col_ident = quote_identifier(col_name); const MaskRule *rule = pgclone_find_mask_rule(opts, col_name); + ColMaskMeta meta = pgclone_maskmeta_from_row(col_res, ci, 1); if (ci > 0) appendStringInfoString(&select_cmd, ", "); - /* issue #17: skip a mask whose output type the column - * cannot store (e.g. random_int on a boolean); emit the - * column unchanged rather than corrupt the COPY stream. */ - if (rule != NULL && - !pgclone_mask_kind_fits(pgclone_mask_out_kind(rule->type), - typcat)) + /* issues #17/#18: skip a mask the column cannot safely + * store (wrong type, would overflow a constraint, ...); + * emit the column unchanged rather than break the clone. */ + if (rule != NULL) { - ereport(WARNING, - (errmsg("pgclone: mask \"%s\" is not compatible with %s column \"%s\"; leaving it unmasked", - pgclone_masktype_name(rule->type), - pgclone_typcat_desc(typcat), - col_name))); - rule = NULL; + const char *why = pgclone_mask_skip_reason(rule, &meta); + if (why != NULL) + { + ereport(WARNING, + (errmsg("pgclone: skipping mask \"%s\" on column \"%s\": %s; leaving it unmasked", + pgclone_masktype_name(rule->type), + col_name, why))); + rule = NULL; + } } if (rule != NULL) { - pgclone_build_mask_expr(&select_cmd, col_ident, rule); - /* Alias to original column name so COPY IN matches */ + /* Length-clamp so the value fits (issue #18), then + * alias to the original name so COPY IN matches. */ + pgclone_append_mask_expr_clamped(&select_cmd, col_ident, + rule, meta.char_maxlen); appendStringInfo(&select_cmd, " AS %s", col_ident); } else @@ -1910,35 +2077,33 @@ pgclone_copy_data(PGconn *source_conn, PGconn *local_conn, if (ci > 0) appendStringInfoString(&select_cmd, ", "); - /* issue #17: skip a mask that the column's type cannot + /* issues #17/#18: skip a mask the column cannot safely * store (looked up per masked column on the source). */ if (rule != NULL) { - char typcat = pgclone_column_typcategory(source_conn, - schema_name, - source_table, - opts->columns[ci]); - if (!pgclone_mask_kind_fits(pgclone_mask_out_kind(rule->type), - typcat)) + ColMaskMeta meta = pgclone_column_maskmeta(source_conn, + schema_name, + source_table, + opts->columns[ci]); + const char *why = pgclone_mask_skip_reason(rule, &meta); + if (why != NULL) { ereport(WARNING, - (errmsg("pgclone: mask \"%s\" is not compatible with %s column \"%s\"; leaving it unmasked", + (errmsg("pgclone: skipping mask \"%s\" on column \"%s\": %s; leaving it unmasked", pgclone_masktype_name(rule->type), - pgclone_typcat_desc(typcat), - opts->columns[ci]))); + opts->columns[ci], why))); rule = NULL; } + else + { + pgclone_append_mask_expr_clamped(&select_cmd, col_ident, + rule, meta.char_maxlen); + appendStringInfo(&select_cmd, " AS %s", col_ident); + continue; + } } - if (rule != NULL) - { - pgclone_build_mask_expr(&select_cmd, col_ident, rule); - appendStringInfo(&select_cmd, " AS %s", col_ident); - } - else - { - appendStringInfoString(&select_cmd, col_ident); - } + appendStringInfoString(&select_cmd, col_ident); } } else @@ -4131,7 +4296,7 @@ pgclone_discover_sensitive(PG_FUNCTION_ARGS) appendStringInfo(&query, "SELECT c.relname AS table_name, " " a.attname AS column_name, " - " t.typcategory AS typcategory " + " " PGCLONE_MASKMETA_COLS " " "FROM pg_catalog.pg_class c " "JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace " "JOIN pg_catalog.pg_attribute a ON a.attrelid = c.oid " @@ -4168,20 +4333,21 @@ pgclone_discover_sensitive(PG_FUNCTION_ARGS) { const char *tbl = PQgetvalue(res, i, 0); const char *col = PQgetvalue(res, i, 1); - const char *typcat_s = PQgetvalue(res, i, 2); - char typcat = (typcat_s != NULL) ? typcat_s[0] : '\0'; + ColMaskMeta meta = pgclone_maskmeta_from_row(res, i, 2); const SensitivityRule *rule; + const char *strategy; rule = pgclone_match_sensitivity(col); if (rule == NULL) continue; - /* issue #17: don't suggest a mask the column type cannot - * store (e.g. random_int for a boolean "..._income_..." - * flag) — such a suggestion, applied verbatim, would abort - * the clone with "invalid input syntax for type ...". */ - if (!pgclone_strategy_fits(rule->strategy, typcat)) + /* issues #17/#18: only suggest a strategy the masking engine + * will actually apply — one that fits the column type and does + * not break a NOT NULL / UNIQUE / FK constraint. Unique/PK and + * NOT NULL sensitive columns are steered to "hash". */ + strategy = pgclone_discover_strategy(rule->strategy, &meta); + if (strategy == NULL) continue; /* New table? Close previous table's object and open new one. */ @@ -4204,7 +4370,7 @@ pgclone_discover_sensitive(PG_FUNCTION_ARGS) appendStringInfoString(&result, ", "); appendStringInfo(&result, "\"%s\": \"%s\"", - col, rule->strategy); + col, strategy); first_col_in_table = false; matched_in_table++; } @@ -4282,36 +4448,34 @@ pgclone_mask_in_place(PG_FUNCTION_ARGS) { const MaskRule *rule = &opts.masks[i]; const char *col_ident = quote_identifier(rule->column); - StringInfoData expr; - char typcat; - - /* issue #17: skip a mask whose output type the column cannot - * store — otherwise the UPDATE fails with a type-mismatch error. */ - typcat = pgclone_column_typcategory(local_conn, schema_name, - table_name, rule->column); - if (!pgclone_mask_kind_fits(pgclone_mask_out_kind(rule->type), typcat)) + ColMaskMeta meta; + const char *why; + + /* issues #17/#18: skip a mask the column cannot safely store — + * otherwise the UPDATE fails (type mismatch, NOT NULL, unique, ...). */ + meta = pgclone_column_maskmeta(local_conn, schema_name, + table_name, rule->column); + why = pgclone_mask_skip_reason(rule, &meta); + if (why != NULL) { ereport(WARNING, - (errmsg("pgclone: mask \"%s\" is not compatible with %s column \"%s\"; leaving it unmasked", + (errmsg("pgclone: skipping mask \"%s\" on column \"%s\": %s; leaving it unmasked", pgclone_masktype_name(rule->type), - pgclone_typcat_desc(typcat), - rule->column))); + rule->column, why))); continue; } if (!first_set) appendStringInfoString(&update_cmd, ", "); - initStringInfo(&expr); - pgclone_build_mask_expr(&expr, col_ident, rule); - - appendStringInfo(&update_cmd, "%s = %s", col_ident, expr.data); - pfree(expr.data); + appendStringInfo(&update_cmd, "%s = ", col_ident); + pgclone_append_mask_expr_clamped(&update_cmd, col_ident, rule, + meta.char_maxlen); first_set = false; applied++; } - /* All masks skipped as type-incompatible — nothing to do. */ + /* Every mask skipped as unsafe — nothing to do. */ if (applied == 0) { PQfinish(local_conn); @@ -4319,7 +4483,7 @@ pgclone_mask_in_place(PG_FUNCTION_ARGS) initStringInfo(&result); appendStringInfo(&result, - "OK: masked 0 rows in %s.%s (0 columns — all mask rules were incompatible with their column types)", + "OK: masked 0 rows in %s.%s (0 columns — all mask rules were skipped as unsafe for their columns)", schema_name, table_name); PG_RETURN_TEXT_P(cstring_to_text(result.data)); } @@ -4459,25 +4623,33 @@ pgclone_create_masking_policy(PG_FUNCTION_ARGS) { const char *col_name = PQgetvalue(col_res, ci, 0); const char *typcat_s = PQgetvalue(col_res, ci, 1); - char typcat = (typcat_s != NULL) ? typcat_s[0] : '\0'; const char *col_ident = quote_identifier(col_name); const MaskRule *rule = pgclone_find_mask_rule(&opts, col_name); + /* A masked view enforces no constraints, so only the type/constant + * checks apply here (leave notnull/unique/fk cleared). */ + ColMaskMeta meta; + + memset(&meta, 0, sizeof(meta)); + if (typcat_s != NULL) + meta.typcat = typcat_s[0]; if (ci > 0) appendStringInfoString(&view_cmd, ", "); - /* issue #17: skip a mask whose output type the column cannot - * store — otherwise the masked view would silently change the - * column's type (e.g. boolean -> integer for random_int). */ - if (rule != NULL && - !pgclone_mask_kind_fits(pgclone_mask_out_kind(rule->type), typcat)) + /* issues #17/#18: skip a mask that would change the column's type + * in the view (e.g. random_int on boolean, or constant 'REDACTED' + * on an integer column). */ + if (rule != NULL) { - ereport(WARNING, - (errmsg("pgclone: mask \"%s\" is not compatible with %s column \"%s\"; leaving it unmasked", - pgclone_masktype_name(rule->type), - pgclone_typcat_desc(typcat), - col_name))); - rule = NULL; + const char *why = pgclone_mask_skip_reason(rule, &meta); + if (why != NULL) + { + ereport(WARNING, + (errmsg("pgclone: skipping mask \"%s\" on column \"%s\": %s; leaving it unmasked", + pgclone_masktype_name(rule->type), + col_name, why))); + rule = NULL; + } } if (rule != NULL) @@ -5368,7 +5540,7 @@ pgclone_masking_report(PG_FUNCTION_ARGS) /* Get all columns in the schema */ initStringInfo(&query); appendStringInfo(&query, - "SELECT c.relname, a.attname, t.typcategory " + "SELECT c.relname, a.attname, " PGCLONE_MASKMETA_COLS " " "FROM pg_catalog.pg_class c " "JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace " "JOIN pg_catalog.pg_attribute a ON a.attrelid = c.oid " @@ -5416,8 +5588,7 @@ pgclone_masking_report(PG_FUNCTION_ARGS) { const char *tbl = PQgetvalue(col_res, i, 0); const char *col = PQgetvalue(col_res, i, 1); - const char *typcat_s = PQgetvalue(col_res, i, 2); - char typcat = (typcat_s != NULL) ? typcat_s[0] : '\0'; + ColMaskMeta meta = pgclone_maskmeta_from_row(col_res, i, 2); const SensitivityRule *rule; ReportRow *row; bool is_sensitive; @@ -5460,22 +5631,24 @@ pgclone_masking_report(PG_FUNCTION_ARGS) snprintf(row->recommendation, sizeof(row->recommendation), "OK - masked via %s_masked view", tbl); } - else if (!pgclone_strategy_fits(rule->strategy, typcat)) - { - /* issue #17: the pattern matched by name but the - * suggested strategy cannot be stored in this column's - * type — flag it for manual review instead of - * recommending a mask that would break the clone. */ - strlcpy(row->mask_status, "UNMASKED", sizeof(row->mask_status)); - snprintf(row->recommendation, sizeof(row->recommendation), - "Review manually: strategy %s does not fit %s column", - rule->strategy, pgclone_typcat_desc(typcat)); - } else { + /* issues #17/#18: recommend the strategy the engine would + * actually apply (steered to "hash" for UNIQUE/PK and + * NOT NULL columns); if none is safe, flag for review. */ + const char *rec = pgclone_discover_strategy(rule->strategy, &meta); + strlcpy(row->mask_status, "UNMASKED", sizeof(row->mask_status)); - snprintf(row->recommendation, sizeof(row->recommendation), - "Apply mask strategy: %s", rule->strategy); + if (rec != NULL) + snprintf(row->recommendation, sizeof(row->recommendation), + "Apply mask strategy: %s", rec); + else if (meta.is_fk) + snprintf(row->recommendation, sizeof(row->recommendation), + "Review manually: foreign-key column — masking breaks referential integrity"); + else + snprintf(row->recommendation, sizeof(row->recommendation), + "Review manually: no strategy fits a %s column here", + pgclone_typcat_desc(meta.typcat)); } state->num_rows++; @@ -5530,7 +5703,7 @@ PG_FUNCTION_INFO_V1(pgclone_version); Datum pgclone_version(PG_FUNCTION_ARGS) { - PG_RETURN_TEXT_P(cstring_to_text("pgclone 4.4.1")); + PG_RETURN_TEXT_P(cstring_to_text("pgclone 4.4.2")); } /* =============================================================== diff --git a/test/fixtures/seed.sql b/test/fixtures/seed.sql index b95eaab..1991bc6 100644 --- a/test/fixtures/seed.sql +++ b/test/fixtures/seed.sql @@ -200,6 +200,38 @@ INSERT INTO test_schema.flags (income_verified, monthly_income, contact_email) V (false, 6000, 'b@example.com'), (true, 7000, 'c@example.com'); +-- ---- Tables for issue #18: constraint- and length-aware masking ---- +-- mask18_parent exercises every #18 hazard: +-- email UNIQUE NOT NULL -> a collapsing mask (name) breaks uniqueness; +-- only the injective "hash" is allowed. +-- code VARCHAR(4) -> constant/partial masks would overflow the +-- declared length ("value too long"). +-- salary INTEGER -> a text "constant" (REDACTED) cannot be stored. +-- ssn NOT NULL -> a "null" mask would violate NOT NULL. +-- mask18_child.parent_id is a FOREIGN KEY -> masking it breaks referential +-- integrity, so it must be left unmasked. +CREATE TABLE test_schema.mask18_parent ( + id SERIAL PRIMARY KEY, + email VARCHAR(255) UNIQUE NOT NULL, + code VARCHAR(4) NOT NULL, + salary INTEGER NOT NULL, + ssn VARCHAR(11) NOT NULL +); + +INSERT INTO test_schema.mask18_parent (email, code, salary, ssn) VALUES + ('alice@example.com', 'AB', 5000, '111-11-1111'), + ('bob@example.com', 'CD', 6000, '222-22-2222'), + ('carol@example.com', 'EF', 7000, '333-33-3333'); + +CREATE TABLE test_schema.mask18_child ( + id SERIAL PRIMARY KEY, + parent_id INTEGER NOT NULL REFERENCES test_schema.mask18_parent(id), + note TEXT +); + +INSERT INTO test_schema.mask18_child (parent_id, note) VALUES + (1, 'x'), (2, 'y'), (3, 'z'); + -- ---- Roles and permissions for clone_roles tests ---- DO $$ BEGIN diff --git a/test/pgclone_test.sql b/test/pgclone_test.sql index c3010d1..c907f0b 100644 --- a/test/pgclone_test.sql +++ b/test/pgclone_test.sql @@ -5,7 +5,7 @@ BEGIN; -SELECT plan(95); +SELECT plan(107); -- ============================================================ -- TEST GROUP 1: Extension loads correctly @@ -681,5 +681,88 @@ SELECT results_eq( ARRAY[2], 'boolean column unchanged after mask_in_place skips its mask'); +-- ============================================================ +-- TEST GROUP 27: issue #18 — constraint- and length-aware masking +-- +-- A mask must not break the clone by overflowing a column's length, +-- feeding a bad literal to a numeric column, or violating a +-- NOT NULL / UNIQUE / FOREIGN KEY constraint. Such masks are skipped +-- (with a WARNING) and the column is left unmasked; length-limited +-- string columns are clamped so the value always fits. +-- ============================================================ +-- 12 tests below — keep in sync with plan() above. + +-- Length + numeric-constant: constant on varchar(4) is clamped, constant +-- "REDACTED" on an integer column is skipped — the clone still succeeds. +SELECT lives_ok( + format('SELECT pgclone.table(%L, %L, %L, true, %L, %L)', + current_setting('app.source_conninfo'), + 'test_schema', 'mask18_parent', 'm18_const', + '{"mask": {"code": {"type":"constant","value":"REDACTED"}, "salary": {"type":"constant","value":"REDACTED"}}}'), + 'constant on varchar(4) + constant on integer: clone succeeds'); + +SELECT has_table('test_schema', 'm18_const', 'm18_const table created'); + +SELECT results_eq( + 'SELECT bool_and(length(code) <= 4)::boolean FROM test_schema.m18_const', + ARRAY[true], + 'constant on varchar(4) clamped to the column length'); + +SELECT results_eq( + 'SELECT sum(salary)::integer FROM test_schema.m18_const', + ARRAY[18000], + 'constant on integer skipped — salary values intact'); + +-- Uniqueness: a collapsing "name" mask on a UNIQUE column is skipped +-- (values stay distinct); "hash" is applied and preserves distinctness. +SELECT lives_ok( + format('SELECT pgclone.table(%L, %L, %L, true, %L, %L)', + current_setting('app.source_conninfo'), + 'test_schema', 'mask18_parent', 'm18_uname', '{"mask": {"email": "name"}}'), + 'name mask on UNIQUE email: clone succeeds (mask skipped)'); + +SELECT results_eq( + 'SELECT count(DISTINCT email)::integer FROM test_schema.m18_uname', + ARRAY[3], + 'UNIQUE email left intact when a collapsing mask is requested'); + +SELECT lives_ok( + format('SELECT pgclone.table(%L, %L, %L, true, %L, %L)', + current_setting('app.source_conninfo'), + 'test_schema', 'mask18_parent', 'm18_uhash', '{"mask": {"email": "hash"}}'), + 'hash mask on UNIQUE email: clone succeeds'); + +SELECT results_eq( + $$SELECT count(DISTINCT email)::integer FROM test_schema.m18_uhash + WHERE email <> 'alice@example.com'$$, + ARRAY[3], + 'hash mask applied to UNIQUE email and stays distinct'); + +-- NOT NULL: a null mask on a NOT NULL column is skipped. +SELECT lives_ok( + format('SELECT pgclone.table(%L, %L, %L, true, %L, %L)', + current_setting('app.source_conninfo'), + 'test_schema', 'mask18_parent', 'm18_null', '{"mask": {"ssn": "null"}}'), + 'null mask on NOT NULL ssn: clone succeeds (mask skipped)'); + +SELECT results_eq( + 'SELECT count(*)::integer FROM test_schema.m18_null WHERE ssn IS NOT NULL', + ARRAY[3], + 'NOT NULL ssn left intact when a null mask is requested'); + +-- Foreign key: masking a FK column is skipped (referential integrity). +SELECT lives_ok( + format('SELECT pgclone.table(%L, %L, %L, true, %L, %L)', + current_setting('app.source_conninfo'), + 'test_schema', 'mask18_child', 'm18_child', '{"mask": {"parent_id": {"type":"random_int"}}}'), + 'random_int mask on FK parent_id: clone succeeds (mask skipped)'); + +-- discover_sensitive steers a UNIQUE sensitive column to hash. +SELECT ok( + (SELECT pgclone.discover_sensitive( + current_setting('app.source_conninfo'), + 'test_schema')::text LIKE '%"email": "hash"%'), + 'discover suggests hash for the UNIQUE email column'); + SELECT * FROM finish(); ROLLBACK;