Skip to content

fix: type-aware masking — skip masks incompatible with a column's typ…#19

Closed
valehdba wants to merge 1 commit into
mainfrom
claude/busy-keller-pqzb10
Closed

fix: type-aware masking — skip masks incompatible with a column's typ…#19
valehdba wants to merge 1 commit into
mainfrom
claude/busy-keller-pqzb10

Conversation

@valehdba

@valehdba valehdba commented Jul 1, 2026

Copy link
Copy Markdown
Owner

…e (issue #17)

A schema clone with masking aborted mid-COPY when a mask produced a value the target column could not parse:

ERROR: pgclone: COPY completed with error:
ERROR: invalid input syntax for type boolean: "60629"
CONTEXT: COPY acquired_right, line 1,
column base_income_replacement_allow: "60629"

The trigger: discover_sensitive() matched a boolean flag column against the %income% financial pattern and suggested the numeric random_int strategy. Every mask strategy emits a value of a fixed kind (text, integer, NULL, or a caller literal); when that value was fed into an incompatible column type the streaming COPY failed.

Fix: every mask-application site now checks the column's pg_type.typcategory before applying a mask and skips an incompatible one with a WARNING, emitting the column unchanged instead of corrupting the COPY stream. Text masks fit string columns; random_int fits numeric or string columns; null and constant are left to the server.

  • pgclone.table / schema / database clones (both the catalog-driven and the explicit-columns COPY paths): skip + WARNING, column passes through.
  • discover_sensitive(): joins pg_type and omits a suggestion whose strategy cannot be stored in the column's type, so a generated mask file is safe to apply verbatim.
  • masking_report(): flags a name-matched but type-incompatible column for manual review instead of recommending a mask that would fail.
  • mask_in_place(): skips incompatible rules; when all rules are incompatible it returns a clear no-op message instead of an empty UPDATE ... SET.
  • create_masking_policy(): leaves an incompatible column unmasked in the view, preserving its original type (was silently changing boolean -> integer).

New internal helpers: pgclone_mask_out_kind / pgclone_strategy_fits / pgclone_mask_kind_fits, pgclone_column_typcategory, pgclone_masktype_name, pgclone_typcat_desc.

Tests: pgTAP group 26 (8 tests, plan 87 -> 95) plus a test_schema.flags fixture with a boolean %income% column. Docs: USAGE type-compatibility table and a "driving masking from a file" workflow. Version bumped to 4.4.1 with sql/pgclone--4.4.0--4.4.1.sql (no SQL signature changes).

Claude-Session: https://claude.ai/code/session_01S219VD9ZDg211aSEZdE3jD

Summary

Closes #

Type of change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 💥 Breaking change (fix or feature that changes existing behavior)
  • 🏗️ Refactor / internal cleanup (no functional change)
  • ⚡ Performance improvement
  • 📖 Documentation only
  • 🔧 Build / CI / tooling
  • 🧪 Test-only change

Description of changes

Database / SQL changes

  • New or modified SQL functions are declared in sql/pgclone--<version>.sql
  • PG_FUNCTION_INFO_V1 names in C match the AS '<lib>', '<sym>' clauses in SQL
  • Return types in .sql match the C PG_RETURN_* paths
  • COMMENT ON FUNCTION ... added for new public functions
  • Function volatility (VOLATILE / STABLE / IMMUTABLE) and PARALLEL safety set correctly

Testing

PostgreSQL versions tested locally:

  • pgTAP tests added or updated in test/pgclone_test.sql
  • plan() count exactly matches the actual number of assertions
  • Shell tests added/updated in test/test_async.sh or test/test_database_create.sh (if applicable)
  • test/fixtures/seed.sql updated if new test objects are needed
  • CI is green on all of PG 14, 15, 16, 17, 18
# Output of pre_deploy_checks.sh (paste the summary line)
# e.g. "22 passed, 0 failed"

C code safety checklist

  • palloc / palloc0 / pfree only — no malloc/free
  • Every PQconnectdb has a matching PQfinish in all paths (success and error)
  • Every PQexec result is PQclear-ed in all paths
  • Dynamic SQL uses quote_literal_cstr() and quote_identifier()
  • StringInfo used for dynamic strings; no fixed stack buffers for SQL
  • strlcpy instead of strcpy / strncpy
  • ereport / elog with appropriate level — connection strings never logged at LOG level or above
  • PG-version-specific APIs are guarded with #if PG_VERSION_NUM >= XXXXXX
  • No new uses of removed APIs (d.adsrc, pre-PG15 shmem-request pattern, etc.)
  • Shared-memory state protected by LWLock where needed

Documentation

  • CHANGELOG.md updated under the appropriate version heading
  • Relevant doc updated: docs/USAGE.md / docs/ASYNC.md / docs/ARCHITECTURE.md / docs/TESTING.md
  • README.md updated if the change affects the user-facing feature list or quick-start

Version bump

  • pgclone.control (default_version)
  • META.json (version and provides.pgclone.version + file)
  • README.md version badge
  • New sql/pgclone--<old>--<new>.sql migration script (if applicable)
  • CHANGELOG.md has a heading for the new version

Backward compatibility

Screenshots / sample output

Additional notes

…e (issue #17)

A schema clone with masking aborted mid-COPY when a mask produced a value
the target column could not parse:

  ERROR: pgclone: COPY completed with error:
         ERROR: invalid input syntax for type boolean: "60629"
  CONTEXT: COPY acquired_right, line 1,
           column base_income_replacement_allow: "60629"

The trigger: discover_sensitive() matched a boolean flag column against the
%income% financial pattern and suggested the numeric random_int strategy.
Every mask strategy emits a value of a fixed kind (text, integer, NULL, or a
caller literal); when that value was fed into an incompatible column type the
streaming COPY failed.

Fix: every mask-application site now checks the column's pg_type.typcategory
before applying a mask and skips an incompatible one with a WARNING, emitting
the column unchanged instead of corrupting the COPY stream. Text masks fit
string columns; random_int fits numeric or string columns; null and constant
are left to the server.

- pgclone.table / schema / database clones (both the catalog-driven and the
  explicit-columns COPY paths): skip + WARNING, column passes through.
- discover_sensitive(): joins pg_type and omits a suggestion whose strategy
  cannot be stored in the column's type, so a generated mask file is safe to
  apply verbatim.
- masking_report(): flags a name-matched but type-incompatible column for
  manual review instead of recommending a mask that would fail.
- mask_in_place(): skips incompatible rules; when all rules are incompatible
  it returns a clear no-op message instead of an empty UPDATE ... SET.
- create_masking_policy(): leaves an incompatible column unmasked in the view,
  preserving its original type (was silently changing boolean -> integer).

New internal helpers: pgclone_mask_out_kind / pgclone_strategy_fits /
pgclone_mask_kind_fits, pgclone_column_typcategory, pgclone_masktype_name,
pgclone_typcat_desc.

Tests: pgTAP group 26 (8 tests, plan 87 -> 95) plus a test_schema.flags
fixture with a boolean %income% column. Docs: USAGE type-compatibility table
and a "driving masking from a file" workflow. Version bumped to 4.4.1 with
sql/pgclone--4.4.0--4.4.1.sql (no SQL signature changes).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S219VD9ZDg211aSEZdE3jD
@valehdba valehdba closed this Jul 1, 2026
@valehdba valehdba deleted the claude/busy-keller-pqzb10 branch July 1, 2026 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants