fix: type-aware masking — skip masks incompatible with a column's typ…#19
Closed
valehdba wants to merge 1 commit into
Closed
fix: type-aware masking — skip masks incompatible with a column's typ…#19valehdba wants to merge 1 commit into
valehdba wants to merge 1 commit into
Conversation
…e (issue #17) A schema clone with masking aborted mid-COPY when a mask produced a value the target column could not parse: ERROR: pgclone: COPY completed with error: ERROR: invalid input syntax for type boolean: "60629" CONTEXT: COPY acquired_right, line 1, column base_income_replacement_allow: "60629" The trigger: discover_sensitive() matched a boolean flag column against the %income% financial pattern and suggested the numeric random_int strategy. Every mask strategy emits a value of a fixed kind (text, integer, NULL, or a caller literal); when that value was fed into an incompatible column type the streaming COPY failed. Fix: every mask-application site now checks the column's pg_type.typcategory before applying a mask and skips an incompatible one with a WARNING, emitting the column unchanged instead of corrupting the COPY stream. Text masks fit string columns; random_int fits numeric or string columns; null and constant are left to the server. - pgclone.table / schema / database clones (both the catalog-driven and the explicit-columns COPY paths): skip + WARNING, column passes through. - discover_sensitive(): joins pg_type and omits a suggestion whose strategy cannot be stored in the column's type, so a generated mask file is safe to apply verbatim. - masking_report(): flags a name-matched but type-incompatible column for manual review instead of recommending a mask that would fail. - mask_in_place(): skips incompatible rules; when all rules are incompatible it returns a clear no-op message instead of an empty UPDATE ... SET. - create_masking_policy(): leaves an incompatible column unmasked in the view, preserving its original type (was silently changing boolean -> integer). New internal helpers: pgclone_mask_out_kind / pgclone_strategy_fits / pgclone_mask_kind_fits, pgclone_column_typcategory, pgclone_masktype_name, pgclone_typcat_desc. Tests: pgTAP group 26 (8 tests, plan 87 -> 95) plus a test_schema.flags fixture with a boolean %income% column. Docs: USAGE type-compatibility table and a "driving masking from a file" workflow. Version bumped to 4.4.1 with sql/pgclone--4.4.0--4.4.1.sql (no SQL signature changes). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S219VD9ZDg211aSEZdE3jD
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…e (issue #17)
A schema clone with masking aborted mid-COPY when a mask produced a value the target column could not parse:
ERROR: pgclone: COPY completed with error:
ERROR: invalid input syntax for type boolean: "60629"
CONTEXT: COPY acquired_right, line 1,
column base_income_replacement_allow: "60629"
The trigger: discover_sensitive() matched a boolean flag column against the %income% financial pattern and suggested the numeric random_int strategy. Every mask strategy emits a value of a fixed kind (text, integer, NULL, or a caller literal); when that value was fed into an incompatible column type the streaming COPY failed.
Fix: every mask-application site now checks the column's pg_type.typcategory before applying a mask and skips an incompatible one with a WARNING, emitting the column unchanged instead of corrupting the COPY stream. Text masks fit string columns; random_int fits numeric or string columns; null and constant are left to the server.
New internal helpers: pgclone_mask_out_kind / pgclone_strategy_fits / pgclone_mask_kind_fits, pgclone_column_typcategory, pgclone_masktype_name, pgclone_typcat_desc.
Tests: pgTAP group 26 (8 tests, plan 87 -> 95) plus a test_schema.flags fixture with a boolean %income% column. Docs: USAGE type-compatibility table and a "driving masking from a file" workflow. Version bumped to 4.4.1 with sql/pgclone--4.4.0--4.4.1.sql (no SQL signature changes).
Claude-Session: https://claude.ai/code/session_01S219VD9ZDg211aSEZdE3jD
Summary
Closes #
Type of change
Description of changes
Database / SQL changes
sql/pgclone--<version>.sqlPG_FUNCTION_INFO_V1names in C match theAS '<lib>', '<sym>'clauses in SQL.sqlmatch the CPG_RETURN_*pathsCOMMENT ON FUNCTION ...added for new public functionsVOLATILE/STABLE/IMMUTABLE) andPARALLELsafety set correctlyTesting
PostgreSQL versions tested locally:
test/pgclone_test.sqlplan()count exactly matches the actual number of assertionstest/test_async.shortest/test_database_create.sh(if applicable)test/fixtures/seed.sqlupdated if new test objects are neededC code safety checklist
palloc/palloc0/pfreeonly — nomalloc/freePQconnectdbhas a matchingPQfinishin all paths (success and error)PQexecresult isPQclear-ed in all pathsquote_literal_cstr()andquote_identifier()StringInfoused for dynamic strings; no fixed stack buffers for SQLstrlcpyinstead ofstrcpy/strncpyereport/elogwith appropriate level — connection strings never logged atLOGlevel or above#if PG_VERSION_NUM >= XXXXXXd.adsrc, pre-PG15 shmem-request pattern, etc.)LWLockwhere neededDocumentation
CHANGELOG.mdupdated under the appropriate version headingdocs/USAGE.md/docs/ASYNC.md/docs/ARCHITECTURE.md/docs/TESTING.mdREADME.mdupdated if the change affects the user-facing feature list or quick-startVersion bump
pgclone.control(default_version)META.json(versionandprovides.pgclone.version+file)README.mdversion badgesql/pgclone--<old>--<new>.sqlmigration script (if applicable)CHANGELOG.mdhas a heading for the new versionBackward compatibility
Screenshots / sample output
Additional notes