fix: constraint- and length-aware data masking (issue #18)#21
Merged
Conversation
Builds on the type-aware masking from #17. A mask could still abort or corrupt a clone in ways type-checking alone did not catch; all are now handled by skipping the offending mask (with a WARNING) or clamping it, so the clone always succeeds: - length: a constant/partial/hash/random_int value longer than a varchar(N)/char(N) column failed with "value too long". Output for a length-limited string column is now clamped with left(..., N). - constant on a numeric column: the default 'REDACTED' (or any non-numeric text) failed to parse. A constant is applied to a non-string column only when the literal is a valid number, otherwise skipped. - null on a NOT NULL column: skipped. - UNIQUE/PRIMARY KEY column: only the injective "hash" strategy is applied; value-collapsing (name/constant/null) or collision-prone (random_int) strategies are skipped. - FOREIGN KEY column: masking is skipped to preserve referential integrity. discover_sensitive() and masking_report() now only suggest strategies the engine will apply: FK columns are omitted, and UNIQUE/PK or NOT NULL sensitive columns are steered to "hash". A masked view (create_masking_policy) enforces no constraints, so only the type/constant checks apply there. New helpers: ColMaskMeta + pgclone_column_maskmeta / pgclone_maskmeta_from_row (shared PGCLONE_MASKMETA_COLS catalog fragment), pgclone_mask_skip_reason (single decision point), pgclone_append_mask_expr_clamped, pgclone_looks_numeric, pgclone_discover_strategy. Tests: pgTAP group 27 (12 tests, plan 95 -> 107) plus mask18_parent/mask18_child fixtures covering length, constant-on-numeric, uniqueness, NOT NULL and FK. Docs: USAGE length/constraint safety section. Version bumped to 4.4.2 with sql/pgclone--4.4.1--4.4.2.sql (no SQL signature changes).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on the type-aware masking from #17. A mask could still abort or corrupt a clone in ways type-checking alone did not catch; all are now handled by skipping the offending mask (with a WARNING) or clamping it, so the clone always succeeds:
discover_sensitive() and masking_report() now only suggest strategies the engine will apply: FK columns are omitted, and UNIQUE/PK or NOT NULL sensitive columns are steered to "hash". A masked view (create_masking_policy) enforces no constraints, so only the type/constant checks apply there.
New helpers: ColMaskMeta + pgclone_column_maskmeta / pgclone_maskmeta_from_row (shared PGCLONE_MASKMETA_COLS catalog fragment), pgclone_mask_skip_reason (single decision point), pgclone_append_mask_expr_clamped, pgclone_looks_numeric, pgclone_discover_strategy.
Tests: pgTAP group 27 (12 tests, plan 95 -> 107) plus mask18_parent/mask18_child fixtures covering length, constant-on-numeric, uniqueness, NOT NULL and FK. Docs: USAGE length/constraint safety section. Version bumped to 4.4.2 with sql/pgclone--4.4.1--4.4.2.sql (no SQL signature changes).
Summary
Closes #
Type of change
Description of changes
Database / SQL changes
sql/pgclone--<version>.sqlPG_FUNCTION_INFO_V1names in C match theAS '<lib>', '<sym>'clauses in SQL.sqlmatch the CPG_RETURN_*pathsCOMMENT ON FUNCTION ...added for new public functionsVOLATILE/STABLE/IMMUTABLE) andPARALLELsafety set correctlyTesting
PostgreSQL versions tested locally:
test/pgclone_test.sqlplan()count exactly matches the actual number of assertionstest/test_async.shortest/test_database_create.sh(if applicable)test/fixtures/seed.sqlupdated if new test objects are neededC code safety checklist
palloc/palloc0/pfreeonly — nomalloc/freePQconnectdbhas a matchingPQfinishin all paths (success and error)PQexecresult isPQclear-ed in all pathsquote_literal_cstr()andquote_identifier()StringInfoused for dynamic strings; no fixed stack buffers for SQLstrlcpyinstead ofstrcpy/strncpyereport/elogwith appropriate level — connection strings never logged atLOGlevel or above#if PG_VERSION_NUM >= XXXXXXd.adsrc, pre-PG15 shmem-request pattern, etc.)LWLockwhere neededDocumentation
CHANGELOG.mdupdated under the appropriate version headingdocs/USAGE.md/docs/ASYNC.md/docs/ARCHITECTURE.md/docs/TESTING.mdREADME.mdupdated if the change affects the user-facing feature list or quick-startVersion bump
pgclone.control(default_version)META.json(versionandprovides.pgclone.version+file)README.mdversion badgesql/pgclone--<old>--<new>.sqlmigration script (if applicable)CHANGELOG.mdhas a heading for the new versionBackward compatibility
Screenshots / sample output
Additional notes