Skip to content

Python: Validate PostgreSQL collection field types - #14611

Merged
Jose Alvarez (jpalvarezl) merged 4 commits into
mainfrom
jpalvarezl-fix-postgres-field-type-validation
Oct 22, 2026
Merged

Jose Alvarez (jpalvarezl) merged 4 commits into
mainfrom
jpalvarezl-fix-postgres-field-type-validation

Conversation

@jpalvarezl

@jpalvarezl Jose Alvarez (jpalvarezl) commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Motivation and Context

PostgreSQL automatic collection creation previously treated an unrecognized field type string as a complete SQL type expression. Automatic schema creation should accept only the field types it can safely and predictably generate.

Description

  • Map supported Python types and PostgreSQL aliases to canonical, trusted type expressions, including common scalar types such as date, UUID, Decimal, BIGINT, SMALLINT, REAL, and TIMESTAMPTZ.
  • Support arrays only when their element type is also supported, including both list[...] and PostgreSQL ...[] notation.
  • Support strictly parsed length and precision modifiers for VARCHAR/CHARACTER VARYING and NUMERIC/DECIMAL.
  • Reject unsupported or malformed type strings before acquiring a database connection or executing SQL.
  • Preserve advanced PostgreSQL type compatibility through pre-created tables: callers using domains, enums, extension types, or other unsupported type expressions should create the table separately and use the collection without calling ensure_collection_exists().
  • Add exact-DDL coverage for decorator-inferred and explicit collection definitions, rejection coverage for delimiters, comments, newlines, malformed generics, arrays, and modifiers, plus a PostgreSQL sentinel integration test.

Validation:

  • uv sync --all-extras --dev --python 3.12 — passed
  • uv run --python 3.12 pytest tests/unit/connectors/memory/test_postgres_store.py — 117 passed
  • uv run --python 3.12 pre-commit run --files semantic_kernel/connectors/postgres.py tests/unit/connectors/memory/test_postgres_store.py tests/integration/memory/postgres/test_postgres_int.py — passed
  • Targeted PostgreSQL integration test collected locally, but the Windows runner could not establish an async psycopg connection because it uses the Proactor event loop; the test is included for the existing Linux/PostgreSQL integration environment.

Contribution Checklist

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Python Test Coverage

Python Test Coverage Report •
FileStmtsMissCoverMissing
connectors
   postgres.py4143990%141, 220, 247, 360–361, 451, 457–459, 464–465, 467–468, 475, 497, 553, 555, 557, 574, 586, 642, 651, 698, 718, 738, 743, 749, 761, 791, 795, 797, 819, 846, 851, 1005, 1009, 1036–1038
TOTAL29159546481% 

Python Unit Test Overview

Tests Skipped Failures Errors Time
4517 22 💤 0 ❌ 0 🔥 1m 38s ⏱️

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Dictionary matching accepts malformed annotations and silently converts dictionary arrays into scalar JSONB columns.

1 open finding
What changed in this PR

Restricts PostgreSQL automatic collection creation to trusted type expressions while preserving custom-type usage through pre-created tables.

Changes:

  • Adds canonical type mappings and recursive array handling.
  • Rejects unsupported field types before database access.
  • Adds exact-DDL, rejection, and sentinel integration coverage.
File Description
python/​tests/​unit/​connectors/​memory/​test_postgres_store.py Tests supported aliases and early rejection.
python/​tests/​integration/​memory/​postgres/​test_postgres_int.py Adds sentinel-table protection coverage.
python/​semantic_kernel/​connectors/​postgres.py Validates type expressions and documents custom-type limitations.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread python/semantic_kernel/connectors/postgres.py Outdated

@VANDRANKI PRABHU KIRAN VANDRANKI (VANDRANKI) left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran this on Python 3.13 with psycopg 3.3.6, comparing main (7a92cd4) with the PR head.

Tests: tests/unit/connectors/memory/test_postgres_store.py gives 97 passed on main and 110 passed on the PR. With the PR's test file on main, 13 tests fail and 97 pass, so the new tests do cover the change. I did not run the integration test, which needs a database.

The injection fix works. I called the type function directly with TEXT); DROP TABLE x; --, list[int and INTEGER[] trailing. Main turns them into SQL text (TEXT); DROP TABLE X; --, LIST[INT, INTEGER[] TRAILING). The PR rejects all three with VectorStoreModelValidationError.

The part I would change. The old code passed any other type name through as upper case SQL, and that path was used by ordinary models. I took decorated dataclass models with @vectorstoremodel and called ensure_collection_exists() with the repo's unit test fixtures (mock cursor). The decorator stores the type as __name__, so these were the type strings:

model field main PR
datetime.datetime TIMESTAMP TIMESTAMP
list[float] | None DOUBLE PRECISION[] DOUBLE PRECISION[]
str | None TEXT TEXT
datetime.date DATE VectorStoreModelValidationError
uuid.UUID data field UUID VectorStoreModelValidationError
decimal.Decimal DECIMAL VectorStoreModelValidationError

date, UUID and Decimal were fields of the same model in my run, so I did not test them one by one. I also called the type function with strings like bigint, varchar(255), numeric(10,2) and timestamptz. Main turns each into the same name in upper case, which is valid PostgreSQL (I did not run it against a database), and the PR rejects them. (A uuid.UUID key field is rejected by the model validator on both main and PR, so that is unchanged.)

So an app that has a date, UUID or Decimal field and relies on ensure_collection_exists() will start to fail after this change. The description says advanced types should use a pre-created table, but date and UUID are not advanced. I would add the common built-in names (date, uuid, decimal, bigint, smallint, real, timestamptz) to POSTGRES_TYPE_ALIASES. Allowing a length or precision in parentheses with a strict pattern like \(\d+(,\d+)?\) would keep varchar(255) working without opening the injection path.

I think this is worth fixing before merge, so I am requesting changes. I like the approach (an allow list and sql.SQL composition) and the tests are good.

@jpalvarezl

Copy link
Copy Markdown
Member Author

PRABHU KIRAN VANDRANKI (@VANDRANKI) Thanks for the detailed compatibility check. I added the common decorator-inferred types (date, UUID, Decimal), safe scalar aliases (BIGINT, SMALLINT, REAL, TIMESTAMPTZ), and strictly parsed/bounded VARCHAR/CHARACTER VARYING length plus NUMERIC/DECIMAL precision modifiers. Arrays of these forms remain supported. The full Postgres connector unit file passes with 117 tests, focused creation coverage passes with 24 tests, and changed-file pre-commit hooks pass. The PR description now reflects the supported compatibility surface.

@SergeyMenshykh
SergeyMenshykh added this pull request to the merge queue Oct 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 21, 2026
@SergeyMenshykh
SergeyMenshykh added this pull request to the merge queue Oct 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 21, 2026
@jpalvarezl
Jose Alvarez (jpalvarezl) added this pull request to the merge queue Oct 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 21, 2026
@jpalvarezl
Jose Alvarez (jpalvarezl) added this pull request to the merge queue Oct 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 21, 2026
@jpalvarezl
Jose Alvarez (jpalvarezl) added this pull request to the merge queue Oct 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 21, 2026
@jpalvarezl
Jose Alvarez (jpalvarezl) added this pull request to the merge queue Oct 22, 2026
Merged via the queue into main with commit 2550183 Oct 22, 2026
33 checks passed
@jpalvarezl
Jose Alvarez (jpalvarezl) deleted the jpalvarezl-fix-postgres-field-type-validation branch October 22, 2026 07:46

This branch was successfully deployed

1 active and 1 inactive deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Pull requests for the Python Semantic Kernel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants