Skip to content

Python: add Milvus connector for the VectorStore abstraction - #14577

Open
Nantha kumar (Nanduu24) wants to merge 1 commit into
microsoft:mainfrom
Nanduu24:python-milvus-vector-connector
Open

Nantha kumar (Nanduu24) wants to merge 1 commit into
microsoft:mainfrom
Nanduu24:python-milvus-vector-connector

Conversation

@Nanduu24

Copy link
Copy Markdown

Motivation and Context

Milvus is one of the most widely used open-source vector databases, but on the Python side it was only available through the deprecated MemoryStore API (connectors/memory_stores/milvus). Every other major backend — Postgres, Qdrant, Redis, Pinecone, Weaviate, Chroma, Faiss, SQL Server, Azure AI Search — has been ported to the modern VectorStore / VectorStoreCollection abstraction, but Milvus had not.

This has been an open, unaddressed request: #13391 ("Why is the Milvus connector not impl VectorStore?"), where another user reported trying to build their own and finding it non-trivial. This PR closes that gap.

Resolves #13391

Description

Adds a new semantic_kernel/connectors/milvus.py with MilvusStore, MilvusCollection and MilvusSettings, following the same structure as the existing qdrant.py / postgres.py connectors:

  • Schema from the data model — the key field becomes the primary key (INT64/VARCHAR), data fields become typed scalar fields, and vector fields become FLOAT_VECTOR fields, each with an index. The Milvus dynamic field is enabled so properties outside the definition are preserved on round-trip.
  • Mappings — DistanceFunction → Milvus metric types (COSINE/L2/IP), IndexKind → Milvus index types (HNSW/FLAT/IVF_FLAT/DISKANN/AUTOINDEX).
  • Full surface — ensure_collection_exists / collection_exists / ensure_collection_deleted, upsert, get, delete, and vector search. Lambda/string filters are translated into Milvus boolean expressions via _lambda_parser (==, !=, <, <=, >, >=, in, not in, and/or/not, chained comparisons).
  • Uses the sync MilvusClient wrapped with asyncio.to_thread so it works consistently across the supported pymilvus range (the MilvusClient API requires pymilvus >= 2.4, which the existing shared milvus extra already resolves to — happy to bump the extra floor from 2.3 to 2.4 if you'd prefer it made explicit).
  • Registered in connectors/memory.py (lazy import) and connectors/memory.pyi (type stub), mirroring the other connectors.

Testing

  • Added tests/unit/connectors/memory/test_milvus.py (28 tests): ingestion, semantic search, filter translation, serialization/deserialization, component lifecycle, and schema creation (hermetic — fake client, no live DB).
  • Verified end-to-end against a real Milvus Lite database (create → index → upsert → semantic search → metadata round-trip → filtered search → get → cross-instance persistence → delete), all passing.
  • ruff format --check, ruff check, and mypy are clean on the changed files; uv lock --locked is unaffected.

I'm also happy to follow up with a concept sample under samples/concepts/memory/ and a docs entry if that would help.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
  • All unit tests pass, and I have added new tests where possible
  • I didn't break anyone 😄

Adds MilvusStore, MilvusCollection and MilvusSettings implementing the
modern VectorStore/VectorStoreCollection/VectorSearch interfaces, so Milvus
is available as an out-of-the-box vector store alongside Postgres, Qdrant,
Redis, Pinecone, Weaviate, etc. Previously Milvus was only available through
the deprecated MemoryStore API.

- Schema, typed scalar fields, FLOAT_VECTOR fields and per-vector indexes are
  derived from the data model definition; the dynamic field is enabled so
  properties outside the definition are preserved.
- Distance functions and index kinds are mapped to Milvus metric/index types.
- Vector search with lambda/string filters translated to Milvus boolean
  expressions, plus upsert, get, delete, and collection lifecycle.
- Registered in connectors.memory for lazy import and the .pyi stub.
- Unit tests covering ingestion, search, filter translation, serialization,
  component lifecycle and schema creation.

Resolves microsoft#13391
@Nanduu24
Nantha kumar (Nanduu24) requested a review from a team as a code owner October 10, 2026 21:01
Copilot AI balanced review requested due to automatic review settings October 10, 2026 21:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@allyson-tcesp

Copy link
Copy Markdown

Well done! Thx

Is this working with the new Microsoft Agent Framework?

@Nanduu24

Copy link
Copy Markdown
Author

Thanks Allyson (@allyson-tcesp)! This PR targets Semantic Kernel's VectorStore. For the new Microsoft Agent Framework I've just opened the counterpart — a Milvus vector store (MilvusStore/MilvusCollection) — in microsoft/agent-framework#9346, so Milvus will be available in both. Appreciate the interest!

This branch was previously deployed

1 inactive deployment
github-app-auth — 154a0ad0 Deployed Oct 10, 2026 by Nanduu24 via team_check #609
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Pull requests for the Python Semantic Kernel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

New Feature: Why Milvus connector not impl VectorStore?

3 participants