Skip to content

Python: Chroma and Weaviate filters double apostrophes in string values, so values like O'Brien never match #14559

Open

Description

What happens

A filter that compares a field to a string containing an apostrophe never matches in the Python Chroma and Weaviate connectors. The apostrophe is doubled (O'Brien becomes O''Brien) before the value is handed to the store, so it no longer equals the stored value.

Where

python/semantic_kernel/connectors/chroma.py ChromaCollection._lambda_parser, and python/semantic_kernel/connectors/weaviate.py WeaviateCollection._lambda_parser, on current main (64005c7). Both have this in the ast.Constant case:

case ast.Constant():
    value = node.value
    if isinstance(value, str):
        return value.replace("'", "''")
    if isinstance(value, bytes):
        return value.decode("utf-8").replace("'", "''")

That is SQL-style escaping. These two connectors do not build a SQL or query string. Chroma gets a where dict ({"name": value}, {"$and": [...]}) and Weaviate gets Filter.by_property(...).equal(value) objects, so the value is sent as data and nothing needs escaping. The other dict-based connectors (MongoDB, Pinecone, Qdrant) return node.value unchanged. Only the string-building connectors (Azure AI Search, Postgres, SQL Server, Oracle) need the doubling.

Steps to reproduce

Real Chroma, no mocks (pip install chromadb):

import asyncio
from dataclasses import dataclass
from typing import Annotated
import chromadb
from semantic_kernel.connectors.chroma import ChromaCollection
from semantic_kernel.data.vector import VectorStoreField, vectorstoremodel

@vectorstoremodel
@dataclass
class Rec:
    id: Annotated[str, VectorStoreField("key")]
    name: Annotated[str, VectorStoreField("data", is_indexed=True)]
    vec: Annotated[list[float] | None, VectorStoreField("vector", dimensions=2)] = None

async def main():
    col = ChromaCollection(record_type=Rec, collection_name="people", client=chromadb.EphemeralClient())
    await col.ensure_collection_exists()
    await col.upsert([Rec(id="1", name="O'Brien", vec=[0.1, 0.2]), Rec(id="2", name="Smith", vec=[0.2, 0.1])])
    print(col._build_filter("lambda x: x.name == \"O'Brien\""))
    for flt in ["lambda x: x.name == \"Smith\"", "lambda x: x.name == \"O'Brien\""]:
        res = await col.search(vector=[0.1, 0.2], top=5, filter=flt)
        print(flt, "->", [r.record.id async for r in res.results])

asyncio.run(main())

Actual

{'name': "O''Brien"}
lambda x: x.name == "Smith" -> ['2']
lambda x: x.name == "O'Brien" -> []

The record with name="O'Brien" exists but is never returned. For Weaviate, _build_filter("lambda x: x.name == \"O'Brien\"") returns _FilterValue(value="O''Brien", operator=Equal, target='name').

Expected

{'name': "O'Brien"}, and the search returns record 1.

I searched open and closed issues and PRs for this and found nothing. I am opening a PR right after filing this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    pythonPull requests for the Python Semantic Kerneltriage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions