Skip to content

Python: Chroma, MongoDB, Pinecone and Weaviate filters raise NotImplementedError for 'in' with a list literal (and Pinecone 'not' raises TypeError) #14557

Open

Description

What happens

In the Python vector store connectors for Chroma, MongoDB Atlas, Pinecone and Weaviate, a filter lambda that uses in or not in with a list literal always fails with NotImplementedError: Unsupported AST node: <class 'ast.List'>.

The ast.In() and ast.NotIn() branches exist in each _lambda_parser, but the right-hand side can never be parsed, because none of these parsers has a case ast.List(). Qdrant has it (qdrant.py, case ast.List(): return [self._lambda_parser(elt) for elt in node.elts]), so the same filter works there.

Pinecone has a second problem on the same path. Its ast.Not() branch calls next(operand.values()), but dict.values() is a view and not an iterator, so any not (...) filter raises TypeError instead of the intended $nin rewrite or the intended NotImplementedError.

Where

python/semantic_kernel/connectors/ on current main (64005c7):

  • chroma.py ChromaCollection._lambda_parser (case ast.In() / case ast.NotIn() at lines 382 and 384)
  • mongodb.py MongoDBAtlasCollection._lambda_parser (lines 462 and 464)
  • pinecone.py PineconeCollection._lambda_parser (lines 513 and 515)
  • weaviate.py WeaviateCollection._lambda_parser (line 441)

Pinecone ast.Not() branch:

if (
    isinstance(operand, dict)
    and len(operand) == 1
    and isinstance(next(operand.values()), dict)  # type: ignore
    and "$in" in next(operand.values())  # type: ignore
):
    field = next(operand.keys())  # type: ignore
    values = next(operand.values())["$in"]  # type: ignore
    return {field: {"$nin": values}}

Redis (redis.py) also lacks case ast.List(), so it fails the same way. I left it out of the fix I am proposing because RedisVL Text and Num filters reject a list on the right side, so it needs a separate decision.

Steps to reproduce

from semantic_kernel.connectors.pinecone import PineconeCollection
from semantic_kernel.data.vector import VectorStoreCollectionDefinition, VectorStoreField

definition = VectorStoreCollectionDefinition(fields=[
    VectorStoreField("key", name="id"),
    VectorStoreField("data", name="tag", type="str", is_indexed=True),
    VectorStoreField("vector", name="vec", dimensions=2, type="list[float]"),
])
c = PineconeCollection.__new__(PineconeCollection)
object.__setattr__(c, "definition", definition)  # only _build_filter is exercised
c._build_filter("lambda x: x.tag in ['a', 'b']")
c._build_filter("lambda x: not (x.tag == 'a')")

The same call on ChromaCollection, MongoDBAtlasCollection and WeaviateCollection gives the same first error.

Actual

chroma    in ['a','b']      NotImplementedError: Unsupported AST node: <class 'ast.List'>
mongodb   in ['a','b']      NotImplementedError: Unsupported AST node: <class 'ast.List'>
pinecone  in ['a','b']      NotImplementedError: Unsupported AST node: <class 'ast.List'>
weaviate  in ['a','b']      NotImplementedError: Unsupported AST node: <class 'ast.List'>
pinecone  not (x.tag == 'a')   TypeError: 'dict_values' object is not an iterator
qdrant    in ['a','b']      FieldCondition(key='tag', match=MatchAny(any=['a', 'b']))

Expected

  • x.tag in ['a', 'b'] becomes {"tag": {"$in": ["a", "b"]}} for Chroma, MongoDB and Pinecone, and contains_any for Weaviate. not in becomes $nin where the store supports it.
  • Pinecone not (x.tag in [...]) becomes {"tag": {"$nin": [...]}}, and any other not expression raises the documented NotImplementedError("$not is only supported over $in ...").

I checked open and closed issues and PRs for these connectors and for filter parsing, and found nothing covering this. I am opening a PR right after filing this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    pythonPull requests for the Python Semantic Kerneltriage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions