Repository navigation
Python: Chroma, MongoDB, Pinecone and Weaviate filters raise NotImplementedError for 'in' with a list literal (and Pinecone 'not' raises TypeError) #14557
Description
What happens
In the Python vector store connectors for Chroma, MongoDB Atlas, Pinecone and Weaviate, a filter lambda that uses in or not in with a list literal always fails with NotImplementedError: Unsupported AST node: <class 'ast.List'>.
The ast.In() and ast.NotIn() branches exist in each _lambda_parser, but the right-hand side can never be parsed, because none of these parsers has a case ast.List(). Qdrant has it (qdrant.py, case ast.List(): return [self._lambda_parser(elt) for elt in node.elts]), so the same filter works there.
Pinecone has a second problem on the same path. Its ast.Not() branch calls next(operand.values()), but dict.values() is a view and not an iterator, so any not (...) filter raises TypeError instead of the intended $nin rewrite or the intended NotImplementedError.
Where
python/semantic_kernel/connectors/ on current main (64005c7):
chroma.pyChromaCollection._lambda_parser(case ast.In()/case ast.NotIn()at lines 382 and 384)mongodb.pyMongoDBAtlasCollection._lambda_parser(lines 462 and 464)pinecone.pyPineconeCollection._lambda_parser(lines 513 and 515)weaviate.pyWeaviateCollection._lambda_parser(line 441)
Pinecone ast.Not() branch:
if (
isinstance(operand, dict)
and len(operand) == 1
and isinstance(next(operand.values()), dict) # type: ignore
and "$in" in next(operand.values()) # type: ignore
):
field = next(operand.keys()) # type: ignore
values = next(operand.values())["$in"] # type: ignore
return {field: {"$nin": values}}Redis (redis.py) also lacks case ast.List(), so it fails the same way. I left it out of the fix I am proposing because RedisVL Text and Num filters reject a list on the right side, so it needs a separate decision.
Steps to reproduce
from semantic_kernel.connectors.pinecone import PineconeCollection
from semantic_kernel.data.vector import VectorStoreCollectionDefinition, VectorStoreField
definition = VectorStoreCollectionDefinition(fields=[
VectorStoreField("key", name="id"),
VectorStoreField("data", name="tag", type="str", is_indexed=True),
VectorStoreField("vector", name="vec", dimensions=2, type="list[float]"),
])
c = PineconeCollection.__new__(PineconeCollection)
object.__setattr__(c, "definition", definition) # only _build_filter is exercised
c._build_filter("lambda x: x.tag in ['a', 'b']")
c._build_filter("lambda x: not (x.tag == 'a')")The same call on ChromaCollection, MongoDBAtlasCollection and WeaviateCollection gives the same first error.
Actual
chroma in ['a','b'] NotImplementedError: Unsupported AST node: <class 'ast.List'>
mongodb in ['a','b'] NotImplementedError: Unsupported AST node: <class 'ast.List'>
pinecone in ['a','b'] NotImplementedError: Unsupported AST node: <class 'ast.List'>
weaviate in ['a','b'] NotImplementedError: Unsupported AST node: <class 'ast.List'>
pinecone not (x.tag == 'a') TypeError: 'dict_values' object is not an iterator
qdrant in ['a','b'] FieldCondition(key='tag', match=MatchAny(any=['a', 'b']))
Expected
x.tag in ['a', 'b']becomes{"tag": {"$in": ["a", "b"]}}for Chroma, MongoDB and Pinecone, andcontains_anyfor Weaviate.not inbecomes$ninwhere the store supports it.- Pinecone
not (x.tag in [...])becomes{"tag": {"$nin": [...]}}, and any othernotexpression raises the documentedNotImplementedError("$not is only supported over $in ...").
I checked open and closed issues and PRs for these connectors and for filter parsing, and found nothing covering this. I am opening a PR right after filing this.
Activity