Repository navigation
[Python] Optionally write chunk content to Qdrant payload - #40502
pangwangshu wants to merge 1 commit into
Conversation
Add QdrantWriteConfig.content_payload_key. When set, the converter stores EmbeddableItem.content_string under that key in the point payload next to the metadata, so points retrieved from a collection written by Beam carry the chunk text. The default (None) keeps the current behavior of writing only the metadata. A key that collides with a metadata key raises ValueError instead of overwriting it, an empty key is rejected at construction time, and the item's metadata dict is copied rather than mutated. Fixes apache#40476 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #40502 +/- ##
============================================
+ Coverage 56.16% 56.23% +0.06%
Complexity 2288 2288
============================================
Files 1124 1124
Lines 178336 178367 +31
Branches 1489 1489
============================================
+ Hits 100170 100304 +134
+ Misses 75645 75542 -103
Partials 2521 2521
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Assigning reviewers: R: @jrmccluskey for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
jrmccluskey
left a comment
There was a problem hiding this comment.
Broadly LGTM from a Python perspective, looping in a second set of eyes that is more familiar with RAG to hopefully answer some of your outstanding questions on the initial implementation
Adds an optional
content_payload_keyfield toQdrantWriteConfig. When set, the converter storesEmbeddableItem.content_stringunder that key in the point payload, next to the metadata, so points retrieved from a collection written by Beam carry the chunk text.BigQueryVectorWriterConfigand the Milvus writer already persist the content; the Qdrant writer stored onlyitem.metadata.None, which writes only the metadata as before, so there is no behavior change for existing users.ValueErrorinstead of overwriting it.__post_init__.metadatadict is copied, not mutated.ValueErrorfromcontent_stringwhen the key is set.The two design questions in #40476 are still open: whether leaving the content out was deliberate, and whether an opt-in key is the right shape compared with always writing to a fixed
"content"key. I am happy to change the name or the default based on review.Testing
qdrant_test.py: 5 new converter tests (default unchanged, key writes content, no metadata, collision, empty key). All 43 tests in the file pass locally.qdrant_it_test.py: newtest_write_with_content_payload_keywrites through the transform to a testcontainers Qdrant instance and asserts the payload read back is the metadata plus the content. All 7 tests in the file pass locally on DirectRunner.CHANGES.mdupdated.AI assistance disclosure
Per the ASF generative tooling guidance: this change was written with the help of Claude Code (Claude Opus 5.5). I have reviewed the code and tests and ran them locally, and to my knowledge the change contains no third-party copyrighted material.
Generated-by: Claude Code (Claude Opus 5.5)
Fixes #40476
🤖 Generated with Claude Code