Skip to content

[Python] Optionally write chunk content to Qdrant payload - #40502

Open
pangwangshu wants to merge 1 commit into
apache:masterfrom
pangwangshu:qdrant-content-payload
Open

pangwangshu wants to merge 1 commit into
apache:masterfrom
pangwangshu:qdrant-content-payload

Conversation

@pangwangshu

Copy link
Copy Markdown
Contributor

Adds an optional content_payload_key field to QdrantWriteConfig. When set, the converter stores EmbeddableItem.content_string under that key in the point payload, next to the metadata, so points retrieved from a collection written by Beam carry the chunk text. BigQueryVectorWriterConfig and the Milvus writer already persist the content; the Qdrant writer stored only item.metadata.

  • The default is None, which writes only the metadata as before, so there is no behavior change for existing users.
  • A key that collides with a metadata key raises ValueError instead of overwriting it.
  • An empty key is rejected in __post_init__.
  • The item's metadata dict is copied, not mutated.
  • As with the BigQuery writer, an item with no text or image URI raises ValueError from content_string when the key is set.

The two design questions in #40476 are still open: whether leaving the content out was deliberate, and whether an opt-in key is the right shape compared with always writing to a fixed "content" key. I am happy to change the name or the default based on review.

Testing

  • qdrant_test.py: 5 new converter tests (default unchanged, key writes content, no metadata, collision, empty key). All 43 tests in the file pass locally.
  • qdrant_it_test.py: new test_write_with_content_payload_key writes through the transform to a testcontainers Qdrant instance and asserts the payload read back is the metadata plus the content. All 7 tests in the file pass locally on DirectRunner.
  • CHANGES.md updated.

AI assistance disclosure

Per the ASF generative tooling guidance: this change was written with the help of Claude Code (Claude Opus 5.5). I have reviewed the code and tests and ran them locally, and to my knowledge the change contains no third-party copyrighted material.

Generated-by: Claude Code (Claude Opus 5.5)

Fixes #40476

🤖 Generated with Claude Code

Add QdrantWriteConfig.content_payload_key. When set, the converter stores
EmbeddableItem.content_string under that key in the point payload next to
the metadata, so points retrieved from a collection written by Beam carry
the chunk text. The default (None) keeps the current behavior of writing
only the metadata.

A key that collides with a metadata key raises ValueError instead of
overwriting it, an empty key is rejected at construction time, and the
item's metadata dict is copied rather than mutated.

Fixes apache#40476

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.23%. Comparing base (76f6e6d) to head (30227d4).
⚠️ Report is 111 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #40502      +/-   ##
============================================
+ Coverage     56.16%   56.23%   +0.06%     
  Complexity     2288     2288              
============================================
  Files          1124     1124              
  Lines        178336   178367      +31     
  Branches       1489     1489              
============================================
+ Hits         100170   100304     +134     
+ Misses        75645    75542     -103     
  Partials       2521     2521              
Flag Coverage Δ
python 79.68% <100.00%> (+0.13%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @jrmccluskey for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@jrmccluskey jrmccluskey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Broadly LGTM from a Python perspective, looping in a second set of eyes that is more familiar with RAG to hopefully answer some of your outstanding questions on the initial implementation

@jrmccluskey
jrmccluskey requested a review from claudevdm October 12, 2026 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request]: Optionally write chunk content to the payload in QdrantWriteConfig

2 participants