Repository navigation
Removes usages of the sentence-transformers package... - #40439
Conversation
|
cc: @damccorm |
|
Assigning reviewers: R: @tvalentyn for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #40439 +/- ##
==========================================
Coverage 59.04% 59.04%
Complexity 15624 15624
==========================================
Files 2797 2798 +1
Lines 280750 280924 +174
Branches 12488 12488
==========================================
+ Hits 165764 165877 +113
- Misses 108540 108601 +61
Partials 6446 6446
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| super().__init__(type_adapter=create_text_adapter(), **kwargs) | ||
| self.model_name = model_name | ||
| self.max_seq_length = max_seq_length | ||
| self.model_class = SentenceTransformer |
There was a problem hiding this comment.
is SentenceTransformer the only possible model class and will always be the only candidate for this codepath? It looks like we are making an assumption that if model_class is not callable, then it is SentenceTransformer. Is it safe to assume it will hold?
There was a problem hiding this comment.
This is an assumption in existing code, right ? I'm not changing it.
It seems to me like we pushed self.model_class to a variable just to provide a better early error (not an import error). We don't support other class values for model_class within huggingface.py in existing code.
There was a problem hiding this comment.
And note that even if a caller (for example, a test) passes a customer model_class it will still be respected since not callable(model_class) will evaluate to false.
| model_handler = huggingface.SentenceTransformerEmbeddings( | ||
| model_name=DEFAULT_MODEL_NAME, | ||
| columns=[test_query_column]).get_model_handler() | ||
| # Older Beam versions stored a reference to the SentenceTransformer class |
There was a problem hiding this comment.
Does older mean 2.77.0 and earlier? how does the current version store the model_class reference in the artifacts on disk when we save the artifact?
There was a problem hiding this comment.
Does older mean 2.77.0 and earlier?
Yes. Updated text.
how does the current version store the model_class reference in the artifacts on disk when we save the artifact?
With the current version, the artifacts will not store the model_class reference. Instead, it will be determined and imported lazily during runtime.
chamikaramj
left a comment
chamikaramj
left a comment
There was a problem hiding this comment.
Thanks. PTAL.
| model_handler = huggingface.SentenceTransformerEmbeddings( | ||
| model_name=DEFAULT_MODEL_NAME, | ||
| columns=[test_query_column]).get_model_handler() | ||
| # Older Beam versions stored a reference to the SentenceTransformer class |
There was a problem hiding this comment.
Does older mean 2.77.0 and earlier?
Yes. Updated text.
how does the current version store the model_class reference in the artifacts on disk when we save the artifact?
With the current version, the artifacts will not store the model_class reference. Instead, it will be determined and imported lazily during runtime.
|
Thanks! |
3 participants
... from the job submission path
This package and dependencies are extremely large. When installed via PyPI, the size is approximately:
Download size: 4 GB
Installed size: 7 GB
This change removes usage of the package for ML transforms from the job submission path so that these package do not have to be available during job submission. The packages just have to be available to workers during execution (for example, provided via a requirements file).
Example Dataflow pipeline where
sentence-transformerswas not installed during job submission: https://console.cloud.google.com/dataflow/jobs/us-central1/2026-10-07_21_55_30-3173716524350011720?project=apache-beam-testingThank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README for a list of workflows and how to trigger them.