Add PostgreSQL search weights/exclude options - #9423
Open
gbp wants to merge 7 commits into
Open
Conversation
The ranking query called ts_rank_cd with no weights array, so tsvector
only used PostgreSQL's built-in default weights and could not be tuned.
Let callers control how much each label contributes to relevance by
adding an optional weights hash through search_scope into hybrid_search.
Example:
Weight hash: { 'A' => 1.0, 'B' => 0.4, 'C' => 0.2, 'D' => 0.1 }
Converts into: '{0.1,0.2,0.4,1.0}'::float4[]'
We merge weights with the default values which are equal to PG defaults.
Values are coerced with Float() before being passed into SQL to ensure
values are injection safe.
Column was added prior to merging the initial implementation of the PG search as there is potential for it to contain PII. Re-add now but weight as with the 'C' label to reduce relevance as the message is mostly boilerplate so doesn't deserve the same weight as name and emails.
Setting a rank weight to zero looks like it should make a tsvector label stop counting, but it only reorders the rows the WHERE clause already matched. A document matching solely through the zero-weighted label is still returned. For the User admin index that means bounce text weighted "C" can still surface a user on mail-server boilerplate alone, however low its weight. Add an except option that removes labels from the index outright. It uses ts_filter over the tsvector to drop the excluded label before either matching or ranking sees them. Exclusion covers the tsvector matching only. Search raw_content with exact_mode can still return matches for excluded columns.
Excluding a field from a search meant knowing which tsvector label it
was weighted with, which leaks an implementation detail of the index.
Let map field names to tsvector labels automatically instead so
search_scope resolves the names and passes the labels to the backend
hybrid search which already accepts labels.
Example:
User.search_scope(query, admin_mode: true,
except: [:email_bounce_message])
To:
SearchDocument.hybrid_search(query, model: 'User', admin_mode: true,
except: ['C'])
Exact mode with except options doesn't work as the fields are concatenated into a single raw_content or raw_admin_content column. This change converts the column to JSONB (actually it drops and re-adds to handle existing content)so we can store keyed concatenated content for each tsvector label. The except option removes tsvector labels from the index the search matches against so we can perform LIKE/ILIKE searches against the labels which don't include the excluded fields content.
Email bounce messages are mail server boilerplate, so an admin searching users would return unexpected results. The exclusion now covers exact mode as well as the tsvector matching, so fragment searches like a partial email address keep working against the fields that are still indexed.
Collaborator
|
As a first thought, what use cases do you foresee for this, besides the exclusion of My thinking being that the added complexity of search may not be worth it if this is the only use case. I'm also wondering how likely it is that users would tune weights at query-time, we would have to provide ready-made options in the UI. I will play a bit with your code, fill in the db with some values and run some As an alternative, building a specific index for that one column (bounce messages), or even just a SQL query for it, might be easier. But I can see how this could become a pattern of "just this one more query" throughout the codebase 🤔 |
Member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issue(s)
What does this do?
Add PostgreSQL search weights/exclude options
Why was this needed?
Makes it possible to:
Implementation notes
Changes the raw_content/raw_admin_content database columns to keyed JSONB columns so we can exclude the relevant fields based on the exclude option passed in to the search.
Notes to reviewer
I had help from Claude to get the PG queries correct.
I believe the performance impact of this change is minor.