Add reindex rake tasks for populating SearchDocuments - #9391
Draft
gbp wants to merge 13 commits into
Draft
Conversation
The admin user search needs to pass admin_mode (and other backend options) when it routes through search_scope, but the concern dropped every argument after the query. Accept and forward keyword options to `Search.search_scope` so callers can scope the backend search. The Xapian adapter already swallows extra keywords, so its path is unchanged.
Prepare hybrid_search to back a `Search::Backend` adapter behind the common search interface. A search_scope needs to run within a caller supplied relation and stay chainable. Accept an optional base relation (defaulting to model.all) and infer the model from it. De-duplicate the joined relation with DISTINCT, fixing duplicate rows (and wrong counts/pagination) when a record matched through several translations or sections in admin or exact search.
The admin user search is about to move onto the PostgreSQL search index, which only finds records that have search documents. Reindex the records the examples create so they pass under both the current ILIKE search and the indexed search, and match the about me URL by its host, which is how PostgreSQL tokenises a link.
Wire up the native PostgreSQL full-text search behind the common `Search::Backend` interface so callers can reach it alongside the existing Xapian adapter. The PG adapter implements `search_scope` and returns the `hybrid_search` joined relation directly. The paginated query interface (search, typeahead, similar) is follow-on work and keeps the base class's NotImplementedError behaviour.
Add the PostgreSQL adapter to the backend registry so a site can select it as the live query backend with SEARCH_BACKEND postgresql.] The initializer already resolves the configured name through `backend_for`, so registration is all that is needed to make the new adapter switchable. Xapian stays the default.
During the migration both the Xapian and PostgreSQL indexes must stay fresh on every write while only one backend answers queries. The interface already fans `reindex_later` and `queued_jobs_count` across `index_backends`, but nothing let a site choose which backends those are. Add a SEARCH_INDEX_BACKENDS config list (default [xapian], so behaviour is unchanged). Backends are resolved by the new `Search.use_configured_backends!` via the initializer, which sets both the query backend and the indexed backends from configuration.
Some callers are tied to a single backend's features regardless of which backend SEARCH_BACKEND configures. Admin search is the driving case: it relies on the PostgreSQL-only admin index, routing it via the configured backend would break using Xapian. Each query method (search, search_scope, typeahead, similar) now takes an optional backend naming a registered backend for that one call.
Exact mode substring matching uses LIKE, so it is case-sensitive. The admin user search which is about to move onto this backend replaces an ILIKE query, and admins search for fragments like partial email addresses where case rarely matters. Now `hybrid_search` has a `case_sensitive` option (default true, keeping current behaviour for existing exact mode callers); when false the exact mode clauses match with ILIKE instead. The PostgreSQL adaptor exposes the option on `search_scope`; other backends ignore it via their option splats.
The admin user listing searched with a raw ILIKE query over name, email, about_me and exact tag names. Move it onto the unified `search_scope` so admin search runs through the search backend interface and its PostgreSQL-only admin index, forced via the new backend option so SEARCH_BACKEND cannot route it to Xapian. Behaviour changes: exact tag-name matching is dropped (the admin tag pages cover that), results are capped at the search_scope limit of 1000 matches, and results now depend on search_documents freshness.
Its only caller, the admin user listing, now searches through the unified search_scope backend interface, so this is now dead code.
The interface gained a per-call backend override and exact mode gained a case sensitivity option. Describe search_scope chaining with its PostgreSQL option set, the backend override with the admin search example, and note the ILIKE variant where the doc describes the LIKE exact search.
The chunked reindex rake tasks need to enumerate searchable models, find records that still lack a search document, and index a record without aborting on a single failure. Add three class helpers to the Searchable concern to support this. `searchable_models` lists the registered models. `not_indexed` returns records with no search document via a NOT EXISTS subquery, so a reindex can resume and skip already-indexed records without relying on id order. `reindex_record` indexes one record, logging and swallowing failures.
A full reindex of a pre-existing database processes millions of records. Doing so in one long-running process lets memory grow unbounded, and basing resumption on an id cursor alone re-does already-indexed work. Add reindex:missing and reindex:all. Both drive a coordinator that does not load the app and instead spawns a fresh reindex:chunk process per chunk, so memory is released between chunks rather than accumulating. Each worker indexes one chunk, advances a per-model id cursor file so a failing record cannot stall the run, then exits. reindex:missing skips records that already have a search document; reindex:all rebuilds every record. The worker also quietens query logging, whose per-upsert writes otherwise dominate disk IO.
gbp
force-pushed
the
pg-search-admin
branch
2 times, most recently
from
July 22, 2026 10:50
227c4be to
1713d9a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this do?
Adds the tooling to populate and rebuild the PostgreSQL search index at production scale:
rake reindex:missing[Model]— indexes records that have no search document yet; resumable and safe to re-runrake reindex:all[Model]— full rebuild of every searchable recordBoth accept an optional model argument (all searchable models when omitted) and are tunable via
CHUNK_SIZE,BATCH_SIZE,STATE_DIRandRESUMEenvironment variables.Have you updated the changelog? If this is not necessary, put square brackets around this: skip changelog