Add an embedded full-text search backend that does not require OpenSearch

[jzonthemtn]

Summary

Arbiter's full-text search is implemented only against OpenSearch. Running it therefore requires an OpenSearch cluster alongside MongoDB and Valkey, which is disproportionate for small and single-box deployments: OpenSearch is a JVM service that typically wants 2-4GB of heap and its own container before it indexes a single document.

Add an embedded search backend that requires no external service, make it the default, and keep OpenSearch as an optional backend for deployments that already run one.

Why this matters

• Small deployments. A ten-person firm reviewing a few thousand documents does not need a search cluster. Today the choice is to run one or to turn full-text search off entirely via the master flag, losing the Search page and Find similar documents.
• Air-gapped installs. Arbiter targets high-security, air-gapped environments. Every additional service is another image to ship on media, another set of credentials to seed, and another thing to patch offline.
• Adoption. Required infrastructure is a real barrier to someone evaluating Arbiter. A single container that works out of the box gets tried; a three-service stack often does not.

Scope

In scope: the internal full-text index over Arbiter's own documents, currently OpenSearchIndexService and FullTextSearchIndexManager.

Explicitly out of scope: the OpenSearch and Elasticsearch data source connectors (OpenSearchDataSource, ElasticsearchDataSource, their repositories, and OpenSearchIngestJobService / ElasticsearchIngestJobService). Those let Arbiter import documents from a customer's existing cluster and should be left alone. This issue is about removing a dependency, not an integration.

Suggested approach

The surface to replace is small. OpenSearchIndexService exposes:

• indexDocument(Document)
• search(query, from, size) and its filtered overload
• findSimilar(documentId, batchId, size)
• the SearchHit and SearchResults records, including highlight snippets with configurable open/close tags

Extract that into an interface (for example DocumentSearchService), keep OpenSearchIndexService as one implementation, and add an embedded one selected by configuration.

Embedded Apache Lucene is the recommended backend rather than MongoDB $text indexes. Although MongoDB is already a dependency and $text would add nothing new, it cannot cover the existing surface: there is no more-like-this equivalent for findSimilar, and highlighting is far weaker. Lucene covers all of it in-process with no server:

• indexing and query parsing from lucene-core and lucene-queryparser
• MoreLikeThis from lucene-queries for findSimilar
• UnifiedHighlighter for the highlight snippets

The Lucene index lives on local disk and is fully rebuildable from MongoDB, so it stays disposable for backup and recovery purposes: losing it costs a reindex, not data.

Acceptance criteria

Abstraction

• A search backend interface is extracted covering indexDocument, both search overloads, and findSimilar, with the existing SearchHit and SearchResults record shapes unchanged
• OpenSearchIndexService implements the interface with no behavior change
• The backend is selected by configuration, and an invalid backend name fails fast at startup rather than at first query

Embedded backend

• An embedded backend implements the full interface and requires no external service
• findSimilar returns relevant results on the embedded backend, verified by a test asserting that a near-duplicate document ranks above an unrelated one
• Highlight snippets are returned with the same configurable open and close tags as the OpenSearch backend
• Highlight output is HTML-escaped exactly as the existing sanitizeHighlight path does, verified by a test that user-supplied markup in document text cannot inject HTML into a snippet
• Filtered search (the search overload) applies the same filters as the OpenSearch backend
• The index is rebuildable from MongoDB by an admin action, and the rebuild is idempotent
• Index files are written under a single configurable directory so they can be excluded from backups or placed on a chosen volume

Defaults and deployment

• The embedded backend is the default for a new installation
• docker-compose.yaml and docker-compose.prod.yaml start a working stack with full-text search enabled and no OpenSearch container
• OpenSearch remains available as an opt-in backend, with its container moved to an optional Compose profile rather than the default stack
• The fullTextSearchEnabled master flag keeps its current semantics, including the FullTextSearchModelAdvice behavior of hiding the Search link and Find similar button when disabled

Compatibility and docs

• The OpenSearch and Elasticsearch data source connectors are unaffected, verified by their existing tests passing unchanged
• An existing OpenSearch-backed deployment can continue on OpenSearch by setting the backend explicitly, with no reindex required
• A documented migration path exists for moving an existing deployment from OpenSearch to the embedded backend
• User documentation is updated: how to choose a backend, what each requires, and the trade-offs between them
• Documentation states plainly which backend a given deployment is using and where the embedded index is stored

Open questions

• Whether the embedded index should be memory-mapped or stored on disk by default, given appliance-class hardware with limited RAM but fast NVMe
• At what document count the embedded backend stops being appropriate, and whether Arbiter should warn when a deployment crosses it
• Whether an existing OpenSearch deployment should be able to run both backends simultaneously during a migration, or whether a clean switch plus reindex is sufficient