From 6c324883104fe0c62f02af5be7e318541503683f Mon Sep 17 00:00:00 2001 From: riddim-developer-bot Date: Mon, 25 May 2026 13:20:37 -0400 Subject: [PATCH] [EPAC-2059]: delete dead search manifest entry and supersede tsvector decision doc - Remove `search` service from backend/manifest/deployment-services.json (no backend/search/ directory ever existed; entry caused Terraform to provision a ghost Lambda on every apply) - Remove /search and /search/speeches from retiredPaths in backend/openapi/main_test.go - Add docs/architecture/search-index-choice-v2.md: SQLite FTS5 in S3 is the actual v1 search architecture (supersedes EPAC-452 tsvector decision) - Mark docs/architecture/search-index-choice-epac452.md as superseded (body retained as historical context) - Update CLAUDE.md project overview to reference v2 decision doc - Deleted orphan epac-search-staging Lambda function in AWS (orphan from prior Terraform apply; confirmed epac-search-production was never created) --- CLAUDE.md | 2 +- backend/manifest/deployment-services.json | 18 ------ backend/openapi/main_test.go | 2 - .../search-index-choice-epac452.md | 8 ++- docs/architecture/search-index-choice-v2.md | 58 +++++++++++++++++++ 5 files changed, 66 insertions(+), 22 deletions(-) create mode 100644 docs/architecture/search-index-choice-v2.md diff --git a/CLAUDE.md b/CLAUDE.md index f7da8676..6135bcd3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ epac is an iOS civic-engagement app that displays Canada's House of Commons Hans Brand and copy decisions live in `docs/brand/brand-brief-v1.md`. Treat that brief as the source of truth for product positioning, tagline, voice, tone, audience, and anti-positioning. -Search backend decisions live in `docs/architecture/search-index-choice-epac452.md`. Use Postgres `tsvector` for v1 search and treat any Meilisearch work as a later migration after canonical records and ranking needs are proven. +Search backend decisions live in `docs/architecture/search-index-choice-v2.md`. Use SQLite FTS5 (stored in S3, queried by a stateless Lambda) for v1 search. The prior Postgres `tsvector` decision (EPAC-452) is superseded; see that doc for historical context. Parsed speech schema decisions live in `docs/architecture/parsed-speech-schema-epac464.md`. Treat backend `speeches.intervention_id` as the canonical source-derived speech identity. diff --git a/backend/manifest/deployment-services.json b/backend/manifest/deployment-services.json index aa46a967..19aece99 100644 --- a/backend/manifest/deployment-services.json +++ b/backend/manifest/deployment-services.json @@ -67,24 +67,6 @@ "production": {"database": false, "artifact": true} } }, - { - "name": "search", - "deploy": {"staging": true, "production": true}, - "http": { - "payload_format_version": "1.0", - "routes": { - "staging": [{"method": "GET", "path": "/search/speeches"}], - "production": [ - {"method": "ANY", "path": "/search"}, - {"method": "GET", "path": "/search/speeches"} - ] - } - }, - "sync": { - "staging": {"database": true, "artifact": false}, - "production": {"database": true, "artifact": false} - } - }, { "name": "member-speeches", "deploy": {"staging": true, "production": true}, diff --git a/backend/openapi/main_test.go b/backend/openapi/main_test.go index ba097ad5..d9421d65 100644 --- a/backend/openapi/main_test.go +++ b/backend/openapi/main_test.go @@ -87,8 +87,6 @@ func TestRetiredLambdaPathsAreAbsent(t *testing.T) { "/api/v1/live", "/device/register", "/api/v1/device/register", - "/search", - "/search/speeches", } for _, path := range retiredPaths { if _, ok := spec.Paths[path]; ok { diff --git a/docs/architecture/search-index-choice-epac452.md b/docs/architecture/search-index-choice-epac452.md index e63c829d..2df6155a 100644 --- a/docs/architecture/search-index-choice-epac452.md +++ b/docs/architecture/search-index-choice-epac452.md @@ -1,6 +1,12 @@ # Search Index Choice (EPAC-452) -**Status:** Accepted for v1 +## Status: Superseded + +This decision has been superseded by [`search-index-choice-v2.md`](search-index-choice-v2.md) (2026-05-25). The Aurora Serverless v2 retirement (see `infra/rds/README.md`) and the EPAC-1914 → EPAC-1917 migration to S3 artifacts made the Postgres `tsvector` approach obsolete before it was fully implemented. The body below is retained as historical context. + +--- + +**Status:** Superseded — see `search-index-choice-v2.md` **Last updated:** 2026-04-27 **Decision:** Use PostgreSQL full-text search (`tsvector`) for the v1 parliamentary search index. Revisit Meilisearch after the canonical Hansard, bill, vote, and member records are flowing through the backend. diff --git a/docs/architecture/search-index-choice-v2.md b/docs/architecture/search-index-choice-v2.md new file mode 100644 index 00000000..486311a4 --- /dev/null +++ b/docs/architecture/search-index-choice-v2.md @@ -0,0 +1,58 @@ +# Search Index Choice (v2) + +**Status:** Accepted for v1 (supersedes EPAC-452) +**Last updated:** 2026-05-25 +**Filename convention:** future supersession lands as `search-index-choice-v3.md`, not with an EPAC-NNN suffix. + +## Decision + +Use SQLite FTS5 with the `porter` tokenizer, stored as a single `.sqlite` file in S3, queried by a stateless Lambda that pulls the file to `/tmp` on cold start. + +The index is built by the `backend/hansard-search-index/` Lambda and queried by the `backend/hansard-search/` Lambda (created in EPAC project issues #4 and #5 respectively). + +## Why this changed + +Two migrations made the Postgres `tsvector` approach (EPAC-452) obsolete before it was fully implemented: + +1. **EPAC-1914 → EPAC-1917**: Hot parliamentary entities (members, sittings, bills, statistics pipelines) were migrated from Aurora to S3 artifacts served via CloudFront. The read-side architecture is now S3-native; a Postgres-backed search index would be the only remaining reason to keep a live database connection in the query path. + +2. **Aurora retirement**: With the read-side move to S3 artifacts, the production Aurora Serverless v2 cluster (`epac-db`) is being retired (see `infra/rds/README.md`). Building a search index on a database that is being decommissioned is not viable. + +A self-contained SQLite file in S3 fits the existing S3-artifact pattern, eliminates the database dependency from the search query path, and keeps operational complexity flat. + +## Trade-offs + +### SQLite FTS5 in S3 vs Postgres `tsvector` + +**SQLite FTS5 pros:** No persistent database connection required; the entire index is a single file that cold-starts in a Lambda's `/tmp`; fits the existing S3-artifact deployment model; FTS5 `porter` tokenizer gives reasonable English stemming out of the box; zero additional AWS infrastructure. + +**SQLite FTS5 cons:** The index is rebuilt in full on each refresh cycle (no incremental updates); cold-start latency is proportional to file size; bilingual French stemming is not built in (see Out of scope); concurrent writes to the index file require coordination at build time. + +### SQLite FTS5 in S3 vs OpenSearch / Meilisearch + +**Managed search service pros:** Better out-of-the-box relevance tuning, typo tolerance, synonym handling, and faceted search; real-time index updates; purpose-built query DSL. + +**Managed search service cons:** Adds another production datastore to operate and monitor; requires a reindexing sync path and drift checks between the canonical S3 artifacts and the search index; increases cost and operational surface before search volume justifies it. OpenSearch in particular adds meaningful per-hour cost for a service that is read-heavy and low-QPS at this stage. + +## Out of scope for v1 + +- **Bilingual French stemming:** FTS5 `porter` covers English. French parliamentary debates require a separate tokenizer configuration or a language-split index. Deferred until English search is live and validated. +- **Scheduled refresh:** Index rebuild triggering, freshness SLAs, and alerting on stale indexes are out of scope for the initial implementation. +- **Cross-session search:** Persistent search history, saved searches, and user-scoped results are product features deferred to a later milestone. + +## Implementation pointers + +- **Index builder:** `backend/hansard-search-index/` — reads canonical S3 artifacts, builds an FTS5 SQLite database, and uploads it to S3. +- **Query Lambda:** `backend/hansard-search/` — pulls the SQLite file to `/tmp` on cold start, accepts `GET /api/v1/search?q=...` requests, and returns paginated source-linked results. +- **Manifest registration:** both services are registered in `backend/manifest/deployment-services.json` as part of their respective implementation PRs. + +## Meilisearch migration trigger + +Reconsider a managed search service when at least two of these are true: + +- Users need typo tolerance or synonym ranking that FTS5 cannot satisfy. +- Search spans enough datasets that relevance tuning dominates backend work. +- Query latency on representative production data exceeds the app target after FTS5 indexing and pagination are tuned. +- Product needs faceted search UX that is awkward with the flat SQLite model. + +If a managed service is introduced later, S3 artifacts remain the canonical source. The managed index should be a derived view rebuilt from those records.