Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ epac is an iOS civic-engagement app that displays Canada's House of Commons Hans

Brand and copy decisions live in `docs/brand/brand-brief-v1.md`. Treat that brief as the source of truth for product positioning, tagline, voice, tone, audience, and anti-positioning.

Search backend decisions live in `docs/architecture/search-index-choice-epac452.md`. Use Postgres `tsvector` for v1 search and treat any Meilisearch work as a later migration after canonical records and ranking needs are proven.
Search backend decisions live in `docs/architecture/search-index-choice-v2.md`. Use SQLite FTS5 (stored in S3, queried by a stateless Lambda) for v1 search. The prior Postgres `tsvector` decision (EPAC-452) is superseded; see that doc for historical context.

Parsed speech schema decisions live in `docs/architecture/parsed-speech-schema-epac464.md`. Treat backend `speeches.intervention_id` as the canonical source-derived speech identity.

Expand Down
18 changes: 0 additions & 18 deletions backend/manifest/deployment-services.json
Original file line number Diff line number Diff line change
Expand Up @@ -67,24 +67,6 @@
"production": {"database": false, "artifact": true}
}
},
{
"name": "search",
"deploy": {"staging": true, "production": true},
"http": {
"payload_format_version": "1.0",
"routes": {
"staging": [{"method": "GET", "path": "/search/speeches"}],
"production": [
{"method": "ANY", "path": "/search"},
{"method": "GET", "path": "/search/speeches"}
]
}
},
"sync": {
"staging": {"database": true, "artifact": false},
"production": {"database": true, "artifact": false}
}
},
{
"name": "member-speeches",
"deploy": {"staging": true, "production": true},
Expand Down
2 changes: 0 additions & 2 deletions backend/openapi/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,6 @@ func TestRetiredLambdaPathsAreAbsent(t *testing.T) {
"/api/v1/live",
"/device/register",
"/api/v1/device/register",
"/search",
"/search/speeches",
}
for _, path := range retiredPaths {
if _, ok := spec.Paths[path]; ok {
Expand Down
8 changes: 7 additions & 1 deletion docs/architecture/search-index-choice-epac452.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
# Search Index Choice (EPAC-452)

**Status:** Accepted for v1
## Status: Superseded

This decision has been superseded by [`search-index-choice-v2.md`](search-index-choice-v2.md) (2026-05-25). The Aurora Serverless v2 retirement (see `infra/rds/README.md`) and the EPAC-1914 → EPAC-1917 migration to S3 artifacts made the Postgres `tsvector` approach obsolete before it was fully implemented. The body below is retained as historical context.

---

**Status:** Superseded — see `search-index-choice-v2.md`
**Last updated:** 2026-04-27
**Decision:** Use PostgreSQL full-text search (`tsvector`) for the v1 parliamentary search index. Revisit Meilisearch after the canonical Hansard, bill, vote, and member records are flowing through the backend.

Expand Down
58 changes: 58 additions & 0 deletions docs/architecture/search-index-choice-v2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Search Index Choice (v2)

**Status:** Accepted for v1 (supersedes EPAC-452)
**Last updated:** 2026-05-25
**Filename convention:** future supersession lands as `search-index-choice-v3.md`, not with an EPAC-NNN suffix.

## Decision

Use SQLite FTS5 with the `porter` tokenizer, stored as a single `.sqlite` file in S3, queried by a stateless Lambda that pulls the file to `/tmp` on cold start.

The index is built by the `backend/hansard-search-index/` Lambda and queried by the `backend/hansard-search/` Lambda (created in EPAC project issues #4 and #5 respectively).

## Why this changed

Two migrations made the Postgres `tsvector` approach (EPAC-452) obsolete before it was fully implemented:

1. **EPAC-1914 → EPAC-1917**: Hot parliamentary entities (members, sittings, bills, statistics pipelines) were migrated from Aurora to S3 artifacts served via CloudFront. The read-side architecture is now S3-native; a Postgres-backed search index would be the only remaining reason to keep a live database connection in the query path.

2. **Aurora retirement**: With the read-side move to S3 artifacts, the production Aurora Serverless v2 cluster (`epac-db`) is being retired (see `infra/rds/README.md`). Building a search index on a database that is being decommissioned is not viable.

A self-contained SQLite file in S3 fits the existing S3-artifact pattern, eliminates the database dependency from the search query path, and keeps operational complexity flat.

## Trade-offs

### SQLite FTS5 in S3 vs Postgres `tsvector`

**SQLite FTS5 pros:** No persistent database connection required; the entire index is a single file that cold-starts in a Lambda's `/tmp`; fits the existing S3-artifact deployment model; FTS5 `porter` tokenizer gives reasonable English stemming out of the box; zero additional AWS infrastructure.

**SQLite FTS5 cons:** The index is rebuilt in full on each refresh cycle (no incremental updates); cold-start latency is proportional to file size; bilingual French stemming is not built in (see Out of scope); concurrent writes to the index file require coordination at build time.

### SQLite FTS5 in S3 vs OpenSearch / Meilisearch

**Managed search service pros:** Better out-of-the-box relevance tuning, typo tolerance, synonym handling, and faceted search; real-time index updates; purpose-built query DSL.

**Managed search service cons:** Adds another production datastore to operate and monitor; requires a reindexing sync path and drift checks between the canonical S3 artifacts and the search index; increases cost and operational surface before search volume justifies it. OpenSearch in particular adds meaningful per-hour cost for a service that is read-heavy and low-QPS at this stage.

## Out of scope for v1

- **Bilingual French stemming:** FTS5 `porter` covers English. French parliamentary debates require a separate tokenizer configuration or a language-split index. Deferred until English search is live and validated.
- **Scheduled refresh:** Index rebuild triggering, freshness SLAs, and alerting on stale indexes are out of scope for the initial implementation.
- **Cross-session search:** Persistent search history, saved searches, and user-scoped results are product features deferred to a later milestone.

## Implementation pointers

- **Index builder:** `backend/hansard-search-index/` — reads canonical S3 artifacts, builds an FTS5 SQLite database, and uploads it to S3.
- **Query Lambda:** `backend/hansard-search/` — pulls the SQLite file to `/tmp` on cold start, accepts `GET /api/v1/search?q=...` requests, and returns paginated source-linked results.
- **Manifest registration:** both services are registered in `backend/manifest/deployment-services.json` as part of their respective implementation PRs.

## Meilisearch migration trigger

Reconsider a managed search service when at least two of these are true:

- Users need typo tolerance or synonym ranking that FTS5 cannot satisfy.
- Search spans enough datasets that relevance tuning dominates backend work.
- Query latency on representative production data exceeds the app target after FTS5 indexing and pagination are tuned.
- Product needs faceted search UX that is awkward with the flat SQLite model.

If a managed service is introduced later, S3 artifacts remain the canonical source. The managed index should be a derived view rebuilt from those records.
Loading