Plan: SQLite3 Migration for gps-routes-api
TL;DR
Migrating from PostgreSQL to SQLite3 is highly feasible for this use case. The dataset size (10k-100k routes), low concurrency requirements, and feature maturity of modern SQLite extensions enable feature parity while dramatically simplifying deployment. This plan uses Spatialite for geographic queries, FTS5 for full-text search, and JSON for array operations. Estimated effort: 80-120 developer hours over 4-6 weeks.
Feasibility Assessment
✅ Fully Supported Features
| PostgreSQL Feature |
SQLite Equivalent |
Confidence |
Notes |
| Full-text search (TSVECTOR) |
FTS5 built-in |
✅ 100% |
Better ranking flexibility; no rewrite |
| PostGIS spatial queries |
Spatialite extension |
✅ 95% |
ST_Within, ST_DWithin, KNN all available |
| Array operations (terrain[], activities[]) |
JSON1 (json_extract, json_each) |
✅ 90% |
Slight schema change; queries need translation |
| Batch transactions |
SQLite transactions |
✅ 100% |
Native support; single-threaded model suits use case |
| Connection pooling |
sqlite3 driver pooling |
✅ 100% |
Works with CGo driver |
| Indexes (B-tree, GIN, GIST) |
B-tree + expression indexes |
✅ 85% |
Fewer index types, but sufficient for workload |
| Upserts (ON CONFLICT DO UPDATE) |
SQLite INSERT OR REPLACE |
✅ 100% |
Native SQL support |
⚠️ Known Limitations & Mitigations
| Limitation |
Impact |
Mitigation |
| Concurrent writes |
Only one writer at a time |
✅ N/A - low concurrency requirement |
| Distance ranking with coordinate transformation (ST_Transform) |
No built-in coordinate systems |
✅ Pre-compute in app layer; use metered distance |
| Array unnesting syntax |
Different operator set |
✅ Use json_each() + CROSS JOIN |
| Full-text ranking tie-breaking |
Different algorithms |
✅ Minor UX difference; acceptable |
| CGo compilation |
Requires C compiler (gcc) |
✅ Pin sqlite3 version; document dependencies |
| Database file locking on network |
Should not serve over network |
✅ Single-machine deployment only |
Implementation Plan
Phase 1: Feasibility Validation & POC (Week 1 | 5-8 days)
Objective: Verify core features work as expected with SQLite + extensions.
Steps:
-
Set up SQLite + extensions test environment
- (Depends on: none)
- Install
go-sqlite3 locally: go get github.com/mattn/go-sqlite3
- Verify CGo compilation works:
go test -v github.com/mattn/go-sqlite3 (ensures gcc is available)
- Document required system packages (gcc, sqlite3-dev on Linux)
-
Create POC database with test schema
- (Parallel with step 1)
- Create minimal SQLite schema mirroring db/migrations/00001_routes.up.sql
- Enable extensions:
PRAGMA load_extension('mod_spatialite'); PRAGMA load_extension('fts5');
- Load 100 sample route records from existing PostgreSQL dump
-
POC: Validate FTS5 query equivalence
- (Depends on step 2)
- Implement FTS5 virtual table for full-text search
- Test weighted search (equivalent to PostgreSQL
setweight() - achievable via column ranking)
- Verify prefix matching (
"hello:*" becomes simpler FTS5 syntax)
- Compare result ranking between PG and SQLite for 10 test queries
-
POC: Validate Spatialite geospatial queries
- (Depends on step 2)
- Load coordinate data for test routes
- Test bounding box:
ST_Within(_geoloc, ST_MakeEnvelope(...))
- Test distance-based nearby:
ST_DWithin(...) with distance in meters
- Test KNN sorting:
ORDER BY distance using Spatialite's distance operator
- Verify coordinate transformation (WGS84 ↔ Web Mercator) works as expected
-
POC: Validate JSON array operations
- (Depends on step 2)
- Create JSON array columns for
terrain, activities, facilities, points_of_interest
- Test array overlap equivalent:
json_extract(activities, '$[*]') LIKE '%keyword%' vs. native JSON functions
- Test unnesting for facet queries:
SELECT ... FROM routes, json_each(activities) equivalent
- Compare performance vs. PostgreSQL native arrays
-
Document findings & decision
- (Depends on steps 3-5)
- Create
/doc/SQLITE_MIGRATION_REPORT.md with POC results
- Decision gate: Proceed if all critical queries validate with <5% ranking differences
Deliverables:
- POC database file
- Test query results comparison document
- CGo compilation troubleshooting guide
Phase 2: Schema Design & Migration Scripts (Week 2 | 5-7 days)
Objective: Design SQLite schema and create reversible migration strategy.
Steps:
-
Design production SQLite schema
- (Depends on Phase 1 completion)
- Adapt db/migrations/00001_routes.up.sql:
- Replace
TEXT[] columns with JSON type
- Replace
TSVECTOR computed column with FTS5 virtual table (separate)
- Keep
GEOMETRY(Point, 4326) geometry column, Spatialite will handle it
- Keep all indexes, translate GIN → expression indexes where needed
- Create db/migrations/00002_sqlite_schema.sql with full SQLite DDL
- Include FTS5 table creation:
CREATE VIRTUAL TABLE routes_fts USING fts5(...)
-
Create bidirectional schema migration tool
- (Depends on step 1)
- Write cmds/migrate_postgres_to_sqlite.go:
- Connect to PostgreSQL source (connection string from env)
- Fetch all route records in batches (1000 at a time)
- Transform arrays → JSON, keep geometry as WKB
- Insert into SQLite target database
- Validate record counts match
- Rollback capability (drop destination tables)
- Include related tables: nearby, images, details
-
Design concurrent migration with validation
- (Parallel with step 2)
- Strategy: Run PostgreSQL & SQLite side-by-side during migration
- Write tests/migration_validation_test.go:
- Random sample query from both databases (FTS, spatial, faceted search)
- Verify results match within acceptable difference (e.g., ranking order tolerance)
- Fail if counts differ or essential records missing
-
Create rollback/revert procedure
- (Depends on steps 1-3)
- Document in doc/SQLITE_MIGRATION_RUNBOOK.md:
- Points-in-time for rollback (pre-migration, mid-migration, post-migration)
- Automated rollback: keep PostgreSQL DB available for N days post-cutover
- Data validation queries: count records, verify spatial integrity
- Switchover playbook: update connection strings, restart services
Deliverables:
Phase 3: Query Layer Refactoring (Week 3-4 | 10-12 days)
Objective: Replace PostgreSQL-specific SQL with SQLite equivalent while maintaining query builder API.
Steps:
-
Create SQLite dialect abstraction
- (Depends on none)
- Add db/dialect.go interface:
type Dialect interface {
BuildArrayOverlapQuery(field string, values []string) string
BuildFullTextQuery(query string) string
BuildGeoQuery(operation string, params...interface{}) string
FormatParam(index int) string // $1 for PG, ? for SQLite
}
- Implement
PostgreSQLDialect and SQLiteDialect structs
- Register in query builder
-
Port db/query_builder.go to dialect-aware
- (Depends on step 1)
- Add
dialect Dialect field to QueryBuilder struct
- Replace hardcoded
$1, $2 with dialect.FormatParam(i)
- Replace array overlap
&& operator:
- PG:
activities && $1::TEXT[]
- SQLite: Use JSON array operations or materialized unnested columns (decision needed: see Further Considerations)
- Replace FTS query operator
@@:
- Both use
fts_table MATCH query (SQLite FTS5 syntax)
- Update db/query_builder_test.go with parameterized tests for both dialects
-
Port spatial queries
- (Depends on step 1)
- Update db/query_builder.go geospatial filter building:
- Bounding box (ST_Within): Identical syntax in Spatialite, no changes needed
- Distance-based nearby (ST_DWithin): Verify Spatialite function signatures match
- KNN sorting: Both support distance operator; update sort field construction
- Create db/spatial_test.go with 10 test cases: bbox, distance, KNN, edge cases (null coords, boundary conditions)
-
Port full-text search & ranking
- (Depends on step 1)
- FTS5 uses different ranking function:
rank instead of ts_rank_cd
- Update sort field in repositories/postgres.go:
- PG:
ts_rank_cd(search_vector, to_tsquery($1), 32) DESC
- SQLite:
rank DESC (FTS5 default; or use bm25() for tuning)
- Test weighted search (A=title, B=address, C=description) via FTS5 column creation options
-
Port array unnesting for facets
- (Depends on step 1)
- Update services/facets.go facet query building:
- PG:
SELECT UNNEST(terrain) AS key, COUNT(*)
- SQLite:
SELECT json_extract(terrain, '$[*]') AS key, COUNT(*) (if JSON) OR use pre-normalized columns
- Decide: (1) Keep JSON + json_each() in queries, (2) Materialize array elements in separate tables, or (3) Use SQLite columns for searchable arrays
- Implement chosen approach; update facet query test cases
-
Update batch insert logic for upserts
- (Depends on step 1)
- Rewrite repositories/postgres.go
batch operations:
- PG:
INSERT ... ON CONFLICT (object_id) DO UPDATE
- SQLite:
INSERT OR REPLACE INTO routes (...) VALUES (...) OR INSERT INTO routes ... ON CONFLICT DO UPDATE SET ...
- Both support upsert; verify behavior matches (PG will update only specified columns; SQLite REPLACE replaces entire row)
- Choose SQLite INSERT...ON CONFLICT if column-level control needed; REPLACE if full-row replacement is acceptable
Deliverables:
Phase 4: Repository Layer & Core Business Logic (Week 4-5 | 10-14 days)
Objective: Rewrite data access layer for SQLite while maintaining API contract.
Steps:
-
Create SQLite repository implementation
- (Depends on Phase 3 completion)
- Copy repositories/postgres.go → repositories/sqlite.go
- Update all query execution methods:
- Replace
pgx.Conn.Query() with *sql.DB.QueryContext()
- Replace
pgx.Batch with standard SQL transactions
- Update connection pooling: use standard
sql.DB pool (configured via SetMaxOpenConns(25), SetMaxIdleConns(5))
- Remove pgx-specific error handling; use
database/sql standard errors
- Implement connection string parsing for SQLite (simpler: just file path):
sqlite3:///path/to/routes.db?cache=shared&mode=rwc&_journal_mode=WAL
- WAL mode enables better concurrency for reads during writes
-
Implement SQLite-specific transaction handling
- (Depends on step 1)
- Update repositories/sqlite.go
Store() method:
- Use
Begin(TxOptions{Isolation: LevelImmediate}) for immediate locking (prevents concurrent writes during transaction)
- Batch inserts in
sql.Stmt prepared statements
- Handle
SQLITE_BUSY errors with exponential backoff retry (for WAL contention)
- Create repositories/sqlite_test.go with concurrent write test (verify single-writer safety)
-
Port connection management
- (Depends on step 1)
- Update db/postgres_connect.go → db/sqlite_connect.go:
- Load Spatialite extension on connection open:
db.Exec("SELECT load_extension('mod_spatialite')")
- Initialize FTS5:
db.Exec("SELECT load_extension('fts5')") (usually built-in)
- Set pragmas for performance & integrity:
PRAGMA foreign_keys = ON (enable cascade deletes)
PRAGMA journal_mode = WAL (write-ahead logging for concurrency)
PRAGMA synchronous = NORMAL (balance safety & performance)
- Document in README.md required extensions and build flags
-
Update search & faceting logic
- (Depends on steps 1-3)
- Port repositories/postgres.go
SearchHits() method → repositories/sqlite.go:
- Update WHERE clause builder for SQLite dialect (Step 3.5 above)
- FTS query execution:
SELECT * FROM routes JOIN routes_fts ON routes.rowid = routes_fts.rowid WHERE routes_fts MATCH $1
- Spatial query execution: verify Spatialite functions work identically
- Port repositories/postgres.go
FacetCounts() method:
- Execute 12 facet queries in parallel goroutines (same pattern as PostgreSQL version)
- Handle array unnesting per decision in Phase 3.5
-
Support legacy PostgreSQL builds (optional feature flag)
- (Parallel with step 4, lower priority)
- Add build tag:
//go:build postgres vs. //go:build sqlite
- Maintain repositories/postgres.go as legacy path
- Document in CONTRIBUTING.md:
go build -tags=sqlite (default: SQLite)
go build -tags=postgres (legacy: PostgreSQL)
Deliverables:
Phase 5: Testing, Performance Validation & Deployment (Week 5-6 | 10-12 days)
Objective: Validate feature parity, performance, and production readiness.
Steps:
-
Create integration test suite
- (Depends on Phase 4 completion)
- Refactor tests/http_api_server_test.go to be database-agnostic:
- Parameterize fixtures to run against both PostgreSQL and SQLite
- Add 50+ test cases covering:
- Full-text search (simple, prefix, multi-word, edge cases)
- Geographic search (bbox, nearby, edge coordinates)
- Faceted search (all 12 facets, combinations)
- Pagination (offset, limit edge cases)
- Upserts (insert, update, conflict handling)
- Bulk operations (batch import of 1000+ records)
- Target: >90% code coverage for repositories/sqlite.go
-
Performance benchmarking
- (Depends on step 1)
- Create tests/benchmark_sqlite_postgres_test.go:
- Load dataset: 50,000 routes with geographic & metadata variety
- Benchmark queries:
- FTS: 100 unique search queries, measure avg latency + p95/p99
- Spatial: 100 bounding box queries, 100 nearby distance queries
- Faceted: Full facet query suite (12 concurrent facets)
- Upsert: Batch insert 10,000 new routes, measure throughput
- Success criteria:
- SQLite within 2x PostgreSQL latency for typical queries
- Sub-100ms for 50k-dataset FTS/spatial queries (API target)
- Document results in
/doc/PERFORMANCE_COMPARISON.md
-
Data migration dry-run in staging
- (Parallel with step 2)
- Run cmds/migrate_postgres_to_sqlite.go on production PostgreSQL dump
- Verify migration validation tests pass (from Phase 2)
- Measure migration time for full dataset (set expectations for cutover)
- Document in runbook: "Staging migration took X minutes for Y records"
-
Update Docker & deployment
- (Depends on step 3)
- Update Dockerfile:
- Add build stage: install gcc, sqlite3-dev for CGo compilation
- Build flags: optimize for SQLite driver size
- Final image: include sqlite3 driver code (no external library needed)
- Update docker-compose.yml:
- Remove PostgreSQL service
- Add SQLite volume mount for persistence:
- ./data/routes.db:/app/data/routes.db
- Add health check: ping
/v1/gps-routes/ref-data endpoint (validates FTS + spatial indexing)
-
Update documentation
- (Depends on steps 1-4)
- Update README.md:
- Database setup section: remove PostgreSQL instructions, add SQLite + Spatialite build requirements
- Environment variables: remove PG_* vars, add
SQLITE_DB_PATH=/app/data/routes.db
- Create doc/SQLITE_SETUP.md: system dependencies, CGo compiler setup, extension loading
- Update CONTRIBUTING.md: local dev setup with SQLite
- Create doc/SQLITE_ARCHITECTURE.md: schema, query patterns, indexing strategy
-
Caching layer compatibility check (optional)
- (Depends on step 1)
- repositories/cache.go should be DB-agnostic (already is)
- Verify memoization works with SQLite queries: no changes needed
- Consider: SQLite's built-in query caching; evaluate if additional application caching is still beneficial
Deliverables:
Relevant Files to Modify
Critical Paths
Supporting Files
Testing & Documentation
Verification Steps
Phase 1 Verification
- ✅ POC database created; schema loads without errors
- ✅ 10 test FTS queries produce results; ranking order within 5% of PostgreSQL
- ✅ 50 spatial test queries return same record counts ±1 (tolerance for precision)
- ✅ Array unnesting produces facet counts matching PostgreSQL (within 1%)
- ✅ No CGo compilation errors on macOS, Linux, Windows (CI/CD)
Phase 2 Verification
- ✅ Migration tool imports 50,000 test records in <5 minutes
- ✅ Migration validation tests: all queries pass (FTS, spatial, facets)
- ✅ Record count matches: PostgreSQL record count == SQLite record count (exact)
- ✅ Rollback procedure tested: revert to PostgreSQL and re-run import
Phase 3 Verification
- ✅ Query builder tests pass for both PostgreSQL & SQLite dialects
- ✅ All spatial test cases pass (10 test cases from db/spatial_test.go)
- ✅ Array query translation produces correct results (facet test suite)
- ✅ Parameterized test suite runs against both databases (100+ test cases)
Phase 4 Verification
- ✅ SQLite repository implements all methods from PostgreSQL counterpart
- ✅ Concurrent write test passes: multiple goroutines attempt writes; only one succeeds per transaction
- ✅ Full API test suite passes against SQLite backend (+50 new test cases)
- ✅ Transaction rollback works: failed writes leave database unchanged
Phase 5 Verification
- ✅ FTS benchmark: SQLite P99 latency <2x PostgreSQL (target <100ms)
- ✅ Spatial benchmark: same latency targets met
- ✅ Faceted search: 12-parallel-query execution time within budget
- ✅ Upsert throughput: >1000 inserts/sec for new records
- ✅ Docker image builds without errors;
docker-compose up runs API successfully
- ✅ Health check passes:
/v1/gps-routes/ref-data returns 200 with valid JSON
- ✅ API compatibility: all existing endpoint tests pass without modification
Decisions & Design Rationale
Array Handling: JSON vs. Materialized Columns
Decision: Use JSON1 extension with json_extract() + json_each() for query translation.
Rationale:
- Minimal schema changes: Keep existing query builder logic mostly intact
- Flexibility: Can easily add new array fields without schema migration
- Performance trade-off acceptable: Faceted search (array unnesting) is secondary feature; latency budget is generous
- Alternative rejected: Materialized lookup tables would require significant schema redesign and trigger complexity
Connection Pooling Strategy
Decision: Use standard sql.DB with WAL mode (pragma journal_mode=WAL) for better read-write concurrency.
Rationale:
- Single writer guarantee: WAL mode serializes writes; aligns with low-concurrency requirement
- Non-blocking reads: Readers don't block writers (and vice versa) in WAL mode
- Simple: No need for external connection pool manager; stdlib handles it
Upsert Strategy
Decision: Use INSERT OR REPLACE for full-row replacement (not column-selective UPDATE).
Rationale:
- Simpler implementation: Reduces migration effort from PostgreSQL
ON CONFLICT DO UPDATE
- Acceptable semantics: Data import workflow replaces entire route objects anyway
- Performance: REPLACE is O(log n) like UPDATE; no meaningful difference
Staged Rollout (Optional Future)
Decision: Maintain PostgreSQL repository layer as legacy code (-tags=postgres) for safe rollback.
Rationale:
- Zero-downtime migration possible: Run new SQLite backend in parallel; compare results before cutover
- Rollback safety: If issues emerge post-migration, revert binary to PostgreSQL version
- Cost: ~20-30 extra developer hours (Phase 4, step 5); defer unless cutover risk is high
Extensibility: FTS Ranking Algorithm
Decision: Use SQLite FTS5 default ranking (bm25); can customize via rank() UDF in future.
Rationale:
- Good enough for initial release: FTS5 built-in ranking is tuned for general queries
- Weighted search: Column assignment weights (title, address, description) can be assigned at FTS table creation
- Future optimization: If search UX needs ranking tuning, implement custom rank() function (Phase 6)
Known Risks & Mitigation
| Risk |
Probability |
Impact |
Mitigation |
| Spatialite distance queries not exact match PG |
Medium |
API returns different nearby results |
Phase 1 POC with 50 test queries; establish tolerance (e.g., ±10m) |
| CGo build environment inconsistent |
Low |
Deployment fails on CI/CD |
Document build dependencies; pin go-sqlite3 version; provide Dockerfile with pre-built binary |
| WAL mode file locking over network (if attempted) |
Low |
Corruption or errors |
Document: SQLite not suitable for network-mounted DBs; single-machine deployment only |
| FTS5 ranking produces significantly different results |
Medium |
Search UX regression |
Phase 1 POC + Phase 5 user acceptance testing; establish ranking tolerance |
| Faceted search latency increases 3x+ |
Low |
API latency SLA miss |
Phase 5 benchmarking; if needed, move facet computation to async background job |
| Data import tool bugs lose records |
Low |
Production data loss |
Phase 2 validation tests (exact record count match) + Phase 3 dry-run on staging |
Mitigation overall: POC (Phase 1) de-risks 80% of technical uncertainty; staged testing (Phase 3-5) catches integration issues early.
Further Considerations
1. Search Result Ranking Tolerance
- Question: How much variance in full-text search result ranking is acceptable?
- Impact: Phase 1 POC ranking comparison determines if FTS5 is production-ready
- Recommendation: Establish 10-query test suite; acceptable if top-3 results match PostgreSQL for 80% of queries; if not, consider bm25 tuning or custom ranking UDF (Phase 6 effort)
2. Array Query Performance vs. Simplicity
- Question: Should array operations use JSON functions (slower, simpler) or materialized lookup tables (complex DDL, faster)?
- Impact: Faceted search latency; Phase 5 benchmarking determines if JSON performance is acceptable
- Recommendation: Start with JSON approach (Phase 3.5); if facet queries exceed 100ms SLA, refactor to lookup tables (Phase 6 optimization)
3. Concurrent Write Handling & Queueing
- Question: Should the app handle SQLite BUSY errors transparently (exponential backoff) or fail fast?
- Impact: User experience during concurrent imports; Phase 4 design choice
- Recommendation: Implement exponential backoff (3 retries, max 5-second delay) in repository layer; log/alert if QPS exceeds SQLite write capacity
4. Spatialite Distance Units & Precision
- Question: Does Spatialite's
ST_DWithin() produce identical distance-meters behavior as PostgreSQL?
- Impact: Nearby route queries; Phase 1 POC must validate
- Recommendation: Test 50 nearby queries with various distances (1km, 10km, 50km); accept if results match within 10 meters
5. API Versioning & Backward Compatibility
- Question: Should migration be transparent to API clients, or is a new version/flag acceptable?
- Impact: Deployment complexity; client coordination
- Recommendation: Keep API contract identical (v1 endpoints unchanged); document database change in
/doc/ARCHITECTURE.md (user-facing, not breaking)
Timeline & Effort Estimate
| Phase |
Duration |
Dev Days |
Risk Level |
Blocker? |
| 1. POC & Feasibility |
Week 1 |
5-8 |
Medium |
🔴 Yes—decision gate |
| 2. Schema & Migration Tools |
Week 2 |
5-7 |
Low |
🟢 No |
| 3. Query Layer Refactoring |
Week 3-4 |
10-12 |
Medium |
🟡 Depends on Phase 1 |
| 4. Repository & Core Logic |
Week 4-5 |
10-14 |
Low |
🟡 Depends on Phase 3 |
| 5. Testing & Deployment |
Week 5-6 |
10-12 |
Low |
🟢 No |
| Reserve (unforeseen) |
— |
5-10 |
— |
— |
| TOTAL |
4-6 weeks |
45-63 days |
— |
— |
Note: Phases 2 and 3 can run in parallel after Phase 1 decision gate. Actual timeline depends on team size (1 dev vs. 2). Recommend: allocate 10-12 weeks for comfortable pacing + validation.
Success Criteria (Go/No-Go Checklist)
- ✅ No breaking changes to REST API (100% backward compatible)
- ✅ FTS search results within acceptable ranking tolerance (Phase 1 gate)
- ✅ Geographic queries produce same nearby/bbox results (within precision tolerance)
- ✅ Faceted search queries complete in <100ms for 50k dataset
- ✅ Upsert throughput >1000 inserts/sec (data import performance maintained)
- ✅ Docker build succeeds on Linux, macOS, Windows (CI/CD)
- ✅ No data loss during migration (record count exact match)
- ✅ Rollback procedure tested in staging
- ✅ Load test: API survives 50 concurrent requests without deadlock or BUSY errors
- ✅ All existing tests pass; coverage >90% for new SQLite code
Out of Scope (Phase 6+)
- Advanced FTS ranking customization (bm25 tuning, BM42 algorithm)
- Materialized array lookup tables (if JSON performance acceptable)
- Read replica / distributed SQLite (e.g., LiteFS)
- Query optimization beyond standard SQLite pragmas
- Alternative spatial libraries (h3geo, geohash)—stick with Spatialite
Document Version: 1.0 | Date: 12 April 2026 | Status: Ready for Phase 1 kickoff
How to Use This Plan
- Phase 1 (Week 1): Assign to senior backend engineer; execute POC; hold decision gate meeting
- Phase 2 (Week 2): Parallel track: schema design + migration tooling
- Phase 3-4 (Weeks 3-5): Core implementation; rotate team members for knowledge sharing
- Phase 5 (Weeks 5-6): QA, performance validation, documentation
- Post-Phase 5: Code review, staging deployment, production cutover (separate playbook)
Plan: SQLite3 Migration for gps-routes-api
TL;DR
Migrating from PostgreSQL to SQLite3 is highly feasible for this use case. The dataset size (10k-100k routes), low concurrency requirements, and feature maturity of modern SQLite extensions enable feature parity while dramatically simplifying deployment. This plan uses Spatialite for geographic queries, FTS5 for full-text search, and JSON for array operations. Estimated effort: 80-120 developer hours over 4-6 weeks.
Feasibility Assessment
✅ Fully Supported Features
Implementation Plan
Phase 1: Feasibility Validation & POC (Week 1 | 5-8 days)
Objective: Verify core features work as expected with SQLite + extensions.
Steps:
Set up SQLite + extensions test environment
go-sqlite3locally:go get github.com/mattn/go-sqlite3go test -v github.com/mattn/go-sqlite3(ensures gcc is available)Create POC database with test schema
PRAGMA load_extension('mod_spatialite'); PRAGMA load_extension('fts5');POC: Validate FTS5 query equivalence
setweight()- achievable via column ranking)"hello:*"becomes simpler FTS5 syntax)POC: Validate Spatialite geospatial queries
ST_Within(_geoloc, ST_MakeEnvelope(...))ST_DWithin(...)with distance in metersORDER BY distanceusing Spatialite's distance operatorPOC: Validate JSON array operations
terrain,activities,facilities,points_of_interestjson_extract(activities, '$[*]') LIKE '%keyword%'vs. native JSON functionsSELECT ... FROM routes, json_each(activities)equivalentDocument findings & decision
/doc/SQLITE_MIGRATION_REPORT.mdwith POC resultsDeliverables:
Phase 2: Schema Design & Migration Scripts (Week 2 | 5-7 days)
Objective: Design SQLite schema and create reversible migration strategy.
Steps:
Design production SQLite schema
TEXT[]columns withJSONtypeTSVECTORcomputed column with FTS5 virtual table (separate)GEOMETRY(Point, 4326)geometry column, Spatialite will handle itCREATE VIRTUAL TABLE routes_fts USING fts5(...)Create bidirectional schema migration tool
Design concurrent migration with validation
Create rollback/revert procedure
Deliverables:
Phase 3: Query Layer Refactoring (Week 3-4 | 10-12 days)
Objective: Replace PostgreSQL-specific SQL with SQLite equivalent while maintaining query builder API.
Steps:
Create SQLite dialect abstraction
PostgreSQLDialectandSQLiteDialectstructsPort db/query_builder.go to dialect-aware
dialect Dialectfield toQueryBuilderstruct$1,$2withdialect.FormatParam(i)&&operator:activities && $1::TEXT[]@@:fts_table MATCH query(SQLite FTS5 syntax)Port spatial queries
Port full-text search & ranking
rankinstead ofts_rank_cdts_rank_cd(search_vector, to_tsquery($1), 32) DESCrank DESC(FTS5 default; or use bm25() for tuning)Port array unnesting for facets
SELECT UNNEST(terrain) AS key, COUNT(*)SELECT json_extract(terrain, '$[*]') AS key, COUNT(*)(if JSON) OR use pre-normalized columnsUpdate batch insert logic for upserts
batchoperations:INSERT ... ON CONFLICT (object_id) DO UPDATEINSERT OR REPLACE INTO routes (...) VALUES (...)ORINSERT INTO routes ... ON CONFLICT DO UPDATE SET ...Deliverables:
Phase 4: Repository Layer & Core Business Logic (Week 4-5 | 10-14 days)
Objective: Rewrite data access layer for SQLite while maintaining API contract.
Steps:
Create SQLite repository implementation
pgx.Conn.Query()with*sql.DB.QueryContext()pgx.Batchwith standard SQL transactionssql.DBpool (configured viaSetMaxOpenConns(25),SetMaxIdleConns(5))database/sqlstandard errorssqlite3:///path/to/routes.db?cache=shared&mode=rwc&_journal_mode=WALImplement SQLite-specific transaction handling
Store()method:Begin(TxOptions{Isolation: LevelImmediate})for immediate locking (prevents concurrent writes during transaction)sql.Stmtprepared statementsSQLITE_BUSYerrors with exponential backoff retry (for WAL contention)Port connection management
db.Exec("SELECT load_extension('mod_spatialite')")db.Exec("SELECT load_extension('fts5')")(usually built-in)PRAGMA foreign_keys = ON(enable cascade deletes)PRAGMA journal_mode = WAL(write-ahead logging for concurrency)PRAGMA synchronous = NORMAL(balance safety & performance)Update search & faceting logic
SearchHits()method → repositories/sqlite.go:SELECT * FROM routes JOIN routes_fts ON routes.rowid = routes_fts.rowid WHERE routes_fts MATCH $1FacetCounts()method:Support legacy PostgreSQL builds (optional feature flag)
//go:build postgresvs.//go:build sqlitego build -tags=sqlite(default: SQLite)go build -tags=postgres(legacy: PostgreSQL)Deliverables:
Phase 5: Testing, Performance Validation & Deployment (Week 5-6 | 10-12 days)
Objective: Validate feature parity, performance, and production readiness.
Steps:
Create integration test suite
Performance benchmarking
/doc/PERFORMANCE_COMPARISON.mdData migration dry-run in staging
Update Docker & deployment
- ./data/routes.db:/app/data/routes.db/v1/gps-routes/ref-dataendpoint (validates FTS + spatial indexing)Update documentation
SQLITE_DB_PATH=/app/data/routes.dbCaching layer compatibility check (optional)
Deliverables:
/doc/PERFORMANCE_COMPARISON.mdRelevant Files to Modify
Critical Paths
Supporting Files
Testing & Documentation
Verification Steps
Phase 1 Verification
Phase 2 Verification
Phase 3 Verification
Phase 4 Verification
Phase 5 Verification
docker-compose upruns API successfully/v1/gps-routes/ref-datareturns 200 with valid JSONDecisions & Design Rationale
Array Handling: JSON vs. Materialized Columns
Decision: Use JSON1 extension with json_extract() + json_each() for query translation.
Rationale:
Connection Pooling Strategy
Decision: Use standard
sql.DBwith WAL mode (pragma journal_mode=WAL) for better read-write concurrency.Rationale:
Upsert Strategy
Decision: Use
INSERT OR REPLACEfor full-row replacement (not column-selective UPDATE).Rationale:
ON CONFLICT DO UPDATEStaged Rollout (Optional Future)
Decision: Maintain PostgreSQL repository layer as legacy code (
-tags=postgres) for safe rollback.Rationale:
Extensibility: FTS Ranking Algorithm
Decision: Use SQLite FTS5 default ranking (bm25); can customize via rank() UDF in future.
Rationale:
Known Risks & Mitigation
Mitigation overall: POC (Phase 1) de-risks 80% of technical uncertainty; staged testing (Phase 3-5) catches integration issues early.
Further Considerations
1. Search Result Ranking Tolerance
2. Array Query Performance vs. Simplicity
3. Concurrent Write Handling & Queueing
4. Spatialite Distance Units & Precision
ST_DWithin()produce identical distance-meters behavior as PostgreSQL?5. API Versioning & Backward Compatibility
/doc/ARCHITECTURE.md(user-facing, not breaking)Timeline & Effort Estimate
Note: Phases 2 and 3 can run in parallel after Phase 1 decision gate. Actual timeline depends on team size (1 dev vs. 2). Recommend: allocate 10-12 weeks for comfortable pacing + validation.
Success Criteria (Go/No-Go Checklist)
Out of Scope (Phase 6+)
Document Version: 1.0 | Date: 12 April 2026 | Status: Ready for Phase 1 kickoff
How to Use This Plan