Linguistic Engine is a self-contained linguistic engine with an embedded database. It does not depend on an external database service — on first run, it automatically creates, initializes, and seeds its own SQL database (SQLite by default, or PostgreSQL if configured).
┌─────────────────────────────────────────────────────────────────┐
│ REST API │
│ (Axum HTTP routes) │
└──────────────────────────┬──────────────────────────────────────┘
│
┌──────────────────────────▼──────────────────────────────────────┐
│ Engine Layer │
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ │
│ │ Hebrew │ │ Greek │ │ Ge'ez │ │
│ │ Engine │ │ Engine │ │ Engine │ │
│ └─────┬──────┘ └─────┬──────┘ └─────┬──────┘ │
│ └──────────────┼──────────────┘ │
│ │ │
│ ┌────────────────────▼──────────────────────────────────────┐ │
│ │ Index Registry (in-memory) │ │
│ │ HashMap | Trie | BKTree | 10 specialized maps │ │
│ └──────────────────────────┬───────────────────────────────┘ │
│ │ │
│ ┌──────────────────────────▼───────────────────────────────┐ │
│ │ Cache Registry (L1-L5) │ │
│ │ L1 Strong | L2 Lemma | L3 Search | L4 Morph | L5 Trans │ │
│ └──────────────────────────┬───────────────────────────────┘ │
│ │ │
│ ┌──────────────────────────▼───────────────────────────────┐ │
│ │ Graph Engine │ │
│ │ Synonyms | Antonyms | Cognates | Roots | Derived │ │
│ └──────────────────────────┬───────────────────────────────┘ │
└─────────────────────────────┼───────────────────────────────────┘
│
┌─────────────────────────────▼───────────────────────────────────┐
│ Storage Layer │
│ (StorageEngine trait) │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ SQLite │ │ Postgres │ │ Memory │ │ Future │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
└─────────────────────────────────────────────────────────────────┘
The engine manages its entire database lifecycle automatically. No manual migrations, no external setup scripts.
1. Load settings (config file → CLI → env vars)
2. Connect to database (create file if SQLite, connect if Postgres)
3. Initialize schema:
├── CREATE TABLE IF NOT EXISTS (6 tables)
├── CREATE INDEX IF NOT EXISTS (15+ indexes)
└── INSERT OR IGNORE default languages (he, grc, gez)
4. Load all WordEntry records into IndexRegistry (HashMap + RadixTrie + BKTree)
5. Warm up CacheRegistry (L1-L5)
6. Start HTTP server
| Table | Purpose |
|---|---|
languages |
Supported language metadata (code, name, RTL, country) |
word_entries |
Core dictionary entries with all linguistic fields |
word_translations |
Translations of entries to target languages |
word_senses |
Polysemous senses per word (multiple meanings) |
data_versions |
SHA-256 versioned snapshots of every mutation |
audit_log |
Full audit trail of all create/update/delete operations |
src/
├── main.rs # CLI + entrypoint
├── lib.rs # Library re-exports
├── common/ # Shared infrastructure
│ ├── settings.rs # Configuration (Django-style priority chain)
│ ├── db.rs # DatabasePool enum + schema initialization
│ ├── models.rs # WordEntry, WordSense, LanguageCode
│ ├── traits.rs # LanguageEngine trait + sub-traits
│ ├── errors.rs # LinguisticError enum
│ ├── similarity.rs # Levenshtein distance (shared)
│ └── ...
├── storage/ # Storage abstraction layer
│ ├── traits.rs # StorageEngine trait
│ ├── sqlite.rs # SQLite implementation
│ ├── memory.rs # In-memory implementation (testing)
│ └── factory.rs # StorageFactory
├── indexes/ # In-memory specialized indexes
│ ├── registry.rs # IndexRegistry (10 HashMaps + search methods)
│ ├── trie.rs # Radix/PATRICIA Trie for prefix search (compressed edges)
│ └── bktree.rs # BKTree for fuzzy search
├── cache/ # Multilevel cache
│ └── registry.rs # CacheRegistry (L1-L5)
├── graph/ # Lexical graph engine
│ ├── types.rs # RelationType, WordRelation, GraphPath
│ └── engine.rs # GraphEngine (BFS, shortest path, clusters)
├── query/ # Query optimizer
│ ├── analyzer.rs # QueryType detection, index selection
│ └── planner.rs # QueryPlan generation, cost estimation
├── wal/ # Write-Ahead Log
│ ├── log.rs # WALFrame (SHA-256 checksums, sequence numbers, configurable fsync)
│ └── recovery.rs # WALRecovery (replay with corruption detection, storage replay)
├── snapshot/ # Snapshot manager
│ ├── types.rs # SnapshotFile (versioned format), SnapshotMeta
│ └── manager.rs # SnapshotManager (disk persistence, load on startup)
├── plugin/ # Plugin API
│ ├── traits.rs # Plugin trait
│ ├── manager.rs # PluginManager
│ └── context.rs # PluginContext
├── hebrew/ # Hebrew language engine
├── greek/ # Greek (Koine) engine
├── geez/ # Ge'ez engine
├── api/ # REST API
│ ├── server.rs # Server startup + DB init
│ └── routes.rs # Axum route handlers
├── import/ # Data import (JSON, CSV, TSV)
└── export/ # Data export (JSON, CSV, TSV)
The engine never depends directly on a database. All persistence goes through the StorageEngine trait:
#[async_trait]
pub trait StorageEngine: Send + Sync {
async fn get_word(&self, id: i64) -> Result<Option<WordEntry>>;
async fn save_word(&self, entry: &WordEntry) -> Result<WordEntry>;
async fn delete_word(&self, id: i64) -> Result<()>;
async fn list_words(&self, lang: &str, limit: i64, offset: i64) -> Result<Vec<WordEntry>>;
async fn search_words(&self, query: &str, lang: &str, limit: i64) -> Result<Vec<WordEntry>>;
async fn get_senses(&self, word_id: i64) -> Result<Vec<WordSense>>;
async fn save_sense(&self, sense: &WordSense) -> Result<WordSense>;
async fn get_translations(&self, word_id: i64) -> Result<Vec<WordTranslation>>;
async fn save_translation(&self, translation: &WordTranslation) -> Result<WordTranslation>;
async fn get_versions(&self, entity_type: &str, entity_id: i64) -> Result<Vec<DataVersion>>;
async fn save_version(&self, version: &DataVersion) -> Result<DataVersion>;
async fn initialize(&self) -> Result<()>;
async fn health_check(&self) -> Result<()>;
}| Backend | File | Use Case |
|---|---|---|
| SQLite | src/storage/sqlite.rs |
Default, zero-config, portable |
| Memory | src/storage/memory.rs |
Testing, no persistence |
| PostgreSQL | (planned) | Production, multi-client |
let storage = StorageFactory::create(&settings).await?;10 specialized in-memory maps for O(1) lookups:
| Map | Key → Value | Purpose |
|---|---|---|
entries |
i64 → WordEntry |
Primary store |
language_entries |
String → Vec<i64> |
Language-scoped queries |
strong_to_entry |
String → Vec<i64> |
Strong's number lookup |
root_to_entry |
String → Vec<i64> |
Root → word lookup |
lemma_to_entry |
String → Vec<i64> |
Lemma → word lookup |
gloss_to_entry |
String → Vec<i64> |
Gloss search |
semantic_to_entry |
String → Vec<i64> |
Semantic domain lookup |
transliteration_to_entry |
String → Vec<i64> |
Transliteration lookup |
stripped_lemma_to_entry |
String → Vec<i64> |
Diacritics-stripped lookup |
entry_to_senses |
i64 → Vec<WordSense> |
Entry → senses lookup |
Plus specialized indexes:
- Trie (
src/indexes/trie.rs): Radix/PATRICIA Trie for prefix search in O(m) where m = query length - BKTree (
src/indexes/bktree.rs): Fuzzy search by Levenshtein distance
5-level cache hierarchy:
| Level | Type | Size | Purpose |
|---|---|---|---|
| L1 | DashMap<String, WordEntry> |
~8K | Strong number → WordEntry |
| L2 | DashMap<String, Vec<i64>> |
~5K | Lemma → entry IDs |
| L3 | LruCache<String, Vec<SearchResult>> |
~10K | Search results |
| L4 | LruCache<String, MorphologyData> |
~5K | Morphology analysis |
| L5 | LruCache<String, WordTranslation> |
~5K | Translations |
Full lexical graph for word relationships:
pub trait GraphEngine {
fn get_synonyms(&self, word_id: &str) -> Vec<WordEntry>;
fn get_antonyms(&self, word_id: &str) -> Vec<WordEntry>;
fn get_cognates(&self, word_id: &str) -> Vec<WordEntry>;
fn get_derivations(&self, root: &str) -> Vec<WordEntry>;
fn shortest_path(&self, from: &str, to: &str) -> Option<GraphPath>;
fn clusters(&self, root: &str) -> Vec<Vec<WordEntry>>;
}Automatically selects the optimal index based on query type, pattern length, and estimated selectivity:
pub fn plan_query_with_stats(request: &SearchRequest, stats: &IndexStats) -> QueryPlan {
let query_type = analyze_query_type(request);
let index = select_index_with_stats(&query_type, request.query.len(), stats);
let selectivity = estimate_selectivity(&query_type, &request.query, stats);
// High-selectivity queries fall back to storage to avoid scanning large result sets
// Low-selectivity queries use in-memory indexes (Trie, BKTree, RadixTree)
}| Query Type | Index | Cost Factor |
|---|---|---|
| Exact | HashMap | 1 (direct lookup) |
| Prefix | Trie | 10 (compressed edge traversal) |
| Fuzzy | BKTree | 50 (Levenshtein tree walk) |
| Substring | RadixTree | 100 (compressed edge scan) |
| Regex/Suffix | SQL | 1000 (full scan fallback) |
See Configuration for full details.
Priority chain: Defaults → Config file → CLI flags → Env vars
# SQLite (default — zero config, DB created automatically)
cargo run
# PostgreSQL
LE_DATABASE="postgresql://user:pass@localhost/mydb" cargo run
# Custom config file
cargo run -- --config /etc/linguistic-engine/settings.toml[features]
default = ["sqlite", "translation"]
sqlite = ["sqlx/sqlite"]
postgres = ["sqlx/postgres"]
translation = ["reqwest"]
simd = []
full = ["sqlite", "postgres", "translation"]| Operation | Complexity | Backend |
|---|---|---|
| Index lookup | O(1) amortized | DashMap |
| Prefix search | O(m) | Trie |
| Fuzzy search | O(n) with pruning | BKTree |
| Lemma search | O(1) + O(k) | Index |
| SQL fallback | O(n) | SQLite/Postgres |
| Full load | O(n) | Database → Index |
- Storage abstraction: Never depend on database directly
- Index-first: All sub-engines check in-memory index before SQL
- Built-in search: Exact, fuzzy, prefix, suffix, substring all run in-memory
- Radix/PATRICIA Trie: Compressed prefix edges — common prefixes share a single node
- WAL with checksums: Per-entry SHA-256 checksums, configurable fsync, corruption detection
- Versioned snapshots: JSON snapshot files with format version for future migration
- Cost-based query planner: Selectivity estimation, automatic fallback for high-cardinality queries
- WAL mode: SQLite uses Write-Ahead Logging for concurrent reads
- Connection pooling: Both backends use async connection pools
- Feature flags: Compile only what you need