A full-text search engine built in Go that ingests and indexes Wikipedia XML dumps using Bleve.
Establishes the core search pipeline on a single node:
- Parse and clean Wikipedia's XML dump format
- Build a full-text index using Bleve
- Serve search queries over HTTP with ~400ms latency
This phase extends the single-node search from Phase 1 into a distributed architecture. The Wikipedia index is split across 8 independent shard servers, with a coordinator handling parallel fan-out and result merging. Query latency dropped from ~250ms to under 100ms.
| Phase 1 | Phase 2 | |
|---|---|---|
| Index | Single Bleve index | 8 independent shard indexes |
| Search | Sequential, single node | Parallel fan-out across 8 shards |
| Entry point | One server | Coordinator + 8 shard servers |
| Query latency | ~250ms | <100ms |
The ingester deterministically assigns each document to a shard using an MD5 hash on the document ID:
shard = MD5(doc_id)[0] % num_shards
This produces 8 separate JSONL files (shard-0.jsonl ... shard-7.jsonl), each indexed independently into its own Bleve index (index-0 ... index-7).
In Phase 2, every request triggered a full fan-out across all 8 shards, regardless of how repetitive the query traffic was. Phase 3 introduces a Redis cache in front of the fan-out, a singleflight layer to collapse duplicate in-flight requests, and Prometheus metrics to observe coordinator behavior under load.
| Phase 2 | Phase 3 | |
|---|---|---|
| Repeated queries | Always fan out to all 8 shards | Served from Redis cache |
| Concurrent identical misses | Each fired its own fan-out | Collapsed into a single fan-out via singleflight |
| Observability | Log lines only | Prometheus metrics (latency, cache hit/miss) |
Phase 3 cached repeated queries in Redis, but every cache miss still fanned out to all 8 shards — including popular queries executed thousands of times per second. Phase 4 introduces hot-term affinity routing: frequently-queried terms are detected and pinned to a subset of shards (2 out of 8), reducing fan-out overhead on the hottest traffic while maintaining quality.
When a cache miss triggers a cold query (all 8 shards), the coordinator examines the returned results and selects the 2 shards with the highest aggregate BM25 scores. The query's tokens are added to an in-memory hot-term map with a 15-minute TTL. Subsequent cold queries for those tokens bypass the full fan-out and query only the pinned shards. Expired entries are cleaned up every 2 minutes by a background goroutine.
| Phase 3 | Phase 4 | |
|---|---|---|
| Cache miss routing | Always fans out to all 8 shards | Fans out to top 2 shards if term is hot |
| Hot term detection | None | Learned automatically from top-scoring shards |
| Hot term expiry | N/A | 15-minute TTL with periodic cleanup |
| Thread safety | None on routing state | RWMutex protects hot-term map during concurrent access |
- Cold (cache miss): query fans out to all 8 shards; top 2 shards by aggregate score are recorded as hot
- Hot (cache miss): same query routes only to the 2 pinned shards — cuts fan-out by 75%
- Expiry: after 15 minutes the entry is removed from the map; next cache miss triggers a fresh cold phase, re-evaluating which shards are best
- Refreshing: each successful hot hit refreshes the TTL, keeping frequently-queried terms hot indefinitely
- The hot-term TTL (15 min) is intentionally longer than the Redis cache TTL (5 min) to ensure the routing optimization survives multiple cache cycles