Hybrid Search

Local hybrid search (BM25 + vector) for MCP-compatible agents and plain HTTP clients. Long-term memory and retrieval over your own notes, code, and docs. See PLAN.md for the design.

What you get

Hybrid retrieval — SQLite FTS5 (BM25) + Qdrant (cosine) fused with Reciprocal Rank Fusion; optional cross-encoder reranker.
MCP server — search, index_path, list_sources, get_chunk tools + lore://sources / lore://source/{path} resources.
HTTP API — POST /search, POST /index, GET /jobs/{id}, DELETE /documents, GET /health.
Incremental indexing — watchdog-backed watcher debounces editor saves, respects .gitignore, prunes common noise dirs.
Single CLI — lore init/up/down/restart/status/projects/tui/index/query/serve-api/serve-mcp/watch.
User-scoped data — everything under ~/.lore/ by default.

Requirements

macOS or Linux.
Python 3.11 or 3.12.
Docker (Docker Desktop on macOS). Qdrant runs as a container.
About 300 MB free disk: ~130 MB for the embedding model cache, the rest for Qdrant + SQLite.

Install

Option A — pipx (recommended once 0.5.0 is on PyPI)

pipx install lore-memory
lore init
lore up --watch

pipx installs Lore in its own virtualenv and puts the lore binary on your $PATH automatically.

Option B — Homebrew (macOS)

brew install reipano/lore/lore

Requires the tap — it mirrors the PyPI release. Auto-updates when you run brew upgrade.

Option C — Docker

docker run -p 8765:8765 -v ~/.lore:/data ghcr.io/reipano/lore:latest \
  lore serve-api --host 0.0.0.0

Multi-arch images are published by the release workflow on every v* tag.

Option D — From source (clone and install)

cd /path/where/you/keep/tools
git clone <repo-url> better-mem
cd better-mem
make install

make install does three things:

Creates .venv/ with Python 3.12.
Installs the package + dev extras.
Runs lore init, which:
- Creates ~/.lore/ and ~/.lore/qdrant/.
- Copies config.yaml to ~/.lore/config.yaml.
- Prints symlink + $PATH hints.

2. Put the CLI on your `$PATH`

The init command suggests a symlink. Example:

mkdir -p ~/.local/bin
ln -s /ABSOLUTE/PATH/TO/better-mem/.venv/bin/lore ~/.local/bin/lore

Ensure ~/.local/bin is in $PATH. If not, for zsh:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

Verify:

which lore
lore --help

3. Sanity-check tests (optional)

cd /path/to/better-mem
make test

First run

lore up --watch          # starts Qdrant + (empty) watcher
lore status              # shows qdrant yes, watcher stopped, 0 chunks

lore projects add ~/notes --name notes
# ^ downloads bge-small-en-v1.5 (~130 MB, one time)
# ^ indexes all supported files under ~/notes

lore status              # chunk count > 0
lore query "something you wrote"

After your first projects add, restart services so the watcher picks up the new project:

lore restart

Daily usage

lore up --watch          # morning
# ... work ...
lore restart             # after editing config, or if something feels stale
lore status              # any time
lore down                # evening

Cheatsheet

Command	What it does
`lore init`	Bootstrap `~/.lore/` and print PATH hints.
`lore up`	Start Qdrant (Docker).
`lore up --watch`	Start Qdrant and watcher daemon.
`lore down`	Stop watcher + Qdrant.
`lore restart`	Stop + restart watcher; start Qdrant if down.
`lore restart --full`	Also cycle Qdrant.
`lore restart --no-watch`	Stop the watcher only.
`lore status`	Table: Qdrant, watcher PID, chunk count, sources.
`lore projects list`	Show registered projects.
`lore projects add <path>`	Register a project and index it.
`lore projects add-all <path>`	Register every direct subfolder of `<path>` as its own project.
`lore projects remove <name>`	Unregister + drop chunks.
`lore projects reindex [<name>]`	Re-run indexing for one or all.
`lore index <path>`	One-shot index of a path (no registration).
`lore query <text>`	Ad-hoc query; prints JSON results.
`lore serve-api`	Run the FastAPI HTTP server.
`lore serve-mcp`	Run the FastMCP stdio server (used by Claude).
`lore watch [<paths>]`	Foreground watcher (blocks).
`lore tui`	Interactive menu.
`lore bench [args...]`	Run the token-usage benchmark (forwards args to `eval.token_bench.run_bench`).

Managing projects

Projects are named paths. Metadata lives in ~/.lore/projects.json.

Add

lore projects add ~/notes --name notes
lore projects add ~/code/app --name app
lore projects add ~/scratch --no-watch            # index once, do not watch
lore projects add ~/big-repo --skip-index         # register only; index later

Name is auto-derived from the path if you omit --name. Adding the same path twice updates in place.

List / remove / reindex

lore projects list
lore projects remove app                          # also drops chunks
lore projects remove app --keep-index             # keep chunks
lore projects reindex                             # all
lore projects reindex notes                       # one

Bulk add

Use add-all to register every direct subfolder of a path as its own project:

lore projects add-all ~/Projects/front-end
lore projects add-all ~/Projects/front-end --no-watch
lore projects add-all ~/Projects/front-end --skip-index   # register only

Each subfolder becomes a project whose name is the folder name. Hidden folders (.foo), built-in pruned dirs (.git, node_modules, …), and anything listed in config.yaml:exclude_dirs are skipped. Files directly under the target path are ignored — only immediate subdirectories are registered.

If you need finer control (custom names, selective watch flags), fall back to the loop form:

for dir in ~/Projects/front-end/*/; do
  lore projects add "$dir" --name "$(basename "$dir")"
done

After changing which projects are watched

Bounce the daemon so it picks up new watched paths:

lore restart

Querying

Terminal

lore query "auth middleware" -k 5
lore query "auth middleware" --full               # full chunk text
lore query "auth middleware" --no-rerank          # skip cross-encoder

Output is one JSON object per line with chunk_id, source_path, score, scores_breakdown (bm25 / vector / rrf / rerank contributions), and preview or text.

Interactive menu

lore tui

Menu-driven wrapper around the same commands.

Claude Code / Claude Desktop (MCP)

Register with Claude Code

claude mcp add lore -- \
  /ABSOLUTE/PATH/TO/better-mem/.venv/bin/lore \
  --config ~/.lore/config.yaml \
  serve-mcp

Order matters: --config goes before serve-mcp. Use absolute paths — Claude Code subprocess spawn does not expand ~ or honor your $PATH.

Verify:

claude mcp list | grep lore       # expect: ✓ Connected

Use Claude Code normally — tools search / index_path / list_sources / get_chunk appear automatically.

Register with Claude Desktop

Copy examples/claude_desktop_config.json, fix the absolute paths, drop it into ~/Library/Application Support/Claude/claude_desktop_config.json (macOS), and restart Claude Desktop.

{
  "mcpServers": {
    "hybrid-search": {
      "command": "/ABSOLUTE/PATH/TO/better-mem/.venv/bin/lore",
      "args": ["--config", "/Users/<you>/.lore/config.yaml", "serve-mcp"]
    }
  }
}

HTTP API

lore serve-api                 # default :8765

curl http://127.0.0.1:8765/health
curl -X POST http://127.0.0.1:8765/search \
  -H 'Content-Type: application/json' \
  -d '{"query":"hybrid search","k":5}'

More examples in examples/curl.sh.

Auth

Set api.auth_token in ~/.lore/config.yaml to enable. Clients send Authorization: Bearer <token>.

LangChain retriever

pip install 'lore-memory[langchain]'

from hybrid_search.integrations.langchain import LoreRetriever

retriever = LoreRetriever(
    base_url="http://127.0.0.1:8765",
    k=5,
    project="notes",          # optional: scope by registered project name
    rerank=True,
)
docs = retriever.invoke("auth middleware")
# Async: docs = await retriever.ainvoke("auth middleware")

Each hit becomes a Document whose metadata carries chunk_id, source_path, score, and the per-component scores_breakdown.

Configuration reference

Edit ~/.lore/config.yaml. Apply changes with lore restart.

Paths

Field	Default	Meaning
`index_path`	`~/.lore/lexical.sqlite3`	SQLite file holding full chunk text + BM25 index.
`log_path`	`~/.lore/lore.log`	Rotating log file (2 MB × 5).
`qdrant_url`	`http://localhost:6333`	Where Qdrant is reachable.
`collection_name`	`hybrid_chunks`	Qdrant collection name.

~ and $HOME are expanded. All paths may be absolute or relative; relative paths are anchored to the cwd where you run the command.

Chunking and retrieval

Field	Default	Meaning
`profile`	`mixed`	Preset that pre-fills the fields below. Valid values: `mixed`, `code`, `docs`, `notes`. Explicit values in the same file always win over the preset.
`chunk_size`	500	Tokens per chunk.
`chunk_overlap`	50	Tokens of overlap between neighbors.
`retrieval_k_per_index`	50	Candidates pulled from BM25 and vector before fusion.
`fusion_k`	60	RRF smoothing constant.
`bm25_weight`	1.0	RRF weight for BM25.
`vector_weight`	1.0	RRF weight for vector.
`rerank_top_n`	20	How many fused candidates the reranker scores.
`rerank_enabled`	true	Toggle cross-encoder reranking.
`default_result_k`	10	`k` when the caller omits it.

Profile summary:

Profile	bm25 / vector	chunk_size / overlap	Best for
`mixed` (default)	1.0 / 1.0	500 / 50	Heterogeneous corpora.
`code`	2.0 / 1.0	400 / 40	Source trees — identifier matches matter.
`docs`	1.0 / 1.5	500 / 50	Long-form prose with headings.
`notes`	1.0 / 2.0	300 / 30	Short journals or jotted thoughts.

Models

Field	Default	Meaning
`embedding_model`	`BAAI/bge-small-en-v1.5`	FastEmbed model. 384 dim, cached at `~/.cache/fastembed`.
`embedding_dim`	384	Must match the model.
`rerank_model`	`BAAI/bge-reranker-v2-m3`	FastEmbed cross-encoder. Automatically falls back to `BAAI/bge-reranker-base` and then `Xenova/ms-marco-MiniLM-L-6-v2` if the primary model fails to download.

Changing the embedding model requires a fresh collection — Qdrant records the model name and refuses mismatches. Rebuild with lore projects reindex after wiping Qdrant data (docker compose down -v).

Files and dirs

Field	Default
`max_file_bytes`	1 MB — files larger are skipped.
`supported_extensions`	Long whitelist including md/js/ts/py/java/go/rs and more.
`exclude_dirs`	Adds to the built-in pruned set (.git, node_modules, …).
`exclude_patterns`	Fnmatch globs run against filename, relative path, and full path.

API

Field	Default
`api.host`	`127.0.0.1`
`api.port`	8765
`api.auth_token`	null

Config resolution order

When any command runs, the config path is resolved in this order:

--config <path> flag.
$HYBRID_SEARCH_CONFIG.
~/.lore/config.yaml (created by init).
./config.yaml in the current directory, walking up through parents.
config.yaml at the repo root (dev checkout fallback).

Exclusions

Built-in pruned directories (always on): .git, .hg, .svn, node_modules, __pycache__, .venv, venv, dist, build, .data, .pytest_cache, .ruff_cache, .mypy_cache.

Default user-configured additions in config.yaml cover React, Angular, Next.js, Nuxt, SvelteKit, Java/Maven/Gradle, JetBrains/VS Code, macOS metadata, and common caches. Edit exclude_dirs / exclude_patterns to taste; see comments inline.

Sensitive-file defaults

config.yaml ships two layers of protection:

exclude_patterns adds common secret-bearing filenames (.env, .env.*, *.pem, *.key, id_rsa*, id_ed25519*, credentials.json, .npmrc, .pypirc, secret.yaml).
exclude_content_patterns is a regex list applied to the first 32 KB of every candidate file. Defaults match sk-ant-…, sk-proj-…, AKIA…, ghp_…, gho_…, github_pat_…, xox[baprs]-…, and inline PEM private keys. Any match makes the file skip indexing entirely and increments files_skipped_sensitive in the run stats.

Opt out by editing either list (or emptying it) in your user config. Opt in to new defaults after a future upgrade by running lore init --force.

Watcher and indexer share these rules, so ignored paths never enter the index even if you create them while the watcher is running.

Architecture

 files → chunker → BM25 (SQLite FTS5)
                 → Vectors (Qdrant via FastEmbed)
                 → RRF fusion → (optional) cross-encoder rerank → results
                                                   ↑
                                 MCP server / HTTP API / CLI

Chunk IDs are deterministic hashes of (source_path, start_offset, text). Reindexing a file diffs new vs. existing and only reembeds the changed chunks.
SQLite stores the full text, offsets, and metadata. Qdrant stores vectors plus a short preview. Full-text lookups always hit SQLite.
The collection metadata point (id 0) records which embedding model built the index. Opening a collection with a different model raises.

Evaluation

Retrieval quality (hit@10 / MRR / NDCG@10)

.venv/bin/python -m eval.run_eval --fake-embedder          # smoke, no quality
.venv/bin/python -m eval.run_eval                           # real embedder
.venv/bin/python -m eval.run_eval --include-rerank          # also download reranker

Output lands in eval/results.md. Metrics: hit@10, MRR, NDCG@10. Extend eval/queries.jsonl with your real queries; relevance is binary substring match against must_contain.

Token-usage benchmark (plain vs MCP vs MCP+caveman)

.venv/bin/python -m eval.token_bench.run_bench --dry-run              # resolve paths only
.venv/bin/python -m eval.token_bench.run_bench --only plain --limit 1 # cheapest real run
.venv/bin/python -m eval.token_bench.run_bench                         # full matrix
lore bench --only plain --limit 1                             # equivalent via CLI

Drives claude-agent-sdk against the same task list under three configs (plain, mcp, mcp+caveman) and records input + output tokens per task. Needs ANTHROPIC_API_KEY set and Qdrant running for the MCP configs. Full instructions + task-file format in eval/token_bench/README.md. Report lands in eval/token_bench/results.md.

Troubleshooting and pitfalls

First-run gotchas

Model download blocks — first projects add or query triggers a one-time ~130 MB download. Subsequent runs use the ~/.cache/fastembed cache. If Ctrl-C'd mid-download you may re-download next time.
no docker-compose.yml found — lifecycle now falls back to the repo root. If that still fails, make sure the repo still exists where you cloned it.
FileNotFoundError: config not found — run lore init first, or pass --config <path>.

MCP reports "✗ Failed to connect"

Run the server manually to see real errors:
```
/ABS/PATH/.venv/bin/lore --config ~/.lore/config.yaml serve-mcp
```
It should block silently (banner and logs are suppressed on stdio). Ctrl-C to exit.
Common root causes:
- --config placed after serve-mcp in the MCP registration. Move it before.
- Qdrant not running → lore up.
- Absolute paths missing — Claude subprocess spawn does not expand ~ or read your shell $PATH.
- First-time model download hung.

Watcher

"no projects to watch; skipping watcher" — you have no registered projects, or none are marked watch: true. Run lore projects list to confirm. After adding projects, run lore restart.
Daemon silently stops — check ~/.lore/watcher.log. Likely an unhandled exception or the parent lore up was killed.
macOS asks for permission to watch a folder — accept. Watchdog uses FSEvents and needs per-folder access rights.

Indexing

Progress bar stuck at some % — a single big file may be embedding. tail -f ~/.lore/lore.log shows current file. Use py-spy dump --pid $(pgrep -f reindex) to see the exact stack if it truly hangs.
project not found — you passed a name that does not match lore projects list. Use the name, not the path.
Index looks empty after restart — Qdrant container storage lives under ~/.lore/qdrant/ (mounted via HYBRID_QDRANT_DATA). If you reset that directory, run lore projects reindex to rebuild.

Qdrant corruption (migration / power loss)

If index runs flood the log with HTTP/1.1 500 Internal Server Error on /collections/.../points, Qdrant's segment directory is probably in a bad state (partial writes, swapped volume mid-run, OS crash). Recover by wiping the Qdrant bind-mount and reindexing from SQLite:

lore down
rm -rf ~/.lore/qdrant
lore up --watch
lore projects reindex

SQLite keeps the full text + chunk IDs, so only the vector side is rebuilt. Starting with v0.3.0 the indexer fails fast after five consecutive vector-write errors so the issue can't hide behind a silent progress bar.

Data orphans

Changing index_path or embedding_model in config.yaml strands old data. Either migrate files manually or run make reset + reindex.
Qdrant keeps old vectors if you drop only the SQLite file. Hits without SQLite hydration are filtered out at query time (safe), but they waste space. lore down && docker compose down -v && lore up --watch is the nuclear cleanup.

Config

Do not comment out mandatory fields (index_path, qdrant_url, collection_name, embedding_model, embedding_dim, chunk_size, chunk_overlap) — load_config raises on missing keys.
Paths support ~ and $VARS. Other fields do not.

Uninstall

lore down
docker compose down -v                    # drop Qdrant volume
rm -rf ~/.lore
rm ~/.local/bin/lore             # if you symlinked
# then `rm -rf` the repo

Or, if the Makefile is available:

make reset         # stops services and wipes ~/.lore
make clean         # removes .venv

License

MIT — see LICENSE.

Name		Name	Last commit message	Last commit date
Latest commit History 5 Commits
.github/workflows		.github/workflows
eval		eval
examples		examples
packaging/homebrew		packaging/homebrew
src/hybrid_search		src/hybrid_search
tests		tests
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
Dockerfile		Dockerfile
Makefile		Makefile
PLAN.md		PLAN.md
README.md		README.md
config.yaml		config.yaml
docker-compose.yml		docker-compose.yml
pyproject.toml		pyproject.toml

Folders and files

Latest commit

History

Repository files navigation