librarian

A local-first knowledge base for books. It ingests books in several formats, normalizes them to a single machine- and human-readable canonical form, stores metadata and provenance, builds full-text and semantic indexes, compares new books against the accumulated base, and answers questions with cited sources.

Everything runs locally by default on a laptop (designed for a 24 GB MacBook). S3-compatible object storage is an optional backend, off by default.

---

Why / Goal

Accumulate knowledge from many books while:

• keeping the original text intact (never replaced by a summary),
• tracking provenance (book, page, chapter, offsets, conversion tool),
• adding only what's missing — new material is compared against the base and classified as new, duplicate, extension, contradiction, uncertain, or fluff before you decide to keep it.

Architecture

The pipeline is a sequence of independent, idempotent stages. Each stage reads and writes SQLite + blob storage and can be re-run safely.

                ┌─────────┐   ┌───────────┐   ┌────────┐   ┌───────────┐   ┌─────────┐
  source file → │ ingest  │ → │ normalize │ → │ index  │ → │ knowledge │ → │ compare │
                └─────────┘   └───────────┘   └────────┘   └───────────┘   └─────────┘
   detect+convert   clean/dehyphenate   FTS5 + vector   extract items    vs. the base
   raw+canonical    drop headers/footers  embeddings    topics           statuses
   metadata         classify blocks                                      report/export

Three knowledge layers are kept side by side and never collapsed into one:

original block  ──►  normalized block  ──►  compressed knowledge item
   (verbatim)         (cleaned text)          (concept + summary + claim)

Package layout

librarian/
  cli.py            Typer CLI (entry point: `librarian`)
  app.py            application context + run_all orchestration
  config.py         YAML/TOML + env config (secrets only via env var names)
  logging.py        structured logging (text or JSON lines)
  models.py         shared dataclasses (blocks, metadata, converted doc)
  ids.py            deterministic ids → idempotency
  storage/          BlobStorage: Local (default), S3, Hybrid (cache+remote)
  db/               SQLite connection, migrations, repositories
  ingest/           format detection + converters (text/markdown, epub, pdf, html, docx)
  normalize/        clean, sectionize (headers/footers), classify blocks
  index/            FTS5 + embeddings (pluggable) + local vector index
  knowledge/        extract items, topics, compare against the base
  rag/              hybrid retrieval, citations, optional LLM answer
  export/           Markdown / JSON(L) reports
  cache/            cache stats + LRU prune

Install

Requires Python ≥ 3.11.

python -m venv .venv && source .venv/bin/activate
pip install -e .                 # core (text/markdown ingest, FTS, vector, RAG)
pip install -e '.[formats]'      # + PDF, EPUB, HTML and DOCX support
pip install -e '.[s3]'           # + optional S3 backend
pip install -e '.[embeddings]'   # + optional sentence-transformers
pip install -e '.[ocr]'          # + OCR for scanned PDFs (needs tesseract+poppler)
pip install -e '.[dev]'          # + pytest, ruff, mypy

Quick start

librarian init
librarian import ./examples/book.md --run all
librarian books list
librarian search "congestion window"
librarian semantic-search "controlling the send rate"
librarian review --book <book_id>
librarian export --book <book_id> --mode missing-only --format md
librarian ask "explain TCP congestion control"

This produces:

data/
  librarian.sqlite          metadata, blocks, knowledge, comparisons, citations
  raw/<book_id>/            original source (via blob storage)
  canonical/<book_id>/      manifest.json, book.json, book.md, blocks.jsonl
  indexes/                  blocks.npy / knowledge.npy vector indexes
  cache/  artifacts/  images/  logs/

Commands

Command	Description
librarian init	Create the data dir and database
librarian ingest <path>	Convert a source to canonical form (no downstream stages)
librarian import <path> --run all	Ingest + normalize + index + knowledge + compare
librarian normalize --book <id>	Re-run normalization
librarian index --book <id>	Rebuild FTS + vector indexes
librarian compare --book <id>	Compare against the base
librarian dedup [--threshold 0.9]	Report near-duplicate knowledge clusters across the base
librarian graph [--book id] --format json|dot|graphml	Export the knowledge graph (relations + topics)
librarian adjudicate --book <id> [--show]	LLM-adjudicate contradiction pairs (verdict + reasoning)
librarian review --book <id>	Show status breakdown (-i for the interactive TUI)
librarian search <q>	Full-text (FTS5) search
librarian semantic-search <q>	Vector search
librarian ask <q>	Retrieval + citations (LLM optional)
librarian export --book/--topic --mode <m>	Export `all
librarian serve [--host --port]	Read-only web UI (browse, search, graph JSON)
librarian books list / books show <id>	Catalog
librarian cache stats / cache prune --max-size 50G [--include-blobs]	Cache management (--include-blobs prunes the local blob cache too; s3-backed only)
librarian backup [--out file]	Online backup of the DB (review statuses, adjudications, metadata)
librarian config show	Effective config

Storage modes

Configured in librarian.yaml (next to the data dir or in the CWD), env-overridable.

storage:
  backend: local          # local | s3-backed
  mode: local-only        # local-only (default) | s3-backed | offline

• local-only (default): everything on disk.
• s3-backed: heavy artifacts (raw sources, canonical bundles) replicated to S3 with a local cache; SQLite and indexes stay local.
• offline: never touch S3 even if configured — serve from the local cache.

S3 credentials are never stored in the config or DB — only the names of the env vars holding them:

storage:
  backend: s3-backed
  mode: s3-backed
  s3:
    endpoint_url: "https://..."
    bucket: "librarian"
    region: "auto"
    access_key_env: "LIBRARIAN_S3_ACCESS_KEY"
    secret_key_env: "LIBRARIAN_S3_SECRET_KEY"

Embeddings & vector index — design choices

• Default embedding provider is hashing: a dependency-free, deterministic hashed bag-of-features (word + character trigram) embedding. It needs no model download, runs instantly, and is reproducible in tests. Swap in sentence-transformers, an OpenAI-compatible endpoint (LM Studio's /v1/embeddings, e.g. nomic-embed), or an Ollama endpoint via config — the EmbeddingProvider interface is a single method. Comparison thresholds are configurable per provider (compare.* in the config).
• Vector index is a numpy brute-force cosine store persisted as .npy. Chosen for the MVP because it is exact, inspectable, dependency-light, and fast enough for a personal library (tens of thousands of blocks) well within 24 GB RAM. The small interface makes swapping in hnswlib/Qdrant a local change.

Interactive review

librarian review --book <id> -i opens a terminal UI (stdlib curses, no extra deps) to triage comparison verdicts: navigate with j/k, set a status directly (n/e/d/c/u/f), space to cycle, tab to filter by status, s to save, q to save+quit. On a non-TTY (piped/redirected) it falls back to the text summary. The navigation/edit logic lives in librarian/review/state.py and is unit-tested independently of curses.

Optional local LLM (Ollama or LM Studio)

ask works without any LLM — it returns the retrieved, cited context. To get a synthesized grounded answer, point it at a local LLM. Two providers are supported:

# Ollama
llm:
  enabled: true
  provider: ollama
  endpoint: "http://localhost:11434"
  model: "llama3"

# LM Studio / any OpenAI-compatible server (llama.cpp, vLLM, OpenAI)
llm:
  enabled: true
  provider: openai
  endpoint: "http://localhost:1234/v1"   # include the /v1 suffix
  model: "your-loaded-model-id"

The answer is generated over the same retrieved context and is instructed to cite sources as [1], [2], …. If the endpoint is unreachable the command degrades gracefully to retrieval-only instead of failing. librarian ask <q> --no-llm forces retrieval-only. The LLMClient seam (librarian/rag/llm.py) makes other backends a local change.

When an LLM is configured it is also used to classify block types during normalization (LLMBlockClassifier), and the knowledge extraction stage uses it (LLMKnowledgeExtractor) to produce cleaner concepts/summaries/claims per block, falling back to the heuristic extractor on any failure or malformed output — so the pipeline never breaks or loses the link to the source block.

The compare stage uses the LLM as a judge (LLMComparisonJudge) to adjudicate only the borderline (uncertain) pairs — refining them into duplicate/extension/contradiction/new and shrinking the manual-review pile. Embedding similarity still does the cheap bulk work; the judge is consulted sparingly and any failure leaves the heuristic verdict intact.

Comparison statuses

For each new knowledge item, the best semantic match in the rest of the base determines a preliminary status (no LLM required):

Status	Meaning
new	nothing sufficiently similar in the base
duplicate	near-identical to an existing item
extension	overlaps but adds substantial detail
contradiction	similar topic, opposing polarity (negation, antonym, or numeric clash)
uncertain	partial overlap — needs review
fluff	low-signal/introductory — skip the knowledge layer

Performance notes

• Blocks stream as JSONL; raw sources and images live as files via blob storage, not in SQLite.
• Embeddings are computed in batches.
• Every heavy stage is idempotent and keyed on deterministic ids, so re-runs update in place instead of duplicating.

Error handling

Failures are recorded in logs and the imports table (status, error). A re-run detects an already-imported source (by sha256) and reuses its book_id. Indexes and canonical artifacts can always be rebuilt from the DB.

Development

pip install -e '.[dev,formats]'
# or reproduce the exact known-good environment:
#   pip install -r requirements.lock && pip install -e . --no-deps
pytest          # test suite (incl. end-to-end smoke + S3-via-moto integration)
ruff check .
mypy librarian
./scripts/smoke.sh   # CLI smoke test against examples/book.md

Notes:

• The S3 backend is covered by integration tests using moto (no real network or credentials) — included in the dev extra.
• The sentence-transformers provider is exercised by an optional test that skips automatically if the package or model is unavailable (offline CI).

MVP limitations & roadmap

Implemented: text/markdown, EPUB, text-layer PDF, HTML and DOCX ingest; OCR for scanned PDFs and standalone page images (optional, via tesseract); normalization; FTS5 + vector search; heuristic knowledge extraction; comparison; RAG with citations; optional S3.

Not yet (architected for, but out of scope for the MVP):

• Additional external metadata providers (Open Library ISBN lookup is implemented and off by default — enable with metadata.lookup_enabled)
• Richer graph analytics (a force-layout graph view ships at /graph in serve; the graph also exports as JSON/DOT/GraphML)

Contradiction pairs can be LLM-adjudicated (librarian adjudicate): verdicts (supports_new / supports_existing / both_valid_in_context / unresolved) with confidence and reasoning are stored per pair — prioritizing review, never silently rewriting knowledge.

The LLM layer (block classification, knowledge extraction, compare reranking, RAG answers) is fully implemented and optional — see "Optional local LLM".

Scanned PDFs (no text layer) are detected and, when OCR is disabled, rejected with a clear error rather than producing empty output. Enable OCR with ocr.enabled: true (plus pip install 'librarian[ocr]' and the tesseract / poppler binaries) to ingest them.