Smart Document Reader

A personal research assistant over a local document corpus.

The distinguishing claim here is not the RAG pipeline — those are commodity. It's that document ingestion is treated as the primary engineering problem, and every design choice is backed by measured numbers rather than assertion.

The failure this addresses: most RAG tools run PDFs through a generic text loader and split on character count. Multi-column papers get read across columns, tables collapse into unordered numbers, scanned pages return nothing, and captions detach from figures. Retrieval then searches corrupted text, and the resulting bad answer gets blamed on the model.

No performance numbers are reported below yet — none have been measured. Phase 9 will fill in a benchmark report against a committed corpus, stating hardware, models, and configurations, including ones that perform worse.

Status

• Phase 0 — Scaffold
• Phase 1 — Document type detection
• Phase 2 — Extraction router
• Phase 3 — Structure-aware chunking
• Phase 4 — Indexing and storage
• Phase 5 — Retrieval
• Phase 6 — Answering with span citations
• Phase 7 — Evaluation harness
• Phase 8 — API and interface
• Phase 9 — Benchmark report

Requirements

• Python 3.11+
• Docker (for Postgres + pgvector; docker compose up)

Setup

cp .env.example .env

# Local (non-docker) dev environment
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

pytest

Running with Docker

docker compose up -d
docker compose exec app pytest

This starts Postgres with the pgvector extension (db) and an app container (app). The app container has no long-running process yet — that lands in Phase 8 with the FastAPI service.

Project layout

src/sdr/            application package
  config.py          env-based settings (pydantic-settings)
  logging_setup.py    logging configuration
  detection/           document-type detection from file content
  extraction/           format-appropriate extraction + quality reporting
scripts/             one-off scripts (e.g. fixture generation)
tests/               pytest suite

Document type detection (Phase 1)

sdr.detection.detect_file(path) / detect_bytes(data) classify a document from its content, never its extension — the API doesn't look at the filename at all. It returns a DetectionResult(doc_type, confidence, signals):

• doc_type is one of pdf_text, pdf_scanned, docx, html, plain_text, unknown.
• confidence is a heuristic 0.0–1.0 score, not a calibrated probability.
• signals is a tuple of human-readable evidence (e.g. which magic bytes matched, how many sampled PDF pages had extractable text) for debugging and for later phases' quality reports.

Detection never raises: unreadable, empty, or corrupt files come back as unknown with low confidence and a signal explaining why, instead of an exception. PDFs are told apart from scanned PDFs by opening them with PyMuPDF and checking whether the first few pages yield extractable text — not by file size or page count.

Tests live in tests/test_detection.py against fixtures in tests/fixtures/detection/, generated by scripts/generate_detection_fixtures.py (rerun it if the fixture set needs to change). The fixtures include deliberately mislabeled files (real PDF bytes saved with a .txt extension and vice versa), a corrupt PDF, a non-DOCX zip renamed to .docx, and an empty file.

Extraction router (Phase 2)

sdr.extraction.extract(path) detects the document (reusing Phase 1), routes it to a format-appropriate extractor, and always returns an ExtractedDocument(source_path, doc_type, blocks, quality) — never raises. Each ExtractedBlock carries block_type (heading / paragraph / table), order (reading-order index), section_path (heading hierarchy), page, from_ocr, and, for tables, table_rows (structured, not flattened) and caption.

• PDF reading order: blocks are ordered by detecting a vertical "gutter" (a strip near the page's horizontal center that no block crosses). If one exists and there's content on both sides, the page is treated as two columns and sorted column-by-column, top-to-bottom; otherwise it falls back to a plain top-to-bottom sort. This is a structural heuristic, not a layout model — it's verified against a synthetic two-column fixture, not against real papers yet.
• PDF tables: PyMuPDF's built-in find_tables() (ruling-line based). Rows are kept structured (table_rows); text under a table's bounding box is excluded from surrounding paragraph blocks so it isn't duplicated.
• PDF headings: a block is treated as a heading if its dominant font size is ≥1.15× the document's median span size and it's short — a font-size heuristic, not real style/outline data (PDFs don't reliably expose either).
• Captions: a paragraph immediately preceding a table and matching ^(table|figure|fig\.)\s*\d+ is detached from the paragraph stream and attached as the table's caption instead of kept as an unrelated floating block. HTML tables prefer a native <caption> tag first.
• OCR fallback: any page (regardless of the document-level detected type) that yields fewer than 20 recovered characters is rendered to an image and sent through pytesseract. If tesseract isn't installed, or OCR finds nothing, that's recorded in quality.failures — never silently swallowed. On this dev machine tesseract isn't installed, so tests/test_extraction.py's OCR test asserts the honest "attempted, unavailable" outcome; the Docker image installs tesseract-ocr so the real recovery path runs there.
• DOCX / HTML: heading hierarchy from native styles (Heading N) / tags (h1–h6); tables via native table structures; same caption handling as PDF.

Deviation from the Phase 0 plan: PDF tables use PyMuPDF's native find_tables() instead of the originally stated pdfplumber, since it already ships with the PyMuPDF dependency Phase 1 added and tested out reliably on ruled tables — one fewer dependency for the same result. Flagging this since it wasn't asked about first.

Tests live in tests/test_extraction.py against fixtures in tests/fixtures/extraction/, generated by scripts/generate_extraction_fixtures.py. Fixtures include a two-column PDF (constructed so a naive top-to-bottom sort would visibly interleave the columns wrong), a captioned ruled-line table PDF, an image-only PDF with real rendered text (for the OCR path), and matching structured DOCX/HTML fixtures.

Design decisions carried forward from planning

• Embeddings: sentence-transformers, local model (BAAI/bge-small-en-v1.5 by default, overridable via EMBEDDING_MODEL).
• Answering LLM: Ollama as the default local backend, to keep ingestion and retrieval runnable with no paid API.
• BM25 implementation (Postgres tsvector vs. a standalone library) is an open decision, deferred to Phase 4/5.