Skip to content

Latest commit

 

History

65 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Witchfinder General woodcut, c. 1600s

Bookfinder General

Search. Download. Extract. Translate. Summarize. Turn any book into an agent skill.

MCP server and research tool. Finds books on Anna's Archive, downloads them, pulls out the text, translates if needed, and writes summaries. Works with any MCP-compatible AI tool (Claude Code, Cursor, Windsurf, Continue.dev, and others).

Python 3.11+ MCP License: MIT


Named after the Witchfinder General. Hunts books, not witches.


Legal Disclaimer: This tool is intended for accessing books and materials you are legally entitled to download. Users are solely responsible for ensuring compliance with applicable copyright laws in their jurisdiction. The authors do not condone or encourage copyright infringement.


Why Bookfinder exists

Three specific failures prompted this tool.

First, Anna's Archive's relevance algorithm matches on author names and incidental keywords as often as on the actual title. Searching "Supplying War Van Creveld" returns an unrelated Patton book on the first page. Searching "Guns of August Tuchman" returns a Chinese translation of something else. Fine if you already know the MD5. Painful if you don't. Bookfinder re-ranks results by query-word overlap in the title before handing them back, so the book you asked for is the book you get.

Second, most PDFs on Anna's Archive are scanned images, not extractable text. pymupdf4llm will happily chew on a 40MB image PDF for an hour and return nothing useful. EPUBs, by contrast, are HTML in a zip: they always extract cleanly. So Bookfinder sorts search results EPUB-first, skips text extraction on PDFs over 25MB by default, and the MCP server's tool descriptions tell the calling model to prefer EPUB. Nothing clever; just avoiding the trap.

Third, LLM summaries of non-fiction default to a recognizable boilerplate: "delve into," "it's worth noting," "navigate the complexities." The summarizer ships with stop-slop rules baked into the prompt, which strip most of the obvious tells. Summaries still read like summaries, but they read less like an LLM wrote them.

Bookfinder doesn't do anything Anna's Archive, pymupdf4llm, and an LLM can't do on their own. It does the glue work: right result, clean text, readable prose. The MCP server exposes ten tools so any MCP-compatible client can call any step directly, or the whole pipeline end-to-end.


What It Does

Search query ──> Anna's Archive ──> Download ──> Extract text ──> Translate ──> Summarize ──> PDF Report
                                        │              │              │              │
                                   mirror fallback   Markdown    Google Translate  stop-slop
                                   (LibGen, IPFS)   (pymupdf4llm)  (if non-English)  rules
Feature Description
Search Search Anna's Archive by title, author, ISBN, language
Download Mirror fallback across LibGen, IPFS, Anna's Archive. Validates every file
Extract PDF/EPUB to Markdown. Prefers EPUBs (real text). OCR fallback for scans
Translate Non-English books auto-translated via Google Translate at download time
Summarize Research summaries with stop-slop rules so they read like a person wrote them
PDF Reports Cover pages, headings, page numbers. Proper typography
Topic Briefs Combine multiple books into one research brief
Library Organized collection with metadata, full-text search, git sync
Book → Skill Turn any downloaded book into a reusable agent skill — frameworks, glossary, chapter index, even a calculator that runs the book's own method (details)

Book → Skill

Downloading a book is the start, not the end. The skills/ directory turns a book you've pulled into the library into a reusable skill your AI loads on demand — built from the actual text, not its training-data memory of the book.

Two kinds come out, depending on the book:

  • A study-skill for any clean book: the author's frameworks, a glossary, and a chapter index that loads the right chapter when you ask about a topic.
  • A methodology-skill for books with a quantitative model: all of that, plus a small calculator that runs the book's method. Every number is tagged with where it came from, and the calculator ships a selftest that has to reproduce the book's own worked figures — if it can't, it isn't trusted.

The catch is extraction quality, and the prepare_book_for_skill tool checks it first. A clean EPUB becomes a real calculator; a scanned-image PDF, whose tables come out as noise, becomes a reference at best — so the probe tells you to re-grab the EPUB before building anything on sand. (Same reason search sorts EPUB-first.)

Two worked examples ship in skills/examples/, both built from library books:

  • macedonian-logistics-advisor (Engels, Alexander the Great and the Logistics of the Macedonian Army) — computes an army's daily grain/water/forage, the pack-animals needed to carry it, how far it can march from a depot, and whether a desert crossing is survivable. Its selftest reproduces Engels's own pack-animal counts to within a rounding error.
  • macedonian-phalanx-advisor (Taylor, The Macedonian Phalanx) — computes formation frontage and depth, how many sarissa-points project from the hedge, the unit breakdown from file to full phalanx, and frontal density against a Roman line.

See skills/README.md for the generator recipe.


Quick Start## Quick Start

Install

git clone https://github.com/lerugray/bookfinder-general.git
cd bookfinder-general
pip install -e .

Optional extras:

pip install -e ".[browser]"    # Browser automation (needed for search — Cloudflare bypass)
playwright install chromium

pip install -e ".[ocr]"        # OCR for scanned PDFs (RapidOCR, works on Python 3.11–3.14)

Then check your install:

python -m bookfinder_general doctor

One table: every dependency, the browser binary, your key, the write paths — each with the exact command to fix it. If something misbehaves later, run this first.

Note: Playwright is required for searching Anna's Archive (Cloudflare protection). An ANNAS_KEY lets you skip Playwright for downloads but not for search.

Use

Web UI

Double-click START.bat or run:

python app.py

Opens at localhost:5000.

MCP Server

After pip install -e ., bookfinder-general is available system-wide. Add to .mcp.json in any project:

{
  "mcpServers": {
    "bookfinder-general": {
      "command": "python",
      "args": ["-m", "bookfinder_general.mcp_server"]
    }
  }
}

No cwd needed — the package is installed, so python -m works from anywhere.

Setting your API key

Set ANNAS_KEY as an environment variable so it's available everywhere:

# Linux/Mac — add to ~/.bashrc or ~/.zshrc
export ANNAS_KEY=your_key_here

# Windows — set permanently
setx ANNAS_KEY your_key_here

For Claude Code specifically, add it to ~/.claude/settings.json:

{
  "env": {
    "ANNAS_KEY": "your_key_here"
  }
}

This injects the key into all MCP server processes automatically — no need to put it in every .mcp.json.

Then ask your AI:

"Find me books about 17th century star fort design"

"Download that second result and tell me what it says about defensive geometry"

"Summarize the Moltke book, focus on tactical observations about artillery placement"

"Create a research brief combining all three books on Italian unification"


MCP Tools

Tool Purpose
search_books Search Anna's Archive with language, format, and content type filters
download_book Download + extract text + translate — saves to your research library
read_book Read a book's extracted text (returns English translation for foreign books)
list_library Browse and filter your research library
search_book_content Full-text search across all downloaded books
summarize_book Prepare book content with summary instructions and stop-slop rules
summarize_topic Prepare multi-book content for topic synthesis
save_book_summary Save a generated summary as Markdown + polished PDF
save_research_brief Save a cross-book topic brief as Markdown + PDF
prepare_book_for_skill Health-check a library book and prep it for the Book → Skill generator

Library Structure

Books are organized in ~/Research/BookFinder/ (configurable):

~/Research/BookFinder/
│
├── der-italienische-feldzug-1859-moltke-11ca3de1/
│   ├── original.pdf              # Downloaded file
│   ├── content.md                # Extracted text as Markdown
│   ├── content_en.md             # English translation (if non-English)
│   ├── summary.md                # Research summary
│   ├── summary.pdf               # Polished PDF report
│   └── metadata.json             # Title, author, year, source, etc.
│
└── _topics/
    └── italian-unification-wars/
        ├── summary.md            # Cross-book synthesis
        ├── summary.pdf           # PDF research brief
        └── metadata.json         # Topic metadata + source book IDs

How It Works

Cloudflare Bypass

Anna's Archive sits behind Cloudflare. With an ANNAS_KEY, downloads go through their fast API and no browser is needed. Without a key, Playwright drives a headless Chromium to get past the challenge. Set BOOKFINDER_HEADLESS=false if you need to see the browser window.

Download Validation

Downloads are checked against magic bytes (%PDF, PK for EPUBs, etc.) to catch HTML error pages and paywalls. If the fast API fails, falls back to LibGen and other mirrors. Tries multiple download servers and storage paths before giving up.

Mirror Rotation

Tries .gd, .gl, .pk mirrors in order. Domains rotate due to legal pressure — update config.py if they change.

Text Extraction

Search results put EPUBs first. EPUBs are HTML in a zip file, so they always have real text regardless of file size. PDFs are a gamble: small ones usually have embedded text, large ones are often scanned page images with nothing to extract.

pymupdf4llm handles text-based PDFs. BeautifulSoup handles EPUBs, including non-standard formats that use .xml content files instead of .xhtml. For scanned PDFs, install the optional [ocr] extra and RapidOCR kicks in when regular extraction comes back empty.

Translation

Non-English books get translated via Google Translate at download time. Good enough for AI analysis — not publication-grade.

Library Sync

Set BOOKFINDER_SYNC=true to auto-commit and push after each download. Keeps extracted text synced across machines without the large original files. Set up:

cd ~/Research/BookFinder
git init && git remote add origin git@github.com:you/your-library.git

Add **/original.* to .gitignore to keep the repo lightweight — only extracted text and metadata get pushed.

Summary Generation

Summaries run through embedded stop-slop rules that strip out AI writing patterns so the output reads like a person wrote it. PDF reports use fpdf2 with cover pages, headings, and page numbers.


Configuration

Variable Description Default
ANNAS_KEY Anna's Archive membership key (recommended) (none)
BOOKFINDER_LIBRARY Path to your research library ~/Research/BookFinder
BOOKFINDER_HEADLESS Set to false for visible browser window true
BOOKFINDER_SYNC Auto-commit library to git after downloads false

Anna's Archive membership key (recommended)

A donation to Anna's Archive (~$5-20) gets you an API key. Downloads go through their fast API instead of scraping download pages (Playwright is still needed for search):

# Windows
set ANNAS_KEY=your_key_here

# Linux/Mac
export ANNAS_KEY=your_key_here

Tech Stack

Component Library Purpose
Browser Playwright Cloudflare bypass
Extraction pymupdf4llm PDF/EPUB to Markdown
OCR RapidOCR Scanned PDF fallback (optional)
Translation deep-translator Google Translate
PDF Reports fpdf2 Summary PDF generation
MCP Server MCP SDK Works with any MCP client
Web UI Flask Browser interface
CLI Rich Terminal interface
Parsing BeautifulSoup HTML parsing
Writing Quality stop-slop Anti-AI-slop rules (embedded)

License

MIT — see LICENSE.


🎸 Witchfinder General — Witchfinder General

About

Research book finder, downloader, and summarizer - hunts books, not witches.

Resources

Stars

8 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages