Project Blue Book

OCR + translation + EPUB pipeline for large physics books (Markdown/LaTeX heavy), with:

• OCR sources (Marker and Docling)
• safe LaTeX-preserving translation via local Ollama (mistral-small3.2)
• chapter splitting / resume support
• EPUB generation with Pandoc
• optional hybrid formula recovery (Marker formulas injected into Docling output)

This repository was built around Landau & Lifshitz, Statistical Physics (Vol. 5), but the pipeline is reusable for other technical books.

What This Project Solves

Large OCR-generated Markdown files from scientific books usually fail in one or more ways:

• OCR damages LaTeX
• LLM translation rewrites or deletes formulas
• image paths break when reassembling EPUB
• long runs fail mid-way (4-6h+) with no resume

pipeline_bluebook.py addresses these issues with:

• regex cleanup for OCR/LaTeX
• chapter splitting + safety sub-splitting
• token masking for formulas / code / Docling structures
• retry + adaptive fallback on token corruption
• checkpoints and resume
• final EPUB assembly and Pandoc compilation

Repository Files

• pipeline_bluebook.py: main OCR Markdown -> translated Markdown -> EPUB pipeline
• run_ocr_marker.py: Marker-based OCR script (good formula extraction)
• run_ocr_docling.py: Docling-based OCR script (good structure/images, self-healing fallback)
• run_epub_to_md.py: EPUB -> Markdown + media extraction (Pandoc-based, no OCR)

Quick Commands (Landau)

Marker OCR (formula-first OCR source)

Use this when you want the best possible math extraction. In the hybrid workflow, this Markdown is the formula source used to fill Docling's formula-not-decoded placeholders during final assembly (--formula-source-md).

python3 .../project_blue_book/run_ocr_marker.py \
  .../05StatisticalPhysicsPart.1.pdf \
  --out-md .../LANDAU_COMPLET.md \
  --temp-dir .../temp_marker \
  --title "Physique Statistique - Landau (Volume 5)" \
  --chunk-size 50 \
  --layout-batch-size 4 \
  --equation-batch-size 4 \
  --quiet-marker

Docling OCR (structure/images-first OCR source)

Use this when you want richer Markdown structure and extracted figures/images. In the hybrid workflow, this Markdown is the base text/layout source (chapters, prose, image references), and Marker is used only to restore missing formulas.

python3 .../project_blue_book/run_ocr_docling.py \
  .../05StatisticalPhysicsPart.1.pdf \
  --out-dir .../project_blue_book/Landau_Docling_FINAL \
  --md-name Landau_COMPLET.md \
  --title "Physique Statistique - Landau (Volume 5)" \
  --chunk-size 40

EPUB -> Markdown (no OCR, native EPUB source)

Use this when the source is already a readable EPUB and you want a simpler translation path (EPUB -> Markdown -> pipeline). This helper can also copy your custom generated cover (for example, a diffusion/Kandinsky-style cover) into the same output folder for reuse in the final rebuild.

python3 .../project_blue_book/run_epub_to_md.py \
  .../some_book.epub \
  --out-dir .../project_blue_book/Epub_Source \
  --md-name SOURCE_FROM_EPUB.md \
  --title "Source Book Title" \
  --cover-image .../my_kandinsky_cover.png \
  --cover-name cover_custom.png

Then translate/rebuild with the main pipeline:

python3 .../project_blue_book/pipeline_bluebook.py \
  .../project_blue_book/Epub_Source/SOURCE_FROM_EPUB.md \
  --asset-source-dir .../project_blue_book/Epub_Source \
  --cover-image .../project_blue_book/Epub_Source/cover_custom.png \
  --model mistral-small3.2 \
  --max-concurrency 1 \
  --translation-chunk-chars 2200 \
  --translation-max-retries 3 \
  --pandoc-math-method mathml

OCR Options (Marker vs Docling)

1) Marker OCR (run_ocr_marker.py)

Best when formula fidelity is the priority.

Strengths:

• usually better math/formula extraction for this Landau PDF
• excellent $$...$$ formula source for hybrid repair of Docling placeholders (--formula-source-md)

Weaknesses:

• image extraction/layout handling can be less convenient than Docling
• depends on the local marker_single CLI behavior/output layout
• less convenient than Docling as the primary source when you need rich images/structure preserved

How to use (Landau example):

python3 .../project_blue_book/run_ocr_marker.py \
  .../05StatisticalPhysicsPart.1.pdf \
  --out-md .../LANDAU_COMPLET.md \
  --temp-dir .../temp_marker \
  --title "Physique Statistique - Landau (Volume 5)" \
  --chunk-size 50 \
  --layout-batch-size 4 \
  --equation-batch-size 4 \
  --quiet-marker

Typical output:

• Marker Markdown file (example): .../LANDAU_COMPLET.md

2) Docling OCR (run_ocr_docling.py)

Best when structure + images are the priority.

Strengths:

• exports richer Markdown structure
• extracts and saves images more conveniently
• self-healing fallback (page-by-page) when a chunk fails
• best base source for final prose/layout/images in the hybrid Docling+Marker workflow

Weaknesses:

• may leave formula-not-decoded placeholders for math-heavy pages
• often needs formula reinjection from Marker (--formula-source-md) for best EPUB quality

How to use (Landau example):

python3 .../project_blue_book/run_ocr_docling.py \
  .../05StatisticalPhysicsPart.1.pdf \
  --out-dir .../project_blue_book/Landau_Docling_FINAL \
  --md-name Landau_COMPLET.md \
  --title "Physique Statistique - Landau (Volume 5)" \
  --chunk-size 40

Outputs:

• Markdown: .../Landau_Docling_FINAL/Landau_COMPLET.md
• Images: .../Landau_Docling_FINAL/images/

Recommended Setup (Conda)

You can use one environment, or separate OCR/runtime environments if you prefer.

Single environment (recommended)

conda create -n bluebook python=3.11 -y
conda activate bluebook

Install Python dependencies:

pip install langchain-ollama langchain-core tqdm PyPDF2
pip install docling

Optional (legacy fallback for Ollama integration):

pip install langchain-community

Install Pandoc (system package):

sudo apt install pandoc

Optional GPU stack notes

• Ollama uses the GPU automatically if available.
• Start ollama serve with a fixed GPU if needed:

CUDA_VISIBLE_DEVICES=0 ollama serve

Check usage:

ollama ps
nvidia-smi

Ollama Model

Current default model in the pipeline:

• mistral-small3.2

Test it manually:

ollama run mistral-small3.2

Main Pipeline (pipeline_bluebook.py)

What It Does

1. OCR Markdown sanitization (LaTeX-safe regex cleanup)
2. chapter detection + split
3. optional translation through Ollama with protected tokens
4. final Markdown assembly
5. EPUB compilation with Pandoc

Important Safety Features

• Formula/code/image/HTML masking before translation
• token integrity validation after translation
• retries + adaptive chunk re-splitting
• chapter skip if already translated
• checkpoint file per chapter (.checkpoint.json)
• resume after Ctrl+C / crash

Translation / EPUB Workflows

A) Full pipeline (Docling source + Marker formulas + translation + EPUB)

This is the recommended Landau workflow. Docling provides the base Markdown (structure, prose, image references), while Marker provides formulas used to replace Docling placeholders during assembly.

python3 .../project_blue_book/pipeline_bluebook.py \
  .../project_blue_book/Landau_Docling_FINAL/Landau_COMPLET.md \
  --formula-source-md .../LANDAU_COMPLET.md \
  --asset-source-dir .../project_blue_book/Landau_Docling_FINAL \
  --chapters-raw-dir .../chapters_raw \
  --chapters-translated-dir .../chapters_translated \
  --final-md .../project_blue_book/LANDAU_FINAL.md \
  --final-epub .../project_blue_book/LANDAU_FINAL.epub \
  --cover-image .../couverture_landau_5.png \
  --model mistral-small3.2 \
  --max-concurrency 1 \
  --translation-chunk-chars 2200 \
  --translation-max-retries 3 \
  --pandoc-math-method mathml

B) OCR cleanup + split + assembly only (no translation)

Useful for testing OCR quality and EPUB rendering.

python3 .../project_blue_book/pipeline_bluebook.py \
  /path/to/OCR_SOURCE.md \
  --skip-translate

C) Rebuild final Markdown + EPUB from existing translated chapters (no Ollama)

Use this after a translation run is already done (or mostly done), to iterate quickly on assembly/Pandoc fixes.

python3 .../project_blue_book/pipeline_bluebook.py \
  .../project_blue_book/Landau_Docling_FINAL/Landau_COMPLET.md \
  --assemble-only \
  --chapters-raw-dir .../chapters_raw \
  --chapters-translated-dir .../chapters_translated \
  --formula-source-md .../LANDAU_COMPLET.md \
  --asset-source-dir .../project_blue_book/Landau_Docling_FINAL \
  --final-md .../project_blue_book/LANDAU_FINAL.md \
  --final-epub .../project_blue_book/LANDAU_FINAL.epub \
  --cover-image .../couverture_landau_5.png \
  --pandoc-math-method mathml

D) Build final Markdown only (skip Pandoc)

python3 .../project_blue_book/pipeline_bluebook.py \
  /path/to/OCR_SOURCE.md \
  --assemble-only \
  --skip-pandoc \
  --chapters-translated-dir /path/to/chapters_translated

E) EPUB source -> Markdown -> translation (no OCR)

Use this when your source is already an EPUB (native text). run_epub_to_md.py extracts a Markdown source and media files, then pipeline_bluebook.py handles translation/reassembly.

1. Convert EPUB to Markdown + media:

python3 .../project_blue_book/run_epub_to_md.py \
  .../some_book.epub \
  --out-dir .../project_blue_book/Epub_Source \
  --md-name SOURCE_FROM_EPUB.md \
  --title "Source Book Title" \
  --cover-image .../my_kandinsky_cover.png \
  --cover-name cover_custom.png

2. Translate and rebuild EPUB:

python3 .../project_blue_book/pipeline_bluebook.py \
  .../project_blue_book/Epub_Source/SOURCE_FROM_EPUB.md \
  --asset-source-dir .../project_blue_book/Epub_Source \
  --chapters-raw-dir .../chapters_raw \
  --chapters-translated-dir .../chapters_translated \
  --final-md .../project_blue_book/BOOK_FINAL.md \
  --final-epub .../project_blue_book/BOOK_FINAL.epub \
  --cover-image .../project_blue_book/Epub_Source/cover_custom.png \
  --model mistral-small3.2 \
  --max-concurrency 1 \
  --translation-chunk-chars 2200 \
  --translation-max-retries 3 \
  --pandoc-math-method mathml

Resume / Restart Behavior (Important)

Translation runs can take several hours. The pipeline supports resume:

• If chapters_translated/<chapter>.md already exists -> chapter is skipped
• If <chapter>.md.checkpoint.json exists and matches source/chunk settings -> resume inside that chapter

To preserve resume behavior:

• keep --chapters-translated-dir stable (prefer absolute paths)
• keep --translation-chunk-chars unchanged between restarts
• do not delete checkpoint files unless you want a fresh restart

Output Files

By default (unless overridden):

• LANDAU_FINAL.md
• LANDAU_FINAL.epub
• LANDAU_FINAL_assets/ (copied image assets for EPUB portability)
• LANDAU_FINAL_epub.css (EPUB stylesheet used by Pandoc)

Output Example

Example rendered EPUB output (Calibre viewer):

Notes on Formulas and Docling Placeholders

Docling may emit hidden placeholders like:

• <!-- formula-not-decoded -->

The pipeline can:

• make them visible (instead of silently disappearing in EPUB)
• replace most of them using --formula-source-md (Marker Markdown with $$...$$ formulas)

If you still see a few placeholders, it usually means:

• Marker and Docling streams do not align perfectly at those positions
• or Marker did not output a matching display formula for that placeholder

Logging Language

Runtime logs are in English for public/general use.

The LLM translation prompts remain in French intentionally (to preserve translation behavior/quality for the target output).

Known Limitations

• Some legacy TeX patterns may still fail Pandoc MathML conversion
• OCR formula recovery is heuristic when merging Marker formulas into Docling placeholders
• run_ocr_marker.py relies on the external marker_single CLI output layout