PDF to Markdown Toolkit

GitHub Repository

A comprehensive toolkit designed to convert PDF documents into structured, high-quality Markdown files. The conversion process is executed in three distinct stages:

1. PDF to Images: High-resolution conversion of PDF pages into image formats.
2. Image to Clean MD (EN): Automated OCR error correction and translation from Khmer to English using AI models.
3. Markdown Merging: Aggregation of individual markdown pages into a consolidated document.

---

Tools Overview

1. 1-pdf-to-images

Utilizes pdftoppm to convert PDF files into various image formats (PNG, JPEG, TIFF).

Prerequisites

• Haskell Stack
• poppler-utils (pdftoppm required)

  # Debian/Ubuntu
  sudo apt-get install poppler-utils
  # macOS
  brew install poppler

Installation

cd 1-pdf-to-images
stack build

Usage

stack run -- <INPUT_PDF> [OPTIONS]

Option	Shorthand	Description	Default
--output-dir DIR	-o	Directory to save output images	.
--prefix PREFIX	-p	Prefix for output filenames	page
--format FORMAT	-f	Output format: png, jpeg, tiff	png

---

2. 2-img-to-clean-md-en

Automates OCR error correction and translates Khmer markdown content into English while maintaining the original document structure.

Prerequisites

• AI Access (e.g., Google Gemini API or a compatible AI assistant).

Usage

Refer to the instructions and implementation prompts located in 2-img-to-clean-md-en/gemini.prompt.md.

---

3. 3-combine-markdown

Aggregates multiple markdown files from a specified directory into a single document, ensuring correct sequential order (e.g., page-001.md, page-002.md).

Installation

cd 3-combine-markdown
stack build

Usage

stack run -- -i <INPUT_DIR> -o <OUTPUT_DIR> [-n <FILE_NAME>]

The output document will be saved as <OUTPUT_DIR>/<FILE_NAME or INPUT_DIR_NAME>.md.

---

Workflow Example (Manual)

To convert a standard PDF (such as Khmer legal text) to clean, translated markdown, follow this recommended procedure:

0. Preparation: Create a dedicated project directory within temp/ and establish the required output subdirectories:

# Replace 'my-pdf-name' with the actual document identifier
mkdir -p temp/my-pdf-name/1-output-images
mkdir -p temp/my-pdf-name/2-clean-markdown
mkdir -p temp/my-pdf-name/2.1-en-markdown
mkdir -p temp/my-pdf-name/3-combine-markdown

# Copy the source PDF into the project directory
cp path/to/your/document.pdf temp/my-pdf-name/

1. Stage 1: Image Generation:

   cd 1-pdf-to-images
   stack run -- ../temp/my-pdf-name/document.pdf \
     -o ../temp/my-pdf-name/1-output-images \
     -p page -f png

2. Stage 2: Processing and Translation: Utilize the provided instructions in 2-img-to-clean-md-en/gemini.prompt.md to process images from temp/my-pdf-name/1-output-images/. Output should be directed to temp/my-pdf-name/2-clean-markdown/ (Khmer) and temp/my-pdf-name/2.1-en-markdown/ (English).

3. Stage 3: Final Document Consolidation: Merge the processed Khmer and English files:

   cd ../3-combine-markdown
   # Consolidate Khmer version
   stack run -- \
     -i ../temp/my-pdf-name/2-clean-markdown \
     -o ../temp/my-pdf-name/3-combine-markdown \
     -n my-pdf-name
   
   # Consolidate English version
   stack run -- \
     -i ../temp/my-pdf-name/2.1-en-markdown \
     -o ../temp/my-pdf-name/3-combine-markdown \
     -n my-pdf-name-en

   This process generates temp/my-pdf-name/3-combine-markdown/my-pdf-name.md and temp/my-pdf-name/3-combine-markdown/my-pdf-name-en.md.

---

Examples

The examples/ directory contains sample legal documents processed with this toolkit, including automation scripts.

---

Automated Workflow with AI Agent

For developers using an agentic AI assistant, the entire digitization cycle can be fully automated using the coordinator prompt.

The coordinator dynamically resolves the next target PDF alphabetically, sets up the workspace, extracts images, coordinates parallel subagents for page digitization, combines the final markdown files, and manages the lifecycle directories.

Setup & Directory Layout

All active workspaces are stored under the temp/ folder:

• temp/progress/: Contains the workspace of the law currently being digitized.
• temp/done/: Contains the workspaces of completed laws (acts as a step memory for successor resolution).

Reusable Tools

The coordinator orchestrates the workflow using these generic scripts in the tools/ directory:

• tools/get-next-pdf.sh: Auto-detects the successor PDF target using temp/done/ as memory.
• tools/init-workspace.sh: Creates folders in temp/progress/ and copies the source PDF.
• tools/pdf-to-images.sh: Converts PDF to page images.
• tools/digitize.procedure.md: The page OCR and English translation instructions template.
• tools/combine-md.sh: Merges individual pages into final unified markdown documents.

How to Use the Coordinator

1. Instruct your agentic AI assistant to read coordinator.procedure.md.
2. The AI assistant will automatically:
   • Identify the next PDF target alphabetically.
   • Run the setup and image extraction scripts.
   • Spawn parallel subagents to process the page chunks.
   • Merge the finished outputs.
   • Move the workspace directory to temp/done/ when completed.

---

License

This project is licensed under the GNU General Public License v2.0. For more details, see the LICENSE file.