OCR Benchmark Workbench

Standalone OCR benchmark tools for finance reimbursement documents.

The workbench compares OCR engines on the same local samples and records:

• OCR text and layout block counts
• per-page and per-file elapsed time
• GPU memory peak from nvidia-smi
• warnings/errors/skips
• Markdown and CSV reports

Local Directory Policy

The repository code is versioned, but local test data and generated artifacts are not.

• data/: local benchmark samples. Ignored by Git.
• storage/: rendered pages and OCR text outputs. Ignored by Git.
• runs/: reports and metrics for each run. Historical run outputs are ignored by Git.
• .model-cache/: downloaded model files. Ignored by Git.
• .venv/: local Python environment. Ignored by Git.

Only runs/.gitkeep and manifests/.gitkeep are tracked so the folders exist after clone.

GPU And Disk Notes

GPU memory is runtime memory. It is used while the Python process is running and is released when the process exits. Frameworks such as PyTorch or Paddle may keep short-lived caches, so use the process boundary as the real cleanup point.

For the RTX 5070 Ti Laptop GPU with about 12GB VRAM, use 8-10GB as the practical soft ceiling. The runner marks a page as warning when peak GPU memory exceeds the configured soft limit. The default is 10240MB.

Disk cache is persistent. Large models should be tested in small batches and removed after the run if needed.

Engines

Implemented:

• native_pdf: embedded PDF text layer baseline
• rapidocr_legacy: current project baseline
• tesseract: native Tesseract wrapper
• paddleocr_ppocrv5: PaddleOCR PP-OCRv5 OCR candidate
• paddleocr_ppocrv5_server_1600: PP-OCRv5 server models with lower VRAM input cap
• paddleocr_ppocrv5_server_1600_noangle: 1600px server ablation without textline orientation
• paddleocr_ppocrv5_server_1280_noangle: 1280px server ablation without textline orientation
• paddleocr_mobile_fast: PaddleOCR PP-OCRv5 mobile fast candidate
• paddle_structure: Paddle PP-StructureV3 layout/table candidate

Reserved optional slots:

• surya
• glm_ocr
• paddleocr_vl

The reserved large-model slots are intentionally opt-in. They should be wired and tested one at a time to avoid downloading many GB blindly.

Setup

cd D:\Workbench\Finance\ocr-benchmark
.\scripts\setup.ps1 -Profile core
.\.venv\Scripts\Activate.ps1
python -m ocr_benchmark.cli list-engines

Profiles:

• core: RapidOCR, EasyOCR, Tesseract wrapper, PDF rendering
• paddle: PaddleOCR / PP-OCRv5 / PP-StructureV3 dependencies
• vlm: large document VLM dependencies
• all: all dependency groups

Run The Local Four-Sample Benchmark

Put the four local sample folders under data/:

data/
  24020103604/
  24100107600/
  25080101347/
  25090102873/

Create a small smoke manifest:

python -m ocr_benchmark.cli make-manifest --preset local_four_smoke --output manifests\local_four_smoke.json

Run a quick baseline:

python -m ocr_benchmark.cli run --manifest manifests\local_four_smoke.json --engines native_pdf,rapidocr_legacy --run-name local-four-smoke

Run the named flow layout:

python -m ocr_benchmark.cli run-flow --flow flow1
python -m ocr_benchmark.cli run-flow --flow flow2
python -m ocr_benchmark.cli run-flow --flow flow5
python -m ocr_benchmark.cli run-flow --flow flow6

The named flows are defined in configs/flows.json.

Retired candidates and their stored results are documented in docs/candidate-record.md.

The processed outputs are stored as:

storage/
  flow1/
    finance_24020103604/
      rapidocr_legacy/
        pages/
        texts/
        blocks/
    finance_24100107600/
    finance_25080101347/
    finance_25090102873/

Run a wider comparison:

python -m ocr_benchmark.cli make-manifest --preset local_four --output manifests\local_four.json
python -m ocr_benchmark.cli run --manifest manifests\local_four.json --engines rapidocr_legacy,paddleocr_ppocrv5 --run-name local-four-ppocrv5

Run table/layout candidate on a small subset:

python -m ocr_benchmark.cli run --manifest manifests\local_four_smoke.json --engines paddle_structure --run-name local-four-structure --max-files 10

Accuracy Expectations

Runtime success only means an engine produced OCR text. Accuracy is evaluated with JSON expectation files.

The starter 47-sample truth file is:

expectations\finance_25080101347_smoke.json

Run the first 6 files of sample 47 and score them:

python -m ocr_benchmark.cli make-manifest --preset local_47_smoke --output manifests\local_47_smoke.json
python -m ocr_benchmark.cli run --manifest manifests\local_47_smoke.json --engines rapidocr_legacy --run-name local-47-smoke --max-pages 1
python -m ocr_benchmark.cli evaluate --run-dir runs\local-47-smoke --expectations expectations\finance_25080101347_smoke.json

The evaluator writes:

• runs\<run-name>\accuracy.json
• runs\<run-name>\accuracy.csv

Outputs

runs/<run-name>/:

• run.json
• results.jsonl
• summary.csv
• sample_summary.csv
• report.md

results.jsonl stores per-page values including elapsed time, source bytes, GPU memory baseline/peak/delta, gpu_verified, text length, block count, average/min/max confidence, low-confidence block count, keyword hits, text path, and block JSON path.

storage/<run-name>/:

• rendered PDF/image pages
• OCR text files
• OCR block JSON files with text boxes and confidence scores

Cleanup

Remove downloaded model cache:

.\scripts\cleanup_models.ps1

Remove PaddleX user cache from earlier runs:

.\scripts\cleanup_models.ps1 -IncludePaddlexUserCache

Remove processed pages/text from storage/:

.\scripts\cleanup_models.ps1 -IncludeStorage