patent-ocr

OCR for scanned patent collections: language selection, column-aware reading, and citable spans.

Import-only — there is no CLI.

Why it exists

The obvious way to OCR a patent archive produces quietly bad data. Every default in this library is set from measurement, not habit:

Failure	What actually happens	Fixed by
Wrong language model	Doesn't degrade — fabricates. English over Cyrillic returns confident Latin lookalikes at 0% correct script.	LanguagePolicy
Naive parallelism	ocrmypdf and tesseract both self-parallelize; N at once oversubscribes the box, pages breach the per-page timeout and are emitted empty with a success exit code. Cost 37% of recovered text on one sample.	SINGLE_THREAD_ENV
Placeholder pages	A collection that fills gaps with synthetic text pages makes ocrmypdf abort whole documents. Silently skipped 24% of one corpus.	--skip-text in DEFAULT_FLAGS
Two-column pages	Tesseract merges columns into full-width lines and smears the baselines, so an address runs into a citation table — and the geometry needed to undo it is gone.	columns / Pipeline.columns
Flat text output	A passage can't be cited: no page, no coordinates.	spans

Install

System binaries first. pip cannot install these, and without them the package imports cleanly and then fails on the first page: tesseract with the language packs your jurisdictions need, plus ghostscript for the searchable-PDF pass.

sudo apt install tesseract-ocr ghostscript      # plus tesseract-ocr-{deu,fra,rus,...}

pip install patent-ocr            # core
pip install patent-ocr[legacy]    # + .doc/.xls bibliographic sidecars
pip install -e .[dev,legacy]      # + the test suite (see Tests)

Importing the package pulls in no heavy dependency: every module imports PyMuPDF, numpy, Pillow and the rest inside the functions that use them, so a package you have not installed breaks only the one function that needs it. The package ships py.typed, so its annotations reach your type checker.

A missing language pack does not raise — it silently returns nothing, so check first:

from patent_ocr import preflight, missing_packs, LanguagePolicy, Corpus

preflight()                                        # binaries + installed langs
corpus = Corpus("/data/patents")
missing_packs(LanguagePolicy(), corpus.jurisdictions())

Two passes

from patent_ocr import Corpus, Pipeline

corpus = Corpus("/data/patents")        # derived output -> /data/patents-ocr
pipeline = Pipeline(corpus, progress=print)

report = pipeline.searchable()          # searchable PDFs + text sidecars (fast)
report = pipeline.columns()             # column-correct text (slower, text only)
print(report.as_dict())

They are separate because they yield different things. searchable gives you a PDF that keeps word-level geometry and reads well on single-column pages. columns re-reads the rasters with the columns separated and is the one to use when sentence integrity matters — parsing claims, chunking for retrieval, anything that breaks if a sentence runs into a neighbouring table.

Measured on US 4336787, against ten strings known from the printed page:

	strings found	words
whole-page OCR	8/10	5,415
column-split	10/10	5,712

Column splitting also fixes characters, not just order ("Giacomo", not "iacomo"), because tesseract finally sees a clean single column.

Both passes resume: re-running skips completed work. columns reuses the language decisions searchable probed.

Confidence, and which documents are hard

Word counts say nothing about whether the right words came out. Every span carries the OCR engine's own confidence, so a corpus can triage itself:

from patent_ocr import diagnose_corpus, summarise, write_report

write_report(corpus.output_root)              # -> triage.json
print(summarise(diagnose_corpus(corpus.output_root))["by_dominant"])

Each document gets a dominant label and the evidence behind it — computed from the OCR pass, never hand-labelled:

flag	meaning
low-confidence / weak-confidence	mean confidence under 70 / 80
no-text-pages	pages that yielded nothing at all
sparse-text	mostly plates rather than prose
mixed-layout	two-column on some pages, single on others
incomplete	synthetic placeholder pages present
empty	no text recovered whatsoever

difficulty (0–1) ranks the triage list. It means "a human should look", not "this is wrong".

Ground truth for fine-tuning

On period typography — Fraktur, pre-war Cyrillic, worn letterpress — stock tesseract models are the ceiling, and preprocessing does not lift it. The way past is a model fine-tuned on the corpus itself:

from patent_ocr import read_columns_document, write_lstmf_pairs, write_labelstudio

res = read_columns_document("patent.pdf", language="deu+frk")

# line crops + transcripts, the pair format tesstrain consumes.
# min_conf is an upper bound: it keeps only the lines the model struggles with,
# because the confident ones teach it nothing.
res.to_lstmf_pairs("patent.pdf", "train/", min_conf=60)

# or round-trip through Label Studio for correction by someone without
# tesseract's tooling
write_labelstudio(res.regions, "task.json", image_url="/data/p0.png",
                  page=0, page_size=res.page_sizes[0])

The transcripts start as the OCR's own guess: a draft to correct, not truth. Training on them uncorrected only teaches the model its existing mistakes. Pair this with the triage list so effort goes to the documents that need it.

Prefer res.to_lstmf_pairs(...) over calling write_lstmf_pairs directly. Line boxes are raster pixels at the dpi the OCR ran at, and cropping at any other dpi still succeeds — it just crops the wrong pixels, mislabelling every pair. Passing the regions and the dpi separately is how the two drift apart; the method carries the dpi from the read, and the function refuses the mismatch:

from patent_ocr import DpiMismatch, write_lstmf_pairs

try:
    write_lstmf_pairs("patent.pdf", res.regions, "train/", dpi=150)
except DpiMismatch as exc:
    print(exc)   # "... recognised at roughly 300 dpi -- pass that dpi, or use
                 #  ColumnOcrResult.to_lstmf_pairs() which carries it for you."

Citable text

from patent_ocr import load_columns

doc = load_columns(corpus, "US-4336787")
span = doc.locate(1200)
print(span.page, span.bbox, span.slice(doc.text))
print(doc.page_text(0))

Text and offsets are built in one pass, so text[start:end] is exact by construction rather than aligned afterwards — a sidecar .txt and a PDF text layer are different renderings and drift apart. verify() checks it per document.

load_columns restores everything the pass recorded — the per-span confidence, and the normalised coordinates and declared units that tell a consumer what the numbers mean. Use spanned_from_dict to rebuild the same object from a .columns.json you loaded yourself.

Single documents

from patent_ocr import OcrEngine, read_columns, read_searchable

OcrEngine().run("in.pdf", "out.pdf", "out.txt", language="deu+frk")

spanned = read_columns("in.pdf", language="rus", skip_pages=[2, 3])
spanned = read_searchable("already-ocred.pdf")

Languages

DEFAULT_LANGUAGES maps ~30 offices to tesseract specs. German-language offices get deu+frk, since publications before roughly 1941 are set in Fraktur.

Offices publishing in several languages (CH, BE, EP, WO, LU, CA) are resolved per document by probe, which reads one interior page with each candidate and keeps the most confident. Stacking candidates into one -l argument is slower and less accurate. Correct models score 65–90 mean confidence on these scans, wrong ones 18–45, so one page decides it.

Override without patching:

from patent_ocr import LanguagePolicy, Pipeline

policy = LanguagePolicy(
    languages={**{"ZA": "eng"}},
    multilingual={"CH": ["deu", "fra"]},
)
Pipeline(corpus, policy=policy).columns()

Layout independence

Collections get restructured. Corpus reads a manifest.jsonl if present and otherwise walks the tree, handling both a directory-per-patent layout (with split parts plus a merged whole) and a flat file-per-patent layout, yielding the same PatentDocument either way. Where a patent has both a merged whole and the parts it came from, the whole wins — so pages are counted and OCRed exactly once.

Derived files default to a sibling directory, not inside the corpus. This is deliberate: a run that wrote output into the tree lost every artifact when the tree was rebuilt. An output_root equal to the corpus root is refused outright — the searchable pass writes a .pdf per source, so it would overwrite the scans.

Measuring quality

Word counts say nothing about whether the right words came out. Where a collection ships bibliographic sidecars, those were transcribed independently and make a gold set:

from patent_ocr.validate import Expected, RecallReport, check
from patent_ocr.legacy import extract_text, parse

report = RecallReport()
rec = parse(extract_text("US-2299073.doc"))
report.add("US-2299073", check(doc.text, Expected(
    number=rec.patent_number, date=rec.publication_date,
    inventors=rec.inventors, applicants=rec.applicants, ipc=rec.ipc,
)), jurisdiction="US", complete=True)
print(report.table())

check returns only fields that are meaningful for the document's script. Sidecars romanise Cyrillic and CJK names (KOROLEV R A) while correct OCR returns the original script, so a naive name comparison scores 0% on Russian documents whose OCR is perfect. Absent keys mean "not comparable", never "failed".

Modules

Module	Responsibility
corpus	layout-independent collection access, placeholder pages, output placement
engine	ocrmypdf invocation and the settings that matter
languages	jurisdiction → language, confidence probing, pack checks
columns	gutter detection and column-aware page OCR
reading_order	geometric reordering (only where OCR geometry survived)
spans	citable text assembly and verification
pipeline	concurrency, resume, reporting
overlay	span-aware overlay PDF/HTML for visual QA
hocr	standard hOCR export, for tools outside this package
triage	which documents are hard, and why
groundtruth	correctable exports for fine-tuning a model
validate	recall against independently-known fields
legacy	Word-97 / Espacenet / BIFF sidecars (optional extra)

Tests

pip install -e .[dev,legacy]
pytest              # 294 tests, no tesseract needed
pytest -m ocr       # 14 end-to-end reads of the fixture scan (needs fra/nld packs)
pytest -m dist      # 11 that build the package and inspect the artifacts
pytest -m ''        # all 319

The default suite runs without tesseract, ghostscript or ocrmypdf. That is not a compromise: what is worth guarding here is geometry, character offsets and parsing, and none of it needs an OCR engine. Region/Line/Word are plain dataclasses, so a recognised page is built by hand and the whole downstream chain — spans, hOCR, triage, overlay coordinates — runs exactly as it would on real output. The ocr-marked tests read static/fixtures/ for real and check the same invariants against real geometry, and the dist-marked ones assert what a consumer actually receives: that the licence ships, that py.typed is in the wheel, that the built version matches patent_ocr.__version__, and that the sdist carries no regenerable output.

Versioning

Semantic versioning; patent_ocr.__version__ is the single source and the build reads it from there. Every release, and the migration notes for it, is recorded in CHANGELOG.md.

Interfaces

There is no CLI and no MCP server: this package is an importable library and nothing else, which is why pyproject.toml declares no [project.scripts]. The public API is exactly patent_ocr.__all__ — anything reachable is importable from the package root, including DpiMismatch, the one exception a caller is expected to catch. patent_ocr.validate and patent_ocr.legacy are imported by module path, as shown above.

Visual QA

from patent_ocr import overlay_pdf, overlay_html, write_span_text

overlay_pdf("scan.pdf", spanned, "overlay.pdf")          # text drawn on the scan
overlay_html("scan.pdf", {"columns": spanned}, "qa.html")  # self-contained page
write_span_text(spanned, "spans.txt")                     # text + its addresses

Three things this got wrong before they were measured, worth knowing if you extend it:

• PDF's built-in Helvetica is Latin-1 only and drops what it cannot encode — curly quotes on a French page, and every character of a Russian or Japanese one. resolve_font embeds a real Unicode font instead.
• insert_textbox writes nothing at all when text overflows, returning a negative number that is easy to ignore. Font size is reduced until it fits, and the box is grown before the type is shrunk to nothing.
• A box far taller than wide holds text printed sideways (margin imprints on patent covers). Poured in horizontally it wraps to one or two characters per line — present, but every word destroyed. It is drawn rotated instead.