Skip to content

README.md

Latest commit

 

History

History
76 lines (61 loc) · 4.07 KB

README.md

File metadata and controls

76 lines (61 loc) · 4.07 KB
Raw

anydoc benchmark harness

Benchmarks anydoc against well-known document-to-markdown converters on the samples/ corpus, for both speed and quality (deterministic metrics + LLM judge). Office, text, and presentation documents; PDFs are out of scope.

The corpus is not redistributable and is not in the repo. The harness reads whatever documents are in samples/.

Competitors

tool formats benchmarked
markitdown docx, xlsx, xls, epub, csv, pptx
pandoc docx, odt, rtf, epub, csv
docling docx, xlsx, csv, pptx
unstructured doc, docx, odt, rtf, epub, xls, xlsx, csv, ppt, pptx
mammoth + turndown docx
LibreOffice headless → HTML → pandoc doc, docx, docm, odt, rtf, xls, xlsx, xlsm, xlsb, ods, ppt, pptx, odp

Missing tools are skipped with a warning, so a partial setup still runs.

Setup

cargo build --release --example convert          # anydoc bench binary
cd bench
python -m venv .venv && .venv\Scripts\activate
pip install -r requirements.txt                  # markitdown, pymupdf, requests, markdownify
pip install -r requirements-heavy.txt            # optional: docling (~2 GB torch), unstructured
npm install                                      # mammoth pipeline

Also needed on PATH: pandoc, node, and LibreOffice (soffice; set the SOFFICE env var if it's not in a standard location). Close any running LibreOffice GUI before benchmarking: it silently swallows headless calls.

Running

python convert.py --iters 1            # convert everything with every tool
python render_truth.py                 # ground-truth page images via soffice + pymupdf
python metrics.py                      # structure counts + trigram containment
python judge.py --limit 25            # pairwise LLM judging (see cost note)
python report.py                       # aggregates everything into out/report.md

All artifacts land in out/ (gitignored). convert.py appends to out/timings.jsonl; report.py keeps only the latest row per (tool, file).

Methodology notes

  • Speed: min of --iters warm runs, and the published numbers use --iters 1, one warm conversion per document. markitdown, docling, unstructured, and mammoth are timed in-process. anydoc is timed from the conversion time its own CLI reports, which likewise leaves out process spawn. pandoc and LibreOffice include process spawn (that's how they're used in practice, and it's called out in the report).
  • Quality, deterministic: heading/table/list/link/footnote counts per output, plus word-trigram containment between each tool and anydoc (a cheap dropped-content detector, not a truth measure).
  • Quality, LLM judge: for each doc, anydoc vs one opponent, judged twice with A/B positions swapped. Ground truth is the LibreOffice-rendered pages (first 6) attached as images; for EPUB it's text extracted from the source XHTML. CSV is excluded (rendering is meaningless). Position-inconsistent verdicts count as ties. Model: claude-sonnet-5 via the Anthropic Message Batches API (50% batch discount; override with JUDGE_MODEL, key in bench/.env as ANTHROPIC_API_KEY). Verdicts journal to out/judge.jsonl and submitted batches to out/batches.jsonl; reruns harvest open batches first and only submit never-judged pairs, so it's safe to interrupt.

Cost control

judge.py --dry-run prints how many judge calls would be made. --limit N caps docs per (format, opponent) bucket; --formats docx and --opponents pandoc,markitdown narrow the matrix. Each call sends up to 6 page images and two ~40k-char markdown outputs.