Skip to content

feat(pdf): port docling's LayoutPostprocessor for #61 classification#64

Merged
artiz merged 24 commits into
docling-project:masterfrom
artiz:claude/issue-61-layout-classification
Jul 13, 2026
Merged

feat(pdf): port docling's LayoutPostprocessor for #61 classification#64
artiz merged 24 commits into
docling-project:masterfrom
artiz:claude/issue-61-layout-classification

Conversation

@artiz

@artiz artiz commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

The RT-DETR layout detections were used almost raw (a flat 0.3 threshold plus a single greedy overlap pass), which over-detected tables and let a high-score picture suppress a lower-score structured region. Port the cleanup docling's LayoutPostprocessor applies before assembly:

  • Per-label confidence thresholds (CONFIDENCE_THRESHOLDS): a picture / table / list-item now needs >= 0.5 (0.45 for headings/code/index/etc), not the 0.3 base. Drops the low-confidence detections that double-emit.
  • Bucketed overlap resolution: regular, picture and wrapper clusters are de-overlapped independently, so a picture no longer suppresses a table or table-of-contents index that scores lower.
  • Cross-type picture-vs-table rule (_handle_cross_type_overlaps): a picture coinciding with a table (IoU > 0.8) is dropped, keeping the structured table.
  • Containment drop: a regular region absorbed by a table/index/picture is that special's child, not a separate block, so it is not emitted twice (form/key-value wrappers excluded — this pipeline renders their content from the contained regions).
  • Render a document_index (table of contents) as a table via TableFormer, instead of skipping it.

vs docling groundtruth: table_mislabeled 108 -> 88 (survey no longer rendered as tables), 2203 183 -> 177, right_to_left_03 66 -> 60, normal_4pages 54 -> 52; every previously-exact PDF stays exact. redp5110's table-of-contents is now correctly a table (was a picture); its remaining byte diff is a TableFormer dot-leader column-matching gap tracked in #60.

Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj

@artiz artiz self-assigned this Jul 10, 2026
@artiz artiz added the enhancement New feature or request label Jul 10, 2026
artiz and others added 3 commits July 10, 2026 18:06
…classification

The RT-DETR layout detections were used almost raw (a flat 0.3 threshold
plus a single greedy overlap pass), which over-detected tables and let a
high-score picture suppress a lower-score structured region. Port the
cleanup docling's LayoutPostprocessor applies before assembly:

- Per-label confidence thresholds (CONFIDENCE_THRESHOLDS): a picture /
  table / list-item now needs >= 0.5 (0.45 for headings/code/index/etc),
  not the 0.3 base. Drops the low-confidence detections that double-emit.
- Bucketed overlap resolution: regular, picture and wrapper clusters are
  de-overlapped independently, so a picture no longer suppresses a table
  or table-of-contents index that scores lower.
- Cross-type picture-vs-table rule (_handle_cross_type_overlaps): a
  picture coinciding with a table (IoU > 0.8) is dropped, keeping the
  structured table.
- Containment drop: a regular region absorbed by a table/index/picture is
  that special's child, not a separate block, so it is not emitted twice
  (form/key-value wrappers excluded — this pipeline renders their content
  from the contained regions).
- Render a document_index (table of contents) as a table via TableFormer,
  instead of skipping it.

vs docling groundtruth: table_mislabeled 108 -> 88 (survey no longer
rendered as tables), 2203 183 -> 177, right_to_left_03 66 -> 60,
normal_4pages 54 -> 52; every previously-exact PDF stays exact. redp5110's
table-of-contents is now correctly a table (was a picture); its remaining
byte diff is a TableFormer dot-leader column-matching gap tracked in docling-project#60.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj
Signed-off-by: Artem Kustikov <artem.kustikov@gmail.com>
docling renders `mAP @0.5` (and emails) with no space after the `@`; the
groundtruth never contains "@ " anywhere. Our TableFormer cell text joined
the PDF's word cells (`mAP` `@` `0.5`) with spaces, producing `mAP @ 0.5`.
That single extra space widened the header cell and, through the column
padding, shifted every row of the table.

Gluing `@` to its follower in the assembled cell text drops 2206.01062 from
164 to 134 diff lines vs docling groundtruth (one header cell re-aligns the
whole column); 2305-pg9 stays cell-for-cell exact and no other snapshot
moves.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj
Signed-off-by: Artem Kustikov <artem.kustikov@gmail.com>
Apply `cargo fmt` (pinned 1.96.0) to the bucketed-resolve and per-label
threshold code so the fmt · clippy gate passes. No behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj
Signed-off-by: Artem Kustikov <artem.kustikov@gmail.com>
@artiz artiz force-pushed the claude/issue-61-layout-classification branch from 6554892 to 42ffd7b Compare July 10, 2026 18:06
artiz and others added 9 commits July 10, 2026 19:05
Replace the geometric two-column reading-order heuristic with a faithful
port of docling-ibm-models' ReadingOrderPredictor (reading_order_rb.py).
Per page it builds an up/down adjacency graph from element geometry (an
element is "below" another that is strictly above it and horizontally
overlapping, unless a third element interrupts the run), horizontally
dilates narrow boxes toward their column neighbours, then depth-first
traverses from the top-most elements. page_header/page_footer are ordered
as their own groups (first/last), matching docling's per-page split.
Everything runs in bottom-left origin with docling's exact BoundingBox
predicates; the disabled l2r/r2l maps are omitted.

This fixes multi-column reading order (author blocks, two-column body) that
the geometric sort got wrong:

  2206.01062  134 -> 106   2203.01017v2  177 -> 169   redp5110  230 -> 210

vs docling groundtruth, with every previously-exact PDF still exact
(2305-pg9, multi_page, picture_classification, right_to_left_01/02) — the
port reproduces docling's order on single-column pages too. normal_4pages
moves +4 (a Korean cover page entangled with the separate furniture /
heading-number issues).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj
Signed-off-by: Artem Kustikov <artem.kustikov@gmail.com>
Add the intra-page half of docling's `ReadingOrderPredictor.predict_merges`:
after ordering, join a text fragment with the following text fragment that
lies strictly to its right when the running tail ends with a lowercase
letter / comma / hyphen and the candidate starts with a letter. This merges
the author columns of an academic title page (`… bpf@zurich.ibm.com
Christoph Auer …`) and a paragraph that wraps into the next column into a
single block, matching docling. page_header/footer/table/picture/caption/
footnote (and already-consumed regions) are skipped when scanning for the
next fragment; cross-page/vertical continuations stay with
`merge_continuations`.

vs docling groundtruth: 2206.01062 106 -> 94, 2203.01017v2 169 -> 161,
redp5110 210 -> 204, with every previously-exact PDF still exact and no
other snapshot moved.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj
Signed-off-by: Artem Kustikov <artem.kustikov@gmail.com>
docling never emits U+2022 (•) in its Markdown; an inline CCS-concept
separator that our text layer reads as `•` is rendered `·` (U+00B7) by
docling. Map it in `clean_text`. List bullets are unaffected — `·` is
already in the list-marker strip set — so only inline bullets change
(2206.01062 92 vs 94).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj
Signed-off-by: Artem Kustikov <artem.kustikov@gmail.com>
The raw document.xml line-diff byte-compares the four <location value>
provenance tokens each laid-out block emits. On the PDF path those come
from docling's layout clusters in the reference and ours in the output;
the same region is boxed a few grid units apart, so geometry lines diff
even when text and structure match.

- dclx_diff.py: line-diff that counts a <location> pair as equal when the
  two values are within --tol grid units; text/structure/unmatched lines
  stay byte-exact (tol=0 reproduces diff | grep -c '^[<>]').
- dclx_conformance.sh: apply DCLX_TOL (default 2) to PDF fixtures only;
  OOXML/other formats whose geometry is read from the same source file
  stay exact. Symlink models/.pdfium into the run cwd so the binary finds
  its assets when invoked from the scratch dir.
- dclx_pdf_tol_sweep.sh: convert each fixture once, re-diff at a ladder of
  tolerances (exact -> +/-2/5/10 -> geometry-ignored).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj
Signed-off-by: Artem Kustikov <artem.kustikov@gmail.com>
docling's PDF .dclx puts four <location> tokens (l,t,r,b on a 0-511
normalized page grid) on every laid-out block; our output emitted none
and collapsed each block to one inline line, so ~half the reference (the
provenance lines) had no counterpart. This wraps each text/heading/table/
picture region in Node::Located with the region bbox normalized exactly
as docling_core's _create_location_tokens_for_bbox does
(round(512*coord/page_dim), clamped 0-511, top-left origin). Our heron
cluster boxes match docling's to within ~1 grid unit.

merge_continuations / hold_start now look through the Located wrapper
(as_paragraph / is_picture_node helpers) so cross-column/page paragraph
stitching still works; a merged paragraph keeps the first fragment's
provenance. Markdown/JSON already render Located transparently, so their
output is unchanged (multi_page markdown stays byte-exact).

multi_page .dclx similarity 45% -> ~72% at +/-5 grid-unit tolerance.
List-item, code, and furniture provenance are follow-ups.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj
Signed-off-by: Artem Kustikov <artem.kustikov@gmail.com>
16 .dclx archives generated from published docling 2.112.0 (document.xml
only, no page images), the reference dclx_conformance.sh scores the PDF
--to dclx output against. Matches the tracked groundtruth for the other
formats.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj
Signed-off-by: Artem Kustikov <artem.kustikov@gmail.com>
Record the .dclx metric (51% exact / 60% at +/-2 grid units, issue docling-project#32
target >=50%), the 0-511 location normalization we match, why the +/-2
tolerance is honest (only PDF geometry from docling's own layout run;
text/structure stay byte-exact; DCLX_TOL=0 reproduces raw diff), the
per-fixture split, and the remaining serialization gaps (list/code/
furniture provenance) vs the model-level table/reading-order blockers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj
Signed-off-by: Artem Kustikov <artem.kustikov@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj
Signed-off-by: Artem Kustikov <artem.kustikov@gmail.com>
Re-port of the PDF list/code/furniture provenance onto issue-61's DocLang
model (which already carries ListItem.location + list_item_overlay, so the
list path reuses the existing field rather than a Located wrapper).

- List items set ListItem.location (emitted after the <ldiv>) and the
  bullet as <marker>·</marker>, matching docling's block.
- Code blocks emit <location> after <code> (emit_code/emit_located gain an
  optional location).
- Page headers/footers become Node::PageFurniture -> <page_header>/
  <page_footer> with a furniture layer + location + text at their
  reading-order position; Markdown/JSON omit them like other furniture.
  merge_continuations/hold_start look past furniture (is_merge_trailer) so
  a paragraph continuing across a page break still stitches.

PDF .dclx similarity 45% -> 63% at +/-2 grid units (exact 52%, ceiling 65%)
on the issue-61 base. Markdown byte-exact (multi_page 0, 2305 32,
normal_4pages 54).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj
Signed-off-by: Artem Kustikov <artem.kustikov@gmail.com>
@artiz artiz force-pushed the claude/issue-61-layout-classification branch from 525d8ee to 62c1fd1 Compare July 11, 2026 10:59
artiz and others added 3 commits July 11, 2026 11:17
The PDF Markdown metric moved with the docling-project#61 layout/reading-order work and the
fixture set shrank to 14 (the two Korean image-only pages lost their text
groundtruth): now 5/14 exact, 6/14 whitespace-normalized. Several heavy docs
improved (2206.01062 164->92, 2203.01017v2 183->161, table_mislabeled 108->88,
right_to_left_03 66->60, 2305.03393v1 32->30); code_and_formula now differs by
5 (code block reflowed + trailing newline). Non-PDF .dclx xlsx 87%->85%; mean
unchanged at 91%.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj
Signed-off-by: Artem Kustikov <artem.kustikov@gmail.com>
Our single space before the 1/4 numerator is the faithful rendering; docling's
extra space comes from embedded-font OS/2 metrics and matching it would degrade
output + risk RTL geometry (a prior trial broke right_to_left_01). Keep the
faithful output; close docling-project#63 without a byte-match.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj
Signed-off-by: Artem Kustikov <artem.kustikov@gmail.com>
…project#62)

docling drops the line-wrap hyphen between two digit runs (a reference page
range `pp. 545-` + `561` -> `545561`); a same-line range (`1162-1167`) never
reaches the continuation path and keeps its hyphen. Extends region_text's
wrap-dehyphenation, previously alphabetic-only, to the digit-digit case.

2305.03393v1 30->28, 2203.01017v2 161->157; all exact fixtures stay exact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fKFbatf55BK5j8vr3dZjj
Signed-off-by: Artem Kustikov <artem.kustikov@gmail.com>
@artiz artiz force-pushed the claude/issue-61-layout-classification branch from 9eb4cae to 6713a1e Compare July 11, 2026 12:46
claude added 6 commits July 13, 2026 12:00
…g-project#60)

Word-to-cell matching now runs docling's full pipeline (docling-ibm-models
tf_cell_matcher.py + matching_post_processor.py, ported bug-for-bug in
tf_match.rs): intersection-over-word matching, cell-class-aware good/bad
split per column, column-median snapping of unmatched cells,
adjacent-column de-duplication, best-intersection final assignment, and
row/column-band orphan pickup, followed by docling's per-grid-position
response merge and row/col index compression. The bbox decoder's cell
classes (previously ignored) feed the post-processor, and the table crop
reproduces docling's exact rounding chain (round(bbox) -> x2 -> x1024/h ->
round, banker's rounding both times).

2203.01017v2 157->150, redp5110_sampled 204->202; every other groundtruth
fixture unchanged (2305-pg9 stays cell-for-cell exact).

DOCLING_RS_TF_SIMPLE_MATCH=1 reverts to the old best-overlap matcher.
DOCLING_RS_TF_MATCH_DUMP=dir dumps matcher inputs for
scripts/test/tf_match_reference.py, which replays them through docling's
Python post-processor; that parity harness shows the port reproduces the
reference on identical inputs, isolating redp5110's remaining TOC gap to
the OTSL predictions themselves (model-level).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Signed-off-by: artiz <artem.kustikov@gmail.com>
…rator

docling sorts reading-order heads/children with
functools.cmp_to_key(PageElement.__lt__), a fuzzy geometric relation that
is not a strict total order on mixed overlap groups. Python's Timsort
quietly produces an order; Rust's slice::sort_by detects the Ord violation
and panics (odf's LibreOffice PDF exports trip it on the first file, which
also left the committed odf snapshots unregenerable and stale). Replace
those three sorts with CPython's binarysort (natural-run detection +
stable binary insertion) - exactly Timsort for n < 64, a plain stable sort
under a consistent comparator, and never panicking.

Refresh the odf snapshots, stale since the reading-order port: they pick
up the Python-parity ordering, the earlier bullet normalization, and
false-picture suppression.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Signed-off-by: artiz <artem.kustikov@gmail.com>
…d table numbers

Conformance table picks up the dehyphenation (docling-project#62) and matcher (docling-project#60)
deltas: 2305.03393v1 28, 2203.01017v2 150, redp5110_sampled 202. The docling-project#60
blocker entry now records what is ported (matching + crop rounding, with
the parity harness) and what remains model-level (the OTSL tag stream on
the hard crops). DocLang avg re-measured unchanged at 63% (+/-2).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Signed-off-by: artiz <artem.kustikov@gmail.com>
…zer (docling-project#62)

The docling-project#62 quote gap was never a font-decoding problem: docling-parse decodes
the subsetted TeX fonts' /Differences + /ToUnicode correctly (quotedblleft
-> U+201C), then its cell sanitizer (pdf_sanitators/constants.h) maps every
typographic quote - single and double alike - to the ASCII apostrophe, and
U+201A to a comma. A double quote in docling output only ever comes from a
literal quotedbl glyph (2206's 'text in the wild" pairs a curly open with
a literal close). Our clean_text mapped curly doubles to '"' with a
Hangul-only exception that patched one symptom; replace it with docling's
table (also covering the full hyphen family U+2010..U+2015).

2206.01062 92->86, 2203.01017v2 150->136; all other fixtures unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Signed-off-by: artiz <artem.kustikov@gmail.com>
…der (docling-project#62)

docling orders a cluster's cells by their docling-parse cell index
(LayoutPostprocessor._sort_cells: sorted(cells, key=c.index)) - the
sanitizer's output order, which our cells slice already is. The geometric
band-sort loses that on off-baseline glyphs: 2206's inline math '>' sits
~2pt above its line's band and drifted into the following line
('( > 10 pages)' became '( 10 pages) ... complex > tables'). Keep the
band-sort only for the legacy pdfium cell path, whose cells carry no
docling-parse order.

2203 136->130, 2206 86->80, 2305 28->26, normal_4pages 56->44,
redp5110 202->194, table_mislabeled 88->86; the five exact fixtures
(both RTL ones included) stay exact.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Signed-off-by: artiz <artem.kustikov@gmail.com>
…order fixes

The pdf table/text fixtures pick up the quote normalization and the
docling-parse index-order cell join; several of them (2206, 2305,
normal_4pages, right_to_left_03, llncsdoc, mets_gbs, scanned/*) had been
stale since PR docling-project#29 - the odf panic (fixed two commits ago) aborted every
full snapshot regeneration before it reached them. The scanned/OCR
fixtures were re-pinned from a source-rebuilt model set (models-v1 assets
are unreachable from this environment); if the release-model environment
disagrees on OCR text, re-pin them there once.

Signed-off-by: artiz <artem.kustikov@gmail.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Signed-off-by: artiz <artem.kustikov@gmail.com>
@artiz artiz force-pushed the claude/issue-61-layout-classification branch from 84f1836 to 84c1e3a Compare July 13, 2026 12:00
artiz and others added 3 commits July 13, 2026 12:01
…reshed numbers

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Signed-off-by: artiz <artem.kustikov@gmail.com>
dump_bbox reverse-engineered the DocLang location normalization (docling-project#32,
done - every laid-out block now carries provenance); probe_page's header
already called itself a throwaway. The remaining examples are the live
debugging toolkit for conformance work.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
Signed-off-by: artiz <artem.kustikov@gmail.com>
@artiz artiz merged commit 823200b into docling-project:master Jul 13, 2026
4 checks passed
@artiz artiz deleted the claude/issue-61-layout-classification branch July 13, 2026 12:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants