diff --git a/MIGRATION.md b/MIGRATION.md index f673e009..711676e1 100644 --- a/MIGRATION.md +++ b/MIGRATION.md @@ -14,7 +14,7 @@ phased plan is kept at the end as history.) > against *live* docling; the PDF/image/METS ML path lives in `docling-pdf` > (a pure-Rust PDF text parser + pdfium rasterization + ONNX > layout/TableFormer/OCR + a port of docling-parse's line sanitizer) and is also -> measured byte-for-byte against live docling — **6 / 16 PDF fixtures exact, 7 / 16 +> measured byte-for-byte against live docling — **5 / 14 PDF fixtures exact, 6 / 14 > whitespace-normalized** (see `PDF_CONFORMANCE.md`), with a snapshot baseline > guarding against regressions. `cargo test` is green (unit tests + a 133-source > output-regression suite). @@ -24,7 +24,7 @@ phased plan is kept at the end as history.) | | | |---|---| | **What** | A Rust port of docling's converter, backends, and discriminative PDF/ASR pipelines; same `convert → DoclingDocument → export_to_markdown()/json()` shape, single static binary, no Python/torch at runtime | -| **Conformance** | Declarative formats byte-for-byte vs *live* PyPI docling (most 100%, see §2); `.dclx` DocLang output ≈91% mean vs docling's own `.dclx` (§2); PDF ML path 6/16 fixtures byte-exact, rest close; every optimization is gated on this not regressing | +| **Conformance** | Declarative formats byte-for-byte vs *live* PyPI docling (most 100%, see §2); `.dclx` DocLang output ≈91% mean vs docling's own `.dclx` (§2); PDF ML path 5/14 fixtures byte-exact, rest close; every optimization is gated on this not regressing | | **Performance** | PDF ML pipeline **4.3× faster warm / 4.7× end-to-end** than Python docling at 2.3–2.6× less peak RAM (INT8 + SIMD, conformance-validated); declarative formats 20–60× warm, ~60× less RAM; details + methodology in [`PDF_PERFORMANCE.md`](./PDF_PERFORMANCE.md) | | **Models** | docling's own checkpoints (layout heron, TableFormer, PP-OCRv3, Whisper tiny), format-converted to ONNX by `scripts/export_*.py` — no retraining; INT8 variants are calibrated post-training quantizations (`scripts/install/quantize_models.py`) | | **Tracking upstream** | See [§9](#9-keeping-up-with-upstream-docling): conformance is measured against the *latest published* docling on demand, so an upstream release that changes output surfaces as a concrete per-fixture diff | @@ -106,7 +106,7 @@ content-type resolution, and image extraction — reused by DOCX/PPTX/XLSX/EPUB. These run docling's *discriminative* PDF pipeline ported to ONNX. They are now measured **byte-for-byte against live docling** (the committed PDF groundtruth is -regenerated from it): **6 / 16 exact (7 / 16 whitespace-normalized)**, the rest +regenerated from it): **5 / 14 exact (6 / 14 whitespace-normalized)**, the rest close — see `PDF_CONFORMANCE.md`. A deterministic snapshot baseline (`scripts/conformance/pdf_conformance.sh`) still guards against regressions. @@ -128,15 +128,18 @@ line-diffed, similarity `= 100·(1 − difflines / max_lines)`. **≈91% mean ov |---|---|---|---| | CSV / AsciiDoc / Email | **100%** | Markdown | 92% | | USPTO | 98% | ODF / LaTeX | 91% | -| DOCX / PPTX | 96% | XLSX | 87% | +| DOCX / PPTX | 96% | XLSX | 85% | | JATS | 95% | HTML | 84% | | | | WebVTT | 81% | The remaining format gaps are tracked under [issue #32](https://github.com/docling-project/docling.rs/issues/32); its children (#38–#41, #44) landed the ODF, USPTO legacy-entity, elife XML, wiki_duck and -APS-plain-text work — `pftaps` is now byte-exact (§5). PDF `.dclx` is byte-exact -on the one fixture with a groundtruth archive. +APS-plain-text work — `pftaps` is now byte-exact (§5). The PDF path now emits +full layout `` provenance (text, headings, tables, pictures, list +items, code, and page-header/footer furniture), scored against a 16-fixture +DocLang groundtruth with a ±2-grid-unit geometry tolerance — **63% mean** (§3, +`PDF_CONFORMANCE.md`). --- @@ -161,7 +164,11 @@ on the one fixture with a groundtruth archive. xlsx/html/webvtt in the 80s (full table in §2). The format-by-format gaps are tracked as [issue #32](https://github.com/docling-project/docling.rs/issues/32) and its children (#38–#41, #44). This is an **output** format; a DocLang *input* backend is - still out of scope (§5). + still out of scope (§5). For **PDF**, where the reference `` geometry + comes from docling's own layout run, the metric is scored with a ±2-grid-unit + geometry tolerance (text/structure still byte-exact): **52% exact · 63% at ±2** + (issue #32 target ≥50%); the remaining gap is model-level (TableFormer/layout/ + reading order), not serialization — see [`PDF_CONFORMANCE.md`](./PDF_CONFORMANCE.md). - **JSON** rebuilds docling's full `body`-tree-of-`$ref`s model from the `Node` tree (texts/groups/tables/pictures, labels, list grouping, table grids, @@ -233,7 +240,7 @@ These are deliberate or unavoidable divergences, not bugs. dropped (Form-XObject text and glyph-name-only fonts were the two classes it surfaced and fixed). - Output is measured **byte-for-byte against live docling** (PDF_CONFORMANCE.md): - **6 / 16 exact, 7 / 16 whitespace-normalized**, the rest close. The remaining + **5 / 14 exact, 6 / 14 whitespace-normalized**, the rest close. The remaining gaps are model-level (TableFormer structure on complex tables, layout classification, title-page reading order) plus `amt`'s fraction spacing — a docling quirk from its embedded-font OS/2 metrics that our single-spaced output diff --git a/PDF_CONFORMANCE.md b/PDF_CONFORMANCE.md index 36e2a097..78fa41fd 100644 --- a/PDF_CONFORMANCE.md +++ b/PDF_CONFORMANCE.md @@ -13,38 +13,90 @@ The groundtruth is regenerated from **live published docling**, so it agrees wit ## Current state -**6 / 16 strict** · **7 / 16 whitespace-normalized.** +**5 / 14 strict** · **6 / 14 whitespace-normalized.** (The two Korean +image-only pages `skipped_1page`/`skipped_2pages` carry no text groundtruth and +are no longer scored.) | PDF | diff | dominant remaining blocker | |---|---:|---| | picture_classification | **exact** | — | -| code_and_formula | **exact** | — | | multi_page | **exact** | — | | 2305.03393v1-pg9 | **exact** | — (TableFormer table, cell-for-cell) | | right_to_left_01 | **exact** | — (RTL period attachment) | | right_to_left_02 | **exact** | — (kashida dedup + page-number layout) | | amt_handbook_sample | 2 *(ws-ok)* | docling's spurious fraction double space — ours is more faithful | -| 2305.03393v1 | 32 | title-page reading order + author-ID run spacing | -| skipped_1page | 40 | image/diagram page (Korean); layout picture-detection | -| skipped_2pages | 44 | image/diagram pages (Korean); layout picture-detection | -| normal_4pages | 54 | reading order (heading numbering, footnote order) | -| right_to_left_03 | 66 | RTL bidi | -| table_mislabeled_as_picture | 108 | layout over-detects tables (survey rendered as tables) | -| 2206.01062 | 164 | TableFormer multi-row headers + title-page reading order | -| redp5110_sampled | 173 | TOC mis-classified as a picture; cover-page ordering | -| 2203.01017v2 | 183 | TableFormer structure + reading order | - -`amt` is the 7th under the whitespace-normalized metric: its only diff is +| code_and_formula | 5 | code block reflowed to multiple lines + trailing newline | +| 2305.03393v1 | 26 | title-page reading order + author-ID run spacing | +| normal_4pages | 44 | reading order (heading numbering, footnote order) | +| right_to_left_03 | 60 | RTL bidi | +| table_mislabeled_as_picture | 86 | layout over-detects tables (survey rendered as tables) | +| 2206.01062 | 80 | TableFormer multi-row headers + title-page reading order | +| 2203.01017v2 | 130 | TableFormer structure + reading order | +| redp5110_sampled | 194 | TOC OTSL structure (model-level); cover-page ordering | + +`amt` is the 6th under the whitespace-normalized metric: its only diff is docling's spurious double space before the `1⁄4` fraction, where our single-spaced output is the more faithful rendering. The remaining non-exact PDFs are heavy multi-column / table docs whose gaps are model-level (TableFormer structure, layout classification, title-page reading order), not text-layer. -The heavy table docs improved with the docling-parse **word-cell** grouping now -feeding TableFormer (2305.03393v1 93→32, 2203.01017v2 209→183, 2206.01062 -198→164, redp5110 226→173): the parser's per-word cells reproduce docling-parse's -`word_cells` byte-for-byte, so cell-to-grid matching tracks docling more closely. -See "Text reconstruction" below. +The heavy table docs improved with the docling-parse **word-cell** grouping +feeding TableFormer and the #61 layout/reading-order postprocessor +(2305.03393v1 93→30, 2203.01017v2 209→161, 2206.01062 198→92): the parser's +per-word cells reproduce docling-parse's `word_cells` byte-for-byte, so +cell-to-grid matching tracks docling more closely. See "Text reconstruction" +below. The #60 matching work (docling's `MatchingPostProcessor` ported to +`tf_match.rs`, plus docling's exact table-crop rounding chain) took +2203 157→150 and redp5110 204→202 with every other fixture unchanged; the +#62 text fixes (docling-parse's quote-normalization table — every curly +quote → `'` — and joining region cells in docling-parse index order +instead of geometric bands) then took 2203 →130, 2206 92→80, 2305 28→26, +normal_4pages 56→44, redp5110 →194, and table_mislabeled 88→86. + +## DocLang (`.dclx`) conformance + +Separate from the Markdown metric above: how close `--to dclx` gets to docling's +DocLang archive, scored on the extracted `document.xml` against the committed +groundtruth (`tests/data/pdf/groundtruth_dclx/*.dclx`, from published docling +2.112.0). Run `scripts/conformance/dclx_conformance.sh pdf`; sweep the tolerance +with `scripts/conformance/dclx_pdf_tol_sweep.sh`. + +**PDF avg similarity: 52 % exact · 63 % at the default ±2-grid-unit tolerance** +(issue #32 target: ≥50 %). The ±2 figure is within a point of the +*geometry-ignored* ceiling (65 %), so essentially all of the coordinate +difference is absorbed by ±2 — a wider tolerance buys almost nothing. + +### What the geometry tolerance is, and why it is honest + +Every laid-out block in a DocLang archive carries four `` provenance +tokens — its bbox as `round(512·coord/page_dim)` on a 0–511 page grid +(docling_core's `_create_location_tokens_for_bbox`). We emit the same tokens +from our layout cluster boxes (`assemble.rs`, `norm_loc`) for text, headings, +tables, pictures, list items (on `ListItem.location`), code, and the +`page_header`/`page_footer` furniture blocks. Because our heron +layout model is docling's, the boxes agree to **~1 grid unit**; the small +residual is the aspect-ratio-stretch-vs-letterbox preprocessing difference, not a +structural gap. `dclx_diff.py` therefore counts a `` pair as matching +when the two values are within `DCLX_TOL` (default **2**) grid units — **text, +tags, nesting, spans, and every non-geometry line stay byte-exact, and unmatched +lines always count against the score**. The tolerance is applied **only to PDF**, +where the reference geometry comes from docling's own layout run; formats whose +geometry is read from the same source file (OOXML slides/sheets) stay exact +(`DCLX_TOL=0`). `DCLX_TOL=0` reproduces a raw `diff` line-for-line. + +### Per-fixture (±2) + +Text/list-heavy pages land high (multi_page 82 %, right_to_left_02 82 %, +code_and_formula 81 %, 2305-pg9 78 %, right_to_left_01 75 %, amt 72 %, +normal_4pages 71 %, redp5110 65 %, 2206 61 %); the low ones are **model-level, +not provenance**: the big table papers (2203 51 %, 2305 52 %) diverge in +TableFormer cell structure (2203 alone is ~19 k table-grid diff lines), +table_mislabeled/picture_classification in layout classification, and +skipped_1/2page (Korean image pages) + right_to_left_03 in picture detection / +bidi — the *same* blockers that cap the Markdown metric. The corpus average is +bounded by these, so raising it further is a model problem, not a serialization +one: every laid-out block kind now carries provenance, so the ±2 figure sits at +the geometry-ignored ceiling. ## How the pipeline works @@ -136,19 +188,51 @@ These yield smaller or uncertain gains than the text-layer work already shipped. Each is tracked as its own issue: 1. **TableFormer structure on complex tables** - ([#60](https://github.com/docling-project/docling.rs/issues/60)). Multi-row - headers / spans on the big papers (2206, 2203) differ from docling's OTSL - prediction; one cell-structure diff cascades through the padded columns into - many row diffs (2206's ~92 table-row diffs trace to ~4 structure diffs). + ([#60](https://github.com/docling-project/docling.rs/issues/60)). The + *matching* half is done: docling's `MatchingPostProcessor` (cell-class-aware + good/bad IOU split, column-median snapping, adjacent-column de-duplication, + best-intersection word assignment, row/column-band orphan pickup) is ported + in `tf_match.rs` and is the default word→cell matcher, and the table crop + reproduces docling's exact rounding chain (`round(bbox) → ×2 → ×1024/h → + round`, banker's rounding) — 2203 157→150, redp5110 204→202, everything else + unchanged. The rest is **model-level**: the OTSL tag stream itself differs + from live docling on the hard crops (redp5110's TOC predicts `ched` where + docling gets `fcel`; multi-row headers / spans on 2206, 2203), so one + cell-structure diff still cascades through the padded columns into many row + diffs (2206's ~92 table-row diffs trace to ~4 structure diffs). A parity + harness (`DOCLING_RS_TF_MATCH_DUMP=dir` + `scripts/test/ref_match.py`-style + replay through docling's Python post-processor) confirmed the ported matcher + reproduces the reference on identical inputs, isolating the residual to the + model predictions. `DOCLING_RS_TF_SIMPLE_MATCH=1` reverts to the pre-port + best-overlap matcher. 2. **Layout classification** - ([#61](https://github.com/docling-project/docling.rs/issues/61)). The layout - ONNX classifies redp5110's table-of-contents as a *picture* (docling renders - it as a table) and table_mislabeled's survey as *tables* (docling renders - lists/text) — opposite classifications, not a text problem. + ([#61](https://github.com/docling-project/docling.rs/issues/61)) — *addressed + by porting docling's `LayoutPostprocessor`.* The raw RT-DETR detections now go + through the cleanup docling applies before assembly: per-label confidence + thresholds (`CONFIDENCE_THRESHOLDS`, stricter than the 0.3 base — a + picture/table/list needs ≥ 0.5), regular/picture/wrapper **bucketed** overlap + resolution (a high-score picture no longer suppresses a lower-score table or + table-of-contents index), the picture-vs-table cross-type rule + (`_handle_cross_type_overlaps`), and dropping a regular region absorbed by a + table/index/picture so it isn't emitted twice. With this, table_mislabeled's + survey is no longer over-detected as tables (108 → 88 vs groundtruth), and + redp5110's table-of-contents is now classified and rendered as a **table** + (`document_index`) instead of a picture. The TOC table's remaining diff is a + TableFormer dot-leader column-matching gap, tracked with the other + table-structure work in + [#60](https://github.com/docling-project/docling.rs/issues/60). *(The + groundtruth byte counts in the table above predate this change; regenerate the + committed snapshots with the `models-v1` models to refresh them.)* 3. **Complex title-page reading order** ([#62](https://github.com/docling-project/docling.rs/issues/62)). Author-block / abstract interleaving on the academic papers (band reading-order handles the - full-width title; the in-column author/abstract order is still off). + full-width title; the in-column author/abstract order is still off). Two + pieces landed: the suspected "TeX-font quote decode" gap turned out to be + docling-parse's *sanitizer* table (every curly quote → `'`; a `"` only ever + comes from a literal `quotedbl` glyph) — no font-program parsing needed — + and region cells now join in docling-parse index order (docling's + `_sort_cells`), which fixes off-baseline glyph drift like 2206's inline + math `>` landing on the wrong line. 4. **amt fraction double space (text-layer, strict-only)** ([#63](https://github.com/docling-project/docling.rs/issues/63)). docling boxes glyphs with the embedded font's OS/2 typographic metrics, not the PDF descriptor's; @@ -157,4 +241,6 @@ Each is tracked as its own issue: (the whitespace-normalized metric credits it); reproducing docling's exact spacing needs an embedded-font metrics layer, which globally entangles with RTL box geometry (a trial that fixed one `¼` regressed `right_to_left_01`). See - `MIGRATION.md` §4. + `MIGRATION.md` §4. **Resolved as by-design:** our single space is the correct + rendering, so #63 is closed without matching docling's spurious extra space — + forcing a byte-match would degrade output and risk the RTL geometry. diff --git a/crates/docling-core/src/doclang.rs b/crates/docling-core/src/doclang.rs index e48a71d2..22753076 100644 --- a/crates/docling-core/src/doclang.rs +++ b/crates/docling-core/src/doclang.rs @@ -529,10 +529,32 @@ static IDENTITY_LABELS: &[&str] = &[ /// forces the block form (matching docling); the code text follows as a text /// child (CDATA/plain glued to the closing tag, ``-wrapped text on its /// own line). Without a language, single-fragment text renders inline. -fn emit_code(out: &mut Out, depth: i32, language: Option<&str>, text: &str) { +fn emit_code( + out: &mut Out, + depth: i32, + language: Option<&str>, + text: &str, + location: Option<&[u16; 4]>, +) { let label = language.and_then(code_lang_label); let escaped = escape_text(text); let is_content_element = escaped.starts_with(""); + // Layout provenance forces the block form: `` tokens follow the + // opening ``, before the (optional) label and the code body. + if let Some(loc) = location { + out.push(depth, "".to_string()); + push_location(out, depth + 1, loc); + if let Some(l) = label { + out.push(depth + 1, format!("".to_string()); + return; + } match (label, is_content_element) { (None, false) => out.push(depth, format!("{escaped}")), (None, true) => { @@ -716,7 +738,26 @@ fn emit_nodes(out: &mut Out, depth: i32, nodes: &[Node], i: &mut usize, level: u *i += 1; } Node::Code { language, text } => { - emit_code(out, depth, language.as_deref(), text); + emit_code(out, depth, language.as_deref(), text, None); + *i += 1; + } + Node::PageFurniture { + footer, + location, + text, + } => { + let tag = if *footer { + "page_footer" + } else { + "page_header" + }; + out.push(depth, format!("<{tag}>")); + out.push(depth + 1, "".to_string()); + push_location(out, depth + 1, location); + if !text.is_empty() { + out.push(depth + 1, escape_text(text)); + } + out.push(depth, format!("")); *i += 1; } Node::Table(t) => { @@ -1289,7 +1330,12 @@ fn emit_located(out: &mut Out, depth: i32, location: &[u16; 4], inner: &Node) { t.location = Some(*location); emit_table(out, depth, &t); } + Node::Code { language, text } => { + emit_code(out, depth, language.as_deref(), text, Some(location)); + } // Other node kinds carry no location today — render them as-is. + // (Located list items are routed to emit_list by emit_nodes so they + // still group into one ``.) other => { let mut i = 0usize; emit_nodes(out, depth, std::slice::from_ref(other), &mut i, 0); diff --git a/crates/docling-core/src/document.rs b/crates/docling-core/src/document.rs index b86304f6..0c12952d 100644 --- a/crates/docling-core/src/document.rs +++ b/crates/docling-core/src/document.rs @@ -145,6 +145,15 @@ pub enum Node { location: [u16; 4], inner: Box, }, + /// A PDF page header or footer (docling's `page_header`/`page_footer` + /// furniture): DocLang emits ``/`` with a + /// `` head, the four `` tokens, then the + /// text. Markdown and JSON omit it like other furniture. + PageFurniture { + footer: bool, + location: [u16; 4], + text: String, + }, /// A page boundary — docling's implicit page break between pages. The PPTX /// backend emits one between consecutive slides. DocLang renders it as /// ``; Markdown and JSON omit it (matching docling's default diff --git a/crates/docling-core/src/json.rs b/crates/docling-core/src/json.rs index d992a326..d6fdd585 100644 --- a/crates/docling-core/src/json.rs +++ b/crates/docling-core/src/json.rs @@ -204,6 +204,7 @@ impl Builder { Node::TextDump(text) => Some(self.add_text("text", text, parent, json!({}))), // Furniture is not emitted into the body/JSON (DocLang-only layer). Node::Furniture { .. } => None, + Node::PageFurniture { .. } => None, // Layout provenance is DocLang-only; emit the wrapped node. Node::Located { inner, .. } => self.add_node(inner, parent), // Page breaks are DocLang-only; docling omits them from the JSON body. diff --git a/crates/docling-core/src/markdown.rs b/crates/docling-core/src/markdown.rs index 7ca65027..141d3ac2 100644 --- a/crates/docling-core/src/markdown.rs +++ b/crates/docling-core/src/markdown.rs @@ -410,6 +410,7 @@ fn render_one(node: &Node, blocks: &mut Vec, ctx: &mut Ctx) { // Furniture (page headers/footers, HTML ``) is excluded from // Markdown by default, mirroring docling. Node::Furniture { .. } => {} + Node::PageFurniture { .. } => {} // Layout provenance is DocLang-only; render the wrapped node. Node::Located { inner, .. } => render_one(inner, blocks, ctx), // Page breaks are DocLang-only; docling omits them from Markdown. diff --git a/crates/docling-pdf/examples/probe_page.rs b/crates/docling-pdf/examples/probe_page.rs deleted file mode 100644 index 508d2784..00000000 --- a/crates/docling-pdf/examples/probe_page.rs +++ /dev/null @@ -1,67 +0,0 @@ -//! Throwaway: dump a page's operator histogram, fonts, and XObject subtypes. -//! Usage: probe_page <pdf> <page_index> -use std::collections::BTreeMap; - -use lopdf::{Document, Object}; - -fn main() { - let args: Vec<String> = std::env::args().collect(); - let path = &args[1]; - let idx: usize = args[2].parse().unwrap(); - let doc = Document::load(path).unwrap(); - let mut pages: Vec<_> = doc.get_pages().into_iter().collect(); - pages.sort_by_key(|(n, _)| *n); - let (_, pid) = pages[idx]; - - let fonts = doc.get_page_fonts(pid).unwrap_or_default(); - println!("fonts:"); - for (name, d) in &fonts { - let st = d - .get(b"Subtype") - .ok() - .and_then(|o| o.as_name().ok()) - .map(|n| String::from_utf8_lossy(n).into_owned()) - .unwrap_or_default(); - let bf = d - .get(b"BaseFont") - .ok() - .and_then(|o| o.as_name().ok()) - .map(|n| String::from_utf8_lossy(n).into_owned()) - .unwrap_or_default(); - println!(" {} {} {}", String::from_utf8_lossy(name), st, bf); - } - - let (res, ids) = doc.get_page_resources(pid).unwrap(); - let resd = res.cloned().or_else(|| { - ids.into_iter() - .find_map(|id| doc.get_dictionary(id).ok().cloned()) - }); - if let Some(rd) = &resd { - if let Some(Object::Dictionary(xo)) = rd.get(b"XObject").ok().map(|o| match o { - Object::Reference(r) => doc.get_object(*r).unwrap().clone(), - other => other.clone(), - }) { - println!("XObjects:"); - for (name, v) in xo.iter() { - let st = match v { - Object::Reference(r) => doc - .get_object(*r) - .ok() - .and_then(|o| o.as_stream().ok()) - .and_then(|s| s.dict.get(b"Subtype").ok().and_then(|o| o.as_name().ok())) - .map(|n| String::from_utf8_lossy(n).into_owned()), - _ => None, - }; - println!(" {} {:?}", String::from_utf8_lossy(name), st); - } - } - } - - let content_bytes = doc.get_page_content(pid).unwrap(); - let content = lopdf::content::Content::decode(&content_bytes).unwrap(); - let mut hist: BTreeMap<String, usize> = BTreeMap::new(); - for op in &content.operations { - *hist.entry(op.operator.clone()).or_default() += 1; - } - println!("operators: {:?}", hist); -} diff --git a/crates/docling-pdf/src/assemble.rs b/crates/docling-pdf/src/assemble.rs index 29b22426..707f8bf9 100644 --- a/crates/docling-pdf/src/assemble.rs +++ b/crates/docling-pdf/src/assemble.rs @@ -23,9 +23,27 @@ fn inter(a: &Region, l: f32, t: f32, r: f32, b: f32) -> f32 { area(il, it, ir, ib) } +/// Wrapper (structured-region) labels, ported from docling +/// `LayoutPostprocessor.WRAPPER_TYPES`: a region that *contains* other regions +/// and renders as a structured block (a table / table-of-contents index), not as +/// its own flat text. +fn is_wrapper(label: &str) -> bool { + matches!( + label, + "table" | "document_index" | "form" | "key_value_region" + ) +} + +/// Labels docling's table-structure (TableFormer) model runs on and that render +/// as a Markdown table: a plain `table` and a `document_index` (a table of +/// contents), which docling assembles as a `TableItem` too. +pub fn is_table_like(label: &str) -> bool { + matches!(label, "table" | "document_index") +} + /// Greedily keep regions by descending score, dropping a region that is mostly /// covered by an already-kept one (RT-DETR emits overlapping duplicates). -pub fn resolve(mut regions: Vec<Region>) -> Vec<Region> { +fn greedy(mut regions: Vec<Region>) -> Vec<Region> { regions.sort_by(|a, b| b.score.total_cmp(&a.score)); let mut kept: Vec<Region> = Vec::new(); for r in regions { @@ -40,10 +58,100 @@ pub fn resolve(mut regions: Vec<Region>) -> Vec<Region> { kept.push(r); } } + kept +} + +/// Resolve overlapping RT-DETR detections, ported from the bucket structure of +/// docling's `LayoutPostprocessor`: regular, picture and wrapper clusters live in +/// **separate** spatial indexes and are de-overlapped independently, so a +/// high-score picture never suppresses a lower-score table or table-of-contents +/// index (the redp5110 TOC that was otherwise replaced by a picture box). A +/// cross-type pass first drops a picture that nearly coincides with a table +/// (`_handle_cross_type_overlaps`), keeping the structured table. +pub fn resolve(regions: Vec<Region>) -> Vec<Region> { + // Cross-type: a region proposed as BOTH a picture and a table survives twice + // (the two buckets de-overlap independently). Keep the structured table and + // drop the coincident picture — IoU (not containment), so a genuine small + // figure fully inside a large table region is not removed. + let tables: Vec<(f32, f32, f32, f32)> = regions + .iter() + .filter(|r| r.label == "table") + .map(|r| (r.l, r.t, r.r, r.b)) + .collect(); + let mut regions = regions; + regions.retain(|r| { + if r.label != "picture" { + return true; + } + let ra = area(r.l, r.t, r.r, r.b).max(1.0); + !tables.iter().any(|&(l, t, rr, b)| { + let i = inter(r, l, t, rr, b); + let u = ra + area(l, t, rr, b) - i; + u > 0.0 && i / u > 0.8 + }) + }); + // De-overlap each bucket on its own. + let pictures = greedy( + regions + .iter() + .filter(|r| r.label == "picture") + .cloned() + .collect(), + ); + let wrappers = greedy( + regions + .iter() + .filter(|r| is_wrapper(r.label)) + .cloned() + .collect(), + ); + let mut kept = greedy( + regions + .iter() + .filter(|r| r.label != "picture" && !is_wrapper(r.label)) + .cloned() + .collect(), + ); dedup_nested_code(&mut kept); + kept.extend(pictures); + kept.extend(wrappers); kept } +/// Drop a regular region that is >80% contained in a surviving special region we +/// render **as a single unit** — a picture, or a table/table-of-contents index — +/// ported from docling's "Remove regular clusters that are included in wrappers" +/// step: the special absorbs it as a child (a table cell, an in-figure label), so +/// it must not also be emitted as its own paragraph/list-item. This stops the +/// survey list-items from appearing both inside the detected table and again as +/// bullets (`table_mislabeled_as_picture`). +/// +/// `form` / `key_value_region` wrappers are deliberately **excluded**: this +/// pipeline does not render them as a structured block (they are skipped), so +/// their textual content comes precisely from the contained regular regions — +/// dropping those would erase the page (e.g. `right_to_left_03`'s form-heavy +/// pages). Runs *after* [`drop_false_pictures`] so a phantom picture can't +/// swallow real text on its way out. +pub fn drop_contained_regulars(regions: &mut Vec<Region>) { + let specials: Vec<(f32, f32, f32, f32)> = regions + .iter() + .filter(|r| r.label == "picture" || is_table_like(r.label)) + .map(|r| (r.l, r.t, r.r, r.b)) + .collect(); + if specials.is_empty() { + return; + } + regions.retain(|r| { + if r.label == "picture" || is_wrapper(r.label) { + return true; + } + let ra = area(r.l, r.t, r.r, r.b).max(1.0); + !specials + .iter() + .any(|&(l, t, rr, b)| inter(r, l, t, rr, b) / ra > 0.8) + }); +} + /// True for a bare, single-token source-code language label (`XML`, `C#`, `JSON`, /// `bash`, …) — the little header the docs render above a code block. Matched /// case-insensitively; anything with whitespace or longer than a token is out. @@ -316,66 +424,37 @@ fn is_skipped(label: &str) -> bool { | "checkbox_unselected" | "form" | "key_value_region" - | "document_index" ) } -/// Reading-order sort of regions, with two-column detection on the page. -fn order_regions<T>(items: &mut [T], page_w: f32, reg: impl Fn(&T) -> &Region) { - let cx = page_w / 2.0; - let band = page_w * 0.08; - let crossing = items +/// Reading-order sort of a page's regions, via the ported rule-based +/// [`reading_order`](crate::reading_order) predictor (docling's +/// `ReadingOrderPredictor`): an up/down geometry graph, horizontal dilation and a +/// depth-first traversal, with `page_header`/`page_footer` ordered as their own +/// groups (first/last) as docling does. +fn order_regions<T: Clone>( + items: &mut Vec<T>, + page_w: f32, + page_h: f32, + reg: impl Fn(&T) -> &Region, +) { + let boxes: Vec<(f32, f32, f32, f32)> = items .iter() - .filter(|t| { - let r = reg(t); - r.l < cx - band && r.r > cx + band + .map(|it| { + let r = reg(it); + (r.l, r.t, r.r, r.b) }) - .count(); - let two_col = !items.is_empty() - && (crossing as f32) / (items.len() as f32) < 0.25 - && items.iter().any(|t| reg(t).r <= cx) - && items.iter().any(|t| reg(t).l >= cx); - if two_col { - // Full-width regions (title, figures, wide tables spanning both columns) - // break the two-column flow into horizontal bands: within a band the left - // column reads fully then the right, and a full-width region reads after - // the band above it and before the band below. Band index = number of - // full-width regions above a region's top; column 1=left, 2=right, - // 3=full-width (so it sorts after that band's columns). - // Only a region spanning *most* of the page width is a band break (a - // title, a full-width figure/table) — a merely wide column region is not. - let full_band = page_w * 0.2; - let is_full = |r: &Region| r.l < cx - full_band && r.r > cx + full_band; - let full_tops: Vec<f32> = items - .iter() - .map(®) - .filter(|r| is_full(r)) - .map(|r| r.t) - .collect(); - let key = |r: &Region| -> (usize, u8) { - let bnd = full_tops.iter().filter(|&&ft| ft < r.t - 1.0).count(); - let col = if is_full(r) { - 3 - } else if (r.l + r.r) / 2.0 >= cx { - 2 - } else { - 1 - }; - (bnd, col) - }; - items.sort_by(|a, b| { - let (a, b) = (reg(a), reg(b)); - key(a) - .cmp(&key(b)) - .then(a.t.total_cmp(&b.t)) - .then(a.l.total_cmp(&b.l)) - }); - } else { - items.sort_by(|a, b| { - let (a, b) = (reg(a), reg(b)); - a.t.total_cmp(&b.t).then(a.l.total_cmp(&b.l)) - }); - } + .collect(); + let is_header: Vec<bool> = items + .iter() + .map(|it| reg(it).label == "page_header") + .collect(); + let is_footer: Vec<bool> = items + .iter() + .map(|it| reg(it).label == "page_footer") + .collect(); + let order = crate::reading_order::order_page(&boxes, &is_header, &is_footer, page_w, page_h); + *items = order.iter().map(|&i| items[i].clone()).collect(); } /// Clean a region's assembled text: undo soft-hyphen line wraps, map curly @@ -414,20 +493,32 @@ fn md_escape(text: &str) -> String { } fn clean_text(text: &str) -> String { - // Korean (Hangul) bodies use single straight quotes where the font's double - // curly glyph maps; docling renders `“ ”` as `'` for these fonts (normal_4pages - // `‘코로나’`), not the Latin `"`. Key on Hangul syllables so Latin docs (2305's - // genuine `quotedbl` → `"`) are unaffected. - let hangul = text.chars().any(|c| ('\u{AC00}'..='\u{D7A3}').contains(&c)); - let dquote = if hangul { "'" } else { "\"" }; + // Typographic-quote normalization follows docling-parse's sanitizer table + // (`pdf_sanitators/constants.h`): every curly quote — single *and double* — + // becomes the ASCII apostrophe `'`, and `‚` a comma. A `"` in docling's + // output only ever comes from a literal `quotedbl` glyph, never from `“ ”` + // (2206's `'text in the wild"` pairs a curly open with a literal-quote + // close). This replaces an earlier Hangul-only special case that patched + // one symptom of mapping `“ ”` to `"`. let replaced = text .replace("\u{2} ", "") .replace("\u{ad} ", "") .replace(['\u{2}', '\u{ad}'], "") // any stray wrap hyphens not at a join - .replace(['\u{2018}', '\u{2019}'], "'") // ‘ ’ → ' - .replace(['\u{201c}', '\u{201d}'], dquote) // “ ” → " (or ' for Hangul) - .replace(['\u{2013}', '\u{2014}', '\u{2212}'], "-") // – — − → - + .replace( + [ + '\u{2018}', '\u{2019}', '\u{201b}', '\u{201c}', '\u{201d}', '\u{201e}', '\u{201f}', + ], + "'", + ) // ‘ ’ ‛ “ ” „ ‟ → ' + .replace('\u{201a}', ",") // ‚ → , + .replace( + [ + '\u{2010}', '\u{2011}', '\u{2012}', '\u{2013}', '\u{2014}', '\u{2015}', '\u{2212}', + ], + "-", + ) // hyphen/dash family → - .replace('\u{2044}', "/") // ⁄ fraction slash → / + .replace('\u{2022}', "\u{b7}") // • → · (docling never emits •; inline CCS-concept separators) .replace('\u{2026}', "..."); // … → ... let out = if crate::pdfium_backend::use_dp_lines() { // The docling-parse sanitizer already placed the correct spacing (e.g. @@ -632,17 +723,26 @@ fn region_text(region: &Region, cells: &[TextCell]) -> String { .filter(|c| c.is_ascii_alphabetic()) .count(); let rtl = arabic > latin; - inside.sort_by_key(|c| { - let x = (c.l * 10.0) as i64; - ((c.t / band).round() as i64, if rtl { -x } else { x }) - }); + let dp = crate::pdfium_backend::use_dp_lines(); + if dp { + // docling orders a cluster's cells by their docling-parse cell index + // (`LayoutPostprocessor._sort_cells`: `sorted(cells, key=c.index)`) — + // the sanitizer's output order, which our `cells` slice already is. A + // geometric band-sort loses that on off-baseline glyphs: 2206's inline + // math `>` sits ~2 pt above its line's band and drifted into the next + // one, `( > 10 pages)` → `( 10 pages) … complex > tables`. + } else { + inside.sort_by_key(|c| { + let x = (c.l * 10.0) as i64; + ((c.t / band).round() as i64, if rtl { -x } else { x }) + }); + } // Join cells in reading order. With the docling-parse sanitizer the cells are // already correctly spaced words/lines, so adjacent cells join with a single // space (docling joins its line cells with a space) — matching e.g. a bold // label and its value, `LABEL` | `: value` → `LABEL : value`. The legacy // reconstruction instead joins same-band cells with a space only across a real // gap, because it can split a word into abutting segments (`الت`|`ي` → `التي`). - let dp = crate::pdfium_backend::use_dp_lines(); let mut joined = String::new(); let mut prev: Option<&&TextCell> = None; for c in &inside { @@ -667,9 +767,7 @@ fn region_text(region: &Region, cells: &[TextCell]) -> String { ); let before = joined.chars().nth_back(1); // char before the dash let next = t.chars().next(); - let dehyph = dp - && ends_dash - && before.is_some_and(|c| c.is_alphabetic()) + let alpha_dehyph = before.is_some_and(|c| c.is_alphabetic()) && next.is_some_and(|n| { // Ordinary hyphenation (lowercase continuation), or a CamelCase // compound name wrapped at the hyphen — a lowercase letter before @@ -679,6 +777,13 @@ fn region_text(region: &Region, cells: &[TextCell]) -> String { n.is_lowercase() || (n.is_uppercase() && before.is_some_and(|b| b.is_lowercase())) }); + // A number range wrapped at its hyphen (`pp. 545-` + `561` → `545561`): + // docling drops the line-wrap hyphen between two digit runs. A same-line + // range (`1162-1167`) never reaches this continuation path, so it keeps + // its hyphen. + let num_dehyph = before.is_some_and(|c| c.is_ascii_digit()) + && next.is_some_and(|n| n.is_ascii_digit()); + let dehyph = dp && ends_dash && (alpha_dehyph || num_dehyph); if dehyph { joined.pop(); } else if dp || !same_band || gap > h * 0.25 { @@ -954,6 +1059,40 @@ fn pair_code_captions(regions: &[Region]) -> Vec<Option<usize>> { /// Assemble one page from its (already overlap-resolved) layout regions and /// text cells. +/// Normalize a layout region (page points, top-left origin) to DocLang's 0–511 +/// location grid: `clamp(round(512 · coord / page_dim), 0, 511)`, per axis, +/// order `[x0, y0, x1, y1]`. Mirrors docling_core's +/// `_doclang_utils._create_location_tokens_for_bbox` (resolution 512) so the +/// emitted `<location>` tokens line up with the Python groundtruth. Our heron +/// cluster boxes match docling's to within ~1 grid unit; the residual (mainly +/// the aspect-ratio-stretch vs letterbox preprocessing difference) is absorbed +/// by the conformance harness's geometry tolerance. +fn norm_loc(region: &Region, page_w: f32, page_h: f32) -> [u16; 4] { + let q = |v: f32, dim: f32| -> u16 { + if dim <= 0.0 { + return 0; + } + let g = (512.0 * (v as f64) / (dim as f64)).round() as i64; + g.clamp(0, 511) as u16 + }; + [ + q(region.l, page_w), + q(region.t, page_h), + q(region.r, page_w), + q(region.b, page_h), + ] +} + +/// Wrap a node in its layout provenance so the DocLang serializer emits the four +/// `<location>` tokens as the element's head (Markdown/JSON render `inner` +/// unchanged). +fn located(loc: [u16; 4], inner: Node) -> Node { + Node::Located { + location: loc, + inner: Box::new(inner), + } +} + pub fn assemble_page( page: &PdfPage, regions: Vec<Region>, @@ -969,7 +1108,7 @@ pub fn assemble_page( .enumerate() .map(|(i, r)| (r, table_rows.get(i).cloned().flatten())) .collect(); - order_regions(&mut items, page.width, |it| &it.0); + order_regions(&mut items, page.width, page.height, |it| &it.0); // Float a margin page number to the front of reading order (docling parity: // right_to_left_02's bottom `11` is its first item). Stable, so everything // else keeps its order; no-op on pages without such a region. @@ -1000,32 +1139,101 @@ pub fn assemble_page( } } + // docling `ReadingOrderPredictor.predict_merges`: join a text fragment with a + // following text fragment strictly to its right (an author column that wraps + // into the next, a paragraph continuing in the next column) into one block — + // the intra-page half of docling's reading-order merges (cross-page/vertical + // continuations stay with [`merge_continuations`]). Already-consumed regions + // (paired captions, code labels) are excluded. + let region_texts: Vec<String> = regions + .iter() + .map(|r| region_text(r, &page.cells)) + .collect(); + let is_text: Vec<bool> = regions + .iter() + .enumerate() + .map(|(i, r)| r.label == "text" && !consumed[i]) + .collect(); + let is_skip: Vec<bool> = regions + .iter() + .enumerate() + .map(|(i, r)| { + consumed[i] + || matches!( + r.label, + "page_header" | "page_footer" | "table" | "picture" | "caption" | "footnote" + ) + }) + .collect(); + let boxes: Vec<(f32, f32, f32, f32)> = regions.iter().map(|r| (r.l, r.t, r.r, r.b)).collect(); + let mut merge_suffix: Vec<String> = vec![String::new(); regions.len()]; + for (head, children) in + crate::reading_order::predict_merges(&boxes, ®ion_texts, &is_text, &is_skip) + .into_iter() + .enumerate() + { + for c in children { + let t = region_texts[c].trim(); + if !t.is_empty() { + merge_suffix[head].push(' '); + merge_suffix[head].push_str(t); + } + consumed[c] = true; + } + } + for (i, region) in regions.iter().enumerate() { - if is_skipped(region.label) || consumed[i] { + if consumed[i] { + continue; + } + // Page headers/footers: docling emits them as furniture blocks + // (`<page_header>`/`<page_footer>` with a layer + location + text) at + // their reading-order position, not as body — emit them, don't skip. + if matches!(region.label, "page_header" | "page_footer") { + let text = region_text(region, &page.cells); + if !text.is_empty() { + nodes.push(Node::PageFurniture { + footer: region.label == "page_footer", + location: norm_loc(region, page.width, page_h), + text: md_escape(&text), + }); + } continue; } + if is_skipped(region.label) { + continue; + } + // Layout provenance for this region, normalized to docling's 0–511 grid. + let loc = norm_loc(region, page.width, page_h); if region.label == "picture" { // The figure pixels are cropped from the page render for image export. let caption = caption_for[i] .map(|ci| region_text(®ions[ci], &page.cells)) .filter(|t| !t.is_empty()); - nodes.push(Node::Picture { - caption, - image: crate::timing::timed("crop_region", || crop_region(page, region)), - }); + nodes.push(located( + loc, + Node::Picture { + caption, + image: crate::timing::timed("crop_region", || crop_region(page, region)), + }, + )); continue; } - let text = region_text(region, &page.cells); + let mut text = region_text(region, &page.cells); + text.push_str(&merge_suffix[i]); if text.is_empty() { continue; } match region.label { // docling renders both the document title and section headers as // `##` (it never emits a top-level `#` for PDFs), so match that. - "title" | "section_header" => nodes.push(Node::Heading { - level: 2, - text: md_escape(&text), - }), + "title" | "section_header" => nodes.push(located( + loc, + Node::Heading { + level: 2, + text: md_escape(&text), + }, + )), // docling drops the rendered bullet glyph; the Markdown serializer // adds its own `- ` marker. An item whose text opens with an `N.` // enumeration marker is an ordered item (rendered `N. text`). @@ -1042,7 +1250,7 @@ pub fn assemble_page( text: md_escape(&rest), level: 0, marker: None, - location: None, + location: Some(loc), dclx: None, href: None, layer: None, @@ -1054,8 +1262,10 @@ pub fn assemble_page( first_in_list: false, text: md_escape(&stripped), level: 0, - marker: None, - location: None, + // docling keeps the bullet as the DocLang list marker + // (`<ldiv><marker>·</marker></ldiv>`); Markdown ignores it. + marker: Some("·".into()), + location: Some(loc), dclx: None, href: None, layer: None, @@ -1065,7 +1275,7 @@ pub fn assemble_page( // TableFormer structure (cells + spans, text matched from word cells) // when available; otherwise geometric grid reconstruction; finally a // single cell. - "table" => { + "table" | "document_index" => { let rows = table_rows[i].clone().unwrap_or_else(|| { let rows = reconstruct_table(region, &page.cells); if rows.iter().any(|r| r.len() > 1) { @@ -1074,12 +1284,15 @@ pub fn assemble_page( vec![vec![text.clone()]] } }); - nodes.push(Node::Table(Table { - rows, - location: None, - structure: None, - cell_blocks: None, - })); + nodes.push(located( + loc, + Node::Table(Table { + rows, + location: None, + structure: None, + cell_blocks: None, + }), + )); } // docling does not decode formulas in the standard pipeline; it emits // a placeholder comment rather than the (garbled) raw glyph text. @@ -1100,10 +1313,13 @@ pub fn assemble_page( } else { code }; - nodes.push(Node::Code { - language: None, - text: code, - }); + nodes.push(located( + loc, + Node::Code { + language: None, + text: code, + }, + )); // docling emits the `Listing N:` caption after the code block. if let Some(ci) = code_caption_for[i] { let cap = region_text(®ions[ci], &page.cells); @@ -1113,9 +1329,12 @@ pub fn assemble_page( } } // text, caption, footnote → paragraph - _ => nodes.push(Node::Paragraph { - text: md_escape(&text), - }), + _ => nodes.push(located( + loc, + Node::Paragraph { + text: md_escape(&text), + }, + )), } } (nodes, links) @@ -1147,10 +1366,52 @@ fn paragraph_is_open(text: &str) -> bool { }) } +/// The paragraph text inside a node, looking through a [`Node::Located`] +/// provenance wrapper (PDF body paragraphs are wrapped since they carry a +/// `<location>`). Returns `None` for non-paragraph nodes. +fn as_paragraph(n: &Node) -> Option<&str> { + match n { + Node::Paragraph { text } => Some(text), + Node::Located { inner, .. } => match inner.as_ref() { + Node::Paragraph { text } => Some(text), + _ => None, + }, + _ => None, + } +} + +/// Whether a node is a picture, looking through a [`Node::Located`] wrapper. +fn is_picture_node(n: &Node) -> bool { + match n { + Node::Picture { .. } => true, + Node::Located { inner, .. } => matches!(inner.as_ref(), Node::Picture { .. }), + _ => false, + } +} + +/// A node a forward paragraph merge looks straight past: a figure the text wraps +/// around, or a page header/footer that falls between the two fragments of a +/// paragraph continuing across a page break (docling merges the body text and +/// keeps the header/footer as a separate furniture layer). +fn is_merge_trailer(n: &Node) -> bool { + is_picture_node(n) + || matches!(n, Node::PageFurniture { .. }) + || as_paragraph(n).is_some_and(looks_like_caption) +} + +/// Rebuild node `i` as a paragraph with `text`, preserving its `<location>` +/// wrapper (and thus provenance) if it had one. +fn reparagraph(node: &Node, text: String) -> Node { + match node { + Node::Located { location, .. } => located(*location, Node::Paragraph { text }), + _ => Node::Paragraph { text }, + } +} + pub(crate) fn merge_continuations(nodes: &mut Vec<Node>) { let mut i = 0; while i + 1 < nodes.len() { - let Node::Paragraph { text: a } = &nodes[i] else { + let Some(a) = as_paragraph(&nodes[i]) else { i += 1; continue; }; @@ -1172,25 +1433,21 @@ pub(crate) fn merge_continuations(nodes: &mut Vec<Node>) { // own paragraph (an above-the-figure caption that didn't pair), since the // body text resumes after the whole figure+caption block. let mut j = i + 1; - while matches!(nodes.get(j), Some(Node::Picture { .. })) - || matches!(nodes.get(j), Some(Node::Paragraph { text }) if looks_like_caption(text)) - { + while nodes.get(j).is_some_and(is_merge_trailer) { j += 1; } - let cont = matches!(nodes.get(j), Some(Node::Paragraph { text: b }) - if b.trim_start().chars().next().is_some_and(char::is_lowercase)); + let cont = nodes.get(j).and_then(as_paragraph).is_some_and(|b| { + b.trim_start() + .chars() + .next() + .is_some_and(char::is_lowercase) + }); if cont { - let a = match &nodes[i] { - Node::Paragraph { text } => text.trim_end().to_string(), - _ => unreachable!(), - }; - let b = match &nodes[j] { - Node::Paragraph { text } => text.trim_start().to_string(), - _ => unreachable!(), - }; - nodes[i] = Node::Paragraph { - text: format!("{a} {b}"), - }; + let a = as_paragraph(&nodes[i]).unwrap().trim_end().to_string(); + let b = as_paragraph(&nodes[j]).unwrap().trim_start().to_string(); + // Keep node i's provenance wrapper; docling's merged paragraph keeps + // the first fragment's geometry as its primary location. + nodes[i] = reparagraph(&nodes[i], format!("{a} {b}")); nodes.remove(j); // Re-check i: the merged paragraph may continue further. } else { @@ -1210,13 +1467,15 @@ pub(crate) fn merge_continuations(nodes: &mut Vec<Node>) { /// the whole buffer is safe to flush. fn hold_start(nodes: &[Node]) -> usize { for k in (0..nodes.len()).rev() { - match &nodes[k] { - // Skippable trailers: a forward merge looks straight past them. - Node::Picture { .. } => continue, - Node::Paragraph { text } if looks_like_caption(text) => continue, + // Skippable trailers (figures, page furniture, captions): a forward merge + // looks straight past them. + if is_merge_trailer(&nodes[k]) { + continue; + } + match as_paragraph(&nodes[k]) { // An open body paragraph might still pull a continuation off the next // page — hold from here to the end. - Node::Paragraph { text } if paragraph_is_open(text) => return k, + Some(text) if paragraph_is_open(text) => return k, // A closed paragraph, heading, table, list, etc. ends the paragraph: // nothing after it can merge backwards across it. Flush everything. _ => return nodes.len(), @@ -1529,10 +1788,11 @@ mod tests { assert_eq!(clean_text("end-to\u{2} end deep"), "end-toend deep"); // A stray wrap hyphen (no following join) is dropped. assert_eq!(clean_text("word\u{2}"), "word"); - // Typographic punctuation → ASCII. + // Typographic punctuation → ASCII: every curly quote becomes `'` + // (docling-parse's sanitizer table), a literal `"` stays. assert_eq!( - clean_text("Graph\u{2019}s \u{201c}x\u{201d}"), - "Graph's \"x\"" + clean_text("Graph\u{2019}s \u{201c}x\u{201d} \"y\""), + "Graph's 'x' \"y\"" ); assert_eq!(clean_text("a\u{2026}"), "a..."); // The dp default (the docling-parse sanitizer) preserves internal spacing diff --git a/crates/docling-pdf/src/layout.rs b/crates/docling-pdf/src/layout.rs index bd7b19f3..806d4fdc 100644 --- a/crates/docling-pdf/src/layout.rs +++ b/crates/docling-pdf/src/layout.rs @@ -43,10 +43,34 @@ pub struct Region { pub b: f32, } -/// Confidence threshold (docling-ibm-models `base_threshold`). +/// Base confidence threshold (docling-ibm-models `base_threshold`): the raw +/// RT-DETR floor before docling's `LayoutPostprocessor` applies its stricter +/// per-label thresholds ([`label_threshold`]). const THRESHOLD: f32 = 0.3; const SIDE: u32 = 640; +/// Per-label confidence threshold, ported from docling's +/// `LayoutPostprocessor.CONFIDENCE_THRESHOLDS`. The raw predictor keeps every +/// detection above the 0.3 base; the postprocessor then drops a cluster whose +/// score is below its label's threshold. Applying it here (equivalent, since +/// every per-label threshold is ≥ the 0.3 base) keeps low-confidence pictures / +/// tables / list-items out of the assembly, matching docling. +pub fn label_threshold(label: &str) -> f32 { + match label { + "section_header" + | "title" + | "code" + | "checkbox_selected" + | "checkbox_unselected" + | "form" + | "key_value_region" + | "document_index" => 0.45, + // caption, footnote, formula, list_item, page_footer, page_header, + // picture, table, text — all 0.5 in docling. + _ => 0.5, + } +} + pub struct LayoutModel { session: Session, } diff --git a/crates/docling-pdf/src/lib.rs b/crates/docling-pdf/src/lib.rs index 848aed72..cbcd2a3a 100644 --- a/crates/docling-pdf/src/lib.rs +++ b/crates/docling-pdf/src/lib.rs @@ -15,9 +15,11 @@ pub mod layout; mod mets; mod ocr; pub mod pdfium_backend; +mod reading_order; pub mod resample; pub mod tableformer; pub mod textparse; +mod tf_match; pub mod timing; use std::collections::BTreeMap; @@ -204,12 +206,22 @@ impl Worker { .predict(&page.image, page.width, page.height) }) .map_err(|e| PdfError::Layout(format!("page {}: {e}", n + 1)))?; + // docling's LayoutPostprocessor drops each detection below its label's + // confidence threshold (stricter than the 0.3 base the predictor keeps), + // before any overlap resolution. This removes the low-confidence tables / + // pictures / list-items that otherwise double-emit or mis-classify. + let mut regions = regions; + regions.retain(|r| r.score >= layout::label_threshold(r.label)); // Resolve overlapping detections once, before OCR. let mut regions = assemble::resolve(regions); // Emit text the detector missed as orphan text regions (docling parity). assemble::add_orphan_regions(&mut regions, &page.cells); // Drop phantom empty low-confidence picture boxes (docling parity). assemble::drop_false_pictures(&mut regions, &page.cells, page.width, page.height); + // A regular region fully inside a surviving table/index/picture is that + // special's child (a cell / in-figure label), not a separate block — + // remove it so it isn't emitted twice (docling parity). + assemble::drop_contained_regulars(&mut regions); // No text layer → recognise text from the page image via OCR. if page.cells.is_empty() { if self.ocr.is_none() { @@ -229,7 +241,7 @@ impl Worker { // has a table, so table-free documents never pay for TableFormer at all. let mut table_rows: Vec<Option<Vec<Vec<String>>>> = vec![None; regions.len()]; if let Some(slot) = self.tables.as_ref() { - if regions.iter().any(|r| r.label == "table") { + if regions.iter().any(|r| assemble::is_table_like(r.label)) { timing::timed("tableformer", || { let mut guard = slot.lock().unwrap(); if matches!(*guard, TfSlot::Unloaded) { @@ -242,10 +254,9 @@ impl Worker { } if let TfSlot::Ready(tf) = &mut *guard { for (i, r) in regions.iter().enumerate() { - if r.label == "table" { + if assemble::is_table_like(r.label) { table_rows[i] = tf.predict_table_rows( &page.image, - page.height, [r.l, r.t, r.r, r.b], &page.word_cells, ); diff --git a/crates/docling-pdf/src/reading_order.rs b/crates/docling-pdf/src/reading_order.rs new file mode 100644 index 00000000..f8d5f32a --- /dev/null +++ b/crates/docling-pdf/src/reading_order.rs @@ -0,0 +1,389 @@ +//! Rule-based reading order, ported from docling-ibm-models +//! `reading_order/reading_order_rb.py` (`ReadingOrderPredictor`). +//! +//! For each page it builds an up/down adjacency graph between elements purely +//! from geometry — an element is "below" another that is *strictly above* it and +//! *horizontally overlapping*, unless a third element interrupts the vertical +//! run between them — then horizontally **dilates** narrow elements toward their +//! column neighbours (so a one-line box widens to its paragraph's column), redoes +//! the graph, and depth-first traverses from the top-most elements to produce the +//! reading sequence. This reproduces docling's multi-column reading order (author +//! blocks, two-column body text) that a purely geometric top-to-bottom sort gets +//! wrong. +//! +//! Everything runs in **bottom-left origin** (y grows upward), matching docling; +//! callers pass top-left page coordinates and the page height. The `l2r`/`r2l` +//! maps are omitted because docling disables them (a `False and …` guard). + +const EPS: f32 = 1.0e-3; +/// Horizontal-dilation threshold, normalized by page width +/// (`_horizontal_dilation_threshold_norm`). +const DILATION_THRESHOLD_NORM: f32 = 0.15; + +/// A page element's box in bottom-left origin: `t > b` (top edge higher). +#[derive(Clone, Copy)] +struct Bl { + l: f32, + b: f32, + r: f32, + t: f32, +} + +impl Bl { + /// `overlaps_horizontally` (docling_core `BoundingBox`). + fn overlaps_h(&self, o: &Bl) -> bool { + !(self.r <= o.l || o.r <= self.l) + } + /// `is_strictly_above` (bottom-left branch): self's bottom edge sits above + /// other's top edge. + fn strictly_above(&self, o: &Bl) -> bool { + (self.b + EPS) > o.t + } + /// `PageElement.__lt__` for same-page elements: a horizontally-overlapping + /// pair reads higher-first (larger bottom edge in bottom-left), otherwise the + /// left-most reads first. Returns whether `self` reads before `other`. + fn before(&self, o: &Bl) -> bool { + if self.overlaps_h(o) { + self.b > o.b + } else { + self.l < o.l + } + } +} + +/// Build the up/down adjacency maps (`_init_ud_maps`). `up[j]` lists elements +/// directly above `j`; `dn[i]` lists elements directly below `i`. The rtree of +/// the original is replaced by a brute-force scan (pages carry few regions) with +/// the identical predicates, so the edge set matches. +fn init_ud(elems: &[Bl]) -> (Vec<Vec<usize>>, Vec<Vec<usize>>) { + let n = elems.len(); + let mut up = vec![Vec::new(); n]; + let mut dn = vec![Vec::new(); n]; + for j in 0..n { + for i in 0..n { + if i == j { + continue; + } + if !(elems[i].strictly_above(&elems[j]) && elems[i].overlaps_h(&elems[j])) { + continue; + } + if has_interruption(elems, i, j) { + continue; + } + dn[i].push(j); + up[j].push(i); + } + } + (up, dn) +} + +/// `_has_sequence_interruption`: a third element `w` between `i` and `j` +/// vertically (strictly below `i`, strictly above `j`) that overlaps either +/// horizontally breaks the direct `i → j` link. +fn has_interruption(elems: &[Bl], i: usize, j: usize) -> bool { + for (w, ew) in elems.iter().enumerate() { + if w == i || w == j { + continue; + } + if (elems[i].overlaps_h(ew) || elems[j].overlaps_h(ew)) + && elems[i].strictly_above(ew) + && ew.strictly_above(&elems[j]) + { + return true; + } + } + false +} + +/// `_do_horizontal_dilation`: widen each element toward its first up- and +/// down-neighbour's horizontal extent, but only while the growth on each side +/// stays under the page-width threshold (else the element is left untouched). +fn dilate(orig: &[Bl], up: &[Vec<usize>], dn: &[Vec<usize>], page_w: f32) -> Vec<Bl> { + let th = DILATION_THRESHOLD_NORM * page_w; + let mut dil = orig.to_vec(); + for i in 0..orig.len() { + let mut x0 = orig[i].l; + let mut x1 = orig[i].r; + let mut skip = false; + if let Some(&u) = up[i].first() { + let x0d = x0.min(orig[u].l); + let x1d = x1.max(orig[u].r); + if (x0 - x0d) > th || (x1d - x1) > th { + skip = true; + } else { + x0 = x0d; + x1 = x1d; + } + } + if !skip { + if let Some(&d) = dn[i].first() { + let x0d = x0.min(orig[d].l); + let x1d = x1.max(orig[d].r); + if (x0 - x0d) > th || (x1d - x1) > th { + skip = true; + } else { + x0 = x0d; + x1 = x1d; + } + } + } + if !skip { + dil[i].l = x0; + dil[i].r = x1; + } + } + dil +} + +/// Iterative `_depth_first_search_upwards`: climb `up` edges to the top-most +/// not-yet-visited ancestor of `j`. +fn dfs_up(j: usize, up: &[Vec<usize>], visited: &[bool]) -> usize { + let mut k = j; + loop { + let mut moved = false; + for &ind in &up[k] { + if !visited[ind] { + k = ind; + moved = true; + break; + } + } + if !moved { + return k; + } + } +} + +/// Iterative `_depth_first_search_downwards` from `start`, appending to `order`. +fn dfs_down( + start: usize, + up: &[Vec<usize>], + dn: &[Vec<usize>], + order: &mut Vec<usize>, + visited: &mut [bool], +) { + // Each frame is (node, next child offset into dn[node]). + let mut stack: Vec<(usize, usize)> = vec![(start, 0)]; + while let Some(&(node, off)) = stack.last() { + let mut found = false; + let mut o = off; + while o < dn[node].len() { + let k = dfs_up(dn[node][o], up, visited); + if !visited[k] { + order.push(k); + visited[k] = true; + stack.last_mut().unwrap().1 = o + 1; + stack.push((k, 0)); + found = true; + break; + } + o += 1; + } + if !found { + stack.pop(); + } + } +} + +/// Reading order of one group of page elements (already in bottom-left origin). +/// Returns the permutation of input indices in reading order. +fn predict(orig: &[Bl], page_w: f32) -> Vec<usize> { + let n = orig.len(); + if n == 0 { + return Vec::new(); + } + // Adjacency from the dilated boxes, but head/child sorting from the original + // geometry (docling's `_find_heads`/`_sort_ud_maps` take `page_elements`). + let (up0, dn0) = init_ud(orig); + let dil = dilate(orig, &up0, &dn0, page_w); + let (up, mut dn) = init_ud(&dil); + + let by_geom = |a: usize, b: usize| orig[a].before(&orig[b]); + + let mut heads: Vec<usize> = (0..n).filter(|&i| up[i].is_empty()).collect(); + py_sort(&mut heads, by_geom); + for children in dn.iter_mut() { + py_sort(children, by_geom); + } + + let mut order = Vec::with_capacity(n); + let mut visited = vec![false; n]; + for &h in &heads { + if !visited[h] { + order.push(h); + visited[h] = true; + dfs_down(h, &up, &dn, &mut order, &mut visited); + } + } + // A malformed graph could leave elements unreached; append them in geometric + // order so nothing is dropped (docling logs an error and returns short). + if order.len() != n { + let mut rest: Vec<usize> = (0..n).filter(|&i| !visited[i]).collect(); + py_sort(&mut rest, by_geom); + order.extend(rest); + } + order +} + +/// CPython's list sort for the sizes a page produces, driven by a `<` +/// comparator: detect the leading natural run (reversing a strictly descending +/// prefix), then stable binary-insert the rest — exactly Timsort for fewer +/// than 64 elements (`minrun == n`). docling sorts with +/// `functools.cmp_to_key(PageElement.__lt__)`, whose fuzzy geometric relation +/// (`before`) is **not** a strict total order on mixed overlap groups: Python +/// quietly produces the Timsort order, while `slice::sort_by` panics on the +/// Ord violation (the Korean OCR pages trip it). For a consistent comparator +/// this is just another stable sort, so already-conformant fixtures are +/// unaffected. +fn py_sort(v: &mut [usize], mut lt: impl FnMut(usize, usize) -> bool) { + let n = v.len(); + if n < 2 { + return; + } + // count_run: extend an ascending (`!lt(next, prev)`) or strictly + // descending run from the start; a descending run is reversed in place. + let mut run = 1; + if lt(v[1], v[0]) { + while run + 1 < n && lt(v[run + 1], v[run]) { + run += 1; + } + v[..=run].reverse(); + } else { + while run + 1 < n && !lt(v[run + 1], v[run]) { + run += 1; + } + } + // binarysort: stable binary insertion of the remainder. + for i in run + 1..n { + let pivot = v[i]; + let (mut lo, mut hi) = (0, i); + while lo < hi { + let mid = (lo + hi) / 2; + if lt(pivot, v[mid]) { + hi = mid; + } else { + lo = mid + 1; + } + } + v.copy_within(lo..i, lo + 1); + v[lo] = pivot; + } +} + +/// `predict_merges` (docling `ReadingOrderPredictor`): given elements already in +/// reading order, return for each head the ordered list of following elements to +/// merge into it. A merge chain starts at a TEXT element and extends to the next +/// non-skipped TEXT element as long as the head is *strictly left of* it +/// (horizontally adjacent — an author column, a wrap into the next column) and +/// the running tail ends with a lowercase letter / comma / hyphen while the +/// candidate starts with a letter. `boxes` are top-left; `strictly_left_of` is +/// horizontal-only, so origin does not matter. The cross-page branch is omitted — +/// this runs per page, where cross-page continuations are handled separately. +pub fn predict_merges( + boxes: &[(f32, f32, f32, f32)], + texts: &[String], + is_text: &[bool], + is_skip: &[bool], +) -> Vec<Vec<usize>> { + let n = boxes.len(); + let mut merges = vec![Vec::new(); n]; + let mut curr: isize = -1; + for ind in 0..n { + if ind as isize <= curr || !is_text[ind] { + continue; + } + let mut check = ind; + loop { + let mut p1 = check + 1; + while p1 < n && is_skip[p1] { + p1 += 1; + } + if p1 < n + && is_text[p1] + && strictly_left_of(boxes[ind], boxes[p1]) + && ends_mergeable(&texts[check]) + && starts_mergeable(&texts[p1]) + { + merges[ind].push(p1); + curr = p1 as isize; + check = p1; + } else { + break; + } + } + } + merges +} + +/// `is_strictly_left_of` (horizontal-only): `a.r + eps < b.l`. +fn strictly_left_of(a: (f32, f32, f32, f32), b: (f32, f32, f32, f32)) -> bool { + a.2 + EPS < b.0 +} + +/// Head/tail of a merge ends with a lowercase letter, comma, hyphen or soft +/// hyphen (docling regex `.+([a-z,\-­])(\s*)` — at least two chars). +fn ends_mergeable(t: &str) -> bool { + let s = t.trim_end(); + s.chars().count() >= 2 + && matches!( + s.chars().next_back(), + Some('a'..='z' | ',' | '-' | '\u{ad}') + ) +} + +/// Merge candidate starts with a (Latin) letter and has more after it (docling +/// regex `(\s*[a-zA-ZÀ-ɏ])(.+)`). +fn starts_mergeable(t: &str) -> bool { + let s = t.trim_start(); + let mut ch = s.chars(); + match ch.next() { + Some(c) if c.is_ascii_alphabetic() || ('\u{c0}'..='\u{24f}').contains(&c) => { + ch.next().is_some() + } + _ => false, + } +} + +/// Order one page's elements (top-left coords) into reading order, returning the +/// input-index permutation. `headers`/`footers` are ordered as their own groups +/// and placed first/last, matching docling's per-page header→body→footer split. +pub fn order_page( + boxes: &[(f32, f32, f32, f32)], + is_header: &[bool], + is_footer: &[bool], + page_w: f32, + page_h: f32, +) -> Vec<usize> { + // Split into the three groups, remembering original indices. + let mut groups: [Vec<usize>; 3] = [Vec::new(), Vec::new(), Vec::new()]; + for i in 0..boxes.len() { + let g = if is_header[i] { + 0 + } else if is_footer[i] { + 2 + } else { + 1 + }; + groups[g].push(i); + } + let mut out = Vec::with_capacity(boxes.len()); + for group in groups { + // Convert this group to bottom-left origin. + let bl: Vec<Bl> = group + .iter() + .map(|&i| { + let (l, t, r, b) = boxes[i]; + Bl { + l, + r, + t: page_h - t, + b: page_h - b, + } + }) + .collect(); + for local in predict(&bl, page_w) { + out.push(group[local]); + } + } + out +} diff --git a/crates/docling-pdf/src/tableformer.rs b/crates/docling-pdf/src/tableformer.rs index 7e7491e1..722aaab5 100644 --- a/crates/docling-pdf/src/tableformer.rs +++ b/crates/docling-pdf/src/tableformer.rs @@ -36,7 +36,8 @@ pub const RHED: i64 = 11; // row header pub const SROW: i64 = 12; // section row /// A predicted table cell: an OTSL grid position (with spans) + its box in the -/// 448 image normalized cxcywh, and the OTSL tag. +/// 448 image normalized cxcywh, the OTSL tag, and the bbox decoder's cell +/// class (docling's `cell_class`; 2 = full, ≤1 = predicted empty). #[derive(Debug, Clone)] pub struct TableCell { pub row: usize, @@ -44,6 +45,7 @@ pub struct TableCell { pub colspan: usize, pub rowspan: usize, pub tag: i64, + pub class: i64, pub cx: f32, pub cy: f32, pub w: f32, @@ -385,34 +387,46 @@ impl TableFormer { .chunks_exact(4) .map(|c| [c[0], c[1], c[2], c[3]]) .collect(); - let merged = merge_spans(&boxes, &merge); - Ok(build_table_cells(&otsl, &merged)) + // Per-cell class logits [n, 3] → argmax (docling's `outputs_class`). + let (_, craw) = bout["classes"] + .try_extract_tensor::<f32>() + .map_err(|e| format!("tableformer: classes: {e}"))?; + let classes: Vec<i64> = craw.chunks_exact(3).map(|c| argmax(c) as i64).collect(); + let (merged, merged_classes) = merge_spans(&boxes, &classes, &merge); + Ok(build_table_cells(&otsl, &merged, &merged_classes)) } /// Predict a table region's Markdown grid: crop the region (docling's - /// page→1024px box-average then bbox crop), run the structure model, map each - /// cell box back to page points, match the page's word cells into cells by - /// intersection-over-word-area, and expand spans into a dense `rows × cols` - /// grid. `region` is `(l, t, r, b)` in page points (top-left). Returns `None` - /// if no structure is predicted. + /// page→1024px box-average then bbox crop), run the structure model, then + /// match the page's word cells into the predicted cells with docling's + /// matching post-processor ([`crate::tf_match`]) and expand spans into a + /// dense `rows × cols` grid. `region` is `(l, t, r, b)` in page points + /// (top-left). Returns `None` if no structure is predicted. pub fn predict_table_rows( &mut self, page_image: &RgbImage, - page_h: f32, region: [f32; 4], words: &[TextCell], ) -> Option<Vec<Vec<String>>> { // page → 1024px height (cv2.INTER_AREA), then crop the table bbox. + // docling's coordinate chain, rounding included: the cluster bbox is + // rounded to integer page points *first* (`round(cluster.bbox.l) * + // scale`, banker's rounding), scaled by 2 (its table-structure page + // scale), then by `1024 / <2x page-image height>`, and the crop indices + // round again. Rounding after scaling instead shifts some crops by a + // pixel — enough to change TableFormer's cell boxes on tall tables + // (redp5110's TOC). let sf = 1024.0 / page_image.height() as f32; let pw = (page_image.width() as f32 * sf) as u32; let page1024 = crate::timing::timed("tableformer.inter_area", || { crate::resample::inter_area(page_image, pw, 1024) }); - let k = 1024.0 / page_h; - let x = (region[0] * k).round().max(0.0) as u32; - let y = (region[1] * k).round().max(0.0) as u32; - let x2 = ((region[2] * k).round() as u32).min(page1024.width()); - let y2 = ((region[3] * k).round() as u32).min(page1024.height()); + let k = 2.0 * 1024.0 / page_image.height() as f64; + let px = |v: f32| (v as f64).round_ties_even() * k; + let x = (px(region[0]).round_ties_even()).max(0.0) as u32; + let y = (px(region[1]).round_ties_even()).max(0.0) as u32; + let x2 = (px(region[2]).round_ties_even() as u32).min(page1024.width()); + let y2 = (px(region[3]).round_ties_even() as u32).min(page1024.height()); if x2 <= x || y2 <= y { return None; } @@ -424,6 +438,34 @@ impl TableFormer { if cells.is_empty() { return None; } + // Words that belong to the table: non-empty text, ≥80 % of the word's + // area inside the table region (docling's `get_cells_in_bbox` ios test). + // Ids stay the page-level word indices so text joins in stream order. + let table_words: Vec<crate::tf_match::PdfWord> = words + .iter() + .enumerate() + .filter(|(_, w)| !w.text.trim().is_empty()) + .filter_map(|(wi, w)| { + let (l, t, r, b) = (w.l as f64, w.t as f64, w.r as f64, w.b as f64); + let area = (r - l) * (b - t); + let iw = (r.min(region[2] as f64) - l.max(region[0] as f64)).max(0.0); + let ih = (b.min(region[3] as f64) - t.max(region[1] as f64)).max(0.0); + if area > 0.0 && iw * ih / area > 0.8 { + Some(crate::tf_match::PdfWord { + id: wi, + bbox: [l, t, r, b], + text: w.text.trim().to_string(), + }) + } else { + None + } + }) + .collect(); + + if !table_words.is_empty() && !simple_match() { + return docling_match_rows(&cells, region, &table_words, words); + } + let (rw, rh) = (region[2] - region[0], region[3] - region[1]); // Cell boxes in page points (top-left), aligned with `cells`. @@ -473,6 +515,7 @@ impl TableFormer { .map(|&i| words[i].text.trim()) .collect::<Vec<_>>() .join(" "); + let text = normalize_cell_text(text); // Spanned cells repeat their text across the covered grid positions. for row in grid.iter_mut().skip(c.row).take(c.rowspan) { for cell in row.iter_mut().skip(c.col).take(c.colspan) { @@ -484,6 +527,230 @@ impl TableFormer { } } +/// Append one JSON line per table into `<dir>/tf_match_dump.jsonl` with the +/// exact matcher inputs (hand-rolled JSON to avoid a serde dependency). +fn dump_match_inputs( + dir: &str, + tf_cells: &[crate::tf_match::TfCell], + words: &[crate::tf_match::PdfWord], +) { + use std::io::Write; + let cells: Vec<String> = tf_cells + .iter() + .map(|c| { + format!( + r#"{{"bbox":[{},{},{},{}],"cell_id":{},"row_id":{},"column_id":{},"cell_class":{},"colspan_val":{},"rowspan_val":{}}}"#, + c.bbox[0], c.bbox[1], c.bbox[2], c.bbox[3], + c.cell_id, c.row_id, c.column_id, c.cell_class, + c.colspan_val, c.rowspan_val + ) + }) + .collect(); + let ws: Vec<String> = words + .iter() + .map(|w| { + format!( + r#"{{"id":{},"bbox":[{},{},{},{}],"text":{}}}"#, + w.id, + w.bbox[0], + w.bbox[1], + w.bbox[2], + w.bbox[3], + serde_json_escape(&w.text) + ) + }) + .collect(); + let line = format!( + r#"{{"table_cells":[{}],"pdf_cells":[{}]}}"#, + cells.join(","), + ws.join(",") + ); + if let Ok(mut f) = std::fs::OpenOptions::new() + .create(true) + .append(true) + .open(format!("{dir}/tf_match_dump.jsonl")) + { + let _ = writeln!(f, "{line}"); + } +} + +/// Minimal JSON string escaping for the parity dump. +fn serde_json_escape(s: &str) -> String { + let mut out = String::with_capacity(s.len() + 2); + out.push('"'); + for ch in s.chars() { + match ch { + '"' => out.push_str("\\\""), + '\\' => out.push_str("\\\\"), + '\n' => out.push_str("\\n"), + '\r' => out.push_str("\\r"), + '\t' => out.push_str("\\t"), + c if (c as u32) < 0x20 => out.push_str(&format!("\\u{:04x}", c as u32)), + c => out.push(c), + } + } + out.push('"'); + out +} + +/// `DOCLING_RS_TF_SIMPLE_MATCH=1` reverts to the pre-#60 best-overlap word +/// assignment (A/B escape hatch for the docling matching post-processor). +fn simple_match() -> bool { + std::env::var("DOCLING_RS_TF_SIMPLE_MATCH").is_ok_and(|v| !v.is_empty() && v != "0") +} + +/// docling glues `@` to whatever follows it (`mAP @0.5`, an email): +/// the PDF's word cells split `@` from the next token, and joining them +/// with a space would widen the cell and — via the column pad — shift +/// every row of the table. The groundtruth never contains "@ ", so this +/// is always the right normalization. +fn normalize_cell_text(text: String) -> String { + text.replace("@ ", "@") +} + +/// docling's matched-cell grid assembly (`tf_predictor.predict` with +/// `do_cell_matching=True`): run the ported matching post-processor, group the +/// word→cell assignments per grid position, compress the surviving row/column +/// ids to sequential indexes, and expand spans into a dense `rows × cols` text +/// grid. Matching runs in docling's coordinate space — the table bbox rounded +/// to integers, everything ×2 (its page scale) — so the post-processor's +/// absolute rounding agrees. +fn docling_match_rows( + cells: &[TableCell], + region: [f32; 4], + table_words: &[crate::tf_match::PdfWord], + words: &[TextCell], +) -> Option<Vec<Vec<String>>> { + const SCALE: f64 = 2.0; // docling's table-structure page scale + let sl = (region[0] as f64).round_ties_even() * SCALE; + let st = (region[1] as f64).round_ties_even() * SCALE; + let sr = (region[2] as f64).round_ties_even() * SCALE; + let sb = (region[3] as f64).round_ties_even() * SCALE; + let (w2, h2) = (sr - sl, sb - st); + + let tf_cells: Vec<crate::tf_match::TfCell> = cells + .iter() + .enumerate() + .map(|(i, c)| { + let (cx, cy) = (c.cx as f64, c.cy as f64); + let (w, h) = (c.w as f64, c.h as f64); + crate::tf_match::TfCell { + bbox: [ + sl + (cx - w / 2.0) * w2, + st + (cy - h / 2.0) * h2, + sl + (cx + w / 2.0) * w2, + st + (cy + h / 2.0) * h2, + ], + cell_id: i, + row_id: c.row, + column_id: c.col, + cell_class: c.class, + colspan_val: if c.colspan > 1 { c.colspan } else { 0 }, + rowspan_val: if c.rowspan > 1 { c.rowspan } else { 0 }, + } + }) + .collect(); + + let scaled_words: Vec<crate::tf_match::PdfWord> = table_words + .iter() + .map(|w| crate::tf_match::PdfWord { + id: w.id, + bbox: [ + w.bbox[0] * SCALE, + w.bbox[1] * SCALE, + w.bbox[2] * SCALE, + w.bbox[3] * SCALE, + ], + text: w.text.clone(), + }) + .collect(); + + // Debug: dump the matcher inputs as JSON lines for a side-by-side run + // against docling's Python post-processor (parity harness, not a feature). + if let Ok(dir) = std::env::var("DOCLING_RS_TF_MATCH_DUMP") { + if !dir.is_empty() { + dump_match_inputs(&dir, &tf_cells, &scaled_words); + } + } + + let (cells_wo, final_matches) = + crate::tf_match::match_and_post_process(tf_cells, &scaled_words); + + // `_merge_tf_output`: group per (column, row) in ascending-pdf-id order; + // the first word's table cell fixes the group's offsets and spans. + struct Merged { + start_row: usize, + start_col: usize, + row_span: usize, + col_span: usize, + word_ids: Vec<usize>, + } + let mut merged: Vec<Merged> = Vec::new(); + let mut key_ix: std::collections::HashMap<(usize, usize), usize> = + std::collections::HashMap::new(); + for (&pdf_id, list) in &final_matches { + let tm = list[0].table_cell_id; + let Some(cell) = cells_wo.iter().find(|c| c.cell_id == tm) else { + continue; + }; + match key_ix.entry((cell.column_id, cell.row_id)) { + std::collections::hash_map::Entry::Occupied(e) => { + merged[*e.get()].word_ids.push(pdf_id); + } + std::collections::hash_map::Entry::Vacant(e) => { + e.insert(merged.len()); + merged.push(Merged { + start_row: cell.row_id, + start_col: cell.column_id, + row_span: cell.rowspan_val.max(1), + col_span: cell.colspan_val.max(1), + word_ids: vec![pdf_id], + }); + } + } + } + if merged.is_empty() { + return None; + } + + // `multi_table_predict`'s sort_row_col_indexes: compress the surviving + // row/column ids to gap-free indexes. + let mut start_cols: Vec<usize> = merged.iter().map(|m| m.start_col).collect(); + start_cols.sort_unstable(); + start_cols.dedup(); + let mut start_rows: Vec<usize> = merged.iter().map(|m| m.start_row).collect(); + start_rows.sort_unstable(); + start_rows.dedup(); + let mut num_rows = 0; + let mut num_cols = 0; + for m in &mut merged { + m.start_col = start_cols.binary_search(&m.start_col).expect("own value"); + m.start_row = start_rows.binary_search(&m.start_row).expect("own value"); + num_cols = num_cols.max(m.start_col + m.col_span); + num_rows = num_rows.max(m.start_row + m.row_span); + } + if num_rows == 0 || num_cols == 0 { + return None; + } + + let mut grid = vec![vec![String::new(); num_cols]; num_rows]; + for m in &merged { + let text = m + .word_ids + .iter() + .map(|&i| words[i].text.trim()) + .collect::<Vec<_>>() + .join(" "); + let text = normalize_cell_text(text); + for row in grid.iter_mut().skip(m.start_row).take(m.row_span) { + for cell in row.iter_mut().skip(m.start_col).take(m.col_span) { + *cell = text.clone(); + } + } + } + Some(grid) +} + /// Note once per process that TableFormer's ONNX graphs weren't found, so tables /// fall back to geometric reconstruction. The default paths are relative /// (`models/tableformer/*.onnx`), which only resolves when the process's current @@ -551,23 +818,32 @@ fn mergebboxes(b1: [f32; 4], b2: [f32; 4]) -> [f32; 4] { } /// Apply docling's span merges: each merge key combines its box with the partner -/// (`-1` → the last box); partners are dropped. -fn merge_spans(boxes: &[[f32; 4]], merge: &std::collections::HashMap<usize, i64>) -> Vec<[f32; 4]> { +/// (`-1` → the last box); partners are dropped. The merged cell keeps the +/// *first* box's class, matching docling's `outputs_class1.append(cls1)`. +fn merge_spans( + boxes: &[[f32; 4]], + classes: &[i64], + merge: &std::collections::HashMap<usize, i64>, +) -> (Vec<[f32; 4]>, Vec<i64>) { let skip: std::collections::HashSet<usize> = merge .values() .filter(|&&v| v >= 0) .map(|&v| v as usize) .collect(); let mut out = Vec::new(); + let mut out_classes = Vec::new(); for (i, &b) in boxes.iter().enumerate() { + let class = classes.get(i).copied().unwrap_or(2); if let Some(&j) = merge.get(&i) { let partner = if j < 0 { boxes.len() - 1 } else { j as usize }; out.push(mergebboxes(b, boxes[partner.min(boxes.len() - 1)])); + out_classes.push(class); } else if !skip.contains(&i) { out.push(b); + out_classes.push(class); } } - out + (out, out_classes) } const CELL_TAGS: [i64; 6] = [FCEL, ECEL, XCEL, CHED, RHED, SROW]; @@ -577,7 +853,7 @@ const CELL_TAGS: [i64; 6] = [FCEL, ECEL, XCEL, CHED, RHED, SROW]; /// (counted toward the column index but not cells). Colspan/rowspan are read off /// the grid (consecutive `lcel`/`ucel` to the right/below). `boxes` are indexed /// by cell order and aligned with the cells. -fn build_table_cells(otsl: &[i64], boxes: &[[f32; 4]]) -> Vec<TableCell> { +fn build_table_cells(otsl: &[i64], boxes: &[[f32; 4]], classes: &[i64]) -> Vec<TableCell> { // 2D grid of tags (rows split on NL) for span lookups. let mut grid: Vec<Vec<i64>> = vec![Vec::new()]; for &t in otsl { @@ -607,12 +883,15 @@ fn build_table_cells(otsl: &[i64], boxes: &[[f32; 4]]) -> Vec<TableCell> { rowspan += 1; } let b = boxes.get(cell_id).copied().unwrap_or([0.0; 4]); + // docling defaults a class-less cell to 2 (full). + let class = classes.get(cell_id).copied().unwrap_or(2); cells.push(TableCell { row: r, col: c, colspan, rowspan, tag, + class, cx: b[0], cy: b[1], w: b[2], diff --git a/crates/docling-pdf/src/tf_match.rs b/crates/docling-pdf/src/tf_match.rs new file mode 100644 index 00000000..64a21bfa --- /dev/null +++ b/crates/docling-pdf/src/tf_match.rs @@ -0,0 +1,616 @@ +//! docling's TableFormer cell matching, ported from docling-ibm-models +//! (`tf_cell_matcher.py` + `matching_post_processor.py`, and the response +//! assembly in `tf_predictor.py`). The predicted table cells and the page's +//! word cells are matched by intersection-over-word-area, then the +//! post-processor snaps unmatched cells to their column's median position, +//! de-duplicates columns, assigns each word to its best cell, and finally +//! picks up "orphan" words (no overlap with any cell — dot leaders in a TOC, +//! stray page numbers) through row/column banding. Everything is kept +//! bug-for-bug faithful, including duplicate `good` cells weighting the +//! column medians and Python's banker's rounding on orphan depths. +//! +//! Coordinates: docling matches in its 2x-scaled page space with the table +//! bbox rounded to integers *before* scaling (`round(cluster.bbox.l) * scale`); +//! the caller reproduces that space so absolute constants (orphan-depth +//! rounding) agree. + +use std::collections::BTreeMap; + +/// docling's `table_cell` dict: an OTSL grid cell with its predicted box in +/// the matching coordinate space. `colspan_val`/`rowspan_val` are 0 when the +/// cell has no span (the dict key is absent in docling). +#[derive(Debug, Clone)] +pub struct TfCell { + pub bbox: [f64; 4], + pub cell_id: usize, + pub row_id: usize, + pub column_id: usize, + pub cell_class: i64, + pub colspan_val: usize, + pub rowspan_val: usize, +} + +/// A page word cell in the matching space (docling's `pdf_cells` token). +#[derive(Debug, Clone)] +pub struct PdfWord { + pub id: usize, + pub bbox: [f64; 4], + pub text: String, +} + +/// One `{table_cell_id, iopdf|post}` match entry. +#[derive(Debug, Clone, PartialEq)] +pub struct MatchEntry { + pub table_cell_id: usize, + pub score: f64, +} + +/// pdf-cell-id → its match list. BTreeMap keeps deterministic ascending-id +/// iteration; docling's insertion-ordered dicts never depend on their order +/// for the final result (the response is re-sorted by pdf cell id). +pub type Matches = BTreeMap<usize, Vec<MatchEntry>>; + +/// docling's `find_intersection`, including the source's harmless +/// `b2[1] > b2[3]` self-comparison typo (never true for a valid box). +fn find_intersection(b1: &[f64; 4], b2: &[f64; 4]) -> Option<[f64; 4]> { + if b1[2] < b2[0] || b2[2] < b1[0] || b1[1] > b2[3] { + return None; + } + Some([ + b1[0].max(b2[0]), + b1[1].max(b2[1]), + b1[2].min(b2[2]), + b1[3].min(b2[3]), + ]) +} + +/// `CellMatcher._intersection_over_pdf_match`: every (table cell, word) pair +/// with a positive intersection-over-word-area becomes a match entry; exact +/// duplicates (same cell id *and* score — duplicated `good` cells) are +/// suppressed like Python's `match not in matches[id]`. +fn intersection_over_pdf_match(table_cells: &[TfCell], pdf_cells: &[PdfWord]) -> Matches { + let mut matches: Matches = BTreeMap::new(); + for cell in table_cells { + for word in pdf_cells { + let Some(ib) = find_intersection(&cell.bbox, &word.bbox) else { + continue; + }; + let warea = (word.bbox[2] - word.bbox[0]) * (word.bbox[3] - word.bbox[1]); + let iarea = (ib[2] - ib[0]) * (ib[3] - ib[1]); + let iopdf = if warea > 0.0 { iarea / warea } else { 0.0 }; + if iopdf > 0.0 { + let entry = MatchEntry { + table_cell_id: cell.cell_id, + score: iopdf, + }; + let list = matches.entry(word.id).or_default(); + if !list.contains(&entry) { + list.push(entry); + } + } + } + } + matches +} + +/// Step 0: `_get_table_dimension` — (columns, rows, max_cell_id). +fn get_table_dimension(table_cells: &[TfCell]) -> (usize, usize, usize) { + let mut columns = 1; + let mut rows = 1; + let mut max_cell_id = 0; + for cell in table_cells { + columns = columns.max(cell.column_id); + rows = rows.max(cell.row_id); + max_cell_id = max_cell_id.max(cell.cell_id); + } + (columns + 1, rows + 1, max_cell_id) +} + +/// Step 1: `_get_good_bad_cells_in_column`. A cell is `good` once per match +/// entry pointing at it (docling appends without breaking, so a cell matched +/// by N words appears N times and weights the medians N-fold); a cell whose +/// predicted class is empty (`cell_class <= 1`) is always `bad`. +fn good_bad_cells_in_column( + table_cells: &[TfCell], + column: usize, + matches: &Matches, +) -> (Vec<TfCell>, Vec<TfCell>) { + let mut good = Vec::new(); + let mut bad = Vec::new(); + for cell in table_cells { + if cell.column_id != column { + continue; + } + let mut bad_match = true; + if cell.cell_class > 1 { + for list in matches.values() { + for m in list { + if m.table_cell_id == cell.cell_id { + good.push(cell.clone()); + bad_match = false; + } + } + } + } + if bad_match { + bad.push(cell.clone()); + } + } + (good, bad) +} + +#[derive(Clone, Copy, PartialEq)] +enum Alignment { + Left, + Middle, + Right, +} + +/// Step 2: `_find_alignment_in_column` — smallest min-max spread of the +/// left / middle / right cell edges decides the column alignment. +fn find_alignment_in_column(cells: &[TfCell]) -> Alignment { + if cells.is_empty() { + return Alignment::Left; + } + let (mut lmin, mut lmax) = (f64::INFINITY, f64::NEG_INFINITY); + let (mut mmin, mut mmax) = (f64::INFINITY, f64::NEG_INFINITY); + let (mut rmin, mut rmax) = (f64::INFINITY, f64::NEG_INFINITY); + for cell in cells { + let l = cell.bbox[0]; + let r = cell.bbox[2]; + let m = (l + r) / 2.0; + lmin = lmin.min(l); + lmax = lmax.max(l); + mmin = mmin.min(m); + mmax = mmax.max(m); + rmin = rmin.min(r); + rmax = rmax.max(r); + } + let deltas = [lmax - lmin, mmax - mmin, rmax - rmin]; + // Python's list.index(min(...)) keeps the first minimum. + let mut best = 0; + for (i, d) in deltas.iter().enumerate() { + if *d < deltas[best] { + best = i; + } + } + match best { + 0 => Alignment::Left, + 1 => Alignment::Middle, + _ => Alignment::Right, + } +} + +/// `statistics.median`: mean of the two middle values for an even count. +fn median(values: &mut [f64]) -> f64 { + values.sort_by(|a, b| a.total_cmp(b)); + let n = values.len(); + if n % 2 == 1 { + values[n / 2] + } else { + (values[n / 2 - 1] + values[n / 2]) / 2.0 + } +} + +/// Step 3: `_get_median_pos_size` — median alignment-edge X / width / height +/// over the good cells, skipping spans and predicted-empty cells. Defaults +/// (0, 1, 1) when nothing qualifies, exactly like the source. +fn get_median_pos_size(cells: &[TfCell], alignment: Alignment) -> (f64, f64, f64) { + let mut xs = Vec::new(); + let mut ws = Vec::new(); + let mut hs = Vec::new(); + for cell in cells { + if cell.rowspan_val > 0 || cell.colspan_val > 0 || cell.cell_class <= 1 { + continue; + } + let x = match alignment { + Alignment::Left => cell.bbox[0], + Alignment::Middle => (cell.bbox[2] + cell.bbox[0]) / 2.0, + Alignment::Right => cell.bbox[2], + }; + xs.push(x); + ws.push(cell.bbox[2] - cell.bbox[0]); + hs.push(cell.bbox[3] - cell.bbox[1]); + } + let median_x = if xs.is_empty() { 0.0 } else { median(&mut xs) }; + let median_w = if ws.is_empty() { 1.0 } else { median(&mut ws) }; + let median_h = if hs.is_empty() { 1.0 } else { median(&mut hs) }; + (median_x, median_w, median_h) +} + +/// Step 4: `_move_cells_to_left_pos` with `rescale=False` (docling's call): +/// slide each bad cell to the column's median alignment edge, keeping its +/// original width. +fn move_cells_to_left_pos(cells: &[TfCell], median_x: f64, alignment: Alignment) -> Vec<TfCell> { + cells + .iter() + .map(|cell| { + let width = cell.bbox[2] - cell.bbox[0]; + let (new_x1, new_x2) = match alignment { + Alignment::Left => (median_x, median_x + width), + // Bit-faithful to the source: `new_x2 = new_x1 + original_width`. + Alignment::Middle => { + let x1 = median_x - width / 2.0; + (x1, x1 + width) + } + Alignment::Right => (median_x - width, median_x), + }; + TfCell { + bbox: [new_x1, cell.bbox[1], new_x2, cell.bbox[3]], + ..cell.clone() + } + }) + .collect() +} + +/// Step 7: `_deduplicate_cells` — when two *adjacent* structural columns share +/// more than 60 % of their matched words, drop the lower-scoring column (its +/// cells and their match entries). +fn deduplicate_cells( + tab_columns: usize, + table_cells: &[TfCell], + iou_matches: &Matches, + ioc_matches: &Matches, +) -> (Vec<TfCell>, Matches) { + use std::collections::BTreeSet; + let mut pdf_cells_in_columns: Vec<BTreeSet<usize>> = Vec::with_capacity(tab_columns); + let mut total_score_in_columns: Vec<f64> = Vec::with_capacity(tab_columns); + for col in 0..tab_columns { + let column_cell_ids: BTreeSet<usize> = table_cells + .iter() + .filter(|c| c.column_id == col) + .map(|c| c.cell_id) + .collect(); + let mut ids = BTreeSet::new(); + let mut score = 0.0; + for matches in [iou_matches, ioc_matches] { + for (&pdf_id, list) in matches { + for m in list { + if column_cell_ids.contains(&m.table_cell_id) { + score += m.score; + ids.insert(pdf_id); + } + } + } + } + pdf_cells_in_columns.push(ids); + total_score_in_columns.push(score); + } + + let mut cols_to_eliminate: Vec<usize> = Vec::new(); + for cl in 0..tab_columns.saturating_sub(1) { + let col_a = &pdf_cells_in_columns[cl]; + let col_b = &pdf_cells_in_columns[cl + 1]; + let int_prc = if col_a.is_empty() { + 0.0 + } else { + col_a.intersection(col_b).count() as f64 / col_a.len() as f64 + }; + if int_prc > 0.6 { + if total_score_in_columns[cl] >= total_score_in_columns[cl + 1] { + cols_to_eliminate.push(cl + 1); + } else { + cols_to_eliminate.push(cl); + } + } + } + + let mut removed_ids: BTreeSet<usize> = BTreeSet::new(); + let mut new_table_cells = Vec::new(); + for cell in table_cells { + if cols_to_eliminate.contains(&cell.column_id) { + removed_ids.insert(cell.cell_id); + } else { + new_table_cells.push(cell.clone()); + } + } + let mut new_matches: Matches = BTreeMap::new(); + for (&pdf_id, list) in ioc_matches { + let kept: Vec<MatchEntry> = list + .iter() + .filter(|m| !removed_ids.contains(&m.table_cell_id)) + .cloned() + .collect(); + if !kept.is_empty() { + new_matches.insert(pdf_id, kept); + } + } + (new_table_cells, new_matches) +} + +/// Step 8: `_do_final_asignment` — each word keeps only its highest-scoring +/// match (Python's `max` keeps the first of equals). +fn do_final_assignment(ioc_matches: &Matches) -> Matches { + let mut new_matches: Matches = BTreeMap::new(); + for (&pdf_id, list) in ioc_matches { + let mut best = &list[0]; + for m in &list[1..] { + if m.score > best.score { + best = m; + } + } + new_matches.insert(pdf_id, vec![best.clone()]); + } + new_matches +} + +/// Step 8.a: `_align_table_cells_to_pdf` — matched cells take the union box +/// of their matched words; cells with no match are dropped. +fn align_table_cells_to_pdf( + table_cells: &[TfCell], + pdf_cells: &[PdfWord], + matches: &Matches, +) -> Vec<TfCell> { + use std::collections::HashMap; + let word_boxes: HashMap<usize, [f64; 4]> = pdf_cells.iter().map(|w| (w.id, w.bbox)).collect(); + let mut boxes_per_cell: BTreeMap<usize, [f64; 4]> = BTreeMap::new(); + let mut order: Vec<usize> = Vec::new(); + for (pdf_id, list) in matches { + let Some(&wb) = word_boxes.get(pdf_id) else { + continue; + }; + let mut seen = std::collections::BTreeSet::new(); + for m in list { + if !seen.insert(m.table_cell_id) { + continue; + } + match boxes_per_cell.entry(m.table_cell_id) { + std::collections::btree_map::Entry::Vacant(e) => { + e.insert(wb); + order.push(m.table_cell_id); + } + std::collections::btree_map::Entry::Occupied(mut e) => { + let b = e.get_mut(); + b[0] = b[0].min(wb[0]); + b[1] = b[1].min(wb[1]); + b[2] = b[2].max(wb[2]); + b[3] = b[3].max(wb[3]); + } + } + } + } + let by_id: HashMap<usize, &TfCell> = table_cells.iter().map(|c| (c.cell_id, c)).collect(); + let mut out = Vec::new(); + for cell_id in order { + if let Some(&cell) = by_id.get(&cell_id) { + let mut cell = cell.clone(); + cell.bbox = boxes_per_cell[&cell_id]; + out.push(cell); + } + } + out +} + +fn merge_two_bboxes(a: &[f64; 4], b: &[f64; 4]) -> [f64; 4] { + [ + a[0].min(b[0]), + a[1].min(b[1]), + a[2].max(b[2]), + a[3].max(b[3]), + ] +} + +/// Python 3 `round()` — banker's rounding to the nearest integer. +fn py_round(v: f64) -> i64 { + v.round_ties_even() as i64 +} + +/// A per-band orphan record: (pdf id, rounded depth, word bbox). +type OrphanRecord = (usize, i64, [f64; 4]); + +/// Shared row/column banding scan of step 9: for each band (min/max of the +/// non-span, non-empty member cells' `lo`/`hi` edges), collect the unmatched +/// words whose edge or centroid falls inside, resolving a word seen in an +/// earlier band by the smaller rounded centroid distance. +fn band_orphans( + n_bands: usize, + cells_in_band: impl Fn(usize) -> Vec<[f64; 4]>, + pdf_cells: &[PdfWord], + matches: &Matches, + lo_ix: usize, + hi_ix: usize, +) -> Vec<Vec<OrphanRecord>> { + let mut bands: Vec<Vec<OrphanRecord>> = Vec::with_capacity(n_bands); + // pdf id → (band index, index within band) of its current record. + let mut used: BTreeMap<usize, usize> = BTreeMap::new(); + for band_ix in 0..n_bands { + let boxes = cells_in_band(band_ix); + let mut lo = -1.0f64; + let mut hi = -1.0f64; + if !boxes.is_empty() { + lo = boxes.iter().map(|b| b[lo_ix]).fold(f64::INFINITY, f64::min); + hi = boxes + .iter() + .map(|b| b[hi_ix]) + .fold(f64::NEG_INFINITY, f64::max); + } + let mut in_band: Vec<OrphanRecord> = Vec::new(); + for word in pdf_cells { + if matches.contains_key(&word.id) { + continue; + } + let centroid_band = (hi + lo) / 2.0; + let centroid_cell = (word.bbox[hi_ix] + word.bbox[lo_ix]) / 2.0; + let within = (word.bbox[lo_ix] >= lo && word.bbox[lo_ix] <= hi) + || (word.bbox[hi_ix] >= lo && word.bbox[hi_ix] <= hi) + || (word.bbox[lo_ix] <= lo && word.bbox[hi_ix] >= hi); + if !within { + continue; + } + let depth = py_round((centroid_band - centroid_cell).abs()); + match used.get(&word.id) { + None => { + used.insert(word.id, band_ix); + in_band.push((word.id, depth, word.bbox)); + } + Some(&old_band) => { + let Some(old_pos) = bands[old_band].iter().position(|r| r.0 == word.id) else { + continue; + }; + if depth < bands[old_band][old_pos].1 { + bands[old_band].remove(old_pos); + used.insert(word.id, band_ix); + in_band.push((word.id, depth, word.bbox)); + } + } + } + } + bands.push(in_band); + } + bands +} + +/// Step 9: `_pick_orphan_cells` — words with no match are placed by the row +/// band containing them vertically and the column band containing them +/// horizontally; the (row, column) either reuses an existing structural cell +/// (growing its box) or creates a new one. +fn pick_orphan_cells( + tab_rows: usize, + tab_cols: usize, + mut max_cell_id: usize, + mut table_cells: Vec<TfCell>, + pdf_cells: &[PdfWord], + mut matches: Matches, +) -> (Matches, Vec<TfCell>, usize) { + // NOTE: docling's row-band pass exposes an aliasing quirk — the band's + // orphan test reads `matches` as it was on entry (orphans found later + // never re-enter it), which we reproduce by collecting all bands before + // assigning anything. + let orphan_rows = band_orphans( + tab_rows, + |row| { + table_cells + .iter() + .filter(|c| c.row_id == row && c.rowspan_val == 0 && c.cell_class > 1) + .map(|c| c.bbox) + .collect() + }, + pdf_cells, + &matches, + 1, + 3, + ); + let orphan_columns = band_orphans( + tab_cols, + |col| { + table_cells + .iter() + .filter(|c| c.column_id == col && c.colspan_val == 0 && c.cell_class > 1) + .map(|c| c.bbox) + .collect() + }, + pdf_cells, + &matches, + 0, + 2, + ); + + // pdf id → row / column of its accepted band record. + let mut row_of: BTreeMap<usize, usize> = BTreeMap::new(); + for (row_id, records) in orphan_rows.iter().enumerate() { + for r in records { + row_of.insert(r.0, row_id); + } + } + let mut col_of: BTreeMap<usize, (usize, i64, [f64; 4])> = BTreeMap::new(); + for (col_id, records) in orphan_columns.iter().enumerate() { + for r in records { + col_of.insert(r.0, (col_id, r.1, r.2)); + } + } + + // Ascending pdf id, matching the sorted C++-parity loop in the source. + for (&pdf_id, &new_row_id) in row_of.iter() { + let Some(&(new_column_id, confidence, pdf_bbox)) = col_of.get(&pdf_id) else { + continue; + }; + let existing = table_cells + .iter() + .position(|c| c.row_id == new_row_id && c.column_id == new_column_id); + let table_cell_id = match existing { + Some(ix) => { + let merged = merge_two_bboxes(&table_cells[ix].bbox, &pdf_bbox); + table_cells[ix].bbox = merged; + table_cells[ix].cell_id + } + None => { + max_cell_id += 1; + table_cells.push(TfCell { + bbox: pdf_bbox, + cell_id: max_cell_id, + column_id: new_column_id, + row_id: new_row_id, + cell_class: 2, + colspan_val: 0, + rowspan_val: 0, + }); + max_cell_id + } + }; + matches.insert( + pdf_id, + vec![MatchEntry { + table_cell_id, + score: confidence as f64, + }], + ); + } + (matches, table_cells, max_cell_id) +} + +/// `MatchingPostProcessor.process` with docling's defaults +/// (`correct_overlapping_cells=False`): returns the post-processed table +/// cells and the final one-cell-per-word matches. +pub fn match_and_post_process( + table_cells: Vec<TfCell>, + pdf_cells: &[PdfWord], +) -> (Vec<TfCell>, Matches) { + let matches = intersection_over_pdf_match(&table_cells, pdf_cells); + + let (tab_columns, tab_rows, max_cell_id) = get_table_dimension(&table_cells); + + // Steps 1-4 per column: snap the unmatched cells to the matched cells' + // median alignment edge. + let mut fixed: Vec<TfCell> = Vec::new(); + for col in 0..tab_columns { + let (good, bad) = good_bad_cells_in_column(&table_cells, col, &matches); + let alignment = find_alignment_in_column(&good); + let (median_x, median_w, median_h) = get_median_pos_size(&good, alignment); + let _ = (median_w, median_h); // rescale=False: medians only steer X. + let moved = move_cells_to_left_pos(&bad, median_x, alignment); + fixed.extend(good); + fixed.extend(moved); + } + fixed.sort_by_key(|c| c.cell_id); + + // Step 5: re-match against the fixed cells. + let ioc = intersection_over_pdf_match(&fixed, pdf_cells); + + // Step 7: drop duplicated columns. + let (dedupl_cells, dedupl_matches) = deduplicate_cells(tab_columns, &fixed, &matches, &ioc); + + // Step 8: one table cell per word. + let final_matches = do_final_assignment(&dedupl_matches); + + // Step 8.a: align cell boxes to their matched words (dropping unmatched + // cells) — skipped for large pages, as in the source. + let mut dedupl_sorted = dedupl_cells; + dedupl_sorted.sort_by_key(|c| c.cell_id); + let aligned = if pdf_cells.len() > 300 { + dedupl_sorted + } else { + align_table_cells_to_pdf(&dedupl_sorted, pdf_cells, &final_matches) + }; + + // Step 9: place the leftover words by row/column banding. docling passes + // the *pre-dedup* column count into the orphan scan. + let (final_matches, cells_wo, _) = pick_orphan_cells( + tab_rows, + tab_columns, + max_cell_id, + aligned, + pdf_cells, + final_matches, + ); + (cells_wo, final_matches) +} diff --git a/scripts/conformance/dclx_conformance.sh b/scripts/conformance/dclx_conformance.sh index 6d713ecf..12241280 100755 --- a/scripts/conformance/dclx_conformance.sh +++ b/scripts/conformance/dclx_conformance.sh @@ -10,8 +10,20 @@ # Prints per-fixture EXACT / diff-line counts and a summary split into the # issue #32 targets: >=90% similarity for non-PDF formats, >=50% for PDF. # "Similarity" per fixture = 100 * (1 - difflines / max(lines_ref, lines_ours)). +# +# PDF geometry tolerance (DCLX_TOL, default 2 grid units): on the PDF path the +# reference `<location>` provenance (l,t,r,b on docling's 0-500 normalized page +# grid) comes from docling's layout clusters while ours come from our own layout +# post-processing, so the same region is boxed a few grid units apart. +# dclx_diff.py counts a location pair as matching when the two values are within +# DCLX_TOL; text, structure, and every non-geometry line stay byte-exact, and +# unmatched lines always count. The tolerance is applied ONLY to PDF fixtures -- +# formats whose geometry is read from the same source file (OOXML slides/sheets) +# are still compared exactly (tol=0). set -uo pipefail cd "$(dirname "$0")/../.." +TOL="${DCLX_TOL:-2}" +DIFF="$(dirname "$0")/dclx_diff.py" BIN="$(pwd)/target/release/docling-rs" [ -x "$BIN" ] || { echo "build first: cargo build --release -p docling-cli" >&2; exit 1; } @@ -19,6 +31,13 @@ BIN="$(pwd)/target/release/docling-rs" tmp=$(mktemp -d) trap 'rm -rf "$tmp"' EXIT +# The binary resolves models/.pdfium relative to its CWD (then next to the exe); +# we run it inside $tmp so the .dclx lands there, so make the assets reachable +# from $tmp. Harmless for declarative formats that load no models. +export PDFIUM_DYNAMIC_LIB_PATH="$(pwd)/.pdfium/lib" +[ -d models ] && ln -sfn "$(pwd)/models" "$tmp/models" +[ -d .pdfium ] && ln -sfn "$(pwd)/.pdfium" "$tmp/.pdfium" + declare -a fmts=() if [ $# -gt 0 ]; then fmts=("$@"); else for d in tests/data/*/groundtruth_dclx; do [ -d "$d" ] && fmts+=("$(basename "$(dirname "$d")")"); done @@ -49,7 +68,8 @@ for fmt in "${fmts[@]}"; do # only a genuine structural difference (missing/extra <src>) then shows up. sed -E -i 's#image_[0-9]{6}_[0-9a-f]+\.png#image_NORM.png#g' \ "$tmp/ref.xml" "$tmp/ours.xml" - d=$(diff "$tmp/ref.xml" "$tmp/ours.xml" | grep -c '^[<>]') + ftol=0; [ "$is_pdf" -eq 1 ] && ftol="$TOL" # geometry tolerance: PDF only + d=$(python3 "$DIFF" "$tmp/ref.xml" "$tmp/ours.xml" --tol "$ftol") lr=$(wc -l < "$tmp/ref.xml"); lo=$(wc -l < "$tmp/ours.xml") max=$(( lr > lo ? lr : lo )); [ "$max" -eq 0 ] && max=1 sim=$(( 100 * (max*2 - d) / (max*2) )) # d counts both sides @@ -66,5 +86,5 @@ for fmt in "${fmts[@]}"; do done done echo "----" -[ "$other_n" -gt 0 ] && echo "non-PDF avg similarity: $((other_sum/other_n))% (target 90%) over $other_n fixtures" -[ "$pdf_n" -gt 0 ] && echo "PDF avg similarity: $((pdf_sum/pdf_n))% (target 50%) over $pdf_n fixtures" +[ "$other_n" -gt 0 ] && echo "non-PDF avg similarity: $((other_sum/other_n))% (target 90%, exact geometry) over $other_n fixtures" +[ "$pdf_n" -gt 0 ] && echo "PDF avg similarity: $((pdf_sum/pdf_n))% (target 50%, +/-${TOL} grid-unit geometry tolerance) over $pdf_n fixtures" diff --git a/scripts/conformance/dclx_diff.py b/scripts/conformance/dclx_diff.py new file mode 100755 index 00000000..167875c8 --- /dev/null +++ b/scripts/conformance/dclx_diff.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python3 +"""Geometry-tolerant line diff for DocLang ``document.xml`` conformance scoring. + +The default ``diff a b | grep -c '^[<>]'`` used by dclx_conformance.sh compares +every line byte-for-byte. That is the right measure for text and structure, but +it is *wrong* for the four ``<location value="N"/>`` provenance tokens each +laid-out block emits (l,t,r,b on docling's 0-500 normalized page grid). On the +PDF path those integers come from docling's layout clusters in the reference and +from ours; the same region is boxed a few grid units apart by the two layout +post-processors, so the geometry lines diff even when text and structure match. + +This tool line-diffs exactly like the shell does, with one change: when a diff +aligns a ``<location value="N"/>`` line against another ``<location value="M"/>`` +line and ``|N-M| <= tol``, the pair counts as equal. Everything else -- text, +tags, nesting, spans, and any location whose value is off by more than ``tol`` -- +is still compared exactly. It never drops unmatched lines: a reference line with +no counterpart always counts against the score. + + dclx_diff.py REF.xml OURS.xml [--tol N] # prints the diff-line count + +With ``--tol 0`` the output is identical to ``diff | grep -c '^[<>]'``. +""" +from __future__ import annotations + +import argparse +import difflib +import re +import sys + +_LOC = re.compile(r'^\s*<location value="(-?\d+)"/>\s*$') + + +def _loc_val(line: str): + m = _LOC.match(line) + return int(m.group(1)) if m else None + + +def diff_lines(ref, ours, tol): + """Count differing lines (both sides), treating location tokens within + ``tol`` grid units of each other as equal. Mirrors ``diff | grep -c '^[<>]'``.""" + sm = difflib.SequenceMatcher(a=ref, b=ours, autojunk=False) + diff = 0 + for tag, i1, i2, j1, j2 in sm.get_opcodes(): + if tag == "equal": + continue + if tag == "replace" and (i2 - i1) == (j2 - j1): + # Equal-length replacement: pair the lines up so a block of four + # geometry tokens is compared coordinate-by-coordinate. + for a, b in zip(ref[i1:i2], ours[j1:j2]): + va, vb = _loc_val(a), _loc_val(b) + if va is not None and vb is not None and abs(va - vb) <= tol: + continue # geometry agrees within tolerance + diff += 2 # one changed line counts on both sides + else: + # Insertions, deletions, and ragged replacements: every unmatched + # line counts, exactly as the raw line diff would score them. + diff += (i2 - i1) + (j2 - j1) + return diff + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("ref") + ap.add_argument("ours") + ap.add_argument("--tol", type=int, default=0, + help="max grid-unit difference for two <location> values to match") + args = ap.parse_args() + with open(args.ref, encoding="utf-8") as f: + ref = f.read().splitlines() + with open(args.ours, encoding="utf-8") as f: + ours = f.read().splitlines() + print(diff_lines(ref, ours, args.tol)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/conformance/dclx_pdf_tol_sweep.sh b/scripts/conformance/dclx_pdf_tol_sweep.sh new file mode 100755 index 00000000..98f07001 --- /dev/null +++ b/scripts/conformance/dclx_pdf_tol_sweep.sh @@ -0,0 +1,50 @@ +#!/usr/bin/env bash +# Sweep the PDF .dclx geometry tolerance and print the average-similarity ladder. +# Converts each PDF fixture ONCE, then re-diffs the cached document.xml at every +# tolerance (the tolerance only affects the diff, not the conversion). Exact +# (0) is the raw byte-for-byte score; the largest step ignores geometry entirely +# (structure + text only) and is the ceiling the tolerance approaches. +# +# scripts/conformance/dclx_pdf_tol_sweep.sh +set -uo pipefail +cd "$(dirname "$0")/../.." +BIN="$(pwd)/target/release/docling-rs" +DIFF="$(dirname "$0")/dclx_diff.py" +export PDFIUM_DYNAMIC_LIB_PATH="$(pwd)/.pdfium/lib" + +tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT +ln -sfn "$(pwd)/models" "$tmp/models"; ln -sfn "$(pwd)/.pdfium" "$tmp/.pdfium" + +n=0 +for ref in tests/data/pdf/groundtruth_dclx/*.dclx; do + [ -f "$ref" ] || continue + base=$(basename "$ref" .dclx) + src=tests/data/pdf/sources/$base + [ -f "$src" ] || continue + ( cd "$tmp" && "$BIN" --to dclx "$OLDPWD/$src" >/dev/null 2>&1 ) + ours=$tmp/"${base%.*}".dclx + [ -f "$ours" ] || { echo "convert FAIL: $base" >&2; continue; } + unzip -p "$ref" document.xml > "$tmp/$n.ref.xml" 2>/dev/null + unzip -p "$ours" document.xml > "$tmp/$n.ours.xml" 2>/dev/null + rm -f "$ours"; n=$((n+1)) +done +echo "converted $n PDF fixtures" >&2 + +printf "%-24s %s\n" "TOLERANCE" "PDF avg similarity" +for tol in 0 2 5 10 100000; do + sum=0 + for k in $(seq 0 $((n-1))); do + d=$(python3 "$DIFF" "$tmp/$k.ref.xml" "$tmp/$k.ours.xml" --tol "$tol") + lr=$(wc -l < "$tmp/$k.ref.xml"); lo=$(wc -l < "$tmp/$k.ours.xml") + max=$(( lr > lo ? lr : lo )); [ "$max" -eq 0 ] && max=1 + sim=$(( 100 * (max*2 - d) / (max*2) )); [ "$sim" -lt 0 ] && sim=0 + sum=$((sum+sim)) + done + [ "$n" -gt 0 ] && avg=$((sum/n)) || avg=0 + case "$tol" in + 0) label="exact (0)";; + 100000) label="geometry ignored";; + *) label="+/-${tol} grid units";; + esac + printf "%-24s %s%%\n" "$label" "$avg" +done diff --git a/scripts/test/tf_match_reference.py b/scripts/test/tf_match_reference.py new file mode 100644 index 00000000..8d0e7dd0 --- /dev/null +++ b/scripts/test/tf_match_reference.py @@ -0,0 +1,106 @@ +"""Parity harness for the tf_match.rs port of docling's table-cell matching. + +Run the Rust pipeline with DOCLING_RS_TF_MATCH_DUMP=<dir> to record each +table's matcher inputs (<dir>/tf_match_dump.jsonl), then replay them through +docling_ibm_models' reference CellMatcher + MatchingPostProcessor and rebuild +the markdown grid the same way the Rust side does, for a side-by-side check: + + python scripts/test/tf_match_reference.py <dir>/tf_match_dump.jsonl [text-filter] + +Needs a Python env with docling-ibm-models installed. If the printed grid +matches the Rust output, a conformance gap is in the model predictions (or +upstream geometry), not in the ported matching. +""" +import json +import sys + +from docling_ibm_models.tableformer.data_management.matching_post_processor import ( + MatchingPostProcessor, +) +from docling_ibm_models.tableformer.data_management.tf_cell_matcher import CellMatcher + +config = {"predict": {"pdf_cell_iou_thres": 0.05}} +matcher = CellMatcher(config) +post = MatchingPostProcessor(config) + +dump_path = sys.argv[1] +want = sys.argv[2] if len(sys.argv) > 2 else None # substring to select a table + +for line_no, line in enumerate(open(dump_path)): + rec = json.loads(line) + words = rec["pdf_cells"] + if want and not any(want in w["text"] for w in words): + continue + table_cells = [] + for c in rec["table_cells"]: + cell = { + "cell_id": c["cell_id"], + "row_id": c["row_id"], + "column_id": c["column_id"], + "bbox": c["bbox"], + "cell_class": c["cell_class"], + "label": "body", + } + if c["colspan_val"] > 0: + cell["colspan_val"] = c["colspan_val"] + if c["rowspan_val"] > 0: + cell["rowspan_val"] = c["rowspan_val"] + table_cells.append(cell) + pdf_cells = [{"id": w["id"], "bbox": w["bbox"], "text": w["text"]} for w in words] + + matches, _ = matcher._intersection_over_pdf_match(table_cells, pdf_cells) + matching_details = { + "iou_threshold": 0.05, + "table_cells": table_cells, + "matches": matches, + "pdf_cells": pdf_cells, + } + out = post.process(matching_details, False) + cells_wo = out["table_cells"] + final = out["matches"] + + # Response assembly identical to the Rust side. + by_id = {} + for c in cells_wo: + by_id.setdefault(c["cell_id"], c) + merged = {} + order = [] + for pdf_id in sorted(int(k) for k in final): + m = final[str(pdf_id)][0] + cell = by_id.get(m["table_cell_id"]) + if cell is None: + continue + key = (cell["column_id"], cell["row_id"]) + if key not in merged: + merged[key] = { + "start_row": cell["row_id"], + "start_col": cell["column_id"], + "row_span": cell.get("rowspan_val", 1), + "col_span": cell.get("colspan_val", 1), + "word_ids": [], + } + order.append(key) + merged[key]["word_ids"].append(pdf_id) + + start_cols = sorted({merged[k]["start_col"] for k in order}) + start_rows = sorted({merged[k]["start_row"] for k in order}) + num_rows = num_cols = 0 + for k in order: + m = merged[k] + m["ci"] = start_cols.index(m["start_col"]) + m["ri"] = start_rows.index(m["start_row"]) + num_cols = max(num_cols, m["ci"] + m["col_span"]) + num_rows = max(num_rows, m["ri"] + m["row_span"]) + + wtext = {w["id"]: w["text"] for w in pdf_cells} + grid = [["" for _ in range(num_cols)] for _ in range(num_rows)] + for k in order: + m = merged[k] + text = " ".join(wtext[i].strip() for i in m["word_ids"]).replace("@ ", "@") + for r in range(m["ri"], min(m["ri"] + m["row_span"], num_rows)): + for c in range(m["ci"], min(m["ci"] + m["col_span"], num_cols)): + grid[r][c] = text + print(f"--- table (dump line {line_no}) {num_rows}x{num_cols}") + for row in grid: + print("| " + " | ".join(row) + " |") + break diff --git a/tests/data/pdf/groundtruth_dclx/2203.01017v2.pdf.dclx b/tests/data/pdf/groundtruth_dclx/2203.01017v2.pdf.dclx new file mode 100644 index 00000000..88398c56 Binary files /dev/null and b/tests/data/pdf/groundtruth_dclx/2203.01017v2.pdf.dclx differ diff --git a/tests/data/pdf/groundtruth_dclx/2206.01062.pdf.dclx b/tests/data/pdf/groundtruth_dclx/2206.01062.pdf.dclx new file mode 100644 index 00000000..6590ce4a Binary files /dev/null and b/tests/data/pdf/groundtruth_dclx/2206.01062.pdf.dclx differ diff --git a/tests/data/pdf/groundtruth_dclx/2305.03393v1-pg9.pdf.dclx b/tests/data/pdf/groundtruth_dclx/2305.03393v1-pg9.pdf.dclx new file mode 100644 index 00000000..274f107d Binary files /dev/null and b/tests/data/pdf/groundtruth_dclx/2305.03393v1-pg9.pdf.dclx differ diff --git a/tests/data/pdf/groundtruth_dclx/2305.03393v1.pdf.dclx b/tests/data/pdf/groundtruth_dclx/2305.03393v1.pdf.dclx new file mode 100644 index 00000000..e7aa2cb9 Binary files /dev/null and b/tests/data/pdf/groundtruth_dclx/2305.03393v1.pdf.dclx differ diff --git a/tests/data/pdf/groundtruth_dclx/amt_handbook_sample.pdf.dclx b/tests/data/pdf/groundtruth_dclx/amt_handbook_sample.pdf.dclx new file mode 100644 index 00000000..d631d34f Binary files /dev/null and b/tests/data/pdf/groundtruth_dclx/amt_handbook_sample.pdf.dclx differ diff --git a/tests/data/pdf/groundtruth_dclx/code_and_formula.pdf.dclx b/tests/data/pdf/groundtruth_dclx/code_and_formula.pdf.dclx new file mode 100644 index 00000000..7b5cca63 Binary files /dev/null and b/tests/data/pdf/groundtruth_dclx/code_and_formula.pdf.dclx differ diff --git a/tests/data/pdf/groundtruth_dclx/multi_page.pdf.dclx b/tests/data/pdf/groundtruth_dclx/multi_page.pdf.dclx new file mode 100644 index 00000000..97c35572 Binary files /dev/null and b/tests/data/pdf/groundtruth_dclx/multi_page.pdf.dclx differ diff --git a/tests/data/pdf/groundtruth_dclx/normal_4pages.pdf.dclx b/tests/data/pdf/groundtruth_dclx/normal_4pages.pdf.dclx new file mode 100644 index 00000000..a0758369 Binary files /dev/null and b/tests/data/pdf/groundtruth_dclx/normal_4pages.pdf.dclx differ diff --git a/tests/data/pdf/groundtruth_dclx/picture_classification.pdf.dclx b/tests/data/pdf/groundtruth_dclx/picture_classification.pdf.dclx new file mode 100644 index 00000000..b7c21d34 Binary files /dev/null and b/tests/data/pdf/groundtruth_dclx/picture_classification.pdf.dclx differ diff --git a/tests/data/pdf/groundtruth_dclx/redp5110_sampled.pdf.dclx b/tests/data/pdf/groundtruth_dclx/redp5110_sampled.pdf.dclx new file mode 100644 index 00000000..06af6ac2 Binary files /dev/null and b/tests/data/pdf/groundtruth_dclx/redp5110_sampled.pdf.dclx differ diff --git a/tests/data/pdf/groundtruth_dclx/right_to_left_01.pdf.dclx b/tests/data/pdf/groundtruth_dclx/right_to_left_01.pdf.dclx new file mode 100644 index 00000000..c1b5ccf4 Binary files /dev/null and b/tests/data/pdf/groundtruth_dclx/right_to_left_01.pdf.dclx differ diff --git a/tests/data/pdf/groundtruth_dclx/right_to_left_02.pdf.dclx b/tests/data/pdf/groundtruth_dclx/right_to_left_02.pdf.dclx new file mode 100644 index 00000000..c2cf2e2f Binary files /dev/null and b/tests/data/pdf/groundtruth_dclx/right_to_left_02.pdf.dclx differ diff --git a/tests/data/pdf/groundtruth_dclx/right_to_left_03.pdf.dclx b/tests/data/pdf/groundtruth_dclx/right_to_left_03.pdf.dclx new file mode 100644 index 00000000..2bff4a85 Binary files /dev/null and b/tests/data/pdf/groundtruth_dclx/right_to_left_03.pdf.dclx differ diff --git a/tests/data/pdf/groundtruth_dclx/skipped_1page.pdf.dclx b/tests/data/pdf/groundtruth_dclx/skipped_1page.pdf.dclx new file mode 100644 index 00000000..17d0a6d3 Binary files /dev/null and b/tests/data/pdf/groundtruth_dclx/skipped_1page.pdf.dclx differ diff --git a/tests/data/pdf/groundtruth_dclx/skipped_2pages.pdf.dclx b/tests/data/pdf/groundtruth_dclx/skipped_2pages.pdf.dclx new file mode 100644 index 00000000..01ea2bda Binary files /dev/null and b/tests/data/pdf/groundtruth_dclx/skipped_2pages.pdf.dclx differ diff --git a/tests/data/pdf/groundtruth_dclx/table_mislabeled_as_picture.pdf.dclx b/tests/data/pdf/groundtruth_dclx/table_mislabeled_as_picture.pdf.dclx new file mode 100644 index 00000000..f2add942 Binary files /dev/null and b/tests/data/pdf/groundtruth_dclx/table_mislabeled_as_picture.pdf.dclx differ diff --git a/tests/snapshots/latex/sources/2305.03393/llncsdoc.pdf.md b/tests/snapshots/latex/sources/2305.03393/llncsdoc.pdf.md index 1ac0e1b9..a12fd0e8 100644 --- a/tests/snapshots/latex/sources/2305.03393/llncsdoc.pdf.md +++ b/tests/snapshots/latex/sources/2305.03393/llncsdoc.pdf.md @@ -24,6 +24,14 @@ The llncs class is invoked by replacing article by llncs in the first line of yo \documentclass{llncs} +If your file is already coded with L A T E X, you can easily adapt it to the llncs document class by replacing + +\documentclass{article} + +with + +\documentclass{llncs} + ``` \begin{document} <Your contribution> @@ -35,25 +43,21 @@ with \documentclass{llncs} ``` -If your file is already coded with L A T E X, you can easily adapt it to the llncs document class by replacing - -\documentclass{article} - -with +\fnmsep -\documentclass{llncs} +\titlerunning ## 3 How to Code the Header of Your Paper ## 3.1 Title -\title Please code the title of your contribution as follows: +Please code the title of your contribution as follows: \title \title{<Your contribution title>} All words in titles should be capitalized except for conjunctions, prepositions (e.g. on, of, by, and, or, but, from, with, without, under), and definite/indefinite articles (the, a, an), unless they appear at the beginning. Formula letters are typeset as in the text. Long titles that run over multiple lines can be wrapped explicitly with \\ . Titles have no end punctuation. -Acknowledgements should generally be placed in an unnumbered subsection at the end of the paper. If you still need to refer to a support or funding program \thanks in a note to the title, you can use the \thanks macro inside the title: +Acknowledgements should generally be placed in an unnumbered subsection at the end of the paper. If you still need to refer to a support or funding program in a note to the title, you can use the \thanks macro inside the title: \thanks \title{<Your contribution title>\thanks{<granted by x>}} @@ -61,33 +65,29 @@ Please do not use \thanks inside \author or \institute as footnotes for these el If you need two or more footnotes please separate them with \fnmsep (i.e. f oot n ote m ark sep arator). -\fnmsep - If a long title does not fit in the single line of the running head, a warning is generated. You can specify an abbreviated title for the running head with the command -\titlerunning - \titlerunning{<Your abbreviated contribution title>} -\subtitle An optional subtitle may also be added: +An optional subtitle may also be added: \subtitle \subtitle{<subtitle of your contribution>} ## 3.2 Author(s) -\author The name(s) of the author(s) are specified by: +The name(s) of the author(s) are specified by: \author \author{<author(s) name(s)>} -\and If there is more than one author, please separate them by \and . This makes sure that correct punctuation is inserted according to the number of authors. +If there is more than one author, please separate them by \and . This makes sure \and that correct punctuation is inserted according to the number of authors. -\inst Numbers referring to different addresses or affiliations should be attached to each author with the \inst{<number>} command. If an author is affiliated with multiple institutions the numbers should be separated by a comma, for example \inst{2,3} . +Numbers referring to different addresses or affiliations should be attached to \inst each author with the \inst{<number>} command. If an author is affiliated with multiple institutions the numbers should be separated by a comma, for example \inst{2,3} . -\orcidID ORCID identifiers can be included with +ORCID identifiers can be included with \orcidID \orcidID{<ORCID identifier>} -The ORCID (Open Researcher and Contributor ID) registry provides authors with unique digital identifiers that distinguish them from other researchers and help them link their research activities to these identifiers. Authors who are not yet registered with ORCID are encouraged to apply for an individual ORCID id at https://www.orcid.org and to include it in their papers. In the final publication, the ORCID id will be replaced by an ORCID icon, which will link from the eBook to the actual ID in the ORCID database. The ORCID icon will also replace the number in the printed book. +\authorrunning The ORCID (Open Researcher and Contributor ID) registry provides authors with unique digital identifiers that distinguish them from other researchers and help them link their research activities to these identifiers. Authors who are not yet registered with ORCID are encouraged to apply for an individual ORCID id at https://www.orcid.org and to include it in their papers. In the final publication, the ORCID id will be replaced by an ORCID icon, which will link from the eBook to the actual ID in the ORCID database. The ORCID icon will also replace the number in the printed book. If you have done this correctly, the author line now reads, for example: @@ -100,8 +100,6 @@ The given name(s) should always be followed by the family name(s). Authors who h As given name(s) are to be shortened to initials in the running heads, specifying an abbreviated author list with the optional command: -\authorrunning - ``` \authorrunning{<abbreviated author list>} ``` @@ -110,11 +108,15 @@ might add some clarity about the correct representation of author names, in the ## 3.3 Affiliations -\institute Addresses of institutes, companies, etc. should be given in \institute . +Addresses of institutes, companies, etc. should be given in \institute . \institute -Multiple affiliations are separated by \and , which automatically assures correct numbering: +\and Multiple affiliations are separated by \and , which automatically assures correct numbering: -\and +Inside \institute you can use \email + +\email{<email address>} + +and \url ``` \institute{<name of an institute> @@ -126,30 +128,25 @@ Inside \institute you can use\email \url{<url>} ``` -\email Inside \institute you can use - -\email{<email address>} - -\url and to provide author email addresses and Web pages. If you need to typeset the tilde character - e.g. for your Web page in your unix system's home directory - the \homedir command will do this. If multiple authors have the same affiliation, please check that the order of email addresses matches the sequence of (affiliated) author names. +to provide author email addresses and Web pages. If you need to typeset the tilde character - e.g. for your Web page in your unix system's home directory - the \homedir command will do this. If multiple authors have the same affiliation, please check that the order of email addresses matches the sequence of (affiliated) author names. Please note that, if email addresses are given in your paper, they will also be included in the metadata of the online version. ## 3.4 Format the Header -\maketitle The command \maketitle formats the header of your paper. If you leave it out the work done so far will produce no text. +The command \maketitle formats the header of your paper. If you leave it out \maketitle the work done so far will produce no text. ## 3.5 Abstract and Keywords -abstract ( env. ) The abstract is coded as follows: +The abstract is coded as follows: abstract ( env. ) ``` -The abstract is coded as follows:abstract(env.) - \begin{abstract} - <Text of the summary of your paper> - \end{abstract} +\begin{abstract} +<Text of the summary of your paper> +\end{abstract} ``` -\keywords Keywords should be specified inside the abstract environment. Please capitalize \and the first letter of each keyword and again separate them with \and : +Keywords should be specified inside the abstract environment. Please capitalize \keywords \and the first letter of each keyword and again separate them with \and : \keywords{First keyword \and Second keyword \and Third keyword} @@ -187,9 +184,9 @@ Please note that all these characters are only available in math mode. ## 5.1 Predefined Theorem-Like Environments -( env. ) Several theorem-like environments are predefined in the llncs document class. corollary definition ( env. ) The following environments have a bold run-in heading, while the following text lemma ( env. ) is in italics: +Several theorem-like environments are predefined in the llncs document class. corollary ( env. ) definition ( env. ) lemma ( env. ) The following environments have a bold run-in heading, while the following text is in italics: -proposition ( env. ) +proposition ``` \begin{corollary} <text> \end{corollary} @@ -199,41 +196,41 @@ proposition ( env. ) \begin{theorem} <text> \end{theorem} ``` -( env. ) Other theorem-like environments render the text in roman, while the run-in case conjecture ( env. ) heading is bold as well: - -example ( env. ) +Other theorem-like environments render the text in roman, while the run-in case ( env. ) conjecture ( env. ) heading is bold as well: ``` exercise(env.) property(env.) question(env.) solution(env.) heading is bold as well: problem(env.) note(env.) \begin{case} <text> \end{case} \begin{conjecture} <text> \end{conjecture} \begin{example} <text> \end{example} \begin{exercise} <text> \end{exercise} \begin{note} <text> \end{note} \begin{problem} <text> \end{problem} remark(env.) \begin{property} <text> \end{property} \begin{question} <text> \end{question} \begin{remark} <text> \end{remark} \begin{solution} <text> \end{solution} ``` -claim ( env. ) Finally, there are also two unnumbered environments that have the run-in headproof ( env. ) ing in italics and the text in upright roman. +Finally, there are also two unnumbered environments that have the run-in headclaim ( env. ) proof ( env. ) ing in italics and the text in upright roman. ``` \begin{claim} <text> \end{claim} \begin{proof} <text> \end{proof} ``` -\qed Proofs may contain an eye catching square, which can be inserted with \qed ) before the environment ends. +Proofs may contain an eye catching square, which can be inserted with \qed ) \qed before the environment ends. ## 5.2 User-Defined Theorem-Like Environments -\spnewtheorem We have enhanced the standard \newtheorem command and slightly changed its syntax to get two new commands \spnewtheorem and \spnewtheorem* that now can be used to define additional environments. They require two additional arguments, namely the font style of the label and the font style of the text of the new environment: +We have enhanced the standard \newtheorem command and slightly changed \spnewtheorem its syntax to get two new commands \spnewtheorem and \spnewtheorem* that now can be used to define additional environments. They require two additional arguments, namely the font style of the label and the font style of the text of the new environment: \spnewtheorem{<env\_nam>}[<num\_like>]{<caption>}{<cap\_font>}{<body\_font>} +For example, + +\spnewtheorem{maintheorem}[theorem]{Main Theorem}{\bfseries}{\itshape} + ``` \spnewtheorem{<env_nam>}[<num_like>]{<caption>}{<cap_font>}{<body_font>} For example, \spnewtheorem{maintheorem}[theorem]{Main Theorem}{\bfseries}{\itshape} ``` -For example, - -\spnewtheorem{maintheorem}[theorem]{Main Theorem}{\bfseries}{\itshape} +\spnewtheorem* -will create a main theorem environment that is numbered together with the predefined theorem . The sharing of the default counter ( [theorem] ) is desired. If you omit the optional second argument of \spnewtheorem , a separate counter for your new environment is used throughout your document. +citeauthoryear will create a main theorem environment that is numbered together with the predefined theorem . The sharing of the default counter ( [theorem] ) is desired. If you omit the optional second argument of \spnewtheorem , a separate counter for your new environment is used throughout your document. In combination with the (obsolete) class option envcountsect (see. Sect. 7), the \spnewtheorem command also supports the syntax: @@ -243,28 +240,18 @@ With the parameter <within> , you can control the sectioning element that If you wish to add an unnumbered environment, please use the syntax -\spnewtheorem* - \spnewtheorem*{<env\_nam>}{<caption>}{<cap\_font>}{<body\_font>} ## 6 References There are three options for citing references: -``` -- arabic numbers, i.e. [1], [3-5], [4-6,9], -- labels, i.e. [CE1], [AB1,XY2], -- author/year system,(Smith et al. 2000),(Miller 1999a, 12; Brown 2018). -``` - - arabic numbers, i.e. [1], [3-5], [4-6,9], - labels, i.e. [CE1], [AB1,XY2], - author/year system, (Smith et al. 2000), (Miller 1999a, 12; Brown 2018). We prefer citations with arabic numbers, i.e. the usage of \bibitem without an optional parameter. If you want to use the author/year system, you can use the class option citeauthoryear , i.e. -citeauthoryear - \documentclass[citeauthoryear]{llncs} Please note that this option does not automatically change your citations to the author/year style. It basically redefines the \bibitem command to take the publication year as an optional parameter that is displayed instead of an arabic number. Author name(s) and, if necessary, parentheses are to be typed manually. If your reference reads @@ -275,14 +262,12 @@ van der Aalst, W.: Process Mining, 2nd ed. Springer, Heidelberg(2016) and is cited as follows: ... is shown by van der Aalst(\cite{vdaalst:2016}) the resulting text will be: - "...is shown by van der Aalst(2016)." + '...is shown by van der Aalst(2016).' ``` -We encourage you to use Bib T E X for typesetting your references. For formatting the bibliography according to Springer's standard (for mathematics, physical sciences, and computer science), please use the bibliography style file splncs04.bst that comes with the llncs document class. You simply need to add \bibliographystyle{splncs04} to your document. DOIs should be provided in the doi field of your .bib database. Bib T E X will then automatically add them to your references. Please note that we do not provide an option to implement splncs04.bst +splncs04.bst We encourage you to use Bib T E X for typesetting your references. For formatting the bibliography according to Springer's standard (for mathematics, physical sciences, and computer science), please use the bibliography style file splncs04.bst that comes with the llncs document class. You simply need to add \bibliographystyle{splncs04} to your document. DOIs should be provided in the doi field of your .bib database. Bib T E X will then automatically add them to your references. Please note that we do not provide an option to implement -\doi If you do not use Bib T E X, you can include a DOI with the \doi command: - -- \doi If you do not use Bib T E X, you can include a DOI with the \doi command: \doi{<DOI>} +If you do not use Bib T E X, you can include a DOI with the \doi command: \doi \doi{<DOI>} @@ -292,21 +277,15 @@ The DOI will be expanded to the URL https://doi.org/<DOI> in accordance wi The llncs document class contains several class options that have become obsolete over the years. We only mention them for completeness: -- The document class changes the formatting of vectors coded with orivec llncs \vec to boldface italics. If you absolutely need the original L A T E X design for vectors, i.e. an arrow above the related variable, you can restore it with the option. orivec -- All theorem-like environments share one counter, i.e. Theorem 1, Lemma 2, Corollary 3, etc. +- The llncs document class changes the formatting of vectors coded with orivec \vec to boldface italics. If you absolutely need the original L A T E X design for vectors, i.e. an arrow above the related variable, you can restore it with the orivec option. envcountsame -- All theorem-like environments are numbered per section, i.e. the related counters are reset to 1 in every section. +- All theorem-like environments share one counter, i.e. Theorem 1, Lemma 2, Corollary 3, etc. -envcountreset +envcountreset envcountsect openbib +- All theorem-like environments are numbered per section, i.e. the related counters are reset to 1 in every section. - All theorem-like environments are numbered per section, and the section number added to the individual counter, i.e. Theorem 1.2, Lemma 2.2, etc. - -envcountsect - -- This option produces the "open" bibliography style, in which each block starts on a new line, and succeeding lines in a block are indented by . \bibindent - -openbib - -- oribibl - This option restores the original L A T E X definitions for the bibliography and the \cite mechanism that some Bib T E X applications rely on. +- This option produces the 'open' bibliography style, in which each block starts on a new line, and succeeding lines in a block are indented by \bibindent . +- This option restores the original L A T E X definitions for the bibliography and oribibl the \cite mechanism that some Bib T E X applications rely on. diff --git a/tests/snapshots/mets_gbs/sources/32044009881525_select.tar.gz.md b/tests/snapshots/mets_gbs/sources/32044009881525_select.tar.gz.md index 87e2c775..b9118616 100644 --- a/tests/snapshots/mets_gbs/sources/32044009881525_select.tar.gz.md +++ b/tests/snapshots/mets_gbs/sources/32044009881525_select.tar.gz.md @@ -1,4 +1,4 @@ -recently become prevalent that he who speaks of military power is a " militarist . " This , howreverse assertion ever , is as great a fallacy as the that he who talks of nothing but peace is a 66 pacifist . " +recently become prevalent that he who speaks of military power is a " militarist . " This , however , is as great a fallacy as the reverse assertion that he who talks of nothing but peace is a pacifist . " 66 Truth , even bitter truth , is better than the most high - minded fallacy . @@ -6,7 +6,7 @@ The author has visited Japan , Siberia , China , the Philippines , the Malay Sta The author wishes to acknowledge his debt to Admiral A. D. Bubnov , who has contributed Chapters VII - X . Admiral Bubnov took part in the Russo - Japanese War , was Professor of the Naval Staff College at Petrograd , and Chief of the Naval Section of the Staff of the Supreme Commander - in - Chief in the Great War . The Admiral is an authoritative student of the questions of naval strategy discussed in the chapters that belong to his pen . -The author also has to thank Mr. C. Nabokoff , the late Russian Chargé d'Affaires in London , book . for undertaking the translation of his +The author also has to thank Mr. C. Nabokoff , the late Russian Chargé d'Affaires in London , for undertaking the translation of his book . ## 62 THE PROBLEM OF THE PACIFIC @@ -26,7 +26,7 @@ copper mines with up - to - date technical equipment in various provinces of Cen | | | | | | | | | | -It has been mentioned in one of the preceding chapters that Japanese industries needed iron . The desire to obtain the necessary supplies of iron is thus perfectly legitimate . Japan's endeavour to acquire concessions for the production of iron and coal is not , therefore , a proof of the Imperialism of her policy . But , as the French saying goes , Le ton fait la chanson . Japan is trying to get hold of the entire iron industry of China . She is doing so by the veiled seizure of political and administrative control of the respective provinces of China . To Europe and America this is presented under the guise of the nebulous formula of 66 special interests in the regions of China adjacent to Japan . " Manchuria and Shantung are first in that list . When Japan succeeds , she will have deprived China +It has been mentioned in one of the preceding chapters that Japanese industries needed iron . The desire to obtain the necessary supplies of iron is thus perfectly legitimate . Japan's endeavour to acquire concessions for the production of iron and coal is not , therefore , a proof of the Imperialism of her policy . But , as the French saying goes , Le ton fait la chanson . Japan is trying to get hold of the entire iron industry of China . She is doing so by the veiled seizure of political and administrative control of the respective provinces of China . To Europe and America this is presented under the guise of the nebulous formula of special interests in the regions of China adjacent to Japan . " Manchuria and Shantung are first in that list . When Japan succeeds , she will have deprived China 66 1 These data relate to 1915 . diff --git a/tests/snapshots/odf/sources/odf_presentation_01.odp.pdf.md b/tests/snapshots/odf/sources/odf_presentation_01.odp.pdf.md index 60c69367..e69de29b 100644 --- a/tests/snapshots/odf/sources/odf_presentation_01.odp.pdf.md +++ b/tests/snapshots/odf/sources/odf_presentation_01.odp.pdf.md @@ -1 +0,0 @@ -<!-- image --> diff --git a/tests/snapshots/odf/sources/odf_presentation_02.odp.pdf.md b/tests/snapshots/odf/sources/odf_presentation_02.odp.pdf.md index 1163b433..d01027fa 100644 --- a/tests/snapshots/odf/sources/odf_presentation_02.odp.pdf.md +++ b/tests/snapshots/odf/sources/odf_presentation_02.odp.pdf.md @@ -3,13 +3,3 @@ <!-- image --> <!-- image --> - -<!-- image --> - -<!-- image --> - -<!-- image --> - -<!-- image --> - -<!-- image --> diff --git a/tests/snapshots/odf/sources/text_document_01.odt.pdf.md b/tests/snapshots/odf/sources/text_document_01.odt.pdf.md index 5a42db36..1e9c057e 100644 --- a/tests/snapshots/odf/sources/text_document_01.odt.pdf.md +++ b/tests/snapshots/odf/sources/text_document_01.odt.pdf.md @@ -1,7 +1,3 @@ -## Title - -## Subtitle - ## heading level 1.0 Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since 1966, when designers at Letraset and James Mosley, the librarian at St Bride Printing Library in London, took a 1914 Cicero translation and scrambled it to make dummy text for Letraset's Body Type sheets. It has survived not only many decades, but also the leap into electronic typesetting, remaining essentially unchanged. @@ -17,9 +13,13 @@ Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots - bullet list 2.2 - bullet list 2.2.1 - bullet list 2.3 -- numbered list 3 • +- numbered list 3 - bullet list 3.0.1 ## Heading level 2: B Contrary to popular belief, Lorem Ipsum is not simply random text . Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia , looked up one of the more obscure Latin words, consectetur, from a Lorem Ipsum passage, and going through the cites of the word in classical literature, discovered the undoubtable source. Lorem Ipsum comes from sections 1.10.32 and 1.10.33 of "de Finibus Bonorum et Malorum" (The Extremes of Good and Evil) by Cicero, written in 45 BC. + +## Title + +## Subtitle diff --git a/tests/snapshots/odf/sources/text_document_02.odt.pdf.md b/tests/snapshots/odf/sources/text_document_02.odt.pdf.md index fd992c8d..4fb30f13 100644 --- a/tests/snapshots/odf/sources/text_document_02.odt.pdf.md +++ b/tests/snapshots/odf/sources/text_document_02.odt.pdf.md @@ -1,5 +1,3 @@ -## Title - ## heading level 1.0 Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since 1966. @@ -8,13 +6,13 @@ Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots in a piece of classical Latin literature from 45 BC, making it over 2000 years old. -| Merged row | Cell 1.3 | Cell 1.4 | Cell 1.5 | Cell 1.6 | Cell 1.7 | -|--------------|-----------------|-----------------|------------|---------------|------------| -| | Merged cell 2x2 | Merged cell 2x2 | | Merged column | | -| | | | | Merged column | | -| | | | | Merged column | | -| | | | | Merged column | | +| Merged row | Cell 1.3 | Cell 1.4 | Cell 1.5 | Cell 1.6 | Cell 1.7 | +|--------------|-----------------|-----------------|------------|------------|------------| +| | Merged cell 2x2 | Merged cell 2x2 | | Merged | | +| | | | | column | | <!-- image --> <!-- image --> + +## Title diff --git a/tests/snapshots/odf/sources/text_document_03.odt.pdf.md b/tests/snapshots/odf/sources/text_document_03.odt.pdf.md index 67d9f1b2..4a111584 100644 --- a/tests/snapshots/odf/sources/text_document_03.odt.pdf.md +++ b/tests/snapshots/odf/sources/text_document_03.odt.pdf.md @@ -15,19 +15,22 @@ Before table | Simple cell upper left | Simple cell upper left | Simple cell upper left | Simple cell with bold and italic text | Simple cell with bold and italic text | Simple cell with bold and italic text | | A | B | C | Rich cell | Rich cell | Rich cell | | Cell 1 | Cell 2 | Cell 3 | A nested table | A nested table | A nested table | -| | | | | | | -| | | | A | B | C | -| | | | Cell 1 | Cell 2 | Cell 3 | +| | | | | B | C | +| | | | | Cell 2 | Cell 3 | After table with bold , underline, strikethrough, and italic formatting ## Table with pictures +<!-- image --> + | Column A | Column B | |------------------|------------| | Only text | | | Text and picture | | +<!-- image --> + ## Lists with same numId in different cells | • Cell 1 item 1 | @@ -46,11 +49,11 @@ After table with bold , underline, strikethrough, and italic formatting ## Multiple columns with lists -| • R1C1 item 1 R1C1 item 2 | • R1C2 item 1 • | -|-----------------------------|-------------------| -| • | R1C2 item 2 | -| • R2C1 item 1 | • R2C2 item 1 | -| • R2C1 item 2 | • R2C2 item 2 | +| • R1C1 item 1 | • R1C2 item 1 | +|-----------------|-----------------| +| • R1C1 item 2 | • R1C2 item 2 | +| • R2C1 item 1 | • R2C2 item 1 | +| • R2C1 item 2 | • R2C2 item 2 | ## Mixed content - list and regular text in different cells diff --git a/tests/snapshots/pdf/sources/2203.01017v2.pdf.md b/tests/snapshots/pdf/sources/2203.01017v2.pdf.md index 4c9df89c..9c2108db 100644 --- a/tests/snapshots/pdf/sources/2203.01017v2.pdf.md +++ b/tests/snapshots/pdf/sources/2203.01017v2.pdf.md @@ -12,28 +12,23 @@ The occurrence of tables in documents is ubiquitous. They often summarise quanti ## a. Picture of a table: -Figure 1: Picture of a table with subtle, complex features such as (1) multi-column headers, (2) cell with multi-row text and (3) cells with no content. Image from PubTabNet evaluation set, filename: 'PMC2944238 004 02'. - -<!-- image --> - -| | 3 | | 1 | | -|----|-----|----|-----|----| -| 2 | | | | | -| 2 | | | | | -| 2 | | | | | +| 1 | +|-----| - b. Red-annotation of bounding boxes, Blue-predictions by TableFormer +Figure 1: Picture of a table with subtle, complex features such as (1) multi-column headers, (2) cell with multi-row text and (3) cells with no content. Image from PubTabNet evaluation set, filename: 'PMC2944238 004 02'. + <!-- image --> -- c. Structure predicted by TableFormer: +c. Structure predicted by TableFormer: -| 0 | 1 | 2 1 | 2 1 | 2 1 | -|-----|-----|-------|-------|-------| -| 3 | 4 | 5 | 6 | 7 | -| 8 2 | 9 | 10 | 11 | 12 | -| 8 2 | 13 | 14 | 15 | 16 | -| 8 2 | 17 | 18 | 19 | 20 | +| 0 | 1 2 | 1 2 | 1 | 1 | +|-----|-------|-------|-----|-----| +| 3 | 4 | 5 | 6 | 7 | +| 8 2 | 9 | 10 | 11 | 12 | +| 8 2 | 13 | 14 | 15 | 16 | +| 8 2 | 17 | 18 | 19 | 20 | Recently, significant progress has been made with vision based approaches to extract tables in documents. For the sake of completeness, the issue of table extraction from documents is typically decomposed into two separate challenges, i.e. (1) finding the location of the table(s) on a document-page and (2) finding the structure of a given table in the document. @@ -43,16 +38,16 @@ The second problem is called table-structure decomposition. The latter is a long In this paper, we want to address these weaknesses and present a robust table-structure decomposition algorithm. The design criteria for our model are the following. First, we want our algorithm to be language agnostic. In this way, we can obtain the structure of any table, irregardless of the language. Second, we want our algorithm to leverage as much data as possible from the original PDF document. For programmatic PDF documents, the text-cells can often be extracted much faster and with higher accuracy compared to OCR methods. Last but not least, we want to have a direct link between the table-cell and its bounding box in the image. -To meet the design criteria listed above, we developed a new model called TableFormer and a synthetically gener- 1 ated table structure dataset called SynthTabNet . In particular, our contributions in this work can be summarised as follows: +To meet the design criteria listed above, we developed a new model called TableFormer and a synthetically generated table structure dataset called SynthTabNet 1 . In particular, our contributions in this work can be summarised as follows: - We propose TableFormer , a transformer based model that predicts tables structure and bounding boxes for the table content simultaneously in an end-to-end approach. - Across all benchmark datasets TableFormer significantly outperforms existing state-of-the-art metrics, while being much more efficient in training and inference to existing works. - We present SynthTabNet a synthetically generated dataset, with various appearance styles and complexity. - An augmented dataset based on PubTabNet [37], FinTabNet [36], and TableBank [17] with generated ground-truth for reproducibility. -The paper is structured as follows. In Sec. 2, we give a brief overview of the current state-of-the-art. In Sec. 3, we describe the datasets on which we train. In Sec. 4, we introduce the TableFormer model-architecture and describe +The paper is structured as follows. In Sec. 2, we give a brief overview of the current state-of-the-art. In Sec. 3, we describe the datasets on which we train. In Sec. 4, we introduce the TableFormer model-architecture and describe its results & performance in Sec. 5. As a conclusion, we describe how this new model-architecture can be re-purposed for other tasks in the computer-vision community. -1 https://github.com/IBM/SynthTabNet its results & performance in Sec. 5. As a conclusion, we describe how this new model-architecture can be re-purposed for other tasks in the computer-vision community. +1 https://github.com/IBM/SynthTabNet ## 2. Previous work and State of the Art @@ -60,7 +55,7 @@ Identifying the structure of a table has been an outstanding problem in the docu Before the rising popularity of deep neural networks, the community relied heavily on heuristic and/or statistical methods to do table structure identification [3, 7, 11, 5, 13, 28]. Although such methods work well on constrained tables [12], a more data-driven approach can be applied due to the advent of convolutional neural networks (CNNs) and the availability of large datasets. To the best-of-our knowledge, there are currently two different types of network architecture that are being pursued for state-of-the-art tablestructure identification. -Image-to-Text networks : In this type of network, one predicts a sequence of tokens starting from an encoded image. Such sequences of tokens can be HTML table tags [37, 17] or LaTeX symbols[10]. The choice of symbols is ultimately not very important, since one can be transformed into the other. There are however subtle variations in the Image-to-Text networks. The easiest network architectures are "image-encoder ! text-decoder" (IETD), similar to network architectures that try to provide captions to images [32]. In these IETD networks, one expects as output the LaTeX/HTML string of the entire table, i.e. the symbols necessary for creating the table with the content of the table. Another approach is the "image-encoder ! dual decoder" (IEDD) networks. In these type of networks, one has two consecutive decoders with different purposes. The first decoder is the tag-decoder , i.e. it only produces the HTM- L/LaTeX tags which construct an empty table. The second content-decoder uses the encoding of the image in combination with the output encoding of each cell-tag (from the tag-decoder ) to generate the textual content of each table cell. The network architecture of IEDD is certainly more elaborate, but it has the advantage that one can pre-train the tag-decoder which is constrained to the table-tags. +Image-to-Text networks : In this type of network, one predicts a sequence of tokens starting from an encoded image. Such sequences of tokens can be HTML table tags [37, 17] or LaTeX symbols[10]. The choice of symbols is ultimately not very important, since one can be transformed into the other. There are however subtle variations in the Image-to-Text networks. The easiest network architectures are 'image-encoder ! text-decoder' (IETD), similar to network architectures that try to provide captions to images [32]. In these IETD networks, one expects as output the LaTeX/HTML string of the entire table, i.e. the symbols necessary for creating the table with the content of the table. Another approach is the 'image-encoder ! dual decoder' (IEDD) networks. In these type of networks, one has two consecutive decoders with different purposes. The first decoder is the tag-decoder , i.e. it only produces the HTM- L/LaTeX tags which construct an empty table. The second content-decoder uses the encoding of the image in combination with the output encoding of each cell-tag (from the tag-decoder ) to generate the textual content of each table cell. The network architecture of IEDD is certainly more elaborate, but it has the advantage that one can pre-train the tag-decoder which is constrained to the table-tags. In practice, both network architectures (IETD and IEDD) require an implicit, custom trained object-characterrecognition (OCR) to obtain the content of the table-cells. In the case of IETD, this OCR engine is implicit in the decoder similar to [24]. For the IEDD, the OCR is solely embedded in the content-decoder. This reliance on a custom, implicit OCR decoder is of course problematic. OCR is a well known and extremely tough problem, that often needs custom training for each individual language. However, the limited availability for non-english content in the current datasets, makes it impractical to apply the IETD and IEDD methods on tables with other languages. Additionally, OCR can be completely omitted if the tables originate from programmatic PDF documents with known positions of each cell. The latter was the inspiration for the work of this paper. @@ -76,7 +71,7 @@ Figure 2: Distribution of the tables across different table dimensions in PubTab <!-- image --> -The PubTabNet dataset contains 509k tables delivered as annotated PNG images. The annotations consist of the table structure represented in HTML format, the tokenized text and its bounding boxes per table cell. Fig. 1 shows the appearance style of PubTabNet. Depending on its complexity, a table is characterized as "simple" when it does not contain row spans or column spans, otherwise it is "complex". The dataset is divided into Train and Val splits (roughly 98% and 2%). The Train split consists of 54% simple and 46% complex tables and the Val split of 51% and 49% respectively. The FinTabNet dataset contains 112k tables delivered as single-page PDF documents with mixed table structures and text content. Similarly to the PubTabNet, the annotations of FinTabNet include the table structure in HTML, the tokenized text and the bounding boxes on a table cell basis. The dataset is divided into Train, Test and Val splits (81%, 9.5%, 9.5%), and each one is almost equally divided into simple and complex tables (Train: 48% simple, 52% complex, Test: 48% simple, 52% complex, Test: 53% simple, 47% complex). Finally the TableBank dataset consists of 145k tables provided as JPEG images. The latter has annotations for the table structure, but only few with bounding boxes of the table cells. The entire dataset consists of simple tables and it is divided into 90% Train, 3% Test and 7% Val splits. +The PubTabNet dataset contains 509k tables delivered as annotated PNG images. The annotations consist of the table structure represented in HTML format, the tokenized text and its bounding boxes per table cell. Fig. 1 shows the appearance style of PubTabNet. Depending on its complexity, a table is characterized as 'simple' when it does not contain row spans or column spans, otherwise it is 'complex'. The dataset is divided into Train and Val splits (roughly 98% and 2%). The Train split consists of 54% simple and 46% complex tables and the Val split of 51% and 49% respectively. The FinTabNet dataset contains 112k tables delivered as single-page PDF documents with mixed table structures and text content. Similarly to the PubTabNet, the annotations of FinTabNet include the table structure in HTML, the tokenized text and the bounding boxes on a table cell basis. The dataset is divided into Train, Test and Val splits (81%, 9.5%, 9.5%), and each one is almost equally divided into simple and complex tables (Train: 48% simple, 52% complex, Test: 48% simple, 52% complex, Test: 53% simple, 47% complex). Finally the TableBank dataset consists of 145k tables provided as JPEG images. The latter has annotations for the table structure, but only few with bounding boxes of the table cells. The entire dataset consists of simple tables and it is divided into 90% Train, 3% Test and 7% Val splits. Due to the heterogeneity across the dataset formats, it was necessary to combine all available data into one homogenized dataset before we could train our models for practical purposes. Given the size of PubTabNet, we adopted its annotation format and we extracted and converted all tables as PNG images with a resolution of 72 dpi. Additionally, we have filtered out tables with extreme sizes due to small amount of such tables, and kept only those ones ranging between 1*1 and 20*10 (rows/columns). @@ -86,7 +81,7 @@ As it is illustrated in Fig. 2, the table distributions from all datasets are sk Motivated by those observations we aimed at generating a synthetic table dataset named SynthTabNet . This approach offers control over: 1) the size of the dataset, 2) the table structure, 3) the table style and 4) the type of content. The complexity of the table structure is described by the size of the table header and the table body, as well as the percentage of the table cells covered by row spans and column spans. A set of carefully designed styling templates provides the basis to build a wide range of table appearances. Lastly, the table content is generated out of a curated collection of text corpora. By controlling the size and scope of the synthetic datasets we are able to train and evaluate our models in a variety of different conditions. For example, we can first generate a highly diverse dataset to train our models and then evaluate their performance on other synthetic datasets which are focused on a specific domain. -In this regard, we have prepared four synthetic datasets, each one containing 150k examples. The corpora to generate the table text consists of the most frequent terms appearing in PubTabNet and FinTabNet together with randomly generated text. The first two synthetic datasets have been fine-tuned to mimic the appearance of the original datasets but encompass more complicated table structures. The third +In this regard, we have prepared four synthetic datasets, each one containing 150k examples. The corpora to generate the table text consists of the most frequent terms appearing in PubTabNet and FinTabNet together with randomly generated text. The first two synthetic datasets have been fine-tuned to mimic the appearance of the original datasets but encompass more complicated table structures. The third one adopts a colorful appearance with high contrast and the last one contains tables with sparse content. Lastly, we have combined all synthetic datasets into one big unified synthetic dataset of 600k examples. | | Tags | Bbox | Size | Format | |--------------------|--------|--------|--------|----------| @@ -97,9 +92,7 @@ In this regard, we have prepared four synthetic datasets, each one containing 15 | Combined(**) | 3 | 3 | 500k | PNG | | SynthTabNet | 3 | 3 | 600k | PNG | -Table 1: Both "Combined-Tabnet" and "CombinedTabnet" are variations of the following: (*) The CombinedTabnet dataset is the processed combination of PubTabNet and Fintabnet. (**) The combined dataset is the processed combination of PubTabNet, Fintabnet and TableBank. - -one adopts a colorful appearance with high contrast and the last one contains tables with sparse content. Lastly, we have combined all synthetic datasets into one big unified synthetic dataset of 600k examples. +Table 1: Both 'Combined-Tabnet' and 'CombinedTabnet' are variations of the following: (*) The CombinedTabnet dataset is the processed combination of PubTabNet and Fintabnet. (**) The combined dataset is the processed combination of PubTabNet, Fintabnet and TableBank. Tab. 1 summarizes the various attributes of the datasets. @@ -121,7 +114,7 @@ Figure 4: Given an input image of a table, the Encoder produces fixed-length fea <!-- image --> -Structure Decoder. The transformer architecture of this component is based on the work proposed in [31]. After extensive experimentation, the Structure Decoder is modeled as a transformer encoder with two encoder layers and a transformer decoder made from a stack of 4 decoder layers that comprise mainly of multi-head attention and feed forward layers. This configuration uses fewer layers and heads in comparison to networks applied to other problems (e.g. "Scene Understanding", "Image Captioning"), something which we relate to the simplicity of table images. +Structure Decoder. The transformer architecture of this component is based on the work proposed in [31]. After extensive experimentation, the Structure Decoder is modeled as a transformer encoder with two encoder layers and a transformer decoder made from a stack of 4 decoder layers that comprise mainly of multi-head attention and feed forward layers. This configuration uses fewer layers and heads in comparison to networks applied to other problems (e.g. 'Scene Understanding', 'Image Captioning'), something which we relate to the simplicity of table images. The transformer encoder receives an encoded image from the CNN Backbone Network and refines it through a multi-head dot-product attention layer, followed by a Feed Forward Network. During training, the transformer decoder receives as input the output feature produced by the transformer encoder, and the tokenized input of the HTML ground-truth tags. Using a stack of multi-head attention layers, different aspects of the tag sequence could be inferred. This is achieved by each attention head on a layer operating in a different subspace, and then combining altogether their attention score. @@ -131,13 +124,13 @@ The encoding generated by the CNN Backbone Network along with the features acqui The output features for each table cell are then fed into the feed-forward network (FFN). The FFN consists of a Multi-Layer Perceptron (3 layers with ReLU activation function) that predicts the normalized coordinates for the bounding box of each table cell. Finally, the predicted bounding boxes are classified based on whether they are empty or not using a linear layer. -Loss Functions. We formulate a multi-task loss Eq. 2 to train our network. The Cross-Entropy loss (denoted as l s ) is used to train the Structure Decoder which predicts the structure tokens. As for the Cell BBox Decoder it is trained with a combination of losses denoted as l box . l box consists of the generally used l 1 loss for object detection and the IoU loss ( l ) to be scale invariant as explained in [25]. In iou comparison to DETR, we do not use the Hungarian algorithm [15] to match the predicted bounding boxes with the ground-truth boxes, as we have already achieved a one-toone match through two steps: 1) Our token input sequence is naturally ordered, therefore the hidden states of the table data cells are also in order when they are provided as input to the Cell BBox Decoder , and 2) Our bounding boxes generation mechanism (see Sec. 3) ensures a one-to-one mapping between the cell content and its bounding box for all post-processed datasets. +Loss Functions. We formulate a multi-task loss Eq. 2 to train our network. The Cross-Entropy loss (denoted as l s ) is used to train the Structure Decoder which predicts the structure tokens. As for the Cell BBox Decoder it is trained with a combination of losses denoted as l box . l box consists of the generally used l 1 loss for object detection and the IoU loss ( l iou ) to be scale invariant as explained in [25]. In comparison to DETR, we do not use the Hungarian algorithm [15] to match the predicted bounding boxes with the ground-truth boxes, as we have already achieved a one-toone match through two steps: 1) Our token input sequence is naturally ordered, therefore the hidden states of the table data cells are also in order when they are provided as input to the Cell BBox Decoder , and 2) Our bounding boxes generation mechanism (see Sec. 3) ensures a one-to-one mapping between the cell content and its bounding box for all post-processed datasets. The loss used to train the TableFormer can be defined as following: <!-- formula-not-decoded --> -where λ 2 [0, 1], and λ ; λ 2 are hyper-parameters. iou l 1 R +where λ 2 [0, 1], and λ iou ; λ l 1 2 R are hyper-parameters. ## 5. Experimental Results @@ -149,7 +142,7 @@ TableFormer uses ResNet-18 as the CNN Backbone Network . The input images are re Although input constraints are used also by other methods, such as EDD, ours are less restrictive due to the improved runtime performance and lower memory footprint of TableFormer. This allows to utilize input samples with longer sequences and images with larger dimensions. -The Transformer Encoder consists of two "Transformer Encoder Layers", with an input feature size of 512, feed forward network of 1024, and 4 attention heads. As for the Transformer Decoder it is composed of four "Transformer Decoder Layers" with similar input and output dimensions as the "Transformer Encoder Layers". Even though our model uses fewer layers and heads than the default implementation parameters, our extensive experimentation has proved this setup to be more suitable for table images. We attribute this finding to the inherent design of table images, which contain mostly lines and text, unlike the more elaborate content present in other scopes (e.g. the COCO dataset). Moreover, we have added ResNet blocks to the inputs of the Structure Decoder and Cell BBox Decoder. This prevents a decoder having a stronger influence over the learned weights which would damage the other prediction task (structure vs bounding boxes), but learn task specific weights instead. Lastly our dropout layers are set to 0.5. +The Transformer Encoder consists of two 'Transformer Encoder Layers', with an input feature size of 512, feed forward network of 1024, and 4 attention heads. As for the Transformer Decoder it is composed of four 'Transformer Decoder Layers' with similar input and output dimensions as the 'Transformer Encoder Layers'. Even though our model uses fewer layers and heads than the default implementation parameters, our extensive experimentation has proved this setup to be more suitable for table images. We attribute this finding to the inherent design of table images, which contain mostly lines and text, unlike the more elaborate content present in other scopes (e.g. the COCO dataset). Moreover, we have added ResNet blocks to the inputs of the Structure Decoder and Cell BBox Decoder. This prevents a decoder having a stronger influence over the learned weights which would damage the other prediction task (structure vs bounding boxes), but learn task specific weights instead. Lastly our dropout layers are set to 0.5. For training, TableFormer is trained with 3 Adam optimizers, each one for the CNN Backbone Network , Structure Decoder , and Cell BBox Decoder . Taking the PubTabNet as an example for our parameter set up, the initializing learning rate is 0.001 for 12 epochs with a batch size of 24, and λ set to 0.5. Afterwards, we reduce the learning rate to 0.0001, the batch size to 18 and train for 12 more epochs or convergence. @@ -167,7 +160,7 @@ The Tree-Edit-Distance-Based Similarity (TEDS) metric was introduced in [37]. It <!-- formula-not-decoded --> -where T and T represent tables in tree structure HTML a b j j format. EditDist denotes the tree-edit distance, and T represents the number of nodes in T . +where T a and T b represent tables in tree structure HTML format. EditDist denotes the tree-edit distance, and j T j represents the number of nodes in T . ## 5.4. Quantitative Analysis @@ -190,7 +183,7 @@ Table 2: Structure results on PubTabNet (PTN), FinTabNet (FTN), TableBank (TB) a FT: Model was trained on PubTabNet then finetuned. -Cell Detection. Like any object detector, our Cell BBox Detector provides bounding boxes that can be improved with post-processing during inference. We make use of the grid-like structure of tables to refine the predictions. A detailed explanation on the post-processing is available in the supplementary material. As shown in Tab. 3, we evaluate our Cell BBox Decoder accuracy for cells with a class label of 'content' only using the PASCAL VOC mAP metric for pre-processing and post-processing. Note that we do not have post-processing results for SynthTabNet as images are only provided. To compare the performance of our proposed approach, we've integrated TableFormer's Cell BBox into EDD architecture. As mentioned previously, Decoder the Structure Decoder provides the Cell BBox Decoder with the features needed to predict the bounding box predictions. Therefore, the accuracy of the Structure Decoder directly influences the accuracy of the Cell BBox Decoder . If the Structure Decoder predicts an extra column, this will result in an extra column of predicted bounding boxes. +Cell Detection. Like any object detector, our Cell BBox Detector provides bounding boxes that can be improved with post-processing during inference. We make use of the grid-like structure of tables to refine the predictions. A detailed explanation on the post-processing is available in the supplementary material. As shown in Tab. 3, we evaluate our Cell BBox Decoder accuracy for cells with a class label of 'content' only using the PASCAL VOC mAP metric for pre-processing and post-processing. Note that we do not have post-processing results for SynthTabNet as images are only provided. To compare the performance of our proposed approach, we've integrated TableFormer's Cell BBox Decoder into EDD architecture. As mentioned previously, the Structure Decoder provides the Cell BBox Decoder with the features needed to predict the bounding box predictions. Therefore, the accuracy of the Structure Decoder directly influences the accuracy of the Cell BBox Decoder . If the Structure Decoder predicts an extra column, this will result in an extra column of predicted bounding boxes. | Model | Dataset | mAP | mAP (PP) | |-------------|-------------|-------|------------| @@ -213,25 +206,16 @@ Cell Content. In this section, we evaluate the entire pipeline of recovering a t Table 4: Results of structure with content retrieved using cell detection on PubTabNet. In all cases the input is PDF documents with cropped tables. -- a. Red - PDF cells, Green - predicted bounding boxes, Blue - post-processed predictions matched to PDF cells Japanese language (previously unseen by TableFormer): Example table from FinTabNet: - Japanese language (previously unseen by TableFormer): +- Japanese language (previously unseen by TableFormer): Example table from FinTabNet: a. Red - PDF cells, Green - predicted bounding boxes, Blue - post-processed predictions matched to PDF cells + Figure 5: One of the benefits of TableFormer is that it is language agnostic, as an example, the left part of the illustration demonstrates TableFormer predictions on previously unseen language (Japanese). Additionally, we see that TableFormer is robust to variability in style and content, right side of the illustration shows the example of the TableFormer prediction from the FinTabNet dataset. <!-- image --> - b. Structure predicted by TableFormer, with superimposed matched PDF cell text: -| | Shares (in millions) | Shares (in millions) | Weighted Average Grant Date Fair Value | Weighted Average Grant Date Fair Value | -|--------------------------|------------------------|------------------------|------------------------------------------|------------------------------------------| -| | RSUs | PSUs | RSUs | PSUs | -| Nonvested on January 1 | 1.1 | 0.3 | 90.10 $ | $ 91.19 | -| Granted | 0.5 | 0.1 | 117.44 | 122.41 | -| Vested | (0.5) | (0.1) | 87.08 | 81.14 | -| Canceled or forfeited | (0.1) | — | 102.01 | 92.18 | -| Nonvested on December 31 | 1.0 | 0.3 | 104.85 $ | $ 104.51 | - | | | 論文ファイル | 論文ファイル | 参考文献 | 参考文献 | |----------------------------------------------------|-------|----------|----------|--------|--------| | 出典 | ファイル数 | 英語 | 日本語 | 英語 | 日本語 | @@ -244,7 +228,14 @@ Figure 5: One of the benefits of TableFormer is that it is language agnostic, as | WWW から収集した論文 | 107 | 73 | 34 | 147 | 96 | | 計 | 945 | 294 | 651 | 1122 | 955 | -<!-- image --> +| | Shares (in millions) | Shares (in millions) | Weighted Average Grant Date Fair Value | Weighted Average Grant Date Fair Value | +|--------------------------|------------------------|------------------------|------------------------------------------|------------------------------------------| +| | RSUs | PSUs | RSUs | PSUs | +| Nonvested on January 1 | 1.1 | 0.3 | 90.10 $ | $ 91.19 | +| Granted | 0.5 | 0.1 | 117.44 | 122.41 | +| Vested | (0.5) | (0.1) | 87.08 | 81.14 | +| Canceled or forfeited | (0.1) | — | 102.01 | 92.18 | +| Nonvested on December 31 | 1.0 | 0.3 | 104.85 $ | $ 104.51 | Text is aligned to match original for ease of viewing @@ -252,13 +243,13 @@ Figure 6: An example of TableFormer predictions (bounding boxes and structure) f <!-- image --> -## 6. Future Work & Conclusion - ## 5.5. Qualitative Analysis -In this paper, we presented TableFormer an end-to-end transformer based approach to predict table structures and bounding boxes of cells from an image. This approach enables us to recreate the table structure, and extract the cell content from PDF or OCR by using bounding boxes. Additionally, it provides the versatility required in real-world scenarios when dealing with various types of PDF documents, and languages. Furthermore, our method outperforms all state-of-the-arts with a wide margin. Finally, we introduce "SynthTabNet" a challenging synthetically generated dataset that reinforces missing characteristics from other datasets. +We showcase several visualizations for the different components of our network on various 'complex' tables within datasets presented in this work in Fig. 5 and Fig. 6 As it is shown, our model is able to predict bounding boxes for all table cells, even for the empty ones. Additionally, our post-processing techniques can extract the cell content by matching the predicted bounding boxes to the PDF cells based on their overlap and spatial proximity. The left part of Fig. 5 demonstrates also the adaptability of our method to any language, as it can successfully extract Japanese text, although the training set contains only English content. We provide more visualizations including the intermediate steps in the supplementary material. Overall these illustrations justify the versatility of our method across a diverse range of table appearances and content type. -We showcase several visualizations for the different components of our network on various "complex" tables within datasets presented in this work in Fig. 5 and Fig. 6 As it is shown, our model is able to predict bounding boxes for all table cells, even for the empty ones. Additionally, our post-processing techniques can extract the cell content by matching the predicted bounding boxes to the PDF cells based on their overlap and spatial proximity. The left part of Fig. 5 demonstrates also the adaptability of our method to any language, as it can successfully extract Japanese text, although the training set contains only English content. We provide more visualizations including the intermediate steps in the supplementary material. Overall these illustrations justify the versatility of our method across a diverse range of table appearances and content type. +## 6. Future Work & Conclusion + +In this paper, we presented TableFormer an end-to-end transformer based approach to predict table structures and bounding boxes of cells from an image. This approach enables us to recreate the table structure, and extract the cell content from PDF or OCR by using bounding boxes. Additionally, it provides the versatility required in real-world scenarios when dealing with various types of PDF documents, and languages. Furthermore, our method outperforms all state-of-the-arts with a wide margin. Finally, we introduce 'SynthTabNet' a challenging synthetically generated dataset that reinforces missing characteristics from other datasets. ## References @@ -293,9 +284,9 @@ end object detection with transformers. In Andrea Vedaldi, Horst Bischof, Thomas Computer Vision and Pattern Recognition , pages 658-666, 2019. 6 -- [26] Sebastian Schreiber, Stefan Agne, Ivo Wolf, Andreas Dengel, and Sheraz Ahmed. Deepdesrt: Deep learning for detection and structure recognition of tables in document images. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) , volume 01, pages 1162- 1167, 2017. 1 +- [26] Sebastian Schreiber, Stefan Agne, Ivo Wolf, Andreas Dengel, and Sheraz Ahmed. Deepdesrt: Deep learning for detection and structure recognition of tables in document images. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) , volume 01, pages 11621167, 2017. 1 - [27] Sebastian Schreiber, Stefan Agne, Ivo Wolf, Andreas Dengel, and Sheraz Ahmed. Deepdesrt: Deep learning for detection and structure recognition of tables in document images. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR) , volume 1, pages 1162-1167. IEEE, 2017. 3 -- [28] Faisal Shafait and Ray Smith. Table detection in heterogeneous documents. In Proceedings of the 9th IAPR International Workshop on Document Analysis Systems , pages 65- 72, 2010. 2 +- [28] Faisal Shafait and Ray Smith. Table detection in heterogeneous documents. In Proceedings of the 9th IAPR International Workshop on Document Analysis Systems , pages 6572, 2010. 2 - [29] Shoaib Ahmed Siddiqui, Imran Ali Fateh, Syed Tahseen Raza Rizvi, Andreas Dengel, and Sheraz Ahmed. Deeptabstr: Deep learning based table structure recognition. In 2019 International Conference on Document Analysis and Recognition (ICDAR) , pages 1403-1409. IEEE, 2019. 3 - [30] Peter W J Staar, Michele Dolfi, Christoph Auer, and Costas Bekas. Corpus conversion service: A machine learning platform to ingest documents at scale. In Proceedings of the 24th ACM SIGKDD , KDD '18, pages 774-782, New York, NY, USA, 2018. ACM. 1 - [31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30 , pages 5998-6008. Curran Associates, Inc., 2017. 5 @@ -308,17 +299,17 @@ Computer Vision and Pattern Recognition , pages 658-666, 2019. 6 - and evaluation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 , pages 564-580, Cham, 2020. Springer International Publishing. 2, 3, 7 - [38] Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: Largest dataset ever for document layout analysis. In 2019 International Conference on Document Analysis and Recognition (ICDAR) , pages 1015-1022, 2019. 1 -## TableFormer: Table Structure Understanding with Transformers Supplementary Material - ## TableFormer: Table Structure Understanding with Transformers Supplementary Material +## TableFormer: Table Structure Understanding with Transformers Supplementary Material + ## 1. Details on the datasets ## 1.1. Data preparation -As a first step of our data preparation process, we have calculated statistics over the datasets across the following dimensions: (1) table size measured in the number of rows and columns, (2) complexity of the table, (3) strictness of the provided HTML structure and (4) completeness (i.e. no omitted bounding boxes). A table is considered to be simple if it does not contain row spans or column spans. Additionally, a table has a strict HTML structure if every row has the same number of columns after taking into account any row or column spans. Therefore a strict HTML structure looks always rectangular. However, HTML is a lenient encoding format, i.e. tables with rows of different sizes might still be regarded as correct due to implicit display rules. These implicit rules leave room for ambiguity, which we want to avoid. As such, we prefer to have "strict" tables, i.e. tables where every row has exactly the same length. +As a first step of our data preparation process, we have calculated statistics over the datasets across the following dimensions: (1) table size measured in the number of rows and columns, (2) complexity of the table, (3) strictness of the provided HTML structure and (4) completeness (i.e. no omitted bounding boxes). A table is considered to be simple if it does not contain row spans or column spans. Additionally, a table has a strict HTML structure if every row has the same number of columns after taking into account any row or column spans. Therefore a strict HTML structure looks always rectangular. However, HTML is a lenient encoding format, i.e. tables with rows of different sizes might still be regarded as correct due to implicit display rules. These implicit rules leave room for ambiguity, which we want to avoid. As such, we prefer to have 'strict' tables, i.e. tables where every row has exactly the same length. We have developed a technique that tries to derive a missing bounding box out of its neighbors. As a first step, we use the annotation data to generate the most fine-grained grid that covers the table structure. In case of strict HTML tables, all grid squares are associated with some table cell and in the presence of table spans a cell extends across multiple grid squares. When enough bounding boxes are known for a rectangular table, it is possible to compute the geometrical border lines between the grid rows and columns. Eventually this information is used to generate the missing bounding boxes. Additionally, the existence of unused grid squares indicates that the table rows have unequal number of columns and the overall structure is non-strict. The generation of missing bounding boxes for non-strict HTML tables is ambiguous and therefore quite challenging. Thus, we have decided to simply discard those tables. In case of PubTabNet we have computed missing bounding boxes for 48% of the simple and 69% of the complex tables. Regarding FinTabNet, 68% of the simple and 98% of the complex tables require the generation of bounding boxes. @@ -353,7 +344,7 @@ Here is a step-by-step description of the prediction postprocessing: 1. Get the minimal grid dimensions - number of rows and columns for the predicted table structure. This represents the most granular grid for the underlying table structure. 2. Generate pair-wise matches between the bounding boxes of the PDF cells and the predicted cells. The Intersection Over Union (IOU) metric is used to evaluate the quality of the matches. -3. Use a carefully selected IOU threshold to designate the matches as "good" ones and "bad" ones. +3. Use a carefully selected IOU threshold to designate the matches as 'good' ones and 'bad' ones. 3. a. If all IOU scores in a column are below the threshold, discard all predictions (structure and bounding boxes) for that column. 4. Find the best-fitting content alignment for the predicted cells with good IOU per each column. The alignment of the column can be identified by the following formula: @@ -392,21 +383,19 @@ Figure 9: Example of a table with big empty distance between cells. Figure 10: Example of a complex table with empty cells. -Figure 13: Table predictions example on colorful table. - -<!-- image --> - Figure 11: Simple table with different style and empty cells. <!-- image --> +Figure 13: Table predictions example on colorful table. + <!-- image --> -Figure 14: Example with multi-line text. +Figure 12: Simple table predictions and post processing. <!-- image --> -Figure 12: Simple table predictions and post processing. +Figure 14: Example with multi-line text. <!-- image --> diff --git a/tests/snapshots/pdf/sources/2206.01062.pdf.md b/tests/snapshots/pdf/sources/2206.01062.pdf.md index e056ec0c..f7095da7 100644 --- a/tests/snapshots/pdf/sources/2206.01062.pdf.md +++ b/tests/snapshots/pdf/sources/2206.01062.pdf.md @@ -1,32 +1,8 @@ ## DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis -Birgit Pfitzmann IBM Research Rueschlikon, Switzerland bpf@zurich.ibm.com +Birgit Pfitzmann IBM Research Rueschlikon, Switzerland bpf@zurich.ibm.com Christoph Auer IBM Research Rueschlikon, Switzerland cau@zurich.ibm.com -Ahmed S. Nassar IBM Research Rueschlikon, Switzerland ahn@zurich.ibm.com - -## ABSTRACT - -Accurate document layout analysis is a key requirement for highquality PDF document conversion. With the recent availability of public, large ground-truth datasets such as PubLayNet and DocBank, deep-learning models have proven to be very effective at layout detection and segmentation. While these datasets are of adequate size to train such models, they severely lack in layout variability since they are sourced from scientific article repositories such as PubMed and arXiv only. Consequently, the accuracy of the layout segmentation drops significantly when these models are applied on more challenging and diverse layouts. In this paper, we present DocLayNet , a new, publicly available, document-layout annotation dataset in COCO format. It contains 80863 manually annotated pages from diverse data sources to represent a wide variability in layouts. For each PDF page, the layout annotations provide labelled bounding-boxes with a choice of 11 distinct classes. DocLayNet also provides a subset of double- and triple-annotated pages to determine the inter-annotator agreement. In multiple experiments, we provide baseline accuracy scores (in mAP) for a set of popular object detection models. We also demonstrate that these models fall approximately 10% behind the inter-annotator agreement. Furthermore, we provide evidence that DocLayNet is of sufficient size. Lastly, we compare models trained on PubLayNet, DocBank and DocLayNet, showing that layout predictions of the DocLayNettrained models are more robust and thus the preferred choice for general-purpose document-layout analysis. - -## CCS CONCEPTS - -• Informationsystems → Documentstructure ; • Appliedcomputing → Document analysis ; • Computing methodologies → Machine learning ; Computer vision ; Object detection ; - -Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). - -KDD '22, August 14-18, 2022, Washington, DC, USA - -© 2022 Copyright held by the owner/author(s). - -ACM ISBN 978-1-4503-9385-0/22/08. - -https://doi.org/10.1145/3534678.3539043 - -Michele Dolfi IBM Research Rueschlikon, Switzerland dol@zurich.ibm.com - -Christoph Auer IBM Research Rueschlikon, Switzerland cau@zurich.ibm.com - -Peter Staar IBM Research Rueschlikon, Switzerland taa@zurich.ibm.com +Ahmed S. Nassar IBM Research Rueschlikon, Switzerland ahn@zurich.ibm.com Michele Dolfi IBM Research Rueschlikon, Switzerland dol@zurich.ibm.com Peter Staar IBM Research Rueschlikon, Switzerland taa@zurich.ibm.com <!-- image --> @@ -48,6 +24,24 @@ PDF document conversion, layout segmentation, object-detection, data set, Machin Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S. Nassar, and Peter Staar. 2022. DocLayNet: A Large Human-Annotated Dataset for DocumentLayout Analysis. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '22), August 14-18, 2022, Washington, DC, USA. ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/ 3534678.3539043 +## ABSTRACT + +Accurate document layout analysis is a key requirement for highquality PDF document conversion. With the recent availability of public, large ground-truth datasets such as PubLayNet and DocBank, deep-learning models have proven to be very effective at layout detection and segmentation. While these datasets are of adequate size to train such models, they severely lack in layout variability since they are sourced from scientific article repositories such as PubMed and arXiv only. Consequently, the accuracy of the layout segmentation drops significantly when these models are applied on more challenging and diverse layouts. In this paper, we present DocLayNet , a new, publicly available, document-layout annotation dataset in COCO format. It contains 80863 manually annotated pages from diverse data sources to represent a wide variability in layouts. For each PDF page, the layout annotations provide labelled bounding-boxes with a choice of 11 distinct classes. DocLayNet also provides a subset of double- and triple-annotated pages to determine the inter-annotator agreement. In multiple experiments, we provide baseline accuracy scores (in mAP) for a set of popular object detection models. We also demonstrate that these models fall approximately 10% behind the inter-annotator agreement. Furthermore, we provide evidence that DocLayNet is of sufficient size. Lastly, we compare models trained on PubLayNet, DocBank and DocLayNet, showing that layout predictions of the DocLayNettrained models are more robust and thus the preferred choice for general-purpose document-layout analysis. + +## CCS CONCEPTS + +· Informationsystems → Documentstructure ; · Appliedcomputing → Document analysis ; · Computing methodologies → Machine learning ; Computer vision ; Object detection ; + +Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). + +KDD '22, August 14-18, 2022, Washington, DC, USA + +© 2022 Copyright held by the owner/author(s). + +ACM ISBN 978-1-4503-9385-0/22/08. + +https://doi.org/10.1145/3534678.3539043 + KDD '22, August 14-18, 2022, Washington, DC, USA Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S. Nassar, and Peter Staar ## 1 INTRODUCTION @@ -83,13 +77,13 @@ Lately, new types of ML models for document-layout analysis have emerged in the DocLayNet contains 80863 PDF pages. Among these, 7059 carry two instances of human annotations, and 1591 carry three. This amounts to 91104 total annotation instances. The annotations provide layout information in the shape of labeled, rectangular boundingboxes. We define 11 distinct labels for layout features, namely Caption , Footnote , Formula , List-item , Page-footer , Page-header , Picture , Section-header , Table , Text , and Title . Our reasoning for picking this particular label set is detailed in Section 4. -In addition to open intellectual property constraints for the source documents, we required that the documents in DocLayNet adhere to a few conditions. Firstly, we kept scanned documents to a minimum, since they introduce difficulties in annotation (see Section 4). As a second condition, we focussed on medium to large documents ( 10 pages) with technical content, dense in complex > tables, figures, plots and captions. Such documents carry a lot of information value, but are often hard to analyse with high accuracy due to their challenging layouts. Counterexamples of documents not included in the dataset are receipts, invoices, hand-written documents or photographs showing "text in the wild". +In addition to open intellectual property constraints for the source documents, we required that the documents in DocLayNet adhere to a few conditions. Firstly, we kept scanned documents to a minimum, since they introduce difficulties in annotation (see Section 4). As a second condition, we focussed on medium to large documents ( > 10 pages) with technical content, dense in complex tables, figures, plots and captions. Such documents carry a lot of information value, but are often hard to analyse with high accuracy due to their challenging layouts. Counterexamples of documents not included in the dataset are receipts, invoices, hand-written documents or photographs showing 'text in the wild". Figure 2: Distribution of DocLayNet pages across document categories. <!-- image --> -The pages in DocLayNet can be grouped into six distinct categories, namely Financial Reports , Manuals , Scientific Articles , Laws & Regulations , Patents and Government Tenders . Each document category was sourced from various repositories. For example, Financial format annual reports 2 Reports contain both free-style which expose company-specific, artistic layouts as well as the more formal SEC filings. The two largest categories ( Financial Reports and Manuals ) contain a large amount of free-style layouts in order to obtain maximum variability. In the other four categories, we boosted the variability by mixing documents from independent providers, such as different government websites or publishers. In Figure 2, we show the document categories contained in DocLayNet with their respective sizes. +The pages in DocLayNet can be grouped into six distinct categories, namely Financial Reports , Manuals , Scientific Articles , Laws & Regulations , Patents and Government Tenders . Each document category was sourced from various repositories. For example, Financial Reports contain both free-style format annual reports 2 which expose company-specific, artistic layouts as well as the more formal SEC filings. The two largest categories ( Financial Reports and Manuals ) contain a large amount of free-style layouts in order to obtain maximum variability. In the other four categories, we boosted the variability by mixing documents from independent providers, such as different government websites or publishers. In Figure 2, we show the document categories contained in DocLayNet with their respective sizes. We did not control the document selection with regard to language. The vast majority of documents contained in DocLayNet (close to 95%) are published in English language. However, DocLayNet also contains a number of documents in other languages such as German (2.5%), French (1.0%) and Japanese (1.0%). While the document language has negligible impact on the performance of computer vision methods such as object detection and segmentation models, it might prove challenging for layout analysis methods which exploit textual features. @@ -101,41 +95,41 @@ Table 1 shows the overall frequency and distribution of the labels among the dif In order to accommodate the different types of models currently in use by the community, we provide DocLayNet in an augmented COCO format [16]. This entails the standard COCO ground-truth file (in JSON format) with the associated page images (in PNG format, 1025 × 1025 pixels). Furthermore, custom fields have been added to each COCO record to specify document category, original document filename and page number. In addition, we also provide the original PDF pages, as well as sidecar files containing parsed PDF text and text-cell coordinates (in JSON). All additional files are linked to the primary page images by their matching filenames. -Despite being cost-intense and far less scalable than automation, human annotation has several benefits over automated groundtruth generation. The first and most obvious reason to leverage human annotations is the freedom to annotate any type of document without requiring a programmatic source. For most PDF documents, the original source document is not available. The latter is not a hard constraint with human annotation, but it is for automated methods. A second reason to use human annotations is that the latter usually provide a more natural interpretation of the page layout. The human-interpreted layout can significantly deviate from the programmatic layout used in typesetting. For example, "invisible" tables might be used solely for aligning text paragraphs on columns. Such typesetting tricks might be interpreted by automated methods incorrectly as an actual table, while the human annotation will interpret it correctly as Text or other styles. The same applies to multi-line text elements, when authors decided to space them as "invisible" list elements without bullet symbols. A third reason to gather ground-truth through human annotation is to estimate a "natural" upper bound on the segmentation accuracy. As we will show in Section 4, certain documents featuring complex layouts can have different but equally acceptable layout interpretations. This natural upper bound for segmentation accuracy can be found by annotating the same pages multiple times by different people and evaluating the inter-annotator agreement. Such a baseline consistency evaluation is very useful to define expectations for a good target accuracy in trained deep neural network models and avoid overfitting (see Table 1). On the flip side, achieving high annotation consistency proved to be a key challenge in human annotation, as we outline in Section 4. +Despite being cost-intense and far less scalable than automation, human annotation has several benefits over automated groundtruth generation. The first and most obvious reason to leverage human annotations is the freedom to annotate any type of document without requiring a programmatic source. For most PDF documents, the original source document is not available. The latter is not a hard constraint with human annotation, but it is for automated methods. A second reason to use human annotations is that the latter usually provide a more natural interpretation of the page layout. The human-interpreted layout can significantly deviate from the programmatic layout used in typesetting. For example, 'invisible' tables might be used solely for aligning text paragraphs on columns. Such typesetting tricks might be interpreted by automated methods incorrectly as an actual table, while the human annotation will interpret it correctly as Text or other styles. The same applies to multi-line text elements, when authors decided to space them as 'invisible' list elements without bullet symbols. A third reason to gather ground-truth through human annotation is to estimate a 'natural' upper bound on the segmentation accuracy. As we will show in Section 4, certain documents featuring complex layouts can have different but equally acceptable layout interpretations. This natural upper bound for segmentation accuracy can be found by annotating the same pages multiple times by different people and evaluating the inter-annotator agreement. Such a baseline consistency evaluation is very useful to define expectations for a good target accuracy in trained deep neural network models and avoid overfitting (see Table 1). On the flip side, achieving high annotation consistency proved to be a key challenge in human annotation, as we outline in Section 4. ## 4 ANNOTATION CAMPAIGN The annotation campaign was carried out in four phases. In phase one, we identified and prepared the data sources for annotation. In phase two, we determined the class labels and how annotations should be done on the documents in order to obtain maximum consistency. The latter was guided by a detailed requirement analysis and exhaustive experiments. In phase three, we trained the annotation staff and performed exams for quality assurance. In phase four, -Table 1: DocLayNet dataset overview. Along with the frequency of each class label, we present the relative occurrence (as % of row "Total") in the train, test and validation sets. The inter-annotator agreement is computed as the mAP@0.5-0.95 metric between pairwise annotations from the triple-annotated pages, from which we obtain accuracy ranges. - -| | | % of Total | % of Total | % of Total | triple inter-annotator mAP @ 0.5-0.95 (%) | triple inter-annotator mAP @ 0.5-0.95 (%) | triple inter-annotator mAP @ 0.5-0.95 (%) | triple inter-annotator mAP @ 0.5-0.95 (%) | triple inter-annotator mAP @ 0.5-0.95 (%) | triple inter-annotator mAP @ 0.5-0.95 (%) | triple inter-annotator mAP @ 0.5-0.95 (%) | -|----------------|---------|--------------|--------------|--------------|---------------------------------------------|---------------------------------------------|---------------------------------------------|---------------------------------------------|---------------------------------------------|---------------------------------------------|---------------------------------------------| -| class label | Count | Train | Test | Val | All | Fin | Man | Sci | Law | Pat | Ten | -| Caption | 22524 | 2.04 | 1.77 | 2.32 | 84-89 | 40-61 | 86-92 | 94-99 | 95-99 | 69-78 | n/a | -| Footnote | 6318 | 0.60 | 0.31 | 0.58 | 83-91 | n/a | 100 | 62-88 | 85-94 | n/a | 82-97 | -| Formula | 25027 | 2.25 | 1.90 | 2.96 | 83-85 | n/a | n/a | 84-87 | 86-96 | n/a | n/a | -| List-item | 185660 | 17.19 | 13.34 | 15.82 | 87-88 | 74-83 | 90-92 | 97-97 | 81-85 | 75-88 | 93-95 | -| Page-footer | 70878 | 6.51 | 5.58 | 6.00 | 93-94 | 88-90 | 95-96 | 100 | 92-97 | 100 | 96-98 | -| Page-header | 58022 | 5.10 | 6.70 | 5.06 | 85-89 | 66-76 | 90-94 | 98-100 | 91-92 | 97-99 | 81-86 | -| Picture | 45976 | 4.21 | 2.78 | 5.31 | 69-71 | 56-59 | 82-86 | 69-82 | 80-95 | 66-71 | 59-76 | -| Section-header | 142884 | 12.60 | 15.77 | 12.85 | 83-84 | 76-81 | 90-92 | 94-95 | 87-94 | 69-73 | 78-86 | -| Table | 34733 | 3.20 | 2.27 | 3.60 | 77-81 | 75-80 | 83-86 | 98-99 | 58-80 | 79-84 | 70-85 | -| Text | 510377 | 45.82 | 49.28 | 45.00 | 84-86 | 81-86 | 88-93 | 89-93 | 87-92 | 71-79 | 87-95 | -| Title | 5071 | 0.47 | 0.30 | 0.50 | 60-72 | 24-63 | 50-63 | 94-100 | 82-96 | 68-79 | 24-56 | -| Total | 1107470 | 941123 | 99816 | 66531 | 82-83 | 71-74 | 79-81 | 89-94 | 86-91 | 71-76 | 68-85 | +Table 1: DocLayNet dataset overview. Along with the frequency of each class label, we present the relative occurrence (as % of row 'Total') in the train, test and validation sets. The inter-annotator agreement is computed as the mAP@0.5-0.95 metric between pairwise annotations from the triple-annotated pages, from which we obtain accuracy ranges. + +| | | % of Total | % of Total | % of Total | triple inter-annotator mAP @0.5-0.95 (%) | triple inter-annotator mAP @0.5-0.95 (%) | triple inter-annotator mAP @0.5-0.95 (%) | triple inter-annotator mAP @0.5-0.95 (%) | triple inter-annotator mAP @0.5-0.95 (%) | triple inter-annotator mAP @0.5-0.95 (%) | triple inter-annotator mAP @0.5-0.95 (%) | +|----------------|---------|--------------|--------------|--------------|--------------------------------------------|--------------------------------------------|--------------------------------------------|--------------------------------------------|--------------------------------------------|--------------------------------------------|--------------------------------------------| +| class label | Count | Train | Test | Val | All | Fin | Man | Sci | Law | Pat | Ten | +| Caption | 22524 | 2.04 | 1.77 | 2.32 | 84-89 | 40-61 | 86-92 | 94-99 | 95-99 | 69-78 | n/a | +| Footnote | 6318 | 0.60 | 0.31 | 0.58 | 83-91 | n/a | 100 | 62-88 | 85-94 | n/a | 82-97 | +| Formula | 25027 | 2.25 | 1.90 | 2.96 | 83-85 | n/a | n/a | 84-87 | 86-96 | n/a | n/a | +| List-item | 185660 | 17.19 | 13.34 | 15.82 | 87-88 | 74-83 | 90-92 | 97-97 | 81-85 | 75-88 | 93-95 | +| Page-footer | 70878 | 6.51 | 5.58 | 6.00 | 93-94 | 88-90 | 95-96 | 100 | 92-97 | 100 | 96-98 | +| Page-header | 58022 | 5.10 | 6.70 | 5.06 | 85-89 | 66-76 | 90-94 | 98-100 | 91-92 | 97-99 | 81-86 | +| Picture | 45976 | 4.21 | 2.78 | 5.31 | 69-71 | 56-59 | 82-86 | 69-82 | 80-95 | 66-71 | 59-76 | +| Section-header | 142884 | 12.60 | 15.77 | 12.85 | 83-84 | 76-81 | 90-92 | 94-95 | 87-94 | 69-73 | 78-86 | +| Table | 34733 | 3.20 | 2.27 | 3.60 | 77-81 | 75-80 | 83-86 | 98-99 | 58-80 | 79-84 | 70-85 | +| Text | 510377 | 45.82 | 49.28 | 45.00 | 84-86 | 81-86 | 88-93 | 89-93 | 87-92 | 71-79 | 87-95 | +| Title | 5071 | 0.47 | 0.30 | 0.50 | 60-72 | 24-63 | 50-63 | 94-100 | 82-96 | 68-79 | 24-56 | +| Total | 1107470 | 941123 | 99816 | 66531 | 82-83 | 71-74 | 79-81 | 89-94 | 86-91 | 71-76 | 68-85 | Figure 3: Corpus Conversion Service annotation user interface. The PDF page is shown in the background, with overlaid text-cells (in darker shades). The annotation boxes can be drawn by dragging a rectangle over each segment with the respective label from the palette on the right. <!-- image --> -include publication repositories such as arXiv 3 , government offices, company websites as well as data directory services for financial reports and patents. Scanned documents were excluded wherever possible because they can be rotated or skewed. This would not allow us to perform annotation with rectangular bounding-boxes and therefore complicate the annotation process. +we distributed the annotation workload and performed continuous quality controls. Phase one and two required a small team of experts only. For phases three and four, a group of 40 dedicated annotators were assembled and supervised. -Preparation work included uploading and parsing the sourced PDF documents in the Corpus Conversion Service (CCS) [22], a cloud-native platform which provides a visual annotation interface and allows for dataset inspection and analysis. The annotation interface of CCS is shown in Figure 3. The desired balance of pages between the different document categories was achieved by selective subsampling of pages with certain desired properties. For example, we made sure to include the title page of each document and bias the remaining page selection to those with figures or tables. The latter was achieved by leveraging pre-trained object detection models from PubLayNet, which helped us estimate how many figures and tables a given page contains. +Phase 1: Data selection and preparation. Our inclusion criteria for documents were described in Section 3. A large effort went into ensuring that all documents are free to use. The data sources include publication repositories such as arXiv 3 , government offices, company websites as well as data directory services for financial reports and patents. Scanned documents were excluded wherever possible because they can be rotated or skewed. This would not allow us to perform annotation with rectangular bounding-boxes and therefore complicate the annotation process. -Phase 2: Label selection and guideline. We reviewed the collected documents and identified the most common structural features they exhibit. This was achieved by identifying recurrent layout elements and lead us to the definition of 11 distinct class labels. These 11 class labels are Caption , Footnote , Formula , List-item , Pagefooter , Page-header , Picture , Section-header , Table , Text , and Title . Critical factors that were considered for the choice of these class labels were (1) the overall occurrence of the label, (2) the specificity of the label, (3) recognisability on a single page (i.e. no need for context from previous or next page) and (4) overall coverage of the page. Specificity ensures that the choice of label is not ambiguous, while coverage ensures that all meaningful items on a page can be annotated. We refrained from class labels that are very specific to a document category, such as Abstract in the Scientific Articles category. We also avoided class labels that are tightly linked to the semantics of the text. Labels such as Author and Affiliation , as seen in DocBank, are often only distinguishable by discriminating on we distributed the annotation workload and performed continuous quality controls. Phase one and two required a small team of experts only. For phases three and four, a group of 40 dedicated annotators were assembled and supervised. +Preparation work included uploading and parsing the sourced PDF documents in the Corpus Conversion Service (CCS) [22], a cloud-native platform which provides a visual annotation interface and allows for dataset inspection and analysis. The annotation interface of CCS is shown in Figure 3. The desired balance of pages between the different document categories was achieved by selective subsampling of pages with certain desired properties. For example, we made sure to include the title page of each document and bias the remaining page selection to those with figures or tables. The latter was achieved by leveraging pre-trained object detection models from PubLayNet, which helped us estimate how many figures and tables a given page contains. -Phase 1: Data selection and preparation. Our inclusion criteria for documents were described in Section 3. A large effort went into ensuring that all documents are free to use. The data sources +Phase 2: Label selection and guideline. We reviewed the collected documents and identified the most common structural features they exhibit. This was achieved by identifying recurrent layout elements and lead us to the definition of 11 distinct class labels. These 11 class labels are Caption , Footnote , Formula , List-item , Pagefooter , Page-header , Picture , Section-header , Table , Text , and Title . Critical factors that were considered for the choice of these class labels were (1) the overall occurrence of the label, (2) the specificity of the label, (3) recognisability on a single page (i.e. no need for context from previous or next page) and (4) overall coverage of the page. Specificity ensures that the choice of label is not ambiguous, while coverage ensures that all meaningful items on a page can be annotated. We refrained from class labels that are very specific to a document category, such as Abstract in the Scientific Articles category. We also avoided class labels that are tightly linked to the semantics of the text. Labels such as Author and Affiliation , as seen in DocBank, are often only distinguishable by discriminating on 3 https://arxiv.org/ @@ -215,11 +209,11 @@ Table 3: Performance of a Mask R-CNN R50 network in mAP@0.5-0.95 scores trained ## Learning Curve -One of the fundamental questions related to any dataset is if it is "large enough". To answer this question for DocLayNet, we performed a data ablation study in which we evaluated a Mask R-CNN model trained on increasing fractions of the DocLayNet dataset. As can be seen in Figure 5, the mAP score rises sharply in the beginning and eventually levels out. To estimate the error-bar on the metrics, we ran the training five times on the entire data-set. This resulted in a 1% error-bar, depicted by the shaded area in Figure 5. In the inset of Figure 5, we show the exact same data-points, but with a logarithmic scale on the x-axis. As is expected, the mAP score increases linearly as a function of the data-size in the inset. The curve ultimately flattens out between the 80% and 100% mark, with the 80% mark falling within the error-bars of the 100% mark. This provides a good indication that the model would not improve significantly by yet increasing the data size. Rather, it would probably benefit more from improved data consistency (as discussed in Section 3), data augmentation methods [23], or the addition of more document categories and styles. +One of the fundamental questions related to any dataset is if it is 'large enough'. To answer this question for DocLayNet, we performed a data ablation study in which we evaluated a Mask R-CNN model trained on increasing fractions of the DocLayNet dataset. As can be seen in Figure 5, the mAP score rises sharply in the beginning and eventually levels out. To estimate the error-bar on the metrics, we ran the training five times on the entire data-set. This resulted in a 1% error-bar, depicted by the shaded area in Figure 5. In the inset of Figure 5, we show the exact same data-points, but with a logarithmic scale on the x-axis. As is expected, the mAP score increases linearly as a function of the data-size in the inset. The curve ultimately flattens out between the 80% and 100% mark, with the 80% mark falling within the error-bars of the 100% mark. This provides a good indication that the model would not improve significantly by yet increasing the data size. Rather, it would probably benefit more from improved data consistency (as discussed in Section 3), data augmentation methods [23], or the addition of more document categories and styles. ## Impact of Class Labels -The choice and number of labels can have a significant effect on the overall model performance. Since PubLayNet, DocBank and DocLayNet all have different label sets, it is of particular interest to understand and quantify this influence of the label set on the model performance. We investigate this by either down-mapping labels into more common ones (e.g. Caption → Text ) or excluding them from the annotations entirely. Furthermore, it must be stressed that all mappings and exclusions were performed on the data before model training. In Table 3, we present the mAP scores for a Mask R-CNN R50 network on different label sets. Where a label is down-mapped, we show its corresponding label, otherwise it was excluded. We present three different label sets, with 6, 5 and 4 different labels respectively. The set of 5 labels contains the same labels as PubLayNet. However, due to the different definition of +The choice and number of labels can have a significant effect on the overall model performance. Since PubLayNet, DocBank and DocLayNet all have different label sets, it is of particular interest to understand and quantify this influence of the label set on the model performance. We investigate this by either down-mapping labels into more common ones (e.g. Caption → Text ) or excluding them from the annotations entirely. Furthermore, it must be stressed that all mappings and exclusions were performed on the data before model training. In Table 3, we present the mAP scores for a Mask R-CNN R50 network on different label sets. Where a label is down-mapped, we show its corresponding label, otherwise it was excluded. We present three different label sets, with 6, 5 and 4 different labels respectively. The set of 5 labels contains the same labels as PubLayNet. However, due to the different definition of lists in PubLayNet (grouped list-items) versus DocLayNet (separate list-items), the label set of size 4 is the closest to PubLayNet, in the assumption that the List is down-mapped to Text in PubLayNet. The results in Table 3 show that the prediction accuracy on the remaining class labels does not change significantly when other classes are merged into them. The overall macro-average improves by around 5%, in particular when Page-footer and Page-header are excluded. Table 4: Performance of a Mask R-CNN R50 network with document-wise and page-wise split for different label sets. Naive page-wise split will result in ~ 10% point improvement. @@ -239,8 +233,6 @@ Table 4: Performance of a Mask R-CNN R50 network with document-wise and page-wis | Title | 77 | 81 | | | | All | 72 | 84 | 78 | 87 | -lists in PubLayNet (grouped list-items) versus DocLayNet (separate list-items), the label set of size 4 is the closest to PubLayNet, in the assumption that the List is down-mapped to Text in PubLayNet. The results in Table 3 show that the prediction accuracy on the remaining class labels does not change significantly when other classes are merged into them. The overall macro-average improves by around 5%, in particular when Page-footer and Page-header are excluded. - ## Impact of Document Split in Train and Test Set Many documents in DocLayNet have a unique styling. In order to avoid overfitting on a particular style, we have split the train-, test- and validation-sets of DocLayNet on document boundaries, i.e. every document contributes pages to only one set. To the best of our knowledge, this was not considered in PubLayNet or DocBank. To quantify how this affects model performance, we trained and evaluated a Mask R-CNN R50 model on a modified dataset version. Here, the train-, test- and validation-sets were obtained by a randomised draw over the individual pages. As can be seen in Table 4, the difference in model performance is surprisingly large: pagewise splitting gains ˜ 10% in mAP over the document-wise splitting. Thus, random page-wise splitting of DocLayNet can easily lead to accidental overestimation of model performance and should be avoided. @@ -306,7 +298,7 @@ Figure 6: Example layout predictions on selected pages from the DocLayNet test-s <!-- image --> -Text Caption List-Item Formula Table Picture Section-Header Page-Header Page-Footer Title +Text Caption List-Item Formula Table Section-Header Picture Page-Header Page-Footer Title Diaconu, Mai Thanh Minh, Marc, albinxavi, fatih, oleg, and wanghao yang. ultralytics/yolov5: v6.0 - yolov5n nano models, roboflow integration, tensorflow export, opencv dnn support, October 2021. @@ -314,7 +306,7 @@ Diaconu, Mai Thanh Minh, Marc, albinxavi, fatih, oleg, and wanghao yang. ultraly - [15] Mingxing Tan, Ruoming Pang, and Quoc V. Le. Efficientdet: Scalable and efficient object detection. CoRR , abs/1911.09070, 2019. - [16] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context, 2014. - [17] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2, 2019. -- [18] Nikolaos Livathinos, Cesar Berrospi, Maksym Lysak, Viktor Kuropiatnyk, Ahmed Nassar, Andre Carvalho, Michele Dolfi, Christoph Auer, Kasper Dinkla, and Peter W. J. Staar. Robust pdf document conversion using recurrent neural networks. In , AAAI, pages 15137- Proceedings of the 35th Conference on Artificial Intelligence 15145, feb 2021. +- [18] Nikolaos Livathinos, Cesar Berrospi, Maksym Lysak, Viktor Kuropiatnyk, Ahmed Nassar, Andre Carvalho, Michele Dolfi, Christoph Auer, Kasper Dinkla, and Peter W. J. Staar. Robust pdf document conversion using recurrent neural networks. In Proceedings of the 35th Conference on Artificial Intelligence , AAAI, pages 1513715145, feb 2021. - [19] Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD, pages 1192-1200, New York, USA, 2020. Association for Computing Machinery. - [20] Shoubin Li, Xuyan Ma, Shuaiqun Pan, Jun Hu, Lin Shi, and Qing Wang. Vtlayout: Fusion of visual and text features for document layout analysis, 2021. - [21] Peng Zhang, Can Li, Liang Qiao, Zhanzhan Cheng, Shiliang Pu, Yi Niu, and Fei Wu. Vsr: A unified framework for document layout analysis combining vision, semantics and relations, 2021. diff --git a/tests/snapshots/pdf/sources/2305.03393v1.pdf.md b/tests/snapshots/pdf/sources/2305.03393v1.pdf.md index 5e62d5e2..2eb735f7 100644 --- a/tests/snapshots/pdf/sources/2305.03393v1.pdf.md +++ b/tests/snapshots/pdf/sources/2305.03393v1.pdf.md @@ -1,6 +1,6 @@ ## Optimized Table Tokenization for Table Structure Recognition -[0000 - 0002 - 3723 - 6960] [0000 - 0002 - 9468 - 0822] Maksym Lysak , Ahmed Nassar , Nikolaos Livathinos [0000 - 0001 - 8513 - 3491] , Christoph Auer [0000 - 0001 - 5761 - 0422] , and Peter Staar [0000 - 0002 - 8088 - 0823] +Maksym Lysak [0000 - 0002 - 3723 - 6960] , Ahmed Nassar [0000 - 0002 - 9468 - 0822] , Nikolaos Livathinos [0000 - 0001 - 8513 - 3491] , Christoph Auer [0000 - 0001 - 5761 - 0422] , and Peter Staar [0000 - 0002 - 8088 - 0823] IBM Research {mly,ahn,nli,cau,taa}@zurich.ibm.com @@ -162,11 +162,12 @@ Secondly, OTSL has more inherent structure and a significantly restricted vocabu ## References 1. Auer, C., Dolfi, M., Carvalho, A., Ramis, C.B., Staar, P.W.J.: Delivering document conversion as a cloud service with high throughput and responsiveness. CoRR abs/2206.00785 (2022). https://doi.org/10.48550/arXiv.2206.00785 , https://doi.org/10.48550/arXiv.2206.00785 -2. Chen, B., Peng, D., Zhang, J., Ren, Y., Jin, L.: Complex table structure recognition in the wild using transformer and identity matrix-based augmentation. In: Porwal, U., Fornés, A., Shafait, F. (eds.) Frontiers in Handwriting Recognition. pp. 545- 561. Springer International Publishing, Cham (2022) +2. Chen, B., Peng, D., Zhang, J., Ren, Y., Jin, L.: Complex table structure recognition in the wild using transformer and identity matrix-based augmentation. In: Porwal, U., Fornés, A., Shafait, F. (eds.) Frontiers in Handwriting Recognition. pp. 545561. Springer International Publishing, Cham (2022) 3. Chi, Z., Huang, H., Xu, H.D., Yu, H., Yin, W., Mao, X.L.: Complicated table structure recognition. arXiv preprint arXiv:1908.04729 (2019) 4. Deng, Y., Rosenberg, D., Mann, G.: Challenges in end-to-end neural scientific table recognition. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 894-901. IEEE (2019) + 5. Kayal, P., Anand, M., Desai, H., Singh, M.: Tables to latex: structure and content extraction from scientific tables. International Journal on Document Analysis and Recognition (IJDAR) pp. 1-10 (2022) -6. Lee, E., Kwon, J., Yang, H., Park, J., Lee, S., Koo, H.I., Cho, N.I.: Table structure recognition based on grid shape graph. In: 2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). pp. 1868- 1873. IEEE (2022) +6. Lee, E., Kwon, J., Yang, H., Park, J., Lee, S., Koo, H.I., Cho, N.I.: Table structure recognition based on grid shape graph. In: 2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). pp. 18681873. IEEE (2022) 7. Li, M., Cui, L., Huang, S., Wei, F., Zhou, M., Li, Z.: Tablebank: A benchmark dataset for table detection and recognition (2019) 8. Livathinos, N., Berrospi, C., Lysak, M., Kuropiatnyk, V., Nassar, A., Carvalho, A., Dolfi, M., Auer, C., Dinkla, K., Staar, P.: Robust pdf document conversion using recurrent neural networks. Proceedings of the AAAI Conference on Artificial Intelligence 35 (17), 15137-15145 (May 2021), https://ojs.aaai.org/index.php/ AAAI/article/view/17777 9. Nassar, A., Livathinos, N., Lysak, M., Staar, P.: Tableformer: Table structure understanding with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4614-4623 (June 2022) @@ -178,8 +179,9 @@ Secondly, OTSL has more inherent structure and a significantly restricted vocabu 15. Staar, P.W.J., Dolfi, M., Auer, C., Bekas, C.: Corpus conversion service: A machine learning platform to ingest documents at scale. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 774-782. KDD '18, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3219819.3219834 , https://doi.org/10. 1145/3219819.3219834 16. Wang, X.: Tabular Abstraction, Editing, and Formatting. Ph.D. thesis, CAN (1996), aAINN09397 17. Xue, W., Li, Q., Tao, D.: Res2tim: Reconstruct syntactic structures from table images. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 749-755. IEEE (2019) + 18. Xue, W., Yu, B., Wang, W., Tao, D., Li, Q.: Tgrnet: A table graph reconstruction network for table structure recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1295-1304 (2021) -19. Ye, J., Qi, X., He, Y., Chen, Y., Gu, D., Gao, P., Xiao, R.: Pingan-vcgroup's solution for icdar 2021 competition on scientific literature parsing task b: Table recognition to html (2021). , https://doi.org/10.48550/ARXIV.2105.01848 https://arxiv.org/abs/2105.01848 +19. Ye, J., Qi, X., He, Y., Chen, Y., Gu, D., Gao, P., Xiao, R.: Pingan-vcgroup's solution for icdar 2021 competition on scientific literature parsing task b: Table recognition to html (2021). https://doi.org/10.48550/ARXIV.2105.01848 , https://arxiv.org/abs/2105.01848 20. Zhang, Z., Zhang, J., Du, J., Wang, F.: Split, embed and merge: An accurate table structure recognizer. Pattern Recognition 126 , 108565 (2022) 21. Zheng, X., Burdick, D., Popa, L., Zhong, X., Wang, N.X.R.: Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context. In: 2021 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 697-706 (2021). https://doi.org/10.1109/WACV48630.2021. 00074 22. Zhong, X., ShafieiBavani, E., Jimeno Yepes, A.: Image-based table recognition: Data, model, and evaluation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) Computer Vision - ECCV 2020. pp. 564-580. Springer International Publishing, Cham (2020) diff --git a/tests/snapshots/pdf/sources/normal_4pages.pdf.md b/tests/snapshots/pdf/sources/normal_4pages.pdf.md index 9c28887b..8fc7bfcb 100644 --- a/tests/snapshots/pdf/sources/normal_4pages.pdf.md +++ b/tests/snapshots/pdf/sources/normal_4pages.pdf.md @@ -4,33 +4,31 @@ 발행일 2020년 4월 2일 -발행일 2020년 4월 2일 발행처 국회입법조사처 발행인 김하중 국회입법조사처장 www.nars.go.kr - 발행처 국회입법조사처 발행인 김하중 국회입법조사처장 www.nars.go.kr -<!-- image --> +발행일 2020년 4월 2일 발행처 국회입법조사처 발행인 김하중 국회입법조사처장 www.nars.go.kr + +## 코로나-19 관련 보험약관상 재해보험금 지급문제 및 개선과제 제1695호 ## 이슈와 논점 -## 코로나-19 관련 보험약관상 재해보험금 지급문제 및 개선과제 - 김 창 호* 2020.1.1. 「감염병예방법」 관련 규정이 변경되어 적용됨에 따라, 세계적 유행(Pandemic) 단계에 돌 입하였다고 세계보건기구(WHO)가 인정한 코로나-19와 관련하여, 보험회사에서 그동안 판매했거나 향후 판매하게 될 보험상품의 입원 및 사망의 재해 인정 여부에 대하여 보험실무상 혼선이 초래되어 이 에 대하여 보험약관의 현황 및 문제점을 살펴보고 그에 따른 개선방안을 제시하고자 하였다. -## 들어가며 - 1 +## 들어가며 + 2020.3.30. 0시 기준 현재 '코로나바이러스감염 증-19(COVID-19, 이하 '코로나-19')'의 국내 확 진자는 9,661명, 사망자는 158명으로 나타났다. 이와 관련하여 세계보건기구(WHO)는 지난 2020. 3.11. 코로나-19가 세계적 유행(Pandemic) 단계 에 돌입하였음을 선언하였다. -한편 코로나-19가 전 세계적 유행단계에 돌입한 와중에 국내 보험업계에서는 코로나-19를 과연 질 1) 2) 병으로 보아야 할지, 상해 나 재해 로 보아야 할지 +한편 코로나-19가 전 세계적 유행단계에 돌입한 와중에 국내 보험업계에서는 코로나-19를 과연 질 병으로 보아야 할지, 상해 1) 나 재해 2) 로 보아야 할지 1) 손해보험의 표준약관 규정에 따르면, 보험기간 중에 발생한 급격하고도 우연한 외래의 사고로 신체에 입은 상해라고 규정하고 있는데, 즉, '급 격성, 우연성, 외래성'을 충족하는 사고를 상해로 정의함 @@ -52,7 +50,7 @@ <!-- image --> -## 2 코로나-19 관련 보험 현황 +## 코로나-19 관련 보험 현황 2 ## (1) 「감염병의 예방 및 관리에 관한 법률」 개정 @@ -85,7 +83,7 @@ 현행 생명보험 표준약관상 재해분류표 5) 는 위 「감염병예방법」 제2조 제2호의 제1급 감염병들을 질병임에도 불구하고 보장대상이 되는 '재해'로 규 정하고 있다. -그러나 제1급 감염병에 포함되는 질병임에도 불 구하고, 생명보험 표준약관 재해분류표의 한국표 6) 준질병·사인분류 (이하 'KCD'라 함)상 U코드 (U00~U99)에 해당하는 질병들(SARS-U04.9, MERS-U19.9 등)은 보장제외 대상으로 분류되어 보험금을 지급하지 않는 재해로 규정되어 있다. +그러나 제1급 감염병에 포함되는 질병임에도 불 구하고, 생명보험 표준약관 재해분류표의 한국표 준질병·사인분류 6) (이하 'KCD'라 함)상 U코드 (U00~U99)에 해당하는 질병들(SARS-U04.9, MERS-U19.9 등)은 보장제외 대상으로 분류되어 보험금을 지급하지 않는 재해로 규정되어 있다. 코로나-19 역시 KCD 수록 정식 명칭은 '코로나 바이러스 질환 2019'로 질병분류기호는 'U07.1' 로 표시하고 있어서 보험금이 지급되지 않는 재해 로 분류되어 재해보험금 지급대상에 포함되지 않는 다고 해석될 수 있다. @@ -105,7 +103,7 @@ 코로나-19와 관련하여 생명보험 업계는 보험상 품은 측정가능한 위험(Measurable Risk)을 보장 하는 금융상품으로 코로나-19와 같은 신종질환의 경우 과거 감염데이터 기반의 위험률을 산출할 수 없으므로, 원칙적으로 보장대상이 아니라는 의견 을 제시하고 있다. -## 3 문제점 +## 문제점 3 ## (1) 상위법에 반하는 보험약관의 해석 @@ -123,7 +121,7 @@ ## (3) 보험사의 보험금 지급실무상 혼선 초래 -보험업계는 보험은 사고 위험의 예측가능성과, 8) 보험의 기본원리인 대수의 법칙 이나 수지상등의 원칙 9) 에 부합하고, 최대 손실을 보험회사가 감당할 수 있어야 한다고 주장한다. +보험업계는 보험은 사고 위험의 예측가능성과, 보험의 기본원리인 대수의 법칙 8) 이나 수지상등의 원칙 9) 에 부합하고, 최대 손실을 보험회사가 감당할 수 있어야 한다고 주장한다. 특히, 신종감염병증후군과 신종인플루엔자는 특 정 질병이 아닌 앞으로 새롭게 발생할 모든 신종감 염병을 포괄하는 개념으로 위험률 측정이 불가능하 고, 담보 범위도 확정할 수도 없다고 주장한다. @@ -139,7 +137,7 @@ 이슈와 논점 -## 4 개선과제 +## 개선과제 4 ## (1) 「약관규제법」에 따른 보험금 지급 검토 필요 @@ -163,7 +161,7 @@ 따라서 감염병 보험, 파라메트릭(Parametric Insurance)보험 10) , 인덱스(Index)보험 등과 같이 실제 발생한 손실액이 아니라 특정 지표에 의해 보 험금이 지급되는 신종보험상품을 개발 보급 판매할 필요가 점진적으로 증가하고 있다. -## 5 맺으며 +## 맺으며 5 지금까지 이 글에서 코로나-19와 관련하여 생명 보험 표준약관상 재해보험금 지급의 문제점 및 개 선과제에 대하여 살펴보았다. diff --git a/tests/snapshots/pdf/sources/redp5110_sampled.pdf.md b/tests/snapshots/pdf/sources/redp5110_sampled.pdf.md index 54357c24..dc6082ca 100644 --- a/tests/snapshots/pdf/sources/redp5110_sampled.pdf.md +++ b/tests/snapshots/pdf/sources/redp5110_sampled.pdf.md @@ -1,28 +1,62 @@ -<!-- image --> - Front cover -## Row and Column Access Control Support in IBM DB2 for i - -Implement roles and separation of duties - <!-- image --> -Leverage row permissions on the database - -Protect columns by defining column masks ibm.com /redbooks +## Row and Column Access Control Support in IBM DB2 for i -Jim Bainbridge Hernando Bedoya Rob Bestgen Mike Cain Dan Cruikshank Jim Denton Doug Mack Tom McKinley Kent Milligan +Implement roles and separation of duties ibm.com /redbooks Redpaper -Redpaper +<!-- image --> ## Contents -DB2 for i Center of Excellence +| Notices | . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii | +|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------| +| Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | viii | +| DB2 for i Center of Excellence | . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix | +| Preface | . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi | +| Authors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi | | +| Now you can become a published author, too! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | xiii | +| Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | xiii | +| Stay connected to IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | xiv | +| Chapter 1. Securing and protecting IBM DB2 data | . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 | +| 1.1 Security fundamentals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 2 | +| 1.2 Current state of IBM i security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 2 | +| 1.3 DB2 for i security controls | . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 | +| 1.3.1 Existing row and column control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 4 | +| 1.3.2 New controls: Row and Column Access Control. . . . . . . . . . . . . . . . . . . . . . . . . . . | 5 | +| Chapter 2. Roles and separation of duties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 7 | +| 2.1 Roles | . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 | +| 2.1.1 DDM and DRDA application server access: QIBM_DB_DDMDRDA . . . . . . . . . . . | 8 | +| 2.1.2 Toolbox application server access: QIBM_DB_ZDA. . . . . . . . . . . . . . . . . . . . . . . . | 8 | +| 2.1.3 Database Administrator function: QIBM_DB_SQLADM . . . . . . . . . . . . . . . . . . . . . | 9 | +| 2.1.4 Database Information function: QIBM_DB_SYSMON | . . . . . . . . . . . . . . . . . . . . . . 9 | +| 2.1.5 Security Administrator function: QIBM_DB_SECADM . . . . . . . . . . . . . . . . . . . . . . | 9 | +| 2.1.6 Change Function Usage CL command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 10 | +| 2.1.7 Verifying function usage IDs for RCAC with the FUNCTION_USAGE view | . . . . . 10 | +| 2.2 Separation of duties | . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 | +| Chapter 3. Row and Column Access Control | . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 | +| 3.1 Explanation of RCAC and the concept of access control . . . . . . . . . . . . . . . . . . . . . . . | 14 | +| 3.1.1 Row permission and column mask definitions 3.1.2 Enabling and activating RCAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 | +| 3.2 Special registers and built-in global variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 16 18 | +| 3.2.1 Special registers | . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 | +| 3.2.2 Built-in global variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 19 | +| 3.3 VERIFY_GROUP_FOR_USER function. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 20 | +| 3.4 Establishing and controlling accessibility by using the RCAC rule text. . . . . . . . . . . . . | 21 | +| 3.5 SELECT, INSERT, and UPDATE behavior with RCAC . . . . . . . . . . . . . . . . . . . . . . . . | 22 | +| . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 22 | +| 3.6 Human resources example 3.6.1 Assigning the QIBM_DB_SECADM function ID to the consultants. . . . . . . . . . . . | 23 | +| 3.6.2 Creating group profiles for the users and their roles. . . . . . . . . . . . . . . . . . . . . . . | 23 | +| 3.6.3 Demonstrating data access without RCAC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 24 | +| 3.6.4 Defining and creating row permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 25 | +| | . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 | +| 3.6.5 Defining and creating column masks | 28 | +| 3.6.6 Activating RCAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 3.6.7 Demonstrating data access with RCAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 | +| 3.6.8 Demonstrating data access with a view and RCAC . . . . . . . . . . . . . . . . . . . . . . . | 32 | -<!-- image --> +DB2 for i Center of Excellence -IBM Systems Lab Services and Training Solution Brief +Solution Brief IBM Systems Lab Services and Training <!-- image --> @@ -69,33 +103,33 @@ This IBM® Redpaper publication provides information about the IBM i 7.2 feature This paper is intended for database engineers, data-centric application developers, and security officers who want to design and implement RCAC as a part of their data control and governance policy. A solid background in IBM i object level security, DB2 for i relational database concepts, and SQL is assumed. -## Authors - This paper was produced by the IBM DB2 for i Center of Excellence team in partnership with the International Technical Support Organization (ITSO), Rochester, Minnesota US. <!-- image --> -Jim Bainbridge is a senior DB2 consultant on the DB2 for i Center of Excellence team in the IBM Lab Services and Training organization. His primary role is training and implementation services for IBM DB2 Web Query for i and business analytics. Jim began his career with IBM 30 years ago in the IBM Rochester Development Lab, where he developed cooperative processing products that paired IBM PCs with IBM S/36 and AS/.400 systems. In the years since, Jim has held numerous technical roles, including independent software vendors technical support on a broad range of IBM technologies and products, and supporting customers in the IBM Executive Briefing Center and IBM Project Office. - <!-- image --> +Jim Bainbridge is a senior DB2 consultant on the DB2 for i Center of Excellence team in the IBM Lab Services and Training organization. His primary role is training and implementation services for IBM DB2 Web Query for i and business analytics. Jim began his career with IBM 30 years ago in the IBM Rochester Development Lab, where he developed cooperative processing products that paired IBM PCs with IBM S/36 and AS/.400 systems. In the years since, Jim has held numerous technical roles, including independent software vendors technical support on a broad range of IBM technologies and products, and supporting customers in the IBM Executive Briefing Center and IBM Project Office. + Hernando Bedoya is a Senior IT Specialist at STG Lab Services and Training in Rochester, Minnesota. He writes extensively and teaches IBM classes worldwide in all areas of DB2 for i. Before joining STG Lab Services, he worked in the ITSO for nine years writing multiple IBM Redbooks® publications. He also worked for IBM Colombia as an IBM AS/400® IT Specialist doing presales support for the Andean countries. He has 28 years of experience in the computing field and has taught database classes in Colombian universities. He holds a Master's degree in Computer Science from EAFIT, Colombia. His areas of expertise are database technology, performance, and data warehousing. Hernando can be contacted at hbedoya@us.ibm.com . +## Authors + <!-- image --> +Chapter 1. + ## 1 ## Securing and protecting IBM DB2 data -Chapter 1. - -Recent news headlines are filled with reports of data breaches and cyber-attacks impacting 1 reports that almost 5000 global businesses of all sizes. The Identity Theft Resource Center data breaches have occurred since 2005, exposing over 600 million records of data. The 2 financial cost of these data breaches is skyrocketing. Studies from the Ponemon Institute revealed that the average cost of a data breach increased in 2013 by 15% globally and resulted in a brand equity loss of $9.4 million per attack. The average cost that is incurred for each lost record containing sensitive information increased more than 9% to $145 per record. +Recent news headlines are filled with reports of data breaches and cyber-attacks impacting global businesses of all sizes. The Identity Theft Resource Center 1 reports that almost 5000 data breaches have occurred since 2005, exposing over 600 million records of data. The financial cost of these data breaches is skyrocketing. Studies from the Ponemon Institute 2 revealed that the average cost of a data breach increased in 2013 by 15% globally and resulted in a brand equity loss of $9.4 million per attack. The average cost that is incurred for each lost record containing sensitive information increased more than 9% to $145 per record. Businesses must make a serious effort to secure their data and recognize that securing information assets is a cost of doing business. In many parts of the world and in many industries, securing the data is required by law and subject to audits. Data security is no longer an option; it is a requirement. This chapter describes how you can secure and protect data in DB2 for i. The following topics are covered in this chapter: -- Security fundamentals /SM590000 +- /SM590000 Security fundamentals - /SM590000 Current state of IBM i security - /SM590000 DB2 for i security controls @@ -107,7 +141,7 @@ This chapter describes how you can secure and protect data in DB2 for i. The fol Before reviewing database security techniques, there are two fundamental steps in securing information assets that must be described: -- First, and most important, is the definition of a company's security policy . Without a /SM590000 security policy, there is no definition of what are acceptable practices for using, accessing, and storing information by who, what, when, where, and how. A security policy should minimally address three things: confidentiality, integrity, and availability. +- /SM590000 First, and most important, is the definition of a company's security policy . Without a security policy, there is no definition of what are acceptable practices for using, accessing, and storing information by who, what, when, where, and how. A security policy should minimally address three things: confidentiality, integrity, and availability. The monitoring and assessment of adherence to the security policy determines whether your security strategy is working. Often, IBM security consultants are asked to perform security assessments for companies without regard to the security policy. Although these assessments can be useful for observing how the system is defined and how data is being accessed, they cannot determine the level of security without a security policy. Without a security policy, it really is not an assessment as much as it is a baseline for monitoring the changes in the security settings that are captured. @@ -168,10 +202,9 @@ To discover who has authorization to define and manage RCAC, you can use the que Example 2-1 Query to determine who has authority to define and manage RCAC -| | SELECT function_id, user_name, usage, user_type | -|----|-----------------------------------------------------------------------------------------| -| | FROM function_usage WHERE function_id=’QIBM_DB_SECADM’ ORDER BY user_name; | -| | | +| SELECT function_id, user_name, usage, user_type | +|-----------------------------------------------------------------------------------------| +| FROM function_usage WHERE function_id=’QIBM_DB_SECADM’ ORDER BY user_name; | ## 2.2 Separation of duties @@ -191,20 +224,20 @@ Table 2-2 shows a comparison of the different function usage IDs and *JOBCTL aut Table 2-2 Comparison of the different function usage IDs and *JOBCTL authority -| User action | | | | | | -|-----------------------------------------------------------------------------|----|----|----|----|----| -| SET CURRENT DEGREE (SQL statement) | X | | X | | | -| CHGQRYA command targeting a different user's job | X | | X | | | -| STRDBMON or ENDDBMON commands targeting a different user's job | X | | X | | | -| STRDBMON or ENDDBMON commands targeting a job that matches the current user | X | | X | X | X | -| QUSRJOBI() API format 900 or System i Navigator's SQL Details for Job | X | | X | X | | -| Visual Explain within Run SQL scripts | X | | X | X | X | -| Visual Explain outside of Run SQL scripts | X | | X | | | -| ANALYZE PLAN CACHE procedure | X | | X | | | -| DUMP PLAN CACHE procedure | X | | X | | | -| MODIFY PLAN CACHE procedure | X | | X | | | -| MODIFY PLAN CACHE PROPERTIES procedure (currently does not check authority) | X | | X | | | -| CHANGE PLAN CACHE SIZE procedure (currently does not check authority) | X | | X | | | +| User action | | | | | +|-----------------------------------------------------------------------------|----|----|----|----| +| SET CURRENT DEGREE (SQL statement) | X | X | | | +| CHGQRYA command targeting a different user's job | X | X | | | +| STRDBMON or ENDDBMON commands targeting a different user's job | X | X | | | +| STRDBMON or ENDDBMON commands targeting a job that matches the current user | X | X | X | X | +| QUSRJOBI() API format 900 or System i Navigator's SQL Details for Job | X | X | X | | +| Visual Explain within Run SQL scripts | X | X | X | X | +| Visual Explain outside of Run SQL scripts | X | X | | | +| ANALYZE PLAN CACHE procedure | X | X | | | +| DUMP PLAN CACHE procedure | X | X | | | +| MODIFY PLAN CACHE procedure | X | X | | | +| MODIFY PLAN CACHE PROPERTIES procedure (currently does not check authority) | X | X | | | +| CHANGE PLAN CACHE SIZE procedure (currently does not check authority) | X | X | | | The SQL CREATE PERMISSION statement that is shown in Figure 3-1 is used to define and initially enable or disable the row access rules. @@ -276,8 +309,6 @@ VERIFY\_GROUP\_FOR\_USER (CURRENT\_USER, 'MGR') VERIFY\_GROUP\_FOR\_USER (CURREN The following function invocation returns a value of 0: -The following function invocation returns a value of 0: VERIFY\_GROUP\_FOR\_USER (CURRENT\_USER, 'JUDY', 'TONY') - VERIFY\_GROUP\_FOR\_USER (CURRENT\_USER, 'JUDY', 'TONY') ``` @@ -363,21 +394,19 @@ Figure 4-69 Index advice with no RCAC WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'ADMIN') = 1 THEN C. CUSTOMER_LOGIN_ID WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'CUSTOMER') = 1 THEN C. CUSTOMER_LOGIN_ID ELSE '*****' END ENABLE; CREATE MASK BANK_SCHEMA.MASK_SECURITY_QUESTION_ON_CUSTOMERS ON BANK_SCHEMA.CUSTOMERS AS C WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'ADMIN') = 1 THEN C. CUSTOMER_SECURITY_QUESTION WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'CUSTOMER') = 1 THEN C. CUSTOMER_SECURITY_QUESTION ELSE '*****' END ENABLE; CREATE MASK BANK_SCHEMA.MASK_SECURITY_QUESTION_ANSWER_ON_CUSTOMERS ON BANK_SCHEMA.CUSTOMERS AS C WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'ADMIN') = 1 THEN C. CUSTOMER_SECURITY_QUESTION_ANSWER WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'CUSTOMER') = 1 THEN C. CUSTOMER_SECURITY_QUESTION_ANSWER ELSE '*****' END ENABLE; ALTER TABLE BANK_SCHEMA.CUSTOMERS FOR COLUMN CUSTOMER_SECURITY_QUESTION RETURN CASE FOR COLUMN CUSTOMER_SECURITY_QUESTION_ANSWER RETURN CASE ACTIVATE ROW ACCESS CONTROL ACTIVATE COLUMN ACCESS CONTROL; ``` -<!-- image --> - Back cover ## Row and Column Access Control Support in IBM DB2 for i Implement roles and separation of duties -This IBM Redpaper publication provides information about the IBM i 7.2 feature of IBM DB2 for i Row and Column Access Control (RCAC). It offers a broad description of the function and advantages of controlling access to data in a comprehensive and transparent way. This publication helps you understand the capabilities of RCAC and provides examples of defining, creating, and implementing the row permissions and column masks in a relational database environment. - Leverage row permissions on the database +Protect columns by defining column masks This IBM Redpaper publication provides information about the IBM i 7.2 feature of IBM DB2 for i Row and Column Access Control (RCAC). It offers a broad description of the function and advantages of controlling access to data in a comprehensive and transparent way. This publication helps you understand the capabilities of RCAC and provides examples of defining, creating, and implementing the row permissions and column masks in a relational database environment. + This paper is intended for database engineers, data-centric application developers, and security officers who want to design and implement RCAC as a part of their data control and governance policy. A solid background in IBM i object level security, DB2 for i relational database concepts, and SQL is assumed. -Protect columns by defining column masks +<!-- image --> <!-- image --> diff --git a/tests/snapshots/pdf/sources/right_to_left_03.pdf.md b/tests/snapshots/pdf/sources/right_to_left_03.pdf.md index 86f296bb..fb06010c 100644 --- a/tests/snapshots/pdf/sources/right_to_left_03.pdf.md +++ b/tests/snapshots/pdf/sources/right_to_left_03.pdf.md @@ -1,12 +1,24 @@ ## اميدنامه پذيرش در بازار اصلی - کالای داخلی +شمشه و شمشال توليد شده به روش ريخته گری پيوسته مورد مصرف در فولادهای سازه ای - مطابق آناليز پيوست + +سازمان ملی استاندارد ايران + <!-- image --> <!-- image --> -## شرکت بورس کالای ايران +## 2 - 5 - استاندارد کالا + +نام استاندارد + +شماره استاندارد ملی + +استاندارد اجباری است؟ + +مرجع صادرکننده استاندارد -## 5 - 2 - استاندارد کالا +آيا توليدکننده محصول، استاندارد مذکور را اخذ نموده است؟ ## -3 پذيرش در بورس diff --git a/tests/snapshots/pdf/sources/table_mislabeled_as_picture.pdf.md b/tests/snapshots/pdf/sources/table_mislabeled_as_picture.pdf.md index 6e709245..c0c2986c 100644 --- a/tests/snapshots/pdf/sources/table_mislabeled_as_picture.pdf.md +++ b/tests/snapshots/pdf/sources/table_mislabeled_as_picture.pdf.md @@ -1,12 +1,3 @@ -| | They work in parallel to State funded private practitioners who take assignments to represent people eligible for legal aid | -|----|--------------------------------------------------------------------------------------------------------------------------------| -| | They coordinate appointments of private practitioners (ex officio, or panel appoint- ments) to legal aid cases | -| | They supervise, coach or mentor private practitioners who take legal aid cases | -| | They conduct or organize training sessions for staff lawyers/paralegals | -| | They conduct or organize training sessions for all providers of legal aid, including both staff and private lawyers/paralegals | -| | Other (Please specify) _____________________________________________________ | -| | Not applicable, there is no institutional legal aid provider | - - „ They work in parallel to State funded private practitioners who take assignments to represent people eligible for legal aid - „ They coordinate appointments of private practitioners (ex officio, or panel appointments) to legal aid cases - „ They supervise, coach or mentor private practitioners who take legal aid cases @@ -17,34 +8,29 @@ 23. If your country has an institutional legal aid provider (e.g. public defender), what is the maximum caseload per lawyer at one time? -| 23. | If your country has an institutional legal aid provider (e.g. public defender), what is the maximum caseload per lawyer at one time? _______ at the national (federal) level | -|-------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| - - \_\_\_\_\_\_\_ at the national (federal) level - \_\_\_\_\_\_\_ at the regional (district) level - \_\_\_\_\_\_\_ at the local (municipal) level -- „ ‐There is no such limitation +- „ -There is no such limitation 24. If your country has an institutional legal aid provider (e.g. public defender), do the staff lawyers coordinate to uniformly challenge common violations of national and international due process rights and human rights? - „ Yes, at the national (federal) level - „ Yes, at regional (district) level - „ Yes, at the local (municipal) level -- „ No -| 25. | If your country has an institutional legal aid provider (e.g. public defender), does it have specialized providers and/or units for representing child victims, child witness- es or suspected and accused children? | -|-------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| „ | Yes, at the national (federal) level | -| „ | Yes, at regional (district) level | -| „ | Yes, at the local (municipal) level | -| „ | No | +„ No 25. If your country has an institutional legal aid provider (e.g. public defender), does it have specialized providers and/or units for representing child victims, child witnesses or suspected and accused children? - „ Yes, at the national (federal) level -- Yes, at regional (district) level + +„ Yes, at regional (district) level + - „ Yes, at the local (municipal) level +„ No + 26. If your country allows legal aid services through university-based student law clinics, are there national guidelines on how students are supervised in providing legal aid services? (Please select all that apply) - „ Yes, there are specific guidelines for non-lawyers providing legal aid services @@ -53,23 +39,21 @@ - „ Don't know - „ There are no university-based student law clinics -| 27. | If your country allows legal aid services through university-based student law clinics, what type of legal aid services is a student authorized to undertake? (Please select all that apply) | -|-------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| „ | There is no limitation; they have the same authority as lawyers | -| „ | They can represent people in administrative or civil law hearings | -| „ | They can provide primary legal aid (legal advice) | -| „ | They can prepare legal documents | -| „ | They can represent people in court in civil and criminal matters | -| „ | They have the same authority as lawyers in criminal cases of low to mid gravity | -| „ | They can provide a full range of legal services in criminal cases regardless of gravity | -| „ | They can conduct mediation | -| „ | They are authorized to provide only those services that a faculty member or practic- ing lawyer supervises | -| „ | Don’t know | -| „ | Other (Please specify) _____________________________________________________ | - -27. If your country allows legal aid services through university-based student law clinics, what type of legal aid services is a student authorized to undertake? (Please select all that apply) - -- specialized legal aid services 28. Are provided focusing on specific disadvantaged population groups? If yes, please indicate to whom these services are provided, and whether they are provided by State-funded legal aid, civil society organizations, or both. +| If your country allows legal aid services through university-based student law clinics, what type of legal aid services is a student authorized to undertake? (Please select all that apply) | +|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| „ There is no limitation; they have the same authority as lawyers | +| „ They can represent people in administrative or civil law hearings | +| „ They can provide primary legal aid (legal advice) | +| „ They can prepare legal documents | +| „ They can represent people in court in civil and criminal matters | +| „ They have the same authority as lawyers in criminal cases of low to mid gravity | +| „ They can provide a full range of legal services in criminal cases regardless of gravity | +| „ They can conduct mediation | +| „ They are authorized to provide only those services that a faculty member or practic- ing lawyer supervises | +| „ Don’t know | +| „ Other (Please specify) _____________________________________________________ | + +28. Are specialized legal aid services provided focusing on specific disadvantaged population groups? If yes, please indicate to whom these services are provided, and whether they are provided by State-funded legal aid, civil society organizations, or both. (Please select all that apply) diff --git a/tests/snapshots/scanned/sources/nemotron_multipage.pdf.md b/tests/snapshots/scanned/sources/nemotron_multipage.pdf.md index 4ccde8ad..a15eb35a 100644 --- a/tests/snapshots/scanned/sources/nemotron_multipage.pdf.md +++ b/tests/snapshots/scanned/sources/nemotron_multipage.pdf.md @@ -1,3 +1,3 @@ Docling bundles PDF document conversion to JSON and Markdown in an easy self contained package aerosaeoe e o a -H W ep 9 ps 1s e. uu P1 po te od nn na sm 88 ek da sd bd p0 M e 00 a K C a p +H W ep 9 ps 1s e. uu P1 po od te nn na sm 88 sd ek da M bd p0 diff --git a/tests/snapshots/scanned/sources/ocr_test_rotated_90.pdf.md b/tests/snapshots/scanned/sources/ocr_test_rotated_90.pdf.md index a8bf7add..221a6a2a 100644 --- a/tests/snapshots/scanned/sources/ocr_test_rotated_90.pdf.md +++ b/tests/snapshots/scanned/sources/ocr_test_rotated_90.pdf.md @@ -1 +1 @@ -te od nn na sm 88 ek da sd bd p0 M e 00 a K C a p +od te nn na sm 88 sd ek da M bd p0 diff --git a/tests/snapshots/scanned/sources/old_newspaper.png.md b/tests/snapshots/scanned/sources/old_newspaper.png.md index bb3003db..ec2c39d7 100644 --- a/tests/snapshots/scanned/sources/old_newspaper.png.md +++ b/tests/snapshots/scanned/sources/old_newspaper.png.md @@ -1,97 +1,73 @@ ## French Institute Creates A French Community At OU -ology.A demonstration class of first year students of French at the high school and junior high schoollevel is to be conducted by Pierre Simonian, the Institute's teacher in charge of demonstrationandmethodology. - -strengthen the teacher's knowledge and control of the langlage, - AnNDEAFrench Institute opened its doors on OU's campuslastweekwith the arrival of its 48participants,eightexperienced professors of French, and four graduate assistants'and natives of France. -Lastweek marked the beginning of an intensive program in the French language,methods of teaching the language,and the culture of France.Faculty and Institute participants meet together instudysessionsallmorning and afternoon,Classes are conducted entirely in French, with the exception of courses in language analysis and method- - -The participants eatlunchand dinner together and speak,it is expected,nothing but French. After dinner,participants share extracurricular activitiestogether--films,music,lectures.Often the arranged program of activi- - Theparticipants in the Institute(14menand34women,10 of them nuns)are teachers of French in junior and senior high schoolsinMichiganand14other states.Director of the Institute Is Ou's assistant professor of French,Don Iodice.Also from OU's staff are Mme Genevieve Prevost, assistant director of the Institute,-and M. Charles Forton,in charge of language Improvement. Familiar to OU's campus as well are M. et Mme Francis Tafoya,who will be workingwithcontemporary French-civilization andlanguage improvementrespectively.M. Tafoya was formerly the head of OU's department of foreign languages. -<!-- image --> - -## MITZELFELD'S - -ROCHESTER - The object of the Institute is twofold.The iirst objective is to bring the teachers up to date In theirsubject area,Because of rapid changes in content,approach,.and teaching techniques, trueofsuchfieldsasmathematics andscience as well asforeign languages,the National Defense EducationAct has established summer institutes all over the country to help teachers keep pace with advances within their own fields.The second goal of theFrench institute will be to <!-- image --> -## HILLS THEATRE - -Rochester - -Friday -Tuesday ONE SHOWING NIGHTLY 7:30 SUNDAY2:30&7:30 Programlnformation651-8311 - -## FIRSTTIME AT POPULARPRICES +OF ROCHESTER 743N.MAIN -<!-- image --> +Try our -<!-- image --> +FIESTAS -IR LaDY +SPLITS -Daiey +Hours Hours -OF ROCHESTER 743N.MAIN +11A.M.to11P.M 11A.M.to11P.M -<!-- image --> +11A.M.to11P.M Hours Hours 11A.M.to11P.M strengthen the teacher's knowledge and control of the langlage, -Try our +Lastweek marked the beginning of an intensive program in the French language,methods of teaching the language,and the culture of France.Faculty and Institute participants meet together instudysessionsallmorning and afternoon,Classes are conducted entirely in French, with the exception of courses in language analysis and method- ology.A demonstration class of first year students of French at the high school and junior high schoollevel is to be conducted by Pierre Simonian, the Institute's teacher in charge of demonstrationandmethodology. -FIESTAS +The participants eatlunchand dinner together and speak,it is expected,nothing but French. After dinner,participants share extracurricular activitiestogether--films,music,lectures.Often the arranged program of activi- -Winner of 8Academy Awards +## MITZELFELD'S -## AUDREY HEPBURN·REXHARRISONSTANLEYHOLLOWAY +ROCHESTER -SPLITS +<!-- image --> -Hours Hours 11A.M.to11P.M 11A.M.to11P.M +<!-- image --> -Hours Hours +<!-- image --> -TECHNICOLOR SUPER PANAVISION 7OFROMWARNER BRO +GETDUNLOPIMPORTQUALITY INTHEAMERICAN MADE GOLD SEAL FULL 4PLY (NOT2PLY)CONSTRUCTION NO THUMP WITHTYREX CORD NYLONALSOAVAILABLE） Certified Safe At A SUSTAINED 100 M.P.H. Wholesale Prices to O.U.Students & Faculty on Passenger Car, Sports Car, Radial Ply & Racing Tires -11A.M.to11P.M 11A.M.to11P.M +Bill Basinger DUNLOP -GETDUNLOPIMPORTQUALITY INTHEAMERICAN MADE GOLD SEAL FULL 4PLY (NOT2PLY)CONSTRUCTION NO THUMP WITHTYREX CORD NYLONALSOAVAILABLE） Certified Safe At A SUSTAINED 100 M.P.H. Wholesale Prices to O.U.Students & Faculty on Passenger Car, Sports Car, Radial Ply & Racing Tires +<!-- image --> BruceRobertson Tom Hill -Bill Basinger - ## R.B. DUNLOP TIRE SALES Phone:651-3422 or 673-9227after5p.m.call 334-6452 -## Arnold Rexall Pharmacy - ties cohtinuesuntil sevenoreight in the evening.When they return tothe dormitory(allparticipants are housed in AnibalHousethis summer),they areencouragedto continue speaking nothingbut French.They are bound together in a linguistic and cultural island, andtheirveryisolationwithinthe French language and their tightnessasa community is meant to Increase theirfacility with the language -Prescriptions Cosmetics SundryItems Liquor,Beer,Wine +## CLASSIFIED ADS -2026OpdykeRd. Corner of Pontiac Road 333-7033 +Wanted:Man or womanfullor part time.Direct Sales Prestige Products.Call Mrs.Lodge,6825540or335-9937forappt -## CLASSIFIED ADS +## Arnold Rexall Pharmacy -Wanted:Man or womanfullor part time.Direct Sales Prestige Products.Call Mrs.Lodge,682- 5540or335-9937forappt +Prescriptions Cosmetics SundryItems Liquor,Beer,Wine -## TUKO +2026OpdykeRd. Corner of Pontiac Road 333-7033 + +<!-- image --> 872E.Auburn,NearJohnR.Rochester UL2-5363 <!-- image --> -Treat* the* eptire family +$ to the happiest entertaipment ofall! - <!-- image --> ## HAV FULL HOUSE? @@ -100,18 +76,10 @@ Then use our storage service for all your clothes. It includes complete protecti ## P.S.FREE... -<!-- image --> - s the wider your choice.Bring your Springcleaning in nov -<!-- image --> - -<!-- image --> - ## M.G.M. Cleaners, Inc. In Business for 2l Years Auburn Rd..at Adams Crooks Rd..at Auburn Mound Rd..at 23 Mile Rd. Also on Campus at Oakland Unirersity B Plants and Stores Serving Oakland and Macomb Counties -DUNLOP - OpenytoMMu My huSat