Skip to content

Layout-aware PDF/OCR conversion: reading order, columns, running headers — scrambled markdown is unfixable downstream #168

Description

@fazpu

From the owner's report (2026-07-28): a PDF with a complicated template converted to markdown with sections intermixed — the document flow was scrambled in the output. Once conversion linearizes a layout in the wrong reading order, no downstream logic can recover: the D79 structure stage (parser or LLM fallback) can only draw a tree over the stream it is given, and chunks inherit the scrambled text. The new skeleton sanity check (PR #167) catches "the tree is junk" and demotes to the LLM route, but explicitly does NOT fix intra-section text order — that is this issue.

Current state: the shipped conversion route is essentially identity-preserving markdown passthrough; real PDF/OCR handling has never been hardened (D38).

Scope:

  • Evaluate layout-aware converters (reading-order models, column detection, running header/footer stripping, table handling) — e.g. docling / marker-class tooling — against the D38 contract (blocks + source map + heading syntax when the source has structure).
  • Acceptance scenario: a complicated-template PDF (owner may supply the original offender) converts to markdown whose section flow matches human reading order, and whose headings survive as heading syntax.
  • Converter choice/version becomes conversion provenance (D12), like every other component.

Relation: D79 assumes conversion "preserves or emits heading syntax when the source has structure" — this issue is what makes that clause true for hostile documents. Independent of #165 (structure stage) and not a blocker for it; born-digital sources are unaffected.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions