PRD: Extract a shared document-layout seam for chunkers

[henryle97]

Problem Statement

As an xrag maintainer, the chunkers currently repeat the same document-layout knowledge in multiple places. Heading depth, page-number extraction, whitespace normalization, table handling, image handling, neighbor-text collection, caption consumption, and RAGFlow compatibility aliases appear in parallel across chunker implementations. To understand one chunker strategy, I often have to separate strategy-specific behavior from layout behavior that is duplicated elsewhere.

This makes the chunker layer shallower than it appears. The real complexity of turning normalized elements into chunkable layout units is not hidden behind one seam; it is spread across multiple chunker implementations. That weakens locality, increases drift risk, and makes tests re-prove the same layout behavior through multiple chunkers.

Solution

Introduce a shared document-layout module that converts normalized parser elements into a stable layout-unit seam used by chunkers. From the user's perspective, chunking should still produce the current families of Chunks, including text, table, section, and image behavior, but each chunker should focus on grouping strategy rather than re-implementing layout interpretation.

The shared module should own transformation from normalized elements into layout units such as headings, text runs, tables, images, captions, and neighboring context. Chunkers should consume that seam and specialize only in how they group or split those units into Chunks.

User Stories

1. As a maintainer, I want document-layout interpretation to live in one module, so that chunkers can focus on strategy instead of duplicating layout rules.
2. As a maintainer, I want heading depth logic shared across chunkers, so that section behavior does not drift by strategy.
3. As a maintainer, I want page-number extraction shared, so that chunk provenance stays consistent.
4. As a maintainer, I want whitespace normalization shared at the layout seam, so that chunkers do not each maintain their own text-cleanup behavior.
5. As a maintainer, I want table element handling shared, so that table units carry the same source meaning across chunkers.
6. As a maintainer, I want image handling shared, so that Image Chunk behavior stays aligned with ADR-0004 regardless of chunker strategy.
7. As a maintainer, I want caption consumption rules shared, so that FigureCaption behavior does not diverge across chunkers.
8. As a maintainer, I want neighboring context collection shared where it is a layout concern, so that pre-text and post-text semantics stay consistent.
9. As a maintainer, I want chunkers to consume stable layout units, so that the difference between section_table, section_token, and title_hierarchy is mostly grouping policy.
10. As a maintainer, I want RAGFlow compatibility alias rules centralized where appropriate, so that compatibility behavior is not copied across chunkers.
11. As a test author, I want layout behavior tested once at the layout seam, so that chunker tests can focus on strategy-specific outcomes.
12. As a maintainer, I want adding a new chunker strategy to reuse document-layout behavior, so that new chunkers do not start from duplicated parsing logic.
13. As a maintainer, I want text, table, and image provenance rules to be consistent, so that downstream retrieval and evaluation logic see predictable Chunk structure.
14. As a maintainer, I want layout handling for images and captions to use the project's domain language, so that Image Description and Image Chunk behavior are obvious in the architecture.
15. As a maintainer, I want parent section behavior to derive from shared layout understanding, so that section families are comparable across chunkers.
16. As a maintainer, I want common layout logic to pass the deletion test, so that removing it would clearly reintroduce complexity across multiple chunkers.
17. As a future contributor, I want to understand what is strategy-specific and what is layout-specific, so that chunker work is easier to navigate.
18. As a maintainer, I want chunker option behavior to remain stable while implementation depth improves, so that current workflows do not break.
19. As a maintainer, I want chunker tests to survive internal refactors, so that layout extraction does not create fragile assertions.
20. As a maintainer, I want the code to make it obvious which behaviors belong to document layout and which belong to chunk grouping, so that future deepening work has a clear seam to build on.

Implementation Decisions

• Build a shared document-layout module as the primary seam between normalized parser elements and chunker strategies.
• Define stable layout units that represent the meaning chunkers currently rediscover independently: headings, text runs, tables, images, captions, and layout context.
• Keep chunker-specific modules focused on grouping, splitting, parent-child shaping, and strategy-specific size rules.
• Preserve existing user-visible chunker behavior unless a deliberate follow-up change is proposed separately.
• Preserve ADR-0004 image responsibilities by keeping image and caption interpretation explicit in the layout seam.
• Centralize common provenance extraction such as page numbers, source element handling, and heading-path inputs.
• Centralize shared compatibility behavior, including any cross-strategy alias rules that currently exist only because they were copied.
• Keep option interpretation split appropriately: layout options should belong to the layout seam, while chunk-size and grouping options should remain strategy concerns.
• Avoid extracting thin helpers with no real ownership. The new layout module should own complexity that would otherwise be duplicated across multiple chunkers.
• Preserve the ability for chunkers to produce different Chunk families and different splitting behavior; the goal is shared interpretation, not forced identical output.

Testing Decisions

• Good tests should verify layout behavior through the shared layout seam and chunker behavior through public chunker interfaces, not through private helper calls.
• The shared layout module should have behavioral tests for heading interpretation, text aggregation inputs, table extraction, image-plus-caption handling, and neighboring context collection.
• Each chunker should keep contract-style tests proving its strategy-specific output shape while relying on the shared layout seam underneath.
• Regression tests should confirm that current chunk outputs remain compatible where that compatibility is intentional, especially for table and image behavior.
• Tests should verify observable chunk behavior, provenance fields, and parent-child relationships rather than internal traversal order.
• Prior art already exists in the codebase: chunk prepare tests, title hierarchy chunker tests, retrieval and dedup tests, and image-related preprocess tests provide the style and coverage patterns to preserve.

Out of Scope

• Changing retrieval or ranking algorithms.
• Redefining Chunk V2.
• Replacing existing chunker strategies with a single unified strategy.
• Reworking parser preprocess behavior outside what is necessary to support the layout seam.
• Re-litigating ADR-0004.
• Introducing new chunk types beyond the currently supported text, table, section, and image behavior.

Further Notes

• This PRD is about architectural depth in the chunker layer, not about changing benchmark targets directly.
• The main success criterion is improved locality: changes to shared layout interpretation should no longer require synchronized edits across multiple chunkers.
• The preferred outcome is a codebase where chunker differences are obvious, intentional strategy choices rather than accidental duplication of document-layout logic.