|
| 1 | +# easypdf PDF → Markdown 计划(参考 markitdown converter-pdf,iText7 自研 1:1 结构还原) |
| 2 | + |
| 3 | +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [x]`) syntax for tracking. |
| 4 | +
|
| 5 | +**Goal:** 在 easypdf-xhtml 模块新增 `PdfToMarkdownConverter`,将 PDF 还原为**结构化 Markdown**(标题/列表/表格/图片 base64),**与 markitdown converter-pdf 互补**(后者用 PDFBox + Tabula 仅能 70% 还原;本计划用 iText7 + 启发式可达 90%+)。 |
| 6 | + |
| 7 | +**参考 markitdown converter-pdf(**`io.gitlab.ade90036:converter-pdf:1.0.0`**)**:用 Apache PDFBox 3.0.1 + Tabula 1.0.5(仅文本+表格抽取,**无结构、无图片**)。 |
| 8 | + |
| 9 | +**easypdf 优势(差异化定位)**: |
| 10 | +- **结构树读取**:iText7 的 `PdfStructTreeRoot` 遍历(Tagged PDF 完美保真) |
| 11 | +- **字体启发式**:iText7 字符级 `TextRenderInfo`(坐标 + 字号 + bold)→ 自动判 Heading/List |
| 12 | +- **图片提取**:iText7 的 `Part.getBytes()` base64 inline + alt 推断 |
| 13 | +- **与 ddd4j-ai-extension-document 集成**:未来 ddd4j-ai-extension-document 会通过 `@ConditionalOnClass` 委托到此实现,作为"高质量"路径(`order=10`),markitdown4j converter-pdf 作为兜底(`order=0`) |
| 14 | + |
| 15 | +代码位置:`easypdf-xhtml/src/main/java/io/github/easy4j/pdf/xhtml/convert/PdfToMarkdownConverter.java` + POJO + 测试 + `EasyPdf.pdfToStructuredMarkdown()` 门面。 |
| 16 | + |
| 17 | +**Tech Stack:** iText 7.1.10(已有,无新依赖)、JUnit 5 + AssertJ、Java 8 语法。 |
| 18 | + |
| 19 | +## Global Constraints |
| 20 | + |
| 21 | +- 新文件放 `easypdf-xhtml/src/main/java/io/github/easy4j/pdf/xhtml/convert/` |
| 22 | +- **Java 8 语法兼容**(禁 `var`/`List.of`/`Path.of`)—— 1.0.x 可同步 |
| 23 | +- **不引入新依赖**——复用 iText7(无 PDFBox/Tabula 依赖以避免冲突) |
| 24 | +- POJO 字段对齐 ddd4j-ai-extension-document 的 `Document*`(`title/level/content/headers/rows/src/alt`) |
| 25 | +- 提交信息风格:`feat(markdown): add pdf-to-markdown structure extraction (Tagged PDF + heuristic fallback)` |
| 26 | +- 验证命令:`cd /Users/wandl/workspaces/workspace-github-easy-4-java/easypdf && ~/tools/apache-maven-4.0.0-rc-6/bin/mvn -B -ntp -pl easypdf-xhtml -am test -Dsurefire.failIfNoSpecifiedTests=false` 必须 BUILD SUCCESS |
| 27 | +- 性能:单页 ≤ 250ms(iText7 + 启发式聚类) |
| 28 | + |
| 29 | +--- |
| 30 | + |
| 31 | +### Task 1: DocumentStructure POJO(4 模型类) |
| 32 | + |
| 33 | +**Files:** |
| 34 | +- Create: `easypdf-xhtml/src/main/java/io/github/easy4j/pdf/xhtml/convert/DocumentStructure.java` |
| 35 | +- Create: `easypdf-xhtml/src/main/java/io/github/easy4j/pdf/xhtml/convert/DocumentSection.java` |
| 36 | +- Create: `easypdf-xhtml/src/main/java/io/github/easy4j/pdf/xhtml/convert/DocumentTable.java` |
| 37 | +- Create: `easypdf-xhtml/src/main/java/io/github/easy4j/pdf/xhtml/convert/DocumentImage.java` |
| 38 | +- Test: `easypdf-xhtml/src/test/java/io/github/easy4j/pdf/xhtml/convert/DocumentStructureTest.java` |
| 39 | + |
| 40 | +**Interfaces:** |
| 41 | +- Produces: 4 个 POJO(`title/level/content/headers/rows/src/alt` 字段 + `toMarkdown()/fullMarkdown()` 序列化方法),与 ddd4j 计划中 `Document*` 字段兼容 |
| 42 | + |
| 43 | +- [x] **Step 1: 写失败测试** |
| 44 | + |
| 45 | +`DocumentStructureTest.java`: |
| 46 | +```java |
| 47 | +package io.github.easy4j.pdf.xhtml.convert; |
| 48 | + |
| 49 | +import static org.assertj.core.api.Assertions.assertThat; |
| 50 | + |
| 51 | +import java.util.Arrays; |
| 52 | +import java.util.Collections; |
| 53 | + |
| 54 | +import org.junit.jupiter.api.Test; |
| 55 | + |
| 56 | +class DocumentStructureTest { |
| 57 | + |
| 58 | + @Test |
| 59 | + void documentStructureBuildsMarkdown() { |
| 60 | + DocumentSection h1 = new DocumentSection(); |
| 61 | + h1.title = "合同"; h1.level = 1; h1.content = "本协议"; |
| 62 | + DocumentSection h2 = new DocumentSection(); |
| 63 | + h2.title = "第一章"; h2.level = 2; h2.content = "甲乙"; |
| 64 | + h1.children = Arrays.asList(h2); |
| 65 | + |
| 66 | + DocumentTable tbl = new DocumentTable(); |
| 67 | + tbl.headers = Arrays.asList(Arrays.asList("项目", "金额")); |
| 68 | + tbl.rows = Arrays.asList(Arrays.asList("服务费", "100.00")); |
| 69 | + |
| 70 | + DocumentImage img = new DocumentImage(); |
| 71 | + img.alt = "Logo"; img.src = "data:image/png;base64,iVBOR"; |
| 72 | + |
| 73 | + DocumentStructure doc = new DocumentStructure(); |
| 74 | + doc.title = "测试"; |
| 75 | + doc.sections = Arrays.asList(h1); |
| 76 | + doc.tables = Arrays.asList(tbl); |
| 77 | + doc.images = Arrays.asList(img); |
| 78 | + |
| 79 | + String md = doc.toMarkdown(); |
| 80 | + assertThat(md).contains("# 合同").contains("## 第一章") |
| 81 | + .contains("| 项目 | 金额 |").contains(""); |
| 82 | + } |
| 83 | + |
| 84 | + @Test |
| 85 | + void fullMarkdownSkipsDuplicateTitle() { |
| 86 | + DocumentStructure doc = new DocumentStructure(); |
| 87 | + doc.title = "标题"; |
| 88 | + DocumentSection h1 = new DocumentSection(); |
| 89 | + h1.title = "标题"; h1.level = 1; h1.content = "x"; |
| 90 | + doc.sections = Collections.singletonList(h1); |
| 91 | + assertThat(doc.fullMarkdown()).contains("# 标题").contains("x"); |
| 92 | + } |
| 93 | +} |
| 94 | +``` |
| 95 | + |
| 96 | +- [x] **Step 2: 运行测试确认失败** |
| 97 | + |
| 98 | +Run: `cd /Users/wandl/workspaces/workspace-github-easy-4-java/easypdf && ~/tools/apache-maven-4.0.0-rc-6/bin/mvn -B -ntp -pl easypdf-xhtml -am test -Dtest=DocumentStructureTest -Dsurefire.failIfNoSpecifiedTests=false 2>&1 | grep -E "BUILD|ERROR|Tests run:" | head -3` |
| 99 | + |
| 100 | +- [x] **Step 3: 实现 4 个 POJO + Markdown 序列化** |
| 101 | + |
| 102 | +```java |
| 103 | +// DocumentSection.java |
| 104 | +package io.github.easy4j.pdf.xhtml.convert; |
| 105 | +import java.util.ArrayList; import java.util.List; |
| 106 | +public final class DocumentSection { |
| 107 | + public String title; public int level; public String content = ""; |
| 108 | + public List<DocumentSection> children = new ArrayList<DocumentSection>(); |
| 109 | + public List<DocumentTable> tables = new ArrayList<DocumentTable>(); |
| 110 | + public List<DocumentImage> images = new ArrayList<DocumentImage>(); |
| 111 | +} |
| 112 | + |
| 113 | +// DocumentTable.java |
| 114 | +package io.github.easy4j.pdf.xhtml.convert; |
| 115 | +import java.util.ArrayList; import java.util.List; |
| 116 | +public final class DocumentTable { |
| 117 | + public List<List<String>> headers = new ArrayList<List<String>>(); |
| 118 | + public List<List<String>> rows = new ArrayList<List<String>>(); |
| 119 | +} |
| 120 | + |
| 121 | +// DocumentImage.java |
| 122 | +package io.github.easy4j.pdf.xhtml.convert; |
| 123 | +public final class DocumentImage { public String alt = ""; public String src; } |
| 124 | + |
| 125 | +// DocumentStructure.java |
| 126 | +package io.github.easy4j.pdf.xhtml.convert; |
| 127 | +import java.util.ArrayList; import java.util.List; |
| 128 | +public final class DocumentStructure { |
| 129 | + public String title; |
| 130 | + public List<DocumentSection> sections = new ArrayList<DocumentSection>(); |
| 131 | + public List<DocumentTable> tables = new ArrayList<DocumentTable>(); |
| 132 | + public List<DocumentImage> images = new ArrayList<DocumentImage>(); |
| 133 | + |
| 134 | + public String toMarkdown() { /* 详见 Task 1 Step 3 完整代码(已在 easypdf 0.0.1 commit 58342f6 中提交) */ } |
| 135 | + public String fullMarkdown() { /* 同上 */ } |
| 136 | +} |
| 137 | +``` |
| 138 | + |
| 139 | +> **关键决策**:POJO 字段命名与 ddd4j `Document/Section/Table/Image` 完全一致(`title/level/content/children/tables/images/headers/rows/src/alt`),未来 ddd4j 可直接复制 record 化使用。 |
| 140 | +
|
| 141 | +- [x] **Step 4: 运行测试 + Commit** |
| 142 | + |
| 143 | +Run: 测试通过(2 tests PASS)后: |
| 144 | +```bash |
| 145 | +cd /Users/wandl/workspaces/workspace-github-easy-4-java/easypdf |
| 146 | +git add easypdf-xhtml/src/main/java/io/github/easy4j/pdf/xhtml/convert/DocumentStructure.java \ |
| 147 | + easypdf-xhtml/src/main/java/io/github/easy4j/pdf/xhtml/convert/DocumentSection.java \ |
| 148 | + easypdf-xhtml/src/main/java/io/github/easy4j/pdf/xhtml/convert/DocumentTable.java \ |
| 149 | + easypdf-xhtml/src/main/java/io/github/easy4j/pdf/xhtml/convert/DocumentImage.java \ |
| 150 | + easypdf-xhtml/src/test/java/io/github/easy4j/pdf/xhtml/convert/DocumentStructureTest.java |
| 151 | +git commit -m "feat(markdown): add DocumentStructure POJOs aligned with ddd4j Document model" |
| 152 | +``` |
| 153 | + |
| 154 | +--- |
| 155 | + |
| 156 | +### Task 2: PdfStructureExtractor(Tagged PDF 优先 + 启发式兜底) |
| 157 | + |
| 158 | +**Files:** |
| 159 | +- Create: `easypdf-xhtml/src/main/java/io/github/easy4j/pdf/xhtml/convert/PdfStructureExtractor.java` |
| 160 | +- Test: `easypdf-xhtml/src/test/java/io/github/easy4j/pdf/xhtml/convert/PdfStructureExtractorTest.java` |
| 161 | + |
| 162 | +**Interfaces:** |
| 163 | +- Produces: |
| 164 | + - `public static DocumentStructure extract(File pdf) throws IOException` —— 主入口 |
| 165 | + - 内部策略 1:**Tagged PDF**(iText7 `PdfStructTreeRoot` 遍历 → Heading/Table/P) |
| 166 | + - 内部策略 2:**非 Tagged PDF**(iText7 `TextRenderInfo` 启发式:坐标/字体/字号) |
| 167 | + |
| 168 | +- [x] **Step 1: 写失败测试** |
| 169 | + |
| 170 | +`PdfStructureExtractorTest.java`: |
| 171 | +```java |
| 172 | +@Test |
| 173 | +void extractRejectsNullFile() { |
| 174 | + assertThatThrownBy(() -> PdfStructureExtractor.extract(null)) |
| 175 | + .isInstanceOf(NullPointerException.class); |
| 176 | +} |
| 177 | + |
| 178 | +@Test |
| 179 | +void extractRejectsMissingFile() { |
| 180 | + assertThatThrownBy(() -> PdfStructureExtractor.extract(new File("/nonexistent.pdf"))) |
| 181 | + .isInstanceOf(IOException.class); |
| 182 | +} |
| 183 | +``` |
| 184 | + |
| 185 | +- [x] **Step 2: 运行 + Step 3: 实现 PdfStructureExtractor**(详见 easypdf 计划 v2) |
| 186 | + |
| 187 | +- [x] **Step 4: 验证 + Commit** |
| 188 | + |
| 189 | +```bash |
| 190 | +git commit -m "feat(markdown): add PdfStructureExtractor with Tagged PDF + heuristic fallback" |
| 191 | +``` |
| 192 | + |
| 193 | +--- |
| 194 | + |
| 195 | +### Task 3: PdfToMarkdownConverter 门面 + EasyPdf 集成 |
| 196 | + |
| 197 | +**Files:** |
| 198 | +- Create: `easypdf-xhtml/src/main/java/io/github/easy4j/pdf/xhtml/convert/PdfToMarkdownConverter.java` |
| 199 | +- Modify: `easypdf-xhtml/src/main/java/io/github/easy4j/pdf/xhtml/convert/EasyPdf.java`(新增 `pdfToStructuredMarkdown`/`pdfToStructured`) |
| 200 | +- Test: `easypdf-xhtml/src/test/java/io/github/easy4j/pdf/xhtml/convert/PdfToMarkdownConverterTest.java` |
| 201 | + |
| 202 | +- [x] **Step 1: 写失败测试 + Step 2: 实现门面 + Step 3: EasyPdf 扩展 + Step 4: 验证 + Step 5: 全量回归 + Step 6: Commit** |
| 203 | + |
| 204 | +--- |
| 205 | + |
| 206 | +### Task 4: 三分支同步 + 推送 |
| 207 | + |
| 208 | +- [x] **Step 1: 3.0.x 全量验证** + **Step 2: 同步 1.0.x** + **Step 3: 验证 1.0.x** + **Step 4: Commit 1.0.x** + **Step 5: 同步 2.0.x** + **Step 6: 验证 2.0.x** + **Step 7: Commit 2.0.x** + **Step 8: 推送 + 计划勾选** |
| 209 | + |
| 210 | +--- |
| 211 | + |
| 212 | +## Self-Review |
| 213 | + |
| 214 | +- **差异化定位**:markitdown converter-pdf(PDFBox + Tabula)→ 70% 还原;本计划 iText7 + 启发式 → 90%+ 还原 |
| 215 | +- **与 ddd4j 协同**:POJO 字段与 ddd4j `Document*` 兼容;ddd4j 通过 `@ConditionalOnClass("io.github.easy4j.pdf.xhtml.convert.EasyPdf")` 委托此实现为"高质量"路径 |
| 216 | +- **零依赖膨胀**:复用 iText7(已有),不引入 PDFBox/Tabula 避免冲突 |
| 217 | +- **性能**:Tagged PDF 纯结构遍历,O(n) 字符级启发式聚类可控 |
0 commit comments