Stream .docx extraction

README.md

@@ -57,7 +57,7 @@ const result = await extractAttachment({
 // result.truncated === false
 ```
 
-The heavy parsers (`unpdf`, `mammoth`, `exceljs`, …) are lazy-loaded per handler, so importing
+The heavy parsers (`unpdf`, `exceljs`, `saxes`, …) are lazy-loaded per handler, so importing
 `extractAttachment` costs nothing until you actually call it on a matching attachment. It's also
 available on its own subpath — `import { extractAttachment } from 'agentextract/attachment'` — if
 you want to reach it without touching the body-extraction entry point.
@@ -90,15 +90,41 @@ guards reduce blast radius; they are **not** a sandbox.
   or ZIP64 metadata is treated as over-budget (fail-closed), not trusted.
 - **Output** — extracted text is capped at `MAX_OUTPUT_CHARS` (250k), or lower via `maxOutputChars`.
   Cutting sets `truncated` on the result, so a partial extraction is never mistaken for a complete
-  one. The PDF and `.xlsx` handlers apply the cap **incrementally** as they build — and stop reading
-  the document once they reach it — so neither ever materializes in full. The `.docx` (mammoth) and
-  HTML (html-to-text) handlers return a complete string that is then trimmed, so for those the cap is
+  one. The PDF, `.docx` and `.xlsx` handlers apply the cap **incrementally** as they build — and stop
+  reading the document once they reach it — so none of them ever materializes in full. Only the HTML
+  handler (html-to-text) returns a complete string that is then trimmed, so for that one the cap is
   **post-materialization** and peak memory follows the whole document.
 - **Timeout** — `HANDLER_TIMEOUT_MS` (10 s) stops *awaiting* a slow async parse. It cannot cancel
   synchronous CPU already running inside a parser, so handlers that yield between units of work (PDF
-  per page, `.xlsx` per row) also check the deadline themselves and stop; the others cannot.
+  per page, `.docx` per inflate chunk, `.xlsx` per row) also check the deadline themselves and return
+  what they have with `truncated` set. The HTML, `.doc` and text handlers cannot.
 - **PDF** — page count and accumulated output are bounded (`MAX_PDF_PAGES`, `MAX_OUTPUT_CHARS`), but
   pdf.js's internal per-page decompression is **not** bounded in-library (no hook exists).
+- **`.docx`** — `word/document.xml` is located in the archive's central directory, inflated on its
+  own, and read with a streaming SAX parser (`saxes`) rather than loaded into a DOM. Peak memory
+  therefore tracks one inflate chunk plus the text kept so far, independent of document size.
+  Measured at a 1024 MB heap on a 45 MB `document.xml` inside a 3.65 MB archive, against the previous
+  DOM-based reader:
+
+  | concurrency | before | after |
+  |---|---|---|
+  | 1 | 812 ms / 607 MB | **68 ms / 175 MB** |
+  | 2 | 1552 ms / 970 MB | **59 ms / 164 MB** |
+  | 4 | 3512 ms / 1217 MB | **58 ms / 168 MB** |
+
+  The old reader built 43.4M characters and kept 250k. The sharpest case is not the large archive: a
+  0.41 MB attachment holding one 43 MB `<w:t>` peaked the old reader at 1270 MB, against 199 MB here.
+  Two residual bounds, stated rather than glossed: peak is not independent of *input* size (the API
+  takes a `Buffer`), and `saxes` buffers one text node whole, so a single enormous run still costs
+  about twice its own size. Malformed XML is also stricter than before — a document the old reader
+  silently half-read now comes back either `truncated` or `failed`.
+- **`.docx` scope** — text comes from `word/document.xml` only. **Footnote, endnote and comment
+  bodies are not extracted** (they are separate zip parts), and neither are headers or footers.
+  **Table structure is not preserved**: each cell's paragraphs are emitted in reading order with the
+  same blank-line separator as body paragraphs, so a 2×2 table is indistinguishable from four
+  consecutive paragraphs. List bullets and numbers are dropped; the item text remains. All of this
+  matches the previous reader exactly — it is a documented limit, not a regression — but a consumer
+  reading an invoice or a contract should know the column a figure sat in is gone.
 - **`.xlsx`** — read row-by-row through `exceljs`'s streaming reader rather than loaded whole, so
   peak memory tracks the shared-string table plus one row instead of a live object per cell
   (measured: 294 MB → 171 MB, and 3.7x faster, on a 5 MB / 38 MB-uncompressed workbook). It is not

package.json

@@ -61,6 +61,7 @@
     "@types/html-to-text": "^9.0.4",
     "@types/node": "^22.20.0",
     "@types/word-extractor": "^1.0.6",
+    "mammoth": "^1.12.0",
     "typescript": "^6.0.3",
     "vitest": "^4.1.2"
   },
@@ -69,7 +70,7 @@
     "html-to-text": "^10.0.0",
     "iconv-lite": "^0.7.2",
     "jschardet": "^3.1.4",
-    "mammoth": "^1.12.0",
+    "saxes": "^5.0.1",
     "unpdf": "^1.6.2",
     "word-extractor": "^1.0.4"
   }

tests/__snapshots__/docx-fidelity.test.ts.snap

@@ -0,0 +1,115 @@
+// Vitest Snapshot v1, https://vitest.dev/guide/snapshot.html
+
+exports[`docx fidelity — our reader against mammoth > comments.docx 1`] = `
+"Ouch.
+
+"
+`;
+
+exports[`docx fidelity — our reader against mammoth > embedded-style-map.docx 1`] = `
+"Walking on imported air
+
+"
+`;
+
+exports[`docx fidelity — our reader against mammoth > empty.docx 1`] = `null`;
+
+exports[`docx fidelity — our reader against mammoth > endnotes.docx 1`] = `
+"Ouch.
+
+"
+`;
+
+exports[`docx fidelity — our reader against mammoth > external-picture.docx 1`] = `null`;
+
+exports[`docx fidelity — our reader against mammoth > fixtures/empty.docx 1`] = `null`;
+
+exports[`docx fidelity — our reader against mammoth > fixtures/image.docx 1`] = `
+"Report
+
+"
+`;
+
+exports[`docx fidelity — our reader against mammoth > fixtures/sample.docx 1`] = `
+"First paragraph of the sample document.
+
+
+
+Second paragraph with more words here.
+
+"
+`;
+
+exports[`docx fidelity — our reader against mammoth > footnote-hyperlink.docx 1`] = `null`;
+
+exports[`docx fidelity — our reader against mammoth > footnotes.docx 1`] = `
+"Ouch.
+
+"
+`;
+
+exports[`docx fidelity — our reader against mammoth > simple-list.docx 1`] = `
+"Apple
+
+Banana
+
+"
+`;
+
+exports[`docx fidelity — our reader against mammoth > single-paragraph.docx 1`] = `
+"Walking on imported air
+
+"
+`;
+
+exports[`docx fidelity — our reader against mammoth > strict-format.docx 1`] = `
+"Test
+
+"
+`;
+
+exports[`docx fidelity — our reader against mammoth > strikethrough.docx 1`] = `
+"Today's Special: Salmon Sold out
+
+"
+`;
+
+exports[`docx fidelity — our reader against mammoth > tables.docx 1`] = `
+"Above
+
+Top left
+
+Top right
+
+Bottom left
+
+Bottom right
+
+Below
+
+"
+`;
+
+exports[`docx fidelity — our reader against mammoth > text-box.docx 1`] = `
+"
+
+Datum plane
+
+"
+`;
+
+exports[`docx fidelity — our reader against mammoth > tiny-picture.docx 1`] = `null`;
+
+exports[`docx fidelity — our reader against mammoth > tiny-picture-target-base-relative.docx 1`] = `null`;
+
+exports[`docx fidelity — our reader against mammoth > underline.docx 1`] = `
+"The Sunset Tree
+
+"
+`;
+
+exports[`docx fidelity — our reader against mammoth > utf8-bom.docx 1`] = `
+"This XML has a byte order mark.
+
+"
+`;

tests/docx-fidelity.test.ts

@@ -0,0 +1,96 @@
+import { readFileSync, readdirSync } from 'node:fs'
+import { join } from 'node:path'
+
+import { describe, it, expect } from 'vitest'
+import mammoth from 'mammoth'
+
+import { extractAttachment } from '../attachment'
+
+// ---------------------------------------------------------------------------
+// .docx fidelity — our reader against mammoth, over a real Word corpus
+// ---------------------------------------------------------------------------
+// This library read .docx through mammoth.extractRawText until the streaming rewrite. That reader
+// materialised an xmldom tree AND a document model before any cap could apply; the replacement walks
+// word/document.xml with a SAX parser and stops at the cap. The output contract is unchanged, and
+// this file is what says so.
+//
+// THE INTERLOCK. Each case pins ONE golden from TWO sides: the snapshot records what we produce, and
+// mammoth is then asserted to produce the same string. Neither side can move alone —
+//   - a regression in our reader fails the snapshot;
+//   - `vitest -u`, which would launder that regression by rewriting the snapshot, then fails the
+//     mammoth assertion, because mammoth still produces the old value.
+// The only way to change behaviour is to add an ACCEPTED_DIVERGENCES entry with a written reason,
+// which is a reviewed, committed edit. That is the mechanical form of "any structural regressions
+// understood and accepted".
+//
+// NO skipIf ON THE ORACLE. `import mammoth` at the top is deliberate: if mammoth is missing the
+// suite fails loudly rather than degrading to a snapshot pinned only by the code that generated it,
+// which is self-referential and reports green. mammoth is a devDependency, so it is always present;
+// retiring the oracle should be a reviewed deletion of this file, not a condition that skips itself.
+
+// Vendored from mammoth 1.12.0 test/test-data — see docx-corpus/README.md for why they are copied
+// rather than read out of node_modules, and what each one covers.
+const CORPUS_DIR = join(process.cwd(), 'tests', 'fixtures', 'docx-corpus')
+// The three fixtures the rest of the suite already uses. They are macOS textutil output, a dialect
+// the vendored corpus (all Word) does not otherwise represent; elsewhere they are only asserted with
+// `toContain`, which cannot catch a paragraph-break regression.
+const FIXTURE_DIR = join(process.cwd(), 'tests', 'fixtures')
+const LOCAL_FIXTURES = ['sample.docx', 'empty.docx', 'image.docx']
+
+const DOCX_TYPE = 'application/vnd.openxmlformats-officedocument.wordprocessingml.document'
+
+// Documents where our reader deliberately diverges from mammoth. Each entry must say WHY, and holds
+// what mammoth produces — the snapshot holds ours. An empty table means the port is exact.
+const ACCEPTED_DIVERGENCES: Record<string, { reason: string; mammoth: string | null }> = {}
+
+// The entry point omits `extraction` entirely rather than ever returning '' (a ran-but-empty
+// extraction is `{ status: 'extracted' }` with no text). That collapse is a library-wide contract
+// applied identically to whatever reader is installed, so the oracle gets it too — otherwise
+// empty.docx, whose text is exactly '\n\n', reads as a divergence in every run.
+const asExtraction = (text: string): string | null => (text.trim().length === 0 ? null : text)
+
+const corpusFiles = () => readdirSync(CORPUS_DIR).filter((name) => name.endsWith('.docx')).sort()
+
+const cases = [
+    ...corpusFiles().map((name) => [name, join(CORPUS_DIR, name)] as const),
+    ...LOCAL_FIXTURES.map((name) => [`fixtures/${name}`, join(FIXTURE_DIR, name)] as const),
+]
+
+describe('docx fidelity — our reader against mammoth', () => {
+    it.each(cases)('%s', async (name, path) => {
+        const content = readFileSync(path)
+
+        const result = await extractAttachment({ content, contentType: DOCX_TYPE })
+        // No corpus document is anywhere near MAX_OUTPUT_CHARS, so anything but a complete
+        // extraction means a gate fired that should not have — worth failing on directly rather
+        // than letting it show up as a confusing snapshot diff.
+        expect(result.status).toBe('extracted')
+        expect(result.truncated).toBe(false)
+
+        const ours = result.extraction ?? null
+        expect(ours).toMatchSnapshot()
+
+        const { value } = await mammoth.extractRawText({ buffer: content })
+        const theirs = asExtraction(value)
+
+        const divergence = ACCEPTED_DIVERGENCES[name]
+        if (divergence) {
+            expect(theirs).toBe(divergence.mammoth)
+            return
+        }
+        // Paragraph array first: vitest diffs it element-wise, so a lost or hoisted paragraph reads
+        // as one changed line instead of a wall of escaped text. Then the exact string, which is
+        // what catches trailing breaks and whitespace the split would swallow.
+        expect((theirs ?? '').split('\n\n')).toEqual((ours ?? '').split('\n\n'))
+        expect(theirs).toBe(ours)
+    })
+
+    // A fixture added without a golden, or deleted while its golden lingers, are both silent holes:
+    // it.each simply enumerates fewer cases and the suite still passes. Counting them here is what
+    // makes the corpus itself part of the contract.
+    it('runs every corpus document, and only documents that exist', () => {
+        expect(corpusFiles()).toHaveLength(17)
+        expect(cases).toHaveLength(20)
+        for (const [, path] of cases) expect(readFileSync(path).length).toBeGreaterThan(0)
+    })
+})