[gulatikrishav]

Stream .docx extraction

Phase C of the streaming-parsers spec. Phase D (API wiring of truncated) is not in this PR.

Stacked on #14. Based on krishav/streaming-xlsx-truncation so the diff is Phase C only — the
seven commits below, +1381/−50. GitHub retargets this to main automatically when #14 merges.
Merge #14 first.

.xlsx was one of two formats that can reach the 50 MB decompression ceiling. This is the other one. It still ran mammoth.extractRawText({ buffer }), which built an xmldom tree and a document model over the whole of word/document.xml before any cap could apply, then let the central trim throw almost all of it away.

The measurement

Through dist/ at a 1024 MB heap, on a 45 MB document.xml inside a 3.65 MB archive — every gate cleared, nothing skipped:

concurrency	mammoth	this
1	812 ms / 607 MB	68 ms / 175 MB
2	1552 ms / 970 MB	59 ms / 164 MB
4	3512 ms / 1217 MB	58 ms / 168 MB

mammoth built 43,420,044 characters every time and kept 250,000 — 0.6%. Its peak tracks the document and multiplies by concurrency; this is flat, because the cap now stops the read rather than trimming the result.

The sharpest case is not the large archive. A 0.41 MB attachment holding one 43 MB <w:t> peaked mammoth at 1270 MB, against 199 MB here.

Fidelity — the acceptance criterion

41 documents, 41 byte-identical, 0 divergences, 0 errors. That is the 17 vendored Word documents, the 3 repo fixtures, 8 real-world Word-authored templates, 10 textutil-dialect documents (tables, lists, unicode, 200-paragraph bodies), and 3 more from disk. ACCEPTED_DIVERGENCES is empty.

The harness pins one golden from two sides: the snapshot records what we produce, and mammoth is asserted to produce the same string. A regression fails the snapshot; vitest -u, which would launder it, then fails the mammoth assertion because mammoth still produces the old value. Verified by injecting a trim into the old handler and running with -u — 14 of 20 documents failed on the mammoth side with the snapshot already rewritten. The only way to change behaviour is an ACCEPTED_DIVERGENCES entry carrying a written reason.

The corpus is committed (tests/fixtures/docx-corpus/, 17 files vendored from mammoth 1.12.0 under BSD-2-Clause with the notice retained) rather than read from node_modules: mammoth declares no files field, so test/ ships by accident, and mammoth is now a devDependency.

Landed before the swap, in its own commit, so it is a real baseline rather than a suite that first appears alongside the code it validates.

Two deviations from the spec, called out deliberately

1. No unzipper/yauzl. The spec called for a zip library as a new dependency. zipEntries already walks the central directory and returns each entry's still-compressed subarray, and zlib is already imported and already stream-inflates one. The only new runtime dependency is saxes — already hoisted as a deduped transitive of both exceljs and word-extractor, so it installs nothing.

2. The mapping is ~205 lines, not "tens of lines sharing Phase B's SAX plumbing". Both halves of that estimate were wrong. Phase B shipped on exceljs's own reader and left no SAX plumbing to share, so this is the library's first hand-written SAX path. And mammoth dispatches from a whitelist (body-reader.js:46-57): an element it has no handler for has its entire subtree dropped, not recursed into. "Emit every w:t, newline on </w:p>" over-extracts on any document with a text box or a field.

Adopting the whitelist is also what deletes work: mammoth's 20-entry ignore list and all its emit-nothing leaves are just the default, and w:sdt→w:sdtContent and mc:AlternateContent→mc:Fallback fall out for free because w:sdtPr and mc:Choice are simply absent from the list.

Also not authorised by the spec: mammoth moves to devDependencies. It follows from the malformed-XML decision — with no fallback, nothing calls it — and removes 10 packages / 3.61 MB from the runtime tree (mammoth, underscore, @xmldom/xmldom, dingbat-to-unicode, argparse, xmlbuilder, lop, sprintf-js, duck, option). jszip and bluebird stay; exceljs needs them regardless.

Malformed XML: the line moves in both directions

saxes is conformant where xmldom recovered, and xmldom threw where saxes copes. Measured on twelve cases: eight agree, two we are stricter on, two we are more available on.

• Stricter: a raw U+000B in a w:t, and the same character as . Neither is a legal XML 1.0 character. Both test cases put it in the first text node, so nothing had been read and the result is failed; the same character further into a document keeps everything before it.
• More available: trailing junk after the root, and an unclosed root. mammoth threw away the whole document for both.

The policy: text read then a broken parse → extracted + truncated: true; nothing read → failed. The loss is bounded, not a cliff — the same illegal character moved off the first text node keeps everything before it. Deliberately not a saxes error handler that reports and parses on: measured, that recovery emits close-tag text as content and descends into elements mammoth drops, i.e. silent wrong output.

Undeclared namespace prefixes are handled as a class, not an allowlist. saxes fails the whole parse on an unbound prefix — for attributes as well as elements — and Word 2013+ stamps w15:paraId on every w:p with w16cid:durableId beside it, so a document carrying those without their xmlns lost everything after its first paragraph. resolvePrefix resolves anything unclaimed to a sentinel URI that OOXML_PREFIXES doesn't map, so the element names {urn:agentextract:unbound}local, misses the whitelist, and its subtree drops — exactly what mammoth does with an unmapped namespace. saxes consults it last, after scope and additionalNamespaces, so the two compose; the assumed-prefix map keeps only w/mc/v, the three URIs where guessing beats dropping.

Zero saxes errors were raised on any of the 41 real documents. That was the evidence that mattered most for shipping without a fallback.

Testing

385 tests green; tsc --noEmit and the build clean.

• 56 new tests across three files. Every contract claim was measured against mammoth, not read off the spec: 21 of 22 synthetic constructs byte-identical, including w:pict hoisting (the picture's text lands after the paragraph's break), the deleted paragraph mark, the deleted table row, w:instrText suppression, the w:fldSimple drop, CDATA vanishing, and NEL/U+2028 normalisation. The one divergence is the declared w:sym one.
• Mutation-tested, not just green. Removing the NEL/U+2028 normalisation fails exactly one test; making unknown elements recurse fails exactly one test; emitting one newline per paragraph instead of two fails 34 of 53. The first two failing precisely one test each is the point — each regression is caught by its own test.
• Fuzz: 1000 seeded iterations, two archives (deflated exercises the zip walk and preflight; stored puts mutated bytes straight in front of saxes). Coverage measured rather than assumed: 516/1000 reach extracted, the stored half alone 357/500; failure reasons span the pipeline — 352 from the zip walk, 21 from the handler's own w:body guard, 19 from routing. The property is that the degraded state is always reachable, always labeled, always inside the deadline, and never ''.
• tests/attachment.test.ts is untouched — its 9 docx sites are routing and result-shape tests and stay green as the compatibility statement.

Known limits, now stated in the README

Inherited from mammoth exactly, so none is a regression — but all are easy to assume away:

• Footnote, endnote and comment bodies are not extracted (separate zip parts); headers and footers are never opened.
• Table structure is not preserved. A 2×2 table is byte-identical to four consecutive paragraphs. For a model reading an invoice, the column a figure sat in is gone.
• w:fldSimple and w:ruby text is dropped — genuine mammoth text loss (cross-references, page numbers, DOCPROPERTY, merge-field results), replicated on purpose so this stays a pure port. Recovering them is one array entry each and belongs in its own change, once the corpus says how often they matter.
• Peak is not independent of input size (the API takes a Buffer), and saxes buffers one text node whole, so a single enormous run still costs about twice its own size.

Knock-on comment rewrites

In commit 2 because they would otherwise be actively wrong. DECOMPRESSION BUDGET no longer has two foreign readers to reason about — both OOXML formats now read through our own zipEntries walk — so Invariant 2 becomes the proof that budget-ok implies zipEntries-ok for both, and Invariant 1 is demoted to a structural check. findEocd's last-match choice is now internal rather than a match for jszip's.

The budget is also noted as knowingly stale: it measures every entry while the handler inflates one, and on real documents word/document.xml is 2–28% of the archive (a Word template measured 66 KB of 3.1 MB). Loosening it is a permissiveness change on a zip-bomb gate that belongs with the input caps, not smuggled in behind a parser swap.

Corpus gap

LibreOffice and pandoc are not installed locally, so that dialect is untested. Production attachment bytes are unreachable from this account (the manifest is metadata only).

Follow-ups

The one-line w:fldSimple / two-line w:ruby recovery, once there is frequency data. The API bump to 0.4.0 and the createLimit(1) comment correction — its evidence is an .xlsx measurement Phase B invalidated, and the .docx half is now fixed too; the limit itself stays at 1 per spec §5. Then Phase D.

[Haakam21]

Superseded by #20, which integrates the XLSX and DOCX streaming work on current main and includes the consolidated review fixes.