Japanese text in PDFs reads in full

DOCS/CHANGELOG.md

@@ -6,6 +6,17 @@ section under a version heading.
 
 ## [Unreleased]
 
+### Fixed
+
+- Japanese text in PDFs reads in full. A PDF made on a Mac (Hiragino and the other Japanese fonts the Mac embeds without a ToUnicode map) read as almost nothing, and one using the standard Japanese CMaps (`UniJIS-UCS2-H`, Shift_JIS `90ms-RKSJ-H`, `EUC-H`) read as nothing; both now read as poppler and PyMuPDF read them. A composite font is read through its own encoding: its CIDs through Adobe-Japan1 when the font is in that collection, the Unicode CMaps (for Chinese and Korean too) as Unicode, and the Shift_JIS, EUC-JP, and JIS CMaps decoded as browsers decode them. Kangxi radicals that a ToUnicode map gives for ideographs (Chrome's PDFs) read as the ideographs.
+- Vertical Japanese reads by column, right to left: text in a vertical (`-V`) font, and characters the Mac sets one under another down a column, which read one character per line before (as they do in PyMuPDF).
+- Marked content with `/ActualText` reads as that text, as viewers copy it: LibreOffice's PDFs of Japanese without a Japanese font read as one repeated character before.
+- Japanese lines that wrap within a paragraph join without a space when a PDF becomes Markdown, Word, HTML, or another document.
+
+### Changed
+
+- The Japanese tables (Adobe-Japan1 CIDs, JIS X 0208, radical equivalents) add 100,768 bytes to the binary on CI's Linux build (5,696,960) and 97,509 to the WebAssembly module (3,048,980); the binary's size budget rises from 5.62 to 5.78 MB.
+
 ## [1.0.0] - 2026-10-21
 
 ### Changed

DOCS/STATE.md

@@ -6,6 +6,7 @@ describes.
 
 ## Now
 
+- 2026-10-30: Japanese PDF text (branch `japanese`, 1.0.1). A user's project found Mac-made PDFs reading 1 Japanese character in 360 and PDFs in the standard Japanese CMaps reading as nothing. Composite fonts now read through their encoding CMap (`src/io/pdf/font.rs`): Identity and embedded CMaps give CIDs, read through Adobe-Japan1 (`src/io/cjk`, Adobe's table) when the font is Japan1; the Unicode CMaps give Unicode; Shift_JIS, EUC-JP, and JIS CMaps are decoded (WHATWG's JIS X 0208 index). Also: radicals normalised to ideographs (Chrome), `/ActualText` (LibreOffice without a Japanese font), vertical writing read by column (a `-V` font, and the Mac's glyph-by-glyph columns), and Japanese lines joined without spaces in documents. On twelve test PDFs from seven producers every character reads as PyMuPDF reads it; on 70 to 100-page documents a whole run is 0.01 to 0.05 s at 8 to 13 MB, against PyMuPDF's extraction alone at 0.06 to 0.14 s and 186 MB. Tables add about 75 KB to the binary.
 - 2026-10-21: v1 wrap-up and 1.0.0 (branch `v1-wrapup`). The converter registry is data: each converter has a `const fn entry()` and the registry is a static table of entries, so planning runs no converter's code, and the 195 image pairs are built only when an image format is an end (`registry::converters_for`). Starting memory 2.24 -> 1.95 MB, a ROM conversion 2.66 -> 2.42 MB, startup 0.15 ms faster; sizes move by a few KB. Every benchmark pair was rerun and recorded (first run on the M1 Max for many pairs, under load); the failing lines are under Tech Debt. The bench harness was fixed for this Mac (`gtime` in `csv-json`, awk helpers in `ico-png`, a bash 5 guard in `jpeg-pdf`). ROADMAP item 9 done; the next cycle is audio and video.
 - 2026-10-20: N64 ROMs between `z64`, `v64`, and `n64` (`src/io/n64.rs`, branch `n64`): six lossless converters permute each 32-bit word in a 256 KB buffer, vectorized as an array shuffle, reading the input's order from where the header's `0x80` lies whatever the name. Detection on the command line now lets content overrule an extension whose signature it fails when another format's matches (`format::detect`, for every format), with a warning; errors without a position drop `line 0, column 0`. Checked on Paper Mario (USA), a 40 MB No-Intro dump in `~/Downloads` (not committed): round trips return its SHA-1, outputs byte-identical to ucon64, `n64romconvert`, and `dd conv=swab`. Pair `z64-v64`: every speed line passes (1.2 to 1.35 times `dd`); the four memory lines against ucon64 and `dd` fail by 0.1 to 0.6 MB, the binary's startup floor (Tech Debt). The bench harness takes `bench_runs` for millisecond pairs. The wrap-up is next.
 - 2026-10-17: OpenDocument spreadsheet both ways (`src/io/ods`, branch `ods`): every sheet reads into rows with cells shown as LibreOffice shows them to an English reader, and rows write as sheets typed by the read-back rule; number formats pass through one model between ODS data styles and Excel codes (`numfmt.rs`), so ODS reaches Excel and Numbers with its formats and back. LibreOffice's built-in dates per document language come from its locale data (`builtin.rs`, MPL-2.0, `scripts/gen-ods-dates.py`). `content.xml` streams in four-megabyte pieces through a new `ZipArchive::stream`, read once, each wanted sheet held as its cells' text. Against LibreOffice's CSV export (`scripts/ods-check`): 303 of 463 development sheets byte-identical, 99.65% of non-formula cells; held out (35 government files, 12.2 million cells), 190 of 240 sheets, 99.999%, with five faults the held-out set found fixed before those numbers; every development CSV written as ODS reads back the same in LibreOffice. Pair `ods-csv` passes every line (calamine 0.36, LibreOffice). Budgets: binary 5.62 MB (Linux 5.59), wasm 3.14 MB (3.11). N64 ROMs are next.
@@ -77,6 +78,7 @@ describes.
 
 ## Tech Debt
 
+- 2026-10-30: Japanese beyond PDF reading. Plain text and HTML in Shift_JIS, EUC-JP, or ISO-2022-JP are refused as invalid UTF-8 (`src/converters/input.rs` reads UTF-8 only; the decoders are in `src/io/cjk`); RTF from Word with `\fcharset128` and `\'xx` byte pairs is untested (the code pages in `src/io/rtf/codepage.rs` are single-byte); the PDF writer embeds TrueType only, so the CFF fonts most CJK systems ship (Noto Sans CJK) are skipped, and Japanese wraps by character without the line-breaking rules (kinsoku). Every other document pair carries Japanese whole (pandoc, textutil, and LibreOffice sources to every document format, 100%). Chinese and Korean CID collections (GB1, CNS1, Korea1) have no CID table yet, so their fonts without ToUnicode read only through Unicode CMaps.
 - 2026-09-24 (updated 2026-10-21, moved from Blockers): The Pages document paths' throughput against the shared Pages goal of 50 MB/s of input (`DOCS/benchmarks/pages-markdown.md`, `pages-html.md`, `pages-text.md`). The 1.0.0 run, on an M1 Max under load: memory passes on every document row (26 to 47 MB against 64), Word passes both ways (Pages -> Word 129 and 140 MB/s of input plus uncompressed output, Word -> Pages 67 and 89), and the text paths fail throughput at 26 to 33 MB/s (styled 29 to 33, prose 26 to 28). A real resume converts in 3 ms at 4.8 MB, inside both goals. Where the time goes on the dense shape, in-process (2026-09-24): package decode 10 ms (7.5 of it Snappy), document build 25 ms, the text writers 11 ms; wall clock adds about 10 ms of process start, file I/O, and first-touch page faults. Levers under Spikes. Pages -> Word's peak memory rose on every row since 2026-10-06 (prose 33.8 -> 46.8 MB), still inside the goal.
 - 2026-09-24 (updated 2026-10-21, moved from Blockers): `pages -> pages-json` misses the 50 MB/s goal at 24 and 26 MB/s of package bytes (`DOCS/benchmarks/pages-json.md`); the reverse direction passes at 146 and 239. Levers: the typed decode of the attribute tables written as JSON directly (the Spikes entry below), then `Tree::decode` itself and the JSON writer's per-field work.
 - 2026-10-21: Benchmark lines failing in the 1.0.0 run that passed in their previous blocks. The run was on the M1 Max with the machine in use (load average 7 to 26 on 10 cores); for several pairs every earlier block was on the Linux i7, so these are not like for like. An alternated A/B of ec68950 against the commit before it showed the registry change caused none of those it covered. Rerun on an idle machine before acting on any of them.

DOCS/formats/pdf.md

@@ -122,8 +122,8 @@ first.
 | Part | Handling |
 |---|---|
 | Content | `src/io/pdf/content.rs` runs the page's content: `q`/`Q`/`cm`, `BT`/`ET`, `Tc Tw Tz TL Tf Ts`, `Td TD Tm T*`, `Tj TJ ' "` (TJ offsets move the text position), form XObjects with their matrix and resources (8 deep), inline images skipped. Each shown code becomes a glyph placed in page space with its size there |
-| Fonts | `src/io/pdf/font.rs`: code lengths from the ToUnicode code-space ranges (two bytes for Type0 without one); Unicode from ToUnicode (bfchar, bfrange with strings or arrays), else a simple font's encoding: Standard, WinAnsi, MacRoman, or the Symbol and Zapf Dingbats built-ins, with `/Differences` names through the glyph list, `uniXXXX`, `uXXXX`, a `.suffix` dropped, and ligatures (fi, fl, ff, ffi, ffl) spelled out. Widths from `/Widths` with `/FirstChar` and `/MissingWidth`, CID `/W` and `/DW`, Type3 `/FontMatrix`, and the Core 14 AFM widths for base fonts without `/Widths` (`src/io/pdf/tables.rs`, generated by `scripts/gen-pdf-tables.py`) |
-| Layout | `src/io/pdf/text.rs`: glyphs on one baseline (within half their size) are a line; a gap over 0.15 of the size is a word break unless a space is already there; runs of spaces collapse. A line of another size or weight starts a block, and so does one further below the last than 1.6 sizes and than the page's most common line spacing (so a font whose glyph units are not the usual em, which Ghostscript's rewrites of CID fonts make, still groups right). Reading order is drawing order |
+| Fonts | `src/io/pdf/font.rs`: a composite (Type0) font's code lengths from its encoding CMap: `Identity-H`/`-V` (two bytes, the CID), the Unicode CMaps (`Uni*-UCS2`, `-UTF16`, `-UTF8`, `-UTF32`, for every CJK collection), Shift_JIS (`*RKSJ*`), EUC-JP (`EUC-H`/`-V`), JIS (`H`/`V`), and an embedded CMap's code-space ranges and CID mappings (with `usecmap`); a simple font's from the ToUnicode code-space ranges. Unicode from ToUnicode (bfchar, bfrange with strings or arrays); else, for a composite font, the code itself for a Unicode CMap, the Shift_JIS, EUC-JP, or JIS bytes decoded (`src/io/cjk`, WHATWG's JIS X 0208 index), or the CID through Adobe-Japan1 (Adobe's CID to Unicode table) when the CID font's `/CIDSystemInfo` is Japan1, as the Mac writes Hiragino without a ToUnicode map; else a simple font's encoding: Standard, WinAnsi, MacRoman, or the Symbol and Zapf Dingbats built-ins, with `/Differences` names through the glyph list, `uniXXXX`, `uXXXX`, a `.suffix` dropped, and ligatures (fi, fl, ff, ffi, ffl) spelled out. Widths from `/Widths` with `/FirstChar` and `/MissingWidth`, CID `/W` and `/DW`, Type3 `/FontMatrix`, and the Core 14 AFM widths for base fonts without `/Widths` (`src/io/pdf/tables.rs`, generated by `scripts/gen-pdf-tables.py`); a CID font's `/W` by CID, a code whose CID the encoding does not say (Shift_JIS, Unicode CMaps) half or full width by its character. CJK radicals and strokes a ToUnicode map gives read as the ideographs they stand for (Unicode's equivalents). Marked content with `/ActualText` reads as that text in place of its glyphs |
+| Layout | `src/io/pdf/text.rs`: glyphs on one baseline (within half their size) are a line; a gap over 0.15 of the size is a word break unless a space is already there; runs of spaces collapse. A line of another size or weight starts a block, and so does one further below the last than 1.6 sizes and than the page's most common line spacing (so a font whose glyph units are not the usual em, which Ghostscript's rewrites of CID fonts make, still groups right). Reading order is drawing order. Vertical writing is read by column: a vertical font's glyphs advance down the page, and Japanese characters set one under another in one column (as the Mac places them) continue a line; columns side by side make a block, in drawing order (right to left) |
 
 Oracle: `tests/pdf_text.rs` against pdftotext on every fixture in
 `tests/fixtures/pdf-text` (PyMuPDF's base-14 and embedded TrueType
@@ -158,9 +158,13 @@ Markdown.
 - WebP, QOI, TGA, Netpbm, and ICO record no resolution a page could use (WebP's lives in EXIF, which is not read), so their pages are a point per pixel.
 - Text is read in drawing order: a page that draws its columns or
   boxes out of reading order reads out of order (pdftotext reorders by
-  blocks). Rotated and vertical text is read glyph by glyph in drawing
-  order and reported. A Type0 font with neither ToUnicode nor a known
-  character collection has only glyph ids and reads as nothing.
+  blocks). Rotated text is read glyph by glyph in drawing order and
+  reported. A Type0 font with neither ToUnicode nor a collection with a
+  table here (Adobe-Japan1 only: not GB1, CNS1, or Korea1) has only glyph
+  ids and reads as nothing, unless its CMap is a Unicode one. A ToUnicode
+  map is taken as written: the Mac's Osaka maps its full-width digits to
+  ASCII ones, which read as ASCII (poppler reads them full width from the
+  font program).
 - Structure is guessed from type: bold or italic inside a line, tables,
   links, and footnotes are not read; a paragraph that runs across a
   page break is two paragraphs; running headers other than page numbers

NOTICE

@@ -7,6 +7,17 @@ option) any later version. See LICENSE and LICENSING.md.
 Releases 0.1.0 through 0.3.0 were published under the Apache License,
 Version 2.0, and remain available under it.
 
+src/io/cjk/tables.rs holds data generated by scripts/gen-cjk-tables.py from:
+- Adobe's PDF mapping resources (Adobe-Japan1-UCS2), Copyright 1990-2023
+  Adobe, under the BSD 3-Clause License: redistribution in source and
+  binary forms is permitted provided the copyright notice, the conditions,
+  and the disclaimer are retained (they are, in the source's header);
+  Adobe's name may not be used to endorse products derived from it.
+- The WHATWG Encoding Standard's index-jis0208, Copyright WHATWG (Apple,
+  Google, Mozilla, Microsoft), under the BSD 3-Clause License.
+- Unicode's EquivalentUnifiedIdeograph.txt, Copyright Unicode, Inc., under
+  the Unicode License v3.
+
 src/io/ods/builtin.rs holds date format codes taken from LibreOffice's
 locale data (Copyright The Document Foundation and contributors), under the
 Mozilla Public License 2.0; that file is also available under it.