Japanese in every document and data format

DOCS/CHANGELOG.md

@@ -6,6 +6,13 @@ section under a version heading.
 
 ## [Unreleased]
 
+### Added
+
+- Japanese in the legacy encodings reads everywhere. Plain text, Markdown, and HTML in Shift_JIS, EUC-JP, or ISO-2022-JP (an HTML `charset` or XML `encoding` is followed; otherwise the encoding is detected), CSV and TSV in Shift_JIS (as Excel saves a Japanese CSV) or EUC-JP, decoded as they stream, and XML declaring Shift_JIS. Detection takes bytes for Japanese only when they read as Japanese does (common kana and kanji), so a Western file is still refused with an error rather than turned into kanji. Word's RTF with Japanese as Shift_JIS byte pairs (`\ansicpg932`, `\fcharset128`) reads in full; it read as replacement characters before.
+- PDFs embed the fonts CJK systems ship: CID-keyed CFF OpenType (Hiragino on the Mac, Noto Sans CJK and Source Han Sans on Linux), subset to the glyphs and subroutines used (a Japanese page with Noto Sans CJK is 264 KB from a 16 MB font). Japanese text prefers a Japanese font (Hiragino, Noto Sans CJK, Yu Gothic, Meiryo, MS Gothic) over the broad fallbacks such as Arial Unicode. `--font` takes a CID-keyed CFF font too.
+- Japanese in PDFs breaks lines between characters with the kinsoku rules: no line starts with closing brackets, punctuation, small kana, or the long-vowel mark, and none ends with an opening bracket.
+- `DOCS/LANGUAGES.md`: which scripts each path handles (text and documents, legacy encodings, PDF to text, text to PDF) and what is not yet supported, linked from the README and `FORMATS.md`.
+
 ## [1.0.1] - 2026-10-30
 
 ### Fixed

DOCS/FORMATS.md

@@ -2,7 +2,7 @@
 
 Generated by `sublime paths --markdown`. Do not edit by hand. CI fails if this file is stale.
 
-Every converter and every conversion path, with what each loses, are in [CONVERSIONS.md](CONVERSIONS.md).
+Every converter and every conversion path, with what each loses, are in [CONVERSIONS.md](CONVERSIONS.md); which languages and scripts each handles is in [LANGUAGES.md](LANGUAGES.md).
 
 ## Map
 

DOCS/LANGUAGES.md

@@ -0,0 +1,35 @@
+# Language support
+
+Text is Unicode throughout, so any language in a UTF-8 or UTF-16 file, or
+in a format that stores Unicode (Word, Pages, Numbers, Excel, ODF, EPUB,
+HTML, XML, JSON, YAML, TOML), converts as written. What differs by
+language is reading older encodings, reading text back out of a PDF, and
+setting it into a new PDF.
+
+| Script | Text and documents | Legacy encodings read | PDF to text | Text to PDF |
+|---|:-:|---|:-:|---|
+| Latin, Greek, Cyrillic | ✓ | RTF code pages (437, 850, 1250-1258, Mac Roman) | ✓ | ✓ |
+| Japanese | ✓ | Shift_JIS, EUC-JP, ISO-2022-JP in text, Markdown, HTML, XML, CSV, TSV; RTF code page 932 | ✓ incl. vertical writing and fonts with no ToUnicode map | ✓ Japanese line breaking (kinsoku), horizontal only |
+| Chinese | ✓ | RTF through `\uN` only | ✓ when the PDF carries a ToUnicode map or a Unicode CMap | ✓ may be set in a Japanese font |
+| Korean | ✓ | RTF through `\uN` only | ✓ when the PDF carries a ToUnicode map or a Unicode CMap | ✓ |
+| Hebrew, Arabic | ✓ | RTF code pages 1255, 1256 | in drawing order | left to right, isolated letter forms |
+| Hindi and other Indic, Thai | ✓ | RTF code page 874 (Thai) | ✓ | glyphs unshaped (no conjuncts); Thai breaks only where a line overflows |
+
+✓ means read back as written in the tests. Text to PDF uses a font
+installed on the machine that has the characters (or `--font`); a
+character no installed font has is `?` and reported.
+
+## Not yet
+
+- Chinese and Korean legacy encodings (GBK, GB18030, Big5, EUC-KR) in
+  text and CSV files: rejected with an error, not misread.
+- Chinese and Korean PDFs whose fonts have no ToUnicode map and a
+  non-Unicode CMap (the Adobe-GB1, -CNS1, -Korea1 tables).
+- Western code page detection in text and CSV files (a Windows-1252 CSV
+  is rejected; save it as UTF-8).
+- Bidirectional reordering and complex-script shaping in PDF output.
+- Vertical Japanese and bold Japanese in PDF output (bold is set in the
+  regular weight).
+
+Per-format detail is in [formats/](formats/): `pdf.md`, `csv.md`,
+`html.md`, `rtf.md`.

DOCS/STATE.md

@@ -6,6 +6,7 @@ describes.
 
 ## Now
 
+- 2026-10-31: Japanese everywhere (branch `japanese-input`). Legacy encodings read in every text format: `read_text_document` decodes UTF-16, a declared charset, or detected Shift_JIS, EUC-JP, or ISO-2022-JP (`src/io/cjk`: common-row plausibility keeps Western bytes out), CSV/TSV and XML decode as they stream (`Utf8Reader`), RTF decodes code page 932. The PDF writer embeds CID-keyed CFF fonts (`src/io/font/cff.rs`: unused glyphs and subroutines blanked in place, CID = glyph id; checked in poppler, Quartz, and MuPDF with Hiragino and Noto Sans CJK), prefers Japanese fonts for Japanese, and breaks Japanese lines with kinsoku. A matrix of 20 Japanese document and 5 data sources to every format reads 100%.
 - 2026-10-30: Japanese PDF text (branch `japanese`, 1.0.1). A user's project found Mac-made PDFs reading 1 Japanese character in 360 and PDFs in the standard Japanese CMaps reading as nothing. Composite fonts now read through their encoding CMap (`src/io/pdf/font.rs`): Identity and embedded CMaps give CIDs, read through Adobe-Japan1 (`src/io/cjk`, Adobe's table) when the font is Japan1; the Unicode CMaps give Unicode; Shift_JIS, EUC-JP, and JIS CMaps are decoded (WHATWG's JIS X 0208 index). Also: radicals normalised to ideographs (Chrome), `/ActualText` (LibreOffice without a Japanese font), vertical writing read by column (a `-V` font, and the Mac's glyph-by-glyph columns), and Japanese lines joined without spaces in documents. On twelve test PDFs from seven producers every character reads as PyMuPDF reads it; on 70 to 100-page documents a whole run is 0.01 to 0.05 s at 8 to 13 MB, against PyMuPDF's extraction alone at 0.06 to 0.14 s and 186 MB. Tables add about 75 KB to the binary.
 - 2026-10-21: v1 wrap-up and 1.0.0 (branch `v1-wrapup`). The converter registry is data: each converter has a `const fn entry()` and the registry is a static table of entries, so planning runs no converter's code, and the 195 image pairs are built only when an image format is an end (`registry::converters_for`). Starting memory 2.24 -> 1.95 MB, a ROM conversion 2.66 -> 2.42 MB, startup 0.15 ms faster; sizes move by a few KB. Every benchmark pair was rerun and recorded (first run on the M1 Max for many pairs, under load); the failing lines are under Tech Debt. The bench harness was fixed for this Mac (`gtime` in `csv-json`, awk helpers in `ico-png`, a bash 5 guard in `jpeg-pdf`). ROADMAP item 9 done; the next cycle is audio and video.
 - 2026-10-20: N64 ROMs between `z64`, `v64`, and `n64` (`src/io/n64.rs`, branch `n64`): six lossless converters permute each 32-bit word in a 256 KB buffer, vectorized as an array shuffle, reading the input's order from where the header's `0x80` lies whatever the name. Detection on the command line now lets content overrule an extension whose signature it fails when another format's matches (`format::detect`, for every format), with a warning; errors without a position drop `line 0, column 0`. Checked on Paper Mario (USA), a 40 MB No-Intro dump in `~/Downloads` (not committed): round trips return its SHA-1, outputs byte-identical to ucon64, `n64romconvert`, and `dd conv=swab`. Pair `z64-v64`: every speed line passes (1.2 to 1.35 times `dd`); the four memory lines against ucon64 and `dd` fail by 0.1 to 0.6 MB, the binary's startup floor (Tech Debt). The bench harness takes `bench_runs` for millisecond pairs. The wrap-up is next.
@@ -78,7 +79,7 @@ describes.
 
 ## Tech Debt
 
-- 2026-10-30: Japanese beyond PDF reading. Plain text and HTML in Shift_JIS, EUC-JP, or ISO-2022-JP are refused as invalid UTF-8 (`src/converters/input.rs` reads UTF-8 only; the decoders are in `src/io/cjk`); RTF from Word with `\fcharset128` and `\'xx` byte pairs is untested (the code pages in `src/io/rtf/codepage.rs` are single-byte); the PDF writer embeds TrueType only, so the CFF fonts most CJK systems ship (Noto Sans CJK) are skipped, and Japanese wraps by character without the line-breaking rules (kinsoku). Every other document pair carries Japanese whole (pandoc, textutil, and LibreOffice sources to every document format, 100%). Chinese and Korean CID collections (GB1, CNS1, Korea1) have no CID table yet, so their fonts without ToUnicode read only through Unicode CMaps.
+- 2026-10-30 (updated 2026-10-31): Japanese, what is left. Read and written across every document and data pair (sources from pandoc, textutil, LibreOffice, Word-style RTF, and Shift_JIS/EUC-JP files; `tests/japanese_input.rs`, `tests/pdf_japanese.rs`). Left: Chinese and Korean CID collections (GB1, CNS1, Korea1) have no CID table, so their fonts without ToUnicode read only through Unicode CMaps, and their RTF double-byte pages only through `\uN`; name-keyed CFF fonts (Latin `.otf`) are not embedded; the PDF writer sets Japanese horizontally only and bold Japanese in the regular weight (one face per file: Hiragino W6 is a separate file); Western code pages (Latin-1, Windows-1252) in text, HTML, and CSV are still refused rather than detected.
 - 2026-09-24 (updated 2026-10-21, moved from Blockers): The Pages document paths' throughput against the shared Pages goal of 50 MB/s of input (`DOCS/benchmarks/pages-markdown.md`, `pages-html.md`, `pages-text.md`). The 1.0.0 run, on an M1 Max under load: memory passes on every document row (26 to 47 MB against 64), Word passes both ways (Pages -> Word 129 and 140 MB/s of input plus uncompressed output, Word -> Pages 67 and 89), and the text paths fail throughput at 26 to 33 MB/s (styled 29 to 33, prose 26 to 28). A real resume converts in 3 ms at 4.8 MB, inside both goals. Where the time goes on the dense shape, in-process (2026-09-24): package decode 10 ms (7.5 of it Snappy), document build 25 ms, the text writers 11 ms; wall clock adds about 10 ms of process start, file I/O, and first-touch page faults. Levers under Spikes. Pages -> Word's peak memory rose on every row since 2026-10-06 (prose 33.8 -> 46.8 MB), still inside the goal.
 - 2026-09-24 (updated 2026-10-21, moved from Blockers): `pages -> pages-json` misses the 50 MB/s goal at 24 and 26 MB/s of package bytes (`DOCS/benchmarks/pages-json.md`); the reverse direction passes at 146 and 239. Levers: the typed decode of the attribute tables written as JSON directly (the Spikes entry below), then `Tree::decode` itself and the JSON writer's per-field work.
 - 2026-10-21: Benchmark lines failing in the 1.0.0 run that passed in their previous blocks. The run was on the M1 Max with the machine in use (load average 7 to 26 on 10 cores); for several pairs every earlier block was on the Linux i7, so these are not like for like. An alternated A/B of ec68950 against the commit before it showed the registry change caused none of those it covered. Rerun on an idle machine before acting on any of them.

DOCS/formats/csv.md

@@ -3,7 +3,10 @@
 Delimited rows, read and written a record at a time in constant memory
 (`src/io/csv`): RFC 4180 quoting (a field with the delimiter, a quote, or
 a line break is quoted, a quote doubled), any line ending read, `\n`
-written, a UTF-8 byte order mark skipped. TSV is the same with a tab.
+written, a UTF-8 byte order mark skipped. Input that is not UTF-8 but
+reads as Japanese (Shift_JIS, as Excel saves a Japanese CSV, or EUC-JP)
+is decoded as it is read (`src/io/cjk`); a Western code page is still
+refused. TSV is the same with a tab.
 Every row format reaches every other (JSON, JSON Lines, Excel, Numbers,
 Markdown tables and the documents through them).
 

DOCS/formats/html.md

@@ -75,4 +75,6 @@ gets one.
   table cells.
 - Character references are decoded only with their `;`, as the writer
   produces them; legacy forms without it stay as written.
-- Encodings other than UTF-8 are not sniffed; the input is read as text.
+- Encodings: UTF-8 and UTF-16 (by its byte order mark), and Japanese
+  (Shift_JIS, EUC-JP, ISO-2022-JP): the `charset` the page declares, else
+  detected when the input is not UTF-8. Western code pages are refused.

DOCS/formats/pdf.md

@@ -71,8 +71,8 @@ are at any effort). Fast is the same as balanced (deflate's default is near the
 |---|---|
 | Input | any reader's Markdown event stream (markdown, html, text directly; docx and pages through the document model) |
 | Page | US Letter, one-inch margins; a page is written when the next line would pass the bottom margin, so one page is held |
-| Fonts | the base-14 Helvetica, Helvetica-Bold, -Oblique, -BoldOblique, Courier, Courier-Bold, WinAnsi, not embedded; widths from their AFM metrics. A character outside WinAnsi is set in a font installed on the machine that has it (`src/io/font/system.rs`: broad sans serifs first, then one per script, symbols, the large CJK fallbacks, then every other font, each probed by its character map alone), embedded as a subset (Type0, Identity-H, CIDFontType2 over a TrueType program holding only the glyphs used, with `/W` widths and a ToUnicode map). `--font file.ttf` (or `ConvertOptions::font`, which the WebAssembly module takes as bytes) sets all body text in that font instead, code staying in Courier. Only fonts with TrueType outlines are embedded; a character no such font has is `?` and reported |
-| Text | paragraphs 11/15 pt, broken first fit on real widths (a word wider than the line is split by characters); headings 20, 16, 13 pt bold (4 to 6 at 11), kept with the two lines after them; soft breaks are spaces, hard breaks end the line |
+| Fonts | the base-14 Helvetica, Helvetica-Bold, -Oblique, -BoldOblique, Courier, Courier-Bold, WinAnsi, not embedded; widths from their AFM metrics. A character outside WinAnsi is set in a font installed on the machine that has it (`src/io/font/system.rs`: broad sans serifs first, then the Japanese fonts (Hiragino Kaku Gothic, Noto Sans CJK, Source Han Sans, Yu Gothic, Meiryo, MS Gothic) and one per script, symbols, the large CJK fallbacks, then every other font, each probed by its character map alone), embedded as a subset (Type0, Identity-H, a CIDFontType2 over a TrueType program, or a CIDFontType0 over a CID-keyed CFF program (`FontFile3 /CIDFontType0C`, `src/io/font/cff.rs`: unused glyphs and subroutines blanked in place, each glyph's CID its id), holding only the glyphs used, with `/W` widths and a ToUnicode map). `--font file.ttf` (or `ConvertOptions::font`, which the WebAssembly module takes as bytes) sets all body text in that font instead, code staying in Courier. Fonts with TrueType outlines and CID-keyed CFF ones (every CJK `.otf`/`.ttc`) are embedded; name-keyed CFF fonts (Latin `.otf`) are not, and a character no embeddable font has is `?` and reported |
+| Text | paragraphs 11/15 pt, broken first fit on real widths (a word wider than the line is split by characters; Japanese breaks between characters, kept from starting a line with closing brackets, punctuation, small kana, or the long-vowel mark, and from ending one with an opening bracket: kinsoku); headings 20, 16, 13 pt bold (4 to 6 at 11), kept with the two lines after them; soft breaks are spaces, hard breaks end the line |
 | Inline | bold, italic, and code change the font; links are blue and carry a URI annotation over each piece; footnote references as `[n]`; task markers as `[x]`/`[ ]` |
 | Blocks | lists indented 18 pt a level with `•` or `n.`, tight ones close; block quotes indented 16 pt with a gray bar; code 9.5 pt Courier on a gray band, lines kept, long ones wrapped; rules; tables sized from their content (natural widths when they fit, else the longest word plus a share of the rest), cells wrapped and aligned, header bold, a grid, rows kept whole across pages |
 | Images | each picture a block of its own at its size, scaled down to the text width and the page, on a new page when it does not fit: Word, Pages, and RTF pictures from the document, at the size it shows each (a picture shown twice at two sizes takes each); Markdown and HTML pictures by path from the input's folder (on the command line; `ConvertOptions::base`) or by `data:` URI, at their recorded resolution or 96 pixels to the inch. A JPEG is embedded unchanged (`/DCTDecode`); PNG, WebP, BMP, TIFF, QOI, and ICO are read whole first (a picture that fails leaves nothing behind) and deflated with PNG predictors, alpha as a soft mask. A picture not found, unreadable (GIF has no reader yet), in a table cell, or by web address (not fetched) keeps its alt text in brackets, and is counted |
@@ -149,10 +149,11 @@ Markdown.
   headers, or footers. Complex scripts are set glyph by glyph in logical
   order: Hebrew and Arabic read left to right (no bidirectional
   reordering), Arabic letters take their isolated forms, and Indic
-  conjuncts are not formed (no shaping). Fonts with CFF outlines (most
-  CJK `.otf`/`.ttc`) are not embedded yet; another installed TrueType
-  font is used when one covers the text. A tab in running text is a
-  space. With `--font`, bold and italic body text is set in the one font
+  conjuncts are not formed (no shaping). Name-keyed CFF fonts (Latin
+  `.otf`) are not embedded; another installed font is used when one
+  covers the text. Japanese is set horizontally only (no vertical
+  writing), and bold Japanese in the regular weight (one face per font
+  file). A tab in running text is a space. With `--font`, bold and italic body text is set in the one font
   given.
 
 - WebP, QOI, TGA, Netpbm, and ICO record no resolution a page could use (WebP's lives in EXIF, which is not read), so their pages are a point per pixel.

DOCS/formats/rtf.md

@@ -38,7 +38,7 @@ as their text, aside).
 | Feature | RTF | Notes |
 |---|---|---|
 | Fonts, colours | `\fonttbl`, `\colortbl` | A font's `\fcharset` picks the code page of its `\'hh` bytes; Symbol and Wingdings bytes map to their private-use characters, and bullets among them to real bullets. |
-| Text | text, `\'hh`, `\uN` with `\ucN` | Code pages 437, 850, 874, 1250-1258, and Mac Roman; surrogate pairs; the Asian double-byte pages only through `\uN` (Word always writes it). |
+| Text | text, `\'hh`, `\uN` with `\ucN` | Code pages 437, 850, 874, 1250-1258, and Mac Roman; Japanese code page 932 (Shift_JIS byte pairs and half-width katakana, under `\ansicpg932` or a `\fcharset128` font); surrogate pairs; the Chinese and Korean double-byte pages only through `\uN` (Word always writes it). |
 | Styles | `\stylesheet`, `\sN`, `\csN`, `\sbasedon` | Style 0 is the default; a paragraph naming no style is in it. Built-in heading names give outline levels where the writer left them out. |
 | Character formatting | `\b \i \ul \strike \f \fs \cf \chcbpat \highlight \super \sub \up \dn \caps \scaps \v \lang \expndtw \charscalex` | Formatting in RTF is absolute: the reader keeps on a run only what differs from its paragraph's style, and gives text RTF's defaults where the style sets something the text does not. |
 | Paragraph formatting | `\q* \li \ri \fi \sb \sa \sl \slmult \keepn \keep \widctlpar \outlinelevel \pagebb \contextualspace \cbpat \brdr* \tx` | |
@@ -65,6 +65,7 @@ the header: LibreOffice spaces paragraphs by older rules without
 
 Equations (written as their text), charts, OLE objects (their result
 picture is kept), drawn shapes with neither fill nor outline, form fields
-(their shown text is kept), and text in double-byte code pages without
-`\uN` escapes (each such character becomes U+FFFD). RTFD (TextEdit's
+(their shown text is kept), and text in the Chinese and Korean
+double-byte code pages without `\uN` escapes (each such character
+becomes U+FFFD; Japanese, code page 932, is decoded). RTFD (TextEdit's
 bundle of RTF and attachments) is a later step.

README.md

@@ -4,7 +4,8 @@ Universal efficient file conversion. One binary, zero runtime dependencies.
 Apple Pages and Numbers, Word, Excel, Markdown, HTML, RTF, PDF, plain text,
 CSV, JSON, and images today; the formats and a map are in
 `DOCS/FORMATS.md`, and every path with its fidelity in
-`DOCS/CONVERSIONS.md`.
+`DOCS/CONVERSIONS.md`, and language and script support in
+`DOCS/LANGUAGES.md`.
 
     sublime convert input.csv output.json
     sublime check csv json