[Chiehx0220]

Summary

Key spec 3 for Chinese, as agreed in #1341. Keys change, so this is one bump (NeuroKeySpec.VERSION 2 to 3, schema 21).

• Hant folds to Hans in NeuroText.fold (resources/neuro/fold/hant-hans.txt, 2965 characters, 1:1, from OpenCC TSCharacters.txt, generated by scripts/neuro/gen_hant_hans.py). The fold is decided once per video: title, channel name and tags decide together, and kana in any of them keeps Han unfolded in every field of that video. The title's own spelling stays the display form.
• Single Han characters left by the word breaker are merged into candidates (Chunks.merged): adjacent singles, a word before them (大谷 + 翔 + 平) or a word after a lone character (李 + 永乐). Runs end at the language's break characters (resources/neuro/runbreak/zh.txt), are 2 to 5 characters and contain no stop or format word. A language without a break list merges nothing.
• Candidates count toward document frequency; the single characters do too. NeuroPhrases.isMergedRun promotes a candidate to a key at 3 documents and at least half of the documents holding its rarest single character.
• NeuroTokenizer(wordBreaker) and NeuroText.words(folded, breaker) take the breaker as a parameter. Tests use ICU4J (testImplementation), so the golden splits Chinese like a device.
• zh format words (NeuroFormats), video-structure stop words (scripts/neuro/add/zh.txt), freq/zh.txt (gen_wordfreq.py, then pruned to words ICU keeps whole by NeuroSpecificityTest with NEURO_WRITE_FREQ_ZH=1).
• REJECTION_BROAD_TOPICS gets 新闻, 音乐, 游戏, 科技, 电影, 教育, 美食, 旅游 (Hans). They are also in unstemmed.txt; no key changes.
• Migration (schema 21, NeuroKeyMigration.toKeySpec3): stored keys fold and merge, document frequency is summed, blocked topics fold, the Hant spelling is kept as the display form. Japanese viewers (languageProfile == "ja") are not folded, in the migration and at runtime (NeuroTokenizer.hanFolds).
• Blocked keywords match with both sides fully folded and kana ignored, so blocking 鬼滅 hides 鬼滅 アニメ 最新話. A blocked topic is stored the way the viewer's keys are (Hans, unchanged for ja).
• Only the zh lists fold Han when they load; the ja and ko stop lists, frequency tables and run breaks keep their kanji.
• Golden: 12 Chinese cases (with merged and singles). A JVM test covers the Chinese channel-name path (TVBS新聞 gives 新闻). NeuroHanFoldDecisionTest covers the per-video decision, description keywords, blocking and Japanese viewers.
• .github/nightly-notes/2026-10-20-neuro-key-spec-3.md.

Related issue

Part of #1341.

Change type

• Bug fix
• Feature
• Refactor or maintenance
• Build, packaging, or CI
• Documentation

Validation

• ./gradlew :app:assembleGithubDebug
• ./gradlew :app:testGithubDebugUnitTest :app:testFossDebugUnitTest: 4303 and 4260 tests, 1 failed in each: TranslationFormatStringsTest, because values-ru section_top_artists_in is %1$ (a Weblate commit on main, not touched here). Run with -Duser.language=en; two tests depend on the host locale. Includes the benchmark tests.
• ./gradlew ktlintCheck spotlessCheck, :app:compileGithubDebugAndroidTestKotlin
• I manually tested the affected behavior through the app UI.

Device checks (Pixel 6, Android 16, NeuroTokenizer called from the built APK through app_process):

• The 12 Chinese golden cases give the same chunks, modelPairs, formats, merged and singles as ICU4J (all 119 probed lines, titles and channel names, match).
• All 103 entries of topical/zh.txt return themselves (folded) from tokenizeChannelName.
• NeuroTokenizer.hanFoldOf gives NONE when the channel name or a tag has kana and ALWAYS otherwise; 鬼滅 無限城編 予告 reads 無限 under NONE and 无限 under ALWAYS.
• A ja viewer has hanFolds == false: 鬼滅 stays and 周杰倫 演唱會 merges nothing. The default viewer folds to 鬼灭 and merges 周杰伦 and 演唱会. stopList("ja") holds 私達, 貴方 and 貴方方.

Measurements

I asked an AI assistant to help collect the data and run the tests and measurements.

Corpus: 654 real titles from YouTube search (zh-TW and zh-CN queries, 425 with Hant characters) and 128 English titles. These are not a viewer's history.

Cost, JVM with ICU4J, median per call (spec 2 / spec 3), second of two runs. The runs differ by up to 7 µs for zh chunk and 5 µs for zh features, and by under 2 µs for the other calls. Spec 2 runs first in each pair, so JIT warm-up favours spec 3 slightly:

call	zh	en
chunk	18.8 / 14.2 µs	5.5 / 4.7 µs
extractFeatures	23.2 / 27.3 µs	10.0 / 10.0 µs
documentTerms	31.4 / 38.0 µs	15.3 / 14.4 µs
plantFilter	12.5 / 15.0 µs	4.3 / 4.4 µs

chunk allocates about 24% more for Chinese (20.6 / 25.5 KB), the same for English. NeuroText.fold on non-ASCII text takes about 360 ns more with the Hant fold (0.85 to 1.07 us / 1.23 to 1.43 us).

Pixel 6, chunk median over two rounds (spec 2 / spec 3): zh about 63 to 70 / 62 to 63 µs, en about 19.4 to 19.5 / 18.8 to 19.0 µs. The phone read slower than in the first measurement (zh spec 2 was about 50 µs), so compare the two columns within one run.

IDF table after 654 zh titles: 6832 / 8804 entries (+28%: 1040 single characters, 1991 merged candidates). English is unchanged. IDF_MAX_KEYS is 20,000, so a Chinese history reaches the cap sooner, and candidates seen once can be capped out before they reach 3 documents. A simulation with the cap scaled down (4000/3200, an AI assistant ran it) leaves candidates at a count of 3 or more about as often as other words: 83.7% against 84.7%.

Promotion on this corpus (3 documents):

min share	max 4	max 5
0	87	90
0.3	80	83
0.5 (chosen)	64	67
0.7	50	51
0.9	38	38

No promoted candidate is longer than 5. I read the 67 promoted at 0.5/5 myself: about 80% are names or terms (周杰伦, 李永乐, 台积电, 黄仁勋, 原神, 零基础), about 20% are fragments across a word (旅游全, 首精选, 科学小). This is my judgement, not labelled data, and the thresholds are not tuned beyond this corpus.

Folding: distinct title words across the zh-TW and zh-CN groups 2817 to 2482 (-12%), 247 more words match across spellings.

Risk and compatibility

• Key spec 3: brain sync pauses against peers on spec 2 until Desktop ships 3 (as discussed in [Bug]: FlowNeuro mishandles scripts written without spaces (Chinese, Japanese, Thai): names are dropped or split, channel names give no signal #1341). NeuroContentStore vectors on spec 2 are dropped and refill.

• Schema 21 migration; a copy of the brain is kept before it (keepCopyBeforeMigration). Folding is not reversible: a stored Hant key becomes Hans, its spelling is kept as a display form.

• Removed from topical/zh.txt: 心理學, 紀錄片, 摩托車, 腳踏車, 自行車, 行銷. Folded to Hans, ICU4J splits them, so they could never match. They were checked in Hant on the device in feat(neuro): split channel names into words and add Chinese topic words #1350.

• Hans keys follow the ICU dictionary of the Hans spelling, which differs from Hant for some words.

• The fold only sees kana. A Japanese title with no kana in its title, channel name or tags folds like Chinese for a non-ja viewer (for example 鬼滅 無限城編 予告). Telling it apart needs more than the video's own text.

• A ja viewer folds nothing, Chinese videos included.

• New test dependency: ICU4J 77.1 (testImplementation only; it was already in the Gradle cache). The OpenCC attribution is added to assets/licenses/notices.txt.

• The change does not introduce secrets, private data, or unexpected telemetry.

• No new user-facing text.

• Dependency changes are limited to the test-only ICU4J.

• No Room schema change.

• Breaking changes are listed above (key spec 3 sync pause).

[A-EDev]

Thanks, this covers everything from #1341: the fold, merged runs with a break list per language, ICU4J in tests, Chinese golden cases, the migration, freq/zh.txt and the Hans broad topics. One thing needs to change before I merge, plus a few smaller points.

Needs a change: decide the fold once per video, not per string

NeuroText.simplified skips the fold when the string it gets has kana. But the title, tags, channel name, description and what the viewer types are folded separately, so Japanese is not unchanged. I ran these on the branch with IcuWordBreaker:

• 鬼滅の刃 無限城編 予告 gives the key 無限, and 鬼滅 無限城編 予告 gives 无限. The same word becomes two keys depending on whether the title has kana.
• Blocking 鬼滅 stores 鬼灭, so 鬼滅 アニメ 最新話 is no longer hidden (isBlockedByText returns false; it is hidden on main). addBlockedTopic folds whatever the viewer's language is.
• The migration keeps kanji keys for languageProfile == "ja", but nothing at runtime checks the profile, so their new kanji-only titles and tags still fold.
• Three kanji-only entries in stop/ja.txt (私達, 貴方, 貴方方) fold when the list loads and stop matching Japanese titles.

What I'd like:

1. Decide the fold once per video (title, channel name and tags together) and use that decision for every string of that video.
2. Match blocked keywords with both sides fully folded, ignoring kana, since that's only text matching and not keys.
3. Make the migration and the runtime agree on Japanese viewers (both check the profile, or neither does), and don't Han-fold the ja/ko lists when they load.

The Japanese part I add later needs the same per-video decision, so it's worth getting right now.

Smaller points

• IDF cap: the 28% growth meets an LFU cap at 20,000 entries. Single characters always survive and young candidates are dropped first, so a heavy Chinese viewer may rarely reach 3 documents for a name. Could candidates get some protection from the cap, or could you show it doesn't matter in practice?
• A promoted run keeps its words: when 李永乐 is promoted, 永乐 still looks like it is learned on its own. English phrases take their words. Is that intended?
• Character-only fold: 乾隆皇帝 gives the merged candidate 干隆. It only affects keys and the display keeps the spelling, so a short exception list is enough if you want to fix it. Not blocking.
• Nightly note: since today, every PR with a change users can notice adds one file in .github/nightly-notes/ (see the README there). This one would be 2026-10-20-neuro-key-spec-3.md with an ## Improved line about recommendations for Chinese titles (#1341). The Checks job validates it.

Timing stays as discussed: I'll merge right after the next release.

[Chiehx0220]

Updated, and merged current main.

Fold decision

• One decision per video: title, channel name and tags decide together (NeuroTokenizer.hanFoldOf). Kana in any of them keeps Han unfolded in every field of that video; otherwise every field folds. The description's context words use the same decision.
• Blocking reads both sides fully folded with kana ignored (HanFold.ALWAYS), so blocking 鬼滅 hides 鬼滅 アニメ 最新話. Matching stays token-based.
• Migration and runtime share NeuroTokenizer.hanFolds (spec 3 and viewer not ja). The ja/ko stop lists, frequency tables and run breaks load without the Han fold, so 私達, 貴方, 貴方方 match again.
• Added .github/nightly-notes/2026-10-20-neuro-key-spec-3.md.

Known limit: the decision only sees kana. A Japanese title written in kanji alone (鬼滅 無限城編 予告) has none, so for a non-ja viewer it folds like Chinese. Telling it apart needs more than the video's own text (channel-level memory or YouTube's language data), so I left it for the Japanese part.

Smaller points

• IDF cap: I could not test a real heavy history, so I asked an AI assistant to run a simulation: the 654 zh titles in random order (30 runs), with the cap scaled down to 4000/3200 and 6000/4800 entries. Share of terms seen in 3 or more titles that still have a count of 3 or more at the end:

  cap	candidates	other words	single characters
  4000/3200	83.7%	84.7%	87.6%
  6000/4800	96.2%	96.0%	96.8%

  The cap drops entries seen once whatever they are, so candidates lose no more than other words. That is all this shows; it says nothing about the real 20,000 limit on a long history. I left the cap alone.

• A promoted run keeps its words: this was not a decision I made, I had not looked at it. I measured it on the same titles: in 150 of 473 promoted-run occurrences (31 distinct runs) a word inside the run is also a key. Mostly that word is the real topic, unlike the words of an English phrase (学钢琴 keeps 钢琴, 买手机 keeps 手机, 上海申花 keeps 上海). For personal names the breaker splits wrongly and the leftover is noise: 李永乐 keeps 永乐, 彼得森 keeps 彼得, 切切里 keeps 切切 (by my reading about 3 of the 31). Dropping the word would lose the first group to fix the second, and telling a name from a compound is not something this PR can do, so I left it. If you want the English behaviour here, it is a small change in extractFeatures, but it would change keys and the golden.

• 乾隆: 乾隆 and 干隆 both fold to 干隆, so Hant and Hans titles match each other; only the key looks odd, and the display keeps the spelling. I know of no 乾X word with a real 干X twin to collide with (from reading, not an exhaustive check), so I did not add an exception list.

Tests: NeuroHanFoldDecisionTest covers the decision, blocking, ja viewer and lists. ktlint and the unit tests pass on both flavors, except TranslationFormatStringsTest: values-ru section_top_artists_in is %1$ (from a Weblate commit on main, not touched here).

Side note, not for this PR: extractDescriptionKeywords drops any line equal to its own uppercase, which holds for every line of an unicased script, so Chinese and Japanese descriptions never yield keywords.

[A-EDev]

Thanks, this does it. I ran my earlier checks again on 084a0ebce: blocking 鬼滅 now hides 鬼滅 アニメ 最新話, a kanji-only title with kana in a tag keeps 無限, a ja viewer folds nothing, and 私達 is back in the Japanese stop list. Your answers on the IDF cap, promoted runs keeping their words and 乾隆 all make sense; leaving them as they are is fine.

Two things I'll pick up myself with the Japanese part:

• NeuroChannelKnowledge.profileFromChannelTags still folds each channel tag on its own, so a kanji-only tag like 鬼滅 becomes 鬼灭 and splits into single characters. It needs the same once-per-channel decision.
• The golden only holds single strings, so the per-video kana rule needs a title + tags case for the Desktop port.

I filed the description bug you found as #1395. Merging now.