Commit fe0e85d
committed
fix(supertonic): decompose Cyrillic й/ё/ї into base letter + combining mark
The Supertonic unicode indexer maps most basic Cyrillic letters, but
й/Й, ё/Ё, and ї/Ї are absent (mapped to -1 in unicode_indexer.json),
causing "unicode indexer has no entry for codepoint N" errors for any
Russian or Ukrainian text containing these letters — both "ru" and
"uk" are listed as supported languages.
The indexer does contain entries for the combining breve (U+0306) and
combining diaeresis (U+0308), matching each letter's canonical NFD
decomposition (й = и + breve, ё = е + diaeresis, ї = і + diaeresis).
Decompose these codepoints the same way the tokenizer already handles
Hangul and Japanese kana dakuten/handakuten forms.1 parent 1a66a3f commit fe0e85d
1 file changed
Lines changed: 34 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
139 | 139 | | |
140 | 140 | | |
141 | 141 | | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
142 | 173 | | |
143 | 174 | | |
144 | 175 | | |
145 | 176 | | |
146 | 177 | | |
147 | 178 | | |
148 | 179 | | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
149 | 183 | | |
150 | 184 | | |
151 | 185 | | |
| |||
0 commit comments