Skip to content

Commit fe0e85d

Browse files
committed
fix(supertonic): decompose Cyrillic й/ё/ї into base letter + combining mark
The Supertonic unicode indexer maps most basic Cyrillic letters, but й/Й, ё/Ё, and ї/Ї are absent (mapped to -1 in unicode_indexer.json), causing "unicode indexer has no entry for codepoint N" errors for any Russian or Ukrainian text containing these letters — both "ru" and "uk" are listed as supported languages. The indexer does contain entries for the combining breve (U+0306) and combining diaeresis (U+0308), matching each letter's canonical NFD decomposition (й = и + breve, ё = е + diaeresis, ї = і + diaeresis). Decompose these codepoints the same way the tokenizer already handles Hangul and Japanese kana dakuten/handakuten forms.
1 parent 1a66a3f commit fe0e85d

1 file changed

Lines changed: 34 additions & 0 deletions

File tree

src/models/supertonic/tokenizer_text.cpp

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -139,13 +139,47 @@ bool append_japanese_kana_decomposition(uint32_t codepoint, std::vector<uint32_t
139139
return false;
140140
}
141141

142+
bool append_cyrillic_decomposition(uint32_t codepoint, std::vector<uint32_t> & out) {
143+
constexpr uint32_t kBreve = 0x0306;
144+
constexpr uint32_t kDiaeresis = 0x0308;
145+
static constexpr std::pair<uint32_t, uint32_t> kBreveLetters[] = {
146+
{0x0439, 0x0438}, // й -> и
147+
{0x0419, 0x0418}, // Й -> И
148+
};
149+
static constexpr std::pair<uint32_t, uint32_t> kDiaeresisLetters[] = {
150+
{0x0451, 0x0435}, // ё -> е
151+
{0x0401, 0x0415}, // Ё -> Е
152+
{0x0457, 0x0456}, // ї -> і
153+
{0x0407, 0x0406}, // Ї -> І
154+
};
155+
156+
for (const auto & [composed, base] : kBreveLetters) {
157+
if (codepoint == composed) {
158+
out.push_back(base);
159+
out.push_back(kBreve);
160+
return true;
161+
}
162+
}
163+
for (const auto & [composed, base] : kDiaeresisLetters) {
164+
if (codepoint == composed) {
165+
out.push_back(base);
166+
out.push_back(kDiaeresis);
167+
return true;
168+
}
169+
}
170+
return false;
171+
}
172+
142173
std::vector<uint32_t> decompose_known_text_codepoints(const std::vector<uint32_t> & codepoints) {
143174
std::vector<uint32_t> out;
144175
out.reserve(codepoints.size() * 2);
145176
for (const uint32_t codepoint : codepoints) {
146177
if (append_japanese_kana_decomposition(codepoint, out)) {
147178
continue;
148179
}
180+
if (append_cyrillic_decomposition(codepoint, out)) {
181+
continue;
182+
}
149183
append_hangul_decomposition(codepoint, out);
150184
}
151185
return out;

0 commit comments

Comments
 (0)