normalize-uk

Ukrainian text normalization, tokenization and sentence splitting, in Rust.

The crate turns machine-readable spellings into the words a Ukrainian speaker would say — numbers, dates, times, ranges, units, currencies, abbreviations, identifiers and web addresses — and splits text into sentences and tokens.

[dependencies]
normalize-uk = "0.5"

use normalize_uk::{rozpodil, uktextnorm};

assert_eq!(uktextnorm::normalize("5 кг"), "п'ять кілограмів");
assert_eq!(uktextnorm::number_to_words(123), "сто двадцять три");

let sentences = rozpodil::split_sentences("Привіт! Це тест.");
assert_eq!(sentences.iter().map(|s| s.text).collect::<Vec<_>>(), ["Привіт!", "Це тест."]);

Normalization

normalize applies the default options. Use normalize_preset for a named bundle, or normalize_with for full control.

use normalize_uk::uktextnorm::{normalize, normalize_preset, normalize_with,
                               NormalizeOptions, NormalizePreset, RangeStyle};

assert_eq!(normalize("01.05.2024"), "перше травня дві тисячі двадцять четвертого року");

// Presets bundle the options for a use case.
let spoken = normalize_preset("5–7 кг", NormalizePreset::TtsFriendly);
assert_eq!(spoken, "від п'яти до семи кілограмів");

// Or start from a preset and adjust.
let options = NormalizeOptions {
    range_style: RangeStyle::Compact,
    ..NormalizeOptions::preset(NormalizePreset::TtsFriendly)
};
assert_eq!(normalize_with("5–7 кг", &options), "п'ять сім кілограмів");

The presets are Default, TtsFriendly (everything spelled out, for speech synthesis), Conservative (changes as little as possible) and SearchIndexing (keeps tokens searchable rather than speakable).

Ambiguity controls

NormalizeOptions keeps backwards-compatible defaults while letting callers resolve ambiguous input explicitly:

• colon_style — contextual clock/ratio detection, forced clock, or forced ratio.
• numeric_date_order — day-month-year, month-day-year, or preserving dates where both fields are at most 12.
• currency_symbol_policy — assume the common currency for $ and ¥, or preserve those ambiguous symbols.

Reporting what had to be guessed

flag_uncertain reports every place the reading involved a judgement call. flag_uncertain_with takes options and omits the warnings they already resolve; invalid-value diagnostics always remain.

use normalize_uk::uktextnorm::flag_uncertain;

let spans = flag_uncertain("Дата 30.02.2024");
assert!(spans.iter().any(|s| s.text == "30.02.2024"));

UncertainSpan::start and stop are byte offsets, so &source[span.start..span.stop] == span.text.

Numbers

use normalize_uk::uktextnorm::{number_to_ordinal_words, number_to_words,
                               number_to_words_case, number_to_words_digit_by_digit,
                               GrammaticalCase, OrdinalForm};

// A bare "одна" before "тисяча" is dropped, as Ukrainian usage requires.
assert_eq!(number_to_words(1_234), "тисяча двісті тридцять чотири");
assert_eq!(number_to_words(2_234), "дві тисячі двісті тридцять чотири");
assert_eq!(number_to_ordinal_words(21, OrdinalForm::NomF), "двадцять перша");
assert_eq!(number_to_words_case(500, GrammaticalCase::Genitive), "п'ятисот");
assert_eq!(number_to_words_digit_by_digit("007"), "нуль нуль сім");

Values above MAX_SPELLED_NUMBER (999_999_999_999_999_999) are read digit by digit instead of spelled out.

Segmentation

Both entry points borrow from the input and report byte offsets, so &text[span.start..span.stop] == span.text always holds.

use normalize_uk::rozpodil::{split_sentences, tokenize};

let text = "м. Київ, вул. Хрещатик, 1. Зустріч о 10:30.";
assert_eq!(split_sentences(text).len(), 2);
assert_eq!(tokenize("П'ять зв'язків.").len(), 3);

Custom vocabulary

Pass a map of preferred readings to override the built-in brand and English-word lexicons for a single call. The normalize_english_words switch also controls custom readings.

use normalize_uk::uktextnorm::{normalize_with, NormalizeOptions};

let options = NormalizeOptions {
    vocabulary: [("google", "гуголь"), ("acme", "акме")]
        .into_iter()
        .map(|(k, v)| (k.to_owned(), v.to_owned()))
        .collect(),
    ..NormalizeOptions::default()
};
assert_eq!(normalize_with("Google і Acme", &options), "гуголь і акме");

Readings can also be loaded from a UTF-8 TSV file with the same columns as data/lexicons/brands.tsv:

latin	cyrillic
Acme	акме
Google	гуголь

use normalize_uk::uktextnorm::load_vocabulary_tsv;

let words = load_vocabulary_tsv("my_words.tsv")?;
# Ok::<(), normalize_uk::uktextnorm::VocabularyError>(())

Latin keys are single ASCII words, matched without regard to case.

Currency and cryptocurrency coverage

Normalization covers 178 ISO 4217 List One codes from the 2026-01-01 data snapshot, including their 0-, 2-, 3- and 4-digit minor-unit rules. More than 70 common cryptocurrency and finance tickers have natural Ukrainian readings. Other 2–10 character uppercase alphanumeric tickers are spelled out after amounts and when paired with a known asset, so newly introduced assets do not require an immediate release. Prefix and suffix amounts, localized thousands separators, signs, decimals and the ₿ symbol are all supported.

Lexicons

The lexicons under data/lexicons/ are the source of truth. They are embedded at compile time with include_str! and parsed once on first use, so editing a TSV and rebuilding is all that is needed to change a reading. cargo test checks each table for duplicate keys, out-of-range values and empty columns.

Development

cargo test
cargo clippy --all-targets
cargo fmt --all

The test suite has four parts:

• tests/conformance.rs — the reference assertion suite, reporting every failure at once rather than stopping at the first.
• tests/golden.rs — the TSV corpora under tests/data/, including an idempotence check on the sentence corpus.
• tests/robustness.rs — awkward input must not panic, empty the text, or produce spans that disagree with the source.
• tests/rozpodil.rs and tests/vocabulary.rs — segmentation and vocabulary loading.

License

MIT. See LICENSE.