Strengthen the abjad `context=True` mode with deterministic context-aware disambiguation

[raeq]

Goal / statement of intent

Turn the abjad context=True mode from a flat dictionary lookup into a context-aware disambiguation layer that measurably improves full-sentence Arabic / Persian / Hebrew (and Arabizi) output without sacrificing translit's determinism, zero runtime dependencies, or linear-time guarantees. Success makes translit the first deterministic library offering context-aware abjad romanization.

Non-goals: a neural diacritizer; full morphological analysis; changing the default (non-context) path or any public API signature.

Success metrics (how we measure done)

• Primary: ≥ 10% relative reduction in word-level error (WER, or diacritic error rate where appropriate) vs. the current flat-dictionary context=True baseline on a held-out abjad eval set. (Reference: Kirov et al. 2024 obtain 18.6% relative mean WER reduction from context.)
• Diagnostics: report CCPD-style Selection Rate and Partial-/Reader-DER so reviewers can see where context helps.
• Invariants preserved: determinism (same input → same output, exhaustively checked), idempotence, and ASCII-output invariants stay green; no new runtime dependency; throughput regression within an agreed bound (e.g. ≤ 20% on the abjad path, 0% elsewhere).
• Reproducibility: any new corpus/dictionary artifact is produced by a pinned, checksummed bootstrap (same contract as the existing dict bootstrap).

Acceptance criteria / definition of done

• Context-contrastive (with- vs without-context) selection implemented behind the existing context=True flag; no API break.
• Held-out eval harness + baseline numbers committed; primary metric target met and reported in the PR.
• Determinism / idempotence / ASCII invariants and the perf bound gated in CI.
• Corpus/dictionary build reproducible (pinned source + SHA256).
• Docs state the measured quality and the (still best-effort) scope.

---

Summary

translit's abjad mode (transliterate(text, context=True) for Arabic / Persian / Hebrew, src/context.rs) is today a dictionary-backed readability aid, explicitly "not a romanization standard." The 2024–2026 literature shows that context-conditioned disambiguation — not larger 1:1 tables — is what closes the abjad quality gap, and that several of these techniques are compatible with translit's deterministic, auditable, zero-runtime-dependency design.

Product–market fit

• Who: library cataloguers, search/IR teams, and NLP/TTS practitioners working with Arabic/Persian/Hebrew (and Romanized Arabizi) who need readable, vowelized Latin output, not just consonant skeletons; downstream LLM/IR pipelines that lose meaning when abjad romanization is ambiguous.
• Pain: abjads omit short vowels, so flat dictionary lookup cannot resolve homographs that only sentence context disambiguates — current output is under-vocalized or wrong on real sentences.
• Why translit: it already ships the abjad mode, a deterministic table/PHF engine, and a machine-checked invariant suite. A context-contrastive layer stays auditable and pure-Rust-friendly, differentiating translit from neural-only diacritizers that carry GB-scale model dependencies and non-deterministic output.
• Wedge / adoption: ship as an opt-in upgrade to the existing context=True path (no API break). Positions translit as the only deterministic library offering context-aware abjad readability.

Academic grounding

• Kirov et al. 2024, "Context-aware Transliteration of Romanized South Asian Languages," Computational Linguistics 50(2):475–534 — full-sentence transliteration using non-parallel mono-script resources + finite-state (FST) ensembles; 18.6% relative mean WER reduction over prior best on all 12 Dakshina languages. Demonstrates context can be added without parallel corpora and largely via finite-state methods — squarely in translit's table/FST world. https://aclanthology.org/2024.cl-2.2/
• ElNokrashy & AlKhamissi 2024, "Partial Diacritization: A Context-Contrastive Inference Approach" (CCPD), ArabicNLP 2024 — run inference with vs. without context and mark only the characters where the two diverge; integrates with existing diacritizers and ships quality indicators (SR, P-DER, R-DER). The contrast step has a natural deterministic analogue. https://arxiv.org/abs/2401.08919
• Elgamal et al. 2024, "Arabic Diacritics in the Wild" — dataset + analyze-and-disambiguate method that exploits naturally occurring partial diacritics across six genres. https://arxiv.org/abs/2406.05760
• Bondok et al. 2025, "Proper Noun Diacritization for Arabic Wikipedia" — the transliterated-named-entity × diacritization intersection (GPT-4o reaches only 73%), motivating a dedicated NE path. https://arxiv.org/abs/2505.02656
• Context: Jayakumar et al. 2026, "Scripts Through Time: A Survey of the Evolving Role of Transliteration in NLP" frames context-conditioning as the field's direction. https://arxiv.org/abs/2604.18722

Full texts are attached in the project literature graph (Consensus sweep, 2026-06-08).

SWOT

Strengths — builds on an already-shipped feature; FST/contrastive methods fit translit's deterministic, no-deps, invariant-driven design; strong external evidence of large WER gains; differentiates from neural-only tools.
Weaknesses — needs per-language mono-script corpora + dictionaries (build, license, binary size); context logic is more complex than table lookup and harder to keep linear/O(1); abjad quality is inherently bounded without morphological analysis.
Opportunities — become the reference deterministic abjad romanizer; reuse Dakshina and "diacritics in the wild" datasets; extends naturally to Arabizi/Greeklish; strong synergy with the benchmarking and reversibility directions.
Threats — neural diacritizers keep improving and may set expectations translit deliberately won't meet; corpus licensing/reproducibility burden (cf. the gitignored dictionary bootstrap contract in CLAUDE.md); scope creep toward a full morphological analyzer.