Skip to content

modernize to 2.0: drop Python 2, refresh Unicode data through 17.0.0, fix collator selection - #30

Open
jtauber wants to merge 6 commits into
masterfrom
modernize-2.0
Open

modernize to 2.0: drop Python 2, refresh Unicode data through 17.0.0, fix collator selection#30
jtauber wants to merge 6 commits into
masterfrom
modernize-2.0

Conversation

@jtauber

@jtauber jtauber commented Jul 25, 2026

Copy link
Copy Markdown
Owner

A modernization pass bringing pyuca up to current Python packaging, tooling, and
Unicode data, released as 2.0.0.

Why

The default Collator was chosen from sys.version_info via a branch that
stopped at Python 3.6, so Python 3.7–3.14 all silently used UCA 9.0.0 (2016)
even though their unicodedata is 14.0–16.0. Because sort_key normalizes with
the standard library's unicodedata, the collator should track the runtime's
Unicode version — which is now what it does.

What changed

  • Version selection driven by unicodedata.unidata_version instead of
    sys.version_info. On Python 3.14 the default is now Collator_16_0_0
    (was Collator_9_0_0).
  • Unicode data refreshed — DUCET tables and Non-ignorable conformance
    suites added for 13.0.0, 14.0.0, 15.0.0, 15.1.0, 16.0.0 and 17.0.0. All
    tables are gzipped; scripts/fetch_ducet.py regenerates them.
  • Generalized implicit weights — the hardcoded per-version CJK boolean
    flags are gone, replaced by fixed Han block boundaries plus the data file's
    own @implicitweights directives (handles Tangut/Nushu/Khitan and 17.0.0's
    reassigned bases).
  • Dropped Python 2; now requires Python 3.11+. Full type hints,
    py.typed, and pyright (standard mode).
  • Packaging moved to pyproject.toml + hatchling + uv; setup.py,
    setup.cfg, tox.ini, .travis.yml, MANIFEST.in removed.
  • Tests migrated to pytest with a parametrized conformance suite; coverage
    is enforced at 100%.
  • CI moved from Travis to GitHub Actions (Python 3.11–3.14, plus macOS and
    Windows spot checks).

Performance

  • Parsed DUCET tables are cached per file: a second Collator() drops from
    ~52 ms to ~1.6 µs.
  • collation_elements is now O(n) (index cursor instead of list.pop(0) and
    slicing); long-string sort keys are ~5× faster and no longer quadratic.

Validation

100% UCA conformance (Non-ignorable) on every version run on its matching
interpreter:

Python Unicode conformance
3.10 13.0.0 207268 / 0 fail
3.11 14.0.0 211448 / 0 fail
3.12 15.0.0 180108 / 0 fail
3.13 15.1.0 180158 / 0 fail
3.14 16.0.0 206315 / 0 fail

ruff, pyright, and pytest --cov (100%) all pass. The built wheel installs
clean and ships the gzipped tables and py.typed.

Breaking changes

  • Requires Python 3.11+.
  • The default collator now tracks the runtime's Unicode version, so sort keys
    for characters added since UCA 9.0.0 will differ from 1.x on modern Pythons.

Known limitations

  • Only the Non-ignorable variable-weighting setting is implemented; the Shifted
    fixtures are bundled but not yet exercised.
  • A version's conformance is only guaranteed on an interpreter whose
    unicodedata matches it. Collator_17_0_0 is included ahead of a CPython
    release shipping Unicode 17.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant