modernize to 2.0: drop Python 2, refresh Unicode data through 17.0.0, fix collator selection - #30
Open
jtauber wants to merge 6 commits into
Open
modernize to 2.0: drop Python 2, refresh Unicode data through 17.0.0, fix collator selection#30jtauber wants to merge 6 commits into
jtauber wants to merge 6 commits into
Conversation
…s, and dropped Python 2
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A modernization pass bringing pyuca up to current Python packaging, tooling, and
Unicode data, released as 2.0.0.
Why
The default
Collatorwas chosen fromsys.version_infovia a branch thatstopped at Python 3.6, so Python 3.7–3.14 all silently used UCA 9.0.0 (2016)
even though their
unicodedatais 14.0–16.0. Becausesort_keynormalizes withthe standard library's
unicodedata, the collator should track the runtime'sUnicode version — which is now what it does.
What changed
unicodedata.unidata_versioninstead ofsys.version_info. On Python 3.14 the default is nowCollator_16_0_0(was
Collator_9_0_0).suites added for 13.0.0, 14.0.0, 15.0.0, 15.1.0, 16.0.0 and 17.0.0. All
tables are gzipped;
scripts/fetch_ducet.pyregenerates them.flags are gone, replaced by fixed Han block boundaries plus the data file's
own
@implicitweightsdirectives (handles Tangut/Nushu/Khitan and 17.0.0'sreassigned bases).
py.typed, and pyright (standard mode).pyproject.toml+ hatchling + uv;setup.py,setup.cfg,tox.ini,.travis.yml,MANIFEST.inremoved.is enforced at 100%.
Windows spot checks).
Performance
Collator()drops from~52 ms to ~1.6 µs.
collation_elementsis now O(n) (index cursor instead oflist.pop(0)andslicing); long-string sort keys are ~5× faster and no longer quadratic.
Validation
100% UCA conformance (Non-ignorable) on every version run on its matching
interpreter:
ruff,pyright, andpytest --cov(100%) all pass. The built wheel installsclean and ships the gzipped tables and
py.typed.Breaking changes
for characters added since UCA 9.0.0 will differ from 1.x on modern Pythons.
Known limitations
fixtures are bundled but not yet exercised.
unicodedatamatches it.Collator_17_0_0is included ahead of a CPythonrelease shipping Unicode 17.