[5.3.6 patch] Fix/royin ru indexerror #1473 - #1480
Closed
wannaphong wants to merge 144 commits into
Closed
Conversation
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 7 to 8. - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](actions/download-artifact@v7...v8) --- updated-dependencies: - dependency-name: actions/download-artifact dependency-version: '8' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
แคคตัส = cactus
…licate Three fixes in soundex module: - Filter karan spaces before truncating to requested length - Add ฏ,ฑ,ถ,ธ to _C2 (dental class, same sound as ท,ด) - Remove duplicate ข in prayut_and_somchaip _C2
nighit() crashed with IndexError when w2 had no consonants.
check_sara("") and check_marttra("") crashed accessing word[-1].
Add input validation with clear error messages.
- Bundle deepcut.onnx model (from LEKCut) in pythainlp/corpus/ - Add deepcut_onnx entry to default_db.json - Rewrite pythainlp/tokenize/deepcut.py with ONNX inference; no longer depends on the TF-based deepcut package - Move deepcut tests from noauto_tensorflow/ to noauto_onnx/ - Remove deepcut from pyproject.toml TF deps and mypy overrides - Update CHANGELOG.md Agent-Logs-Url: https://github.com/PyThaiNLP/pythainlp/sessions/ff10e57e-b105-4263-9f34-12b2e67cd4ab Co-authored-by: wannaphong <8536487+wannaphong@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Bumps [dieghernan/cff-validator](https://github.com/dieghernan/cff-validator) from 4 to 5. - [Release notes](https://github.com/dieghernan/cff-validator/releases) - [Changelog](https://github.com/dieghernan/cff-validator/blob/main/CHANGELOG.md) - [Commits](dieghernan/cff-validator@114aae5...54c8e73) --- updated-dependencies: - dependency-name: dieghernan/cff-validator dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
…ption-type fix: catch TypeError instead of ValueError in sent_tokenize
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Added type checks and string handling for inputs.
PR PyThaiNLP#1377 adds guardrails to nighit(). This PR add tests to those guardrails. Signed-off-by: Arthit Suriyawongkul <arthit@gmail.com>
Signed-off-by: Arthit Suriyawongkul <arthit@gmail.com>
Signed-off-by: Arthit Suriyawongkul <arthit@gmail.com>
fix: guard nighit, check_sara, check_marttra against empty input
Signed-off-by: Arthit Suriyawongkul <arthit@gmail.com>
Signed-off-by: Arthit Suriyawongkul <arthit@gmail.com>
Add ValueError tests for nighit (PR 1377)
… ReDoS exponential backtracking (PyThaiNLP#1400)
Bumps [DavidAnson/markdownlint-cli2-action](https://github.com/davidanson/markdownlint-cli2-action) from 23 to 23.1.0. - [Release notes](https://github.com/davidanson/markdownlint-cli2-action/releases) - [Commits](DavidAnson/markdownlint-cli2-action@v23...v23.1.0) --- updated-dependencies: - dependency-name: DavidAnson/markdownlint-cli2-action dependency-version: 23.1.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
…tions/DavidAnson/markdownlint-cli2-action-23.1.0 Bump DavidAnson/markdownlint-cli2-action from 23 to 23.1.0
The second .replace mid-tone removal on line 31 is identical to line 30 and is dead code. After line 30 removes all mid-tone characters, line 31 has no effect.
…tions/actions/stale-10.4.0 Bump actions/stale from 10.3.0 to 10.4.0
…formers-5.13.0 Bump transformers from 5.12.1 to 5.13.0
Bumps [transformers](https://github.com/huggingface/transformers) from 5.13.0 to 5.14.1. - [Release notes](https://github.com/huggingface/transformers/releases) - [Commits](huggingface/transformers@v5.13.0...v5.14.1) --- updated-dependencies: - dependency-name: transformers dependency-version: 5.14.1 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [dieghernan/cff-validator](https://github.com/dieghernan/cff-validator) from 5.0.1 to 5.1.1. - [Release notes](https://github.com/dieghernan/cff-validator/releases) - [Changelog](https://github.com/dieghernan/cff-validator/blob/main/CHANGELOG.md) - [Commits](dieghernan/cff-validator@54c8e73...d8f8582) --- updated-dependencies: - dependency-name: dieghernan/cff-validator dependency-version: 5.1.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [DavidAnson/markdownlint-cli2-action](https://github.com/davidanson/markdownlint-cli2-action) from 23.2.0 to 24.0.0. - [Release notes](https://github.com/davidanson/markdownlint-cli2-action/releases) - [Commits](DavidAnson/markdownlint-cli2-action@v23.2.0...v24.0.0) --- updated-dependencies: - dependency-name: DavidAnson/markdownlint-cli2-action dependency-version: 24.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [sentencepiece](https://github.com/google/sentencepiece) from 0.2.1 to 0.2.2. - [Release notes](https://github.com/google/sentencepiece/releases) - [Commits](google/sentencepiece@v0.2.1...v0.2.2) --- updated-dependencies: - dependency-name: sentencepiece dependency-version: 0.2.2 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [symspellpy](https://github.com/mammothb/symspellpy) from 6.9.0 to 6.10.0. - [Release notes](https://github.com/mammothb/symspellpy/releases) - [Changelog](https://github.com/mammothb/symspellpy/blob/master/CHANGELOG.md) - [Commits](mammothb/symspellpy@v6.9.0...v6.10.0) --- updated-dependencies: - dependency-name: symspellpy dependency-version: 6.10.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.0 to 4.37.1. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v4.37.0...v4.37.1) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.37.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
…tions/DavidAnson/markdownlint-cli2-action-24.0.0 Bump DavidAnson/markdownlint-cli2-action from 23.2.0 to 24.0.0
…tions/dieghernan/cff-validator-5.1.1 Bump dieghernan/cff-validator from 5.0.1 to 5.1.1
…formers-5.14.1 Bump transformers from 5.13.0 to 5.14.1
…ellpy-6.10.0 Bump symspellpy from 6.9.0 to 6.10.0
…ncepiece-0.2.2 Bump sentencepiece from 0.2.1 to 0.2.2
…tions/github/codeql-action-4.37.1 Bump github/codeql-action from 4.37.0 to 4.37.1
…tions/actions/download-artifact-8 Bump actions/download-artifact from 7 to 8
…orcing fix: remove unconditional overwrite that disables teacher forcing in Seq2Seq
Change loop limit from 20 to dec_maxlen
Upstreaming from v5.3 (v5.3.5)
Bumps [DavidAnson/markdownlint-cli2-action](https://github.com/davidanson/markdownlint-cli2-action) from 24.0.0 to 24.1.0. - [Release notes](https://github.com/davidanson/markdownlint-cli2-action/releases) - [Commits](DavidAnson/markdownlint-cli2-action@v24.0.0...v24.1.0) --- updated-dependencies: - dependency-name: DavidAnson/markdownlint-cli2-action dependency-version: 24.1.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [actions/checkout](https://github.com/actions/checkout) from 7.0.0 to 7.0.1. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v7.0.0...v7.0.1) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.1 to 4.37.3. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v4.37.1...v4.37.3) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.37.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
…tions/actions/checkout-7.0.1 Bump actions/checkout from 7.0.0 to 7.0.1
…tions/DavidAnson/markdownlint-cli2-action-24.1.0 Bump DavidAnson/markdownlint-cli2-action from 24.0.0 to 24.1.0
…tions/github/codeql-action-4.37.3 Bump github/codeql-action from 4.37.1 to 4.37.3
romanize(..., engine="royin") raised IndexError on words containing ฤ (U+0E24), e.g. ฤดู, ฤก, ฤทธิ์. _romanize() built its consonant list with _RE_CONSONANT, which matches only thai_consonants and so excludes ฤ, while _replace_consonants() treats every key of _CONSONANTS - including ฤ - as a consonant and advances its index for each one. The two went out of sync and the index ran past the end of the list. The consonant list is now built with the same membership test the loop uses, so ฤ is romanized through its existing _CONSONANTS entry. Adds RU_TESTS and test_romanize_royin_ru to tests/core/test_transliterate.py. Closes PyThaiNLP#1444
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(transliterate): romanize royin words containing ฤ - #1473