Add asian distance and time units - #1
Arararararagi wants to merge 11 commits into
Conversation
…harness - Energy/Pressure: hp↔kW, BTU↔kJ, calories↔kJ, psi↔kPa, atm↔kPa, mmHg↔kPa (both directions) - Height notation: curly-quote primes (6′4″), bare compound (6'4), fractional inches, speech-close guard - New units: ton (with context classifier), carat (with gold guard), verst/arshin/pood, gill - Test harness: 98 tests covering units, notation, guards, compounds, rewriter, energy/pressure - Menu: energy/pressure categories with toggles, fixed trailing-space Time key - Removed FOOTCREAM_DEBUG scaffolding - Updated README with new units, tons note, height notation
|
A for effort on this. The problem with adding all these units is guarding against false positives. Do you have anything in place for that here? @Arararararagi |
The short spellings (go, li, mu, ri, sun, tan, ken...) are ordinary English words, so 'in one go' and 'she got a tan' previously risked converting. Gate them in _finishScan: CLUSTER (>=2 distinct transliteration units in the book) + DIGIT-ONLY (number written as digits), so only genuine wuxia/ Japanese/Korean books convert. Longer spellings (zhang, shichen, koku) remain unambiguous and convert freely. Also fixes the dead trailing-space keys/targets/cats on the Asian units (they never matched and printed '500 m ' with broken category lookup), removes the colliding 'acres ' duplicate, and adds the missing jin/kan suffixes. Adds tests/test_asian_units.lua plus atmosphere/gill no-match locks and README notes. Suite: 114 passed, 0 failed.
|
You are absolutely right. I tested some units against false positives, but not the asian units. My b. I pushed a new commit that should take care of that, although it's more complex. |
|
Okay, good! Just so I understand the use case.. @Arararararagi Wouldn't these words be written with kanji and not romaji for japanese and chinese in an asian book? I know korean doesn't have kanji, but still a different "alphabet". What type of books would write this with latin alphabet? Or do your proposed units show up in translated works? |
|
Asian units are quite common in translated works. "A journey of a thousand li begins with a single step", measuring areas in tatami (4.5 tatami legendary chronicles, aka tatami galaxy), etc. Historical fiction or works until the first decades of the 20th century frequently have these units. |
Brings the original repo's v1.7.0 release into the fork: the whole UI is now translatable (_()/T() + l10n/ for 30 languages, AGPL-3.0 LICENSE, Crowdin setup, CHANGELOG), plus upstream's new false-positive guards: curly/smart-quote heights (6'9"), year/decade possessive guard (2001's is not 2001 ft), bare 'degrees' temperature-cue classifier, and hyphenated square-unit area detection. Kept both sides in main.lua: - fork units (carat, ton classifier, verst/arshin/pood, gill, hp/BTU/psi, Asian transliterations) and the Asian cluster+digit gate - upstream's possessive-'s, curly-prime and degrees guards, merged with the fork's prime/speech-close/carat guards - fork's _PRIME_MARKS approach subsumes upstream's _CAPOS/_CDQ curly marks i18n: wrapped the fork's new menu/unit-list strings in _() with TRANSLATORS notes. CACHE_VERSION bumped to 65 (upstream had 64). Tests updated for i18n stubs plus new regression tests for the merged guards. Suite: 119 passed, 0 failed.
Implements the remaining units from Fank1/foot-cream issue Fank1#3: span, rod, pole, perch, chain, ell, hand, pace (cubit/league/furlong/stone already existed). Values are common English standards, approximate where the unit varied by time/region (pace = 30 in walking pace). False-positive defense: the short homographs (span/rod/pole/ell/hand/pace) join the existing transliteration gate, now generalized to FootFree._GATED_UNITS with a FootFree._GATED_LONG set for the 5-letter perch (fish) and chain (common noun). All gated units need a book cluster of >=2 distinct gated units and a literal digit, so 'life span', 'fishing rod', 'two hands of cards', 'keep pace', 'chain of events' and 'caught two perch' never convert while '3 ell and 5 span' / '15 hands and 4 chain' convert normally. Plural forms normalize to one gate key ('hands'/'hand'). Length help text + README updated. Suite: 133 passed, 0 failed.
Recognizes the twelve named Chinese two-hour periods in the English form
novels actually use — pinyin + "Hour" ("the Shen Hour", "the Chou Hour",
"Hai Hour", ...) with broad pinyin/Wade-Giles romanizations (Chen/Ch'en,
Ssu/Szu/Si, Yu/You/Yiu, Hsu/Xu). Each named period is ONE shichen = 2 h, so a
dedicated _SHICHEN_PATS literal pass converts it to the fixed 120 min even
without a count ("the Shen Hour" — the fast path requires a number and would
drop it). Capitalized on purpose so lowercase clock "hour" never matches.
Also adds the capitalized English renderings of the generic units — Hour
(=shichen), Watch (=geng), Mark (=dian), Ke (=ke) — on the fast path, gated
like the other short homographs (literal digit + book cluster of >=2 distinct
gated units), so a lone "3 Watch" is suppressed while a wuxia book mixing
several converts. The named periods count toward that cluster (r._unit tagged
as "shichen" via a new pass-entry unit field).
run_passes now honors an optional e.unit to tag literal-pass matches. The
suffix entries are placed AFTER "miles per hour"/"miles an hour" so
_identify_unit never shadows them with the generic "Hour". CACHE_VERSION
bumped to 66. Suite: 142 passed, 0 failed.
|
I've tested your code, and even though it adds more units - which is kinda a good thing - the downsides outweights the benefits with all these units. I regularly run tests with hundred of books to combat this wild creature which is human written text. Basically, every new unit adds scan time, and not to mention the guarding against false positives. An feasible option would be to add "unit packages" that are opt-in for the user. Could we do a pure "Asian unit" package, a "fantasy unit" package etc? That would be easier to handle, both to quality assure and for the user to actively activate, as to not get a lot of false positives. |
Maintainer request (Fank1#1): the fork's Asian transliteration and historical/fantasy English units add scan time and false-positive surface. Package them as opt-in toggles (off by default) so users who never read wuxia or epic fantasy skip them entirely: - FootFree._PKG_UNITS: maps each gated unit to its package (asian or fantasy), plus _UNIT_PKG reverse lookup and _PKG_GATED membership set. - _pkg_enabled[pkg] persisted via footcream_pkg_<pkg>, default false. - Per-alias scan loop (ACTIVE_SUFS) skips suffixes belonging to a disabled package; the pinyin shichen period pass is gated by the asian package. - Menu: 'Unit packages' submenu under Advanced with two tickable entries. - CLUSTER+DIGIT guard still applies inside an enabled package. - CACHE_VERSION 68 → 69 (disabling a package changes match output). - Tests: harness defaults both packages ON (142 existing pass), plus 9 new package-gate tests: OFF suppresses even in a cluster, ON restores, and packages are independent.
|
The Asian and historical/fantasy unit sets are now off by default. You only switch them on if you read the kind of book that uses them. Thanks for the suggestion, I think that's a good compromise! |
|
@Arararararagi Cool, I'll have a look! I'll need to run some tests on it and optimize, but I'll of course give you credit if I manage to get your features into a release. |
2 participants
No description provided.