refactor(reading): single data_hex word identity, hash the token (#237)

CHANGELOG.md

@@ -7,6 +7,19 @@ ones are marked like "v1.0.0-fork".
 
 ## [Unreleased]
 
+### Changed
+
+* **Single `data_hex` word identity in the reading view** (#237): every word
+  occurrence now carries its term identity solely as a `data_hex` attribute,
+  and the redundant `TERM<hex>` CSS class has been dropped. The token is now a
+  short SHA-256-derived hex string (`StringUtils::toClassName()`) — pure
+  `[0-9a-f]`, so it is selector-safe and the occurrence-lookup regexes become
+  correct by construction. This replaces the original 2011 `¤`/hex encoder
+  whose per-byte vs. per-codepoint confusion PHP 8.5 surfaced by deprecating
+  `ord()` on multi-byte strings. There is no API wire-format change (the `hex`
+  field keeps its role, recomputed identically on the PHP and JS sides) and no
+  styling change (the `TERM` class carried no CSS rules).
+
 ## [3.2.1-fork] - 2026-06-30
 
 ### Fixed

docs-src/developer/word-identity-data-hex.md

@@ -5,12 +5,10 @@ description: Replace the reading-view TERM<hex> CSS class-as-index with a data_h
 
 # Proposal: `data_hex` Word Identity
 
-**Status:** Proposed — deferred until after the next release (a major feature lands
-first; this is a broad cross-cutting refactor we don't want to collide with it).
+**Status:** Implemented (#237), on the post-3.2.1 development line.
 Tracked in [issue #237](https://github.com/HugoFara/lwt/issues/237).
 
-A design proposal, not shipped work. It records the agreed approach so it isn't lost
-between releases.
+This document records the design and rationale of the change as it was carried out.
 
 ## Problem
 
@@ -73,28 +71,31 @@ keeps its exact role, just a hash string. So there is **no wire-format ripple**
 The only thing given up is human readability of the token in devtools
 (`data_hex="3a7f9c2e1b0d4f88"` instead of a mostly-readable string) — accepted.
 
-## Scope sketch
-
-When picked up post-release:
+## What changed
 
 - **PHP token:** `src/Shared/Infrastructure/Utilities/StringUtils.php::toClassName()`
-  → hash. Keep `toHex()` (independent, tested utility).
-- **PHP emit (5 spans, 2 files):** drop `'TERM' . toClassName(...)` from the `class`
-  and add `'data_hex' => toClassName(...)` in
+  → `substr(hash('sha256', $s), 0, 16)`. `toHex()` kept (independent, tested utility).
+- **PHP emit (5 spans, 2 files):** dropped `'TERM' . toClassName(...)` from the `class`
+  and added `'data_hex' => toClassName(...)` in
   `Modules/Text/Application/Services/TextReadingService.php` (×3) and
-  `Modules/Vocabulary/Application/Services/ExpressionService.php` (×2).
-- **JS emit:** remove the `TERM${word.hex}` push in
-  `modules/text/pages/reading/text_renderer.ts` (`data_hex` is already emitted).
-- **JS selectors (~9):** `.TERM${hex}` → `[data_hex="${hex}"]` in
-  `modules/vocabulary/services/word_dom_updates.ts`,
-  `modules/vocabulary/pages/word_result_init.ts`, and `text_renderer.ts`.
-- **JS extractors (4):** read `data_hex` instead of parsing the class in
-  `text_reader.ts`, `text_keyboard.ts`, `word_actions.ts`, `text_events.ts`.
-- **Tests:** update `toClassName` assertions in
-  `tests/backend/Core/IntegrationTest.php` and `tests/backend/Core/Text/TextProcessingTest.php`
-  (assert hash shape, not the old `¤` output); migrate frontend fixtures from
+  `Modules/Vocabulary/Application/Services/ExpressionService.php` (×2). The
+  server-rendered spans previously carried the token *only* as the class, so this
+  is an add-`data_hex` change, not just a drop.
+- **JS emit:** removed the `TERM${word.hex}` push in
+  `modules/text/pages/reading/text_renderer.ts` (`data_hex` was already emitted).
+- **JS selectors (9):** `.TERM${hex}` → `[data_hex="${hex}"]` in
+  `modules/vocabulary/services/word_dom_updates.ts` (×5),
+  `modules/vocabulary/pages/word_result_init.ts` (×2), and `text_renderer.ts` (×2).
+- **JS extractors (2):** dropped the `TERM([0-9A-Fa-f]+)` class-regex fallback and now
+  read `data_hex` directly in `text_reader.ts` and `text_keyboard.ts`. (The
+  `word_actions.ts` / `text_events.ts` extractors named in the original proposal had
+  already been refactored away by the time this landed.)
+- **Tests:** rewrote the `toClassName` assertions in
+  `tests/backend/Core/IntegrationTest.php` to assert the hash shape (16-char lowercase
+  hex) instead of the old `¤` output; migrated the frontend fixtures from
   `class="… TERM<x>"` + `.TERM<x>` to `data_hex="<x>"` + `[data_hex="<x>"]`
-  (`tests/frontend/reading/*`, `tests/frontend/words/*`, `tests/frontend/texts/text_reader.test.ts`).
+  (`tests/frontend/reading/*`, `tests/frontend/words/*`, `tests/frontend/texts/text_reader.test.ts`)
+  and removed the now-obsolete class-name-fallback extractor test.
 
 ### Out of scope
 

src/Modules/Text/Application/Services/TextReadingService.php

@@ -83,10 +83,10 @@ public function echoTerm(
                         [
                             $hidetag, "click", "mword", ($showAll ? 'mwsty' : 'wsty'),
                             "order" . $ti2Order,
-                            'word' . $woId, 'status' . $woStatus,
-                            'TERM' . StringUtils::toClassName($tiTextLC)
+                            'word' . $woId, 'status' . $woStatus
                         ]
                     ),
+                    'data_hex' => StringUtils::toClassName($tiTextLC),
                     'data_pos' => $currcharcount,
                     'data_order' => $ti2Order,
                     'data_wid' => $woId,
@@ -129,10 +129,10 @@ public function echoTerm(
                         " ",
                         [
                             $hidetag, "click", "word", "wsty", "word" . $woId,
-                            'status' . $woStatus,
-                            'TERM' . StringUtils::toClassName($tiTextLC)
+                            'status' . $woStatus
                         ]
                     ),
+                    'data_hex' => StringUtils::toClassName($tiTextLC),
                     'data_pos' => $currcharcount,
                     'data_order' => $ti2Order,
                     'data_wid' => $woId,
@@ -152,10 +152,10 @@ public function echoTerm(
                     'class' => implode(
                         " ",
                         [
-                            $hidetag, "click", "word", "wsty", "status0",
-                            "TERM" . StringUtils::toClassName($tiTextLC)
+                            $hidetag, "click", "word", "wsty", "status0"
                         ]
                     ),
+                    'data_hex' => StringUtils::toClassName($tiTextLC),
                     'data_pos' => $currcharcount,
                     'data_order' => $ti2Order,
                     'data_trans' => '',

src/Modules/Vocabulary/Application/Services/ExpressionService.php

@@ -368,7 +368,8 @@ public function newMultiWordInteractable(string $hex, array $multiwords, int $wi
 
         $woStatus = (int)($record["WoStatus"] ?? 1);
         $attrs = [
-            "class" => "click mword {$showType}wsty TERM$hex word$wid status" . $woStatus,
+            "class" => "click mword {$showType}wsty word$wid status" . $woStatus,
+            "data_hex" => $hex,
             "data_trans" => (string)($record["WoTranslation"] ?? ''),
             "data_rom" => (string)($record["WoRomanization"] ?? ''),
             "data_code" => $len,
@@ -410,7 +411,8 @@ public function newExpressionInteractable2(string $hex, array $appendtext, int $
 
         $woStatus = (int)($record["WoStatus"] ?? 1);
         $attrs = [
-            "class" => "click mword {$showType}wsty TERM$hex word$wid status" . $woStatus,
+            "class" => "click mword {$showType}wsty word$wid status" . $woStatus,
+            "data_hex" => $hex,
             "data_trans" => (string)($record["WoTranslation"] ?? ''),
             "data_rom" => (string)($record["WoRomanization"] ?? ''),
             "data_code" => $len,

src/Shared/Infrastructure/Utilities/StringUtils.php

@@ -68,44 +68,22 @@ public static function toHex(string $string): string
     }
 
     /**
-     * Escape a string for use as a CSS class name.
+     * Derive an opaque identity token for a term's lowercase text.
      *
-     * Escapes everything to "¤xx" (where xx is hex) except:
-     * - 0-9 (ASCII 48-57)
-     * - a-z (ASCII 97-122)
-     * - A-Z (ASCII 65-90)
-     * - Unicode characters >= 165 (hex 00A5)
+     * Used in the reading view as the `data_hex` attribute so every occurrence of
+     * the same term can be restyled in one pass when its status changes. The token
+     * is computed per render and never stored or reversed back to text, so a short
+     * one-way hash is sufficient. A pure `[0-9a-f]` hash is also selector-safe — no
+     * `CSS.escape` needed — and is recomputed identically on both the PHP and JS
+     * sides, so the API `hex` field keeps its exact role.
      *
-     * @param string $string String to escape
+     * @param string $string String to derive the token from (typically WoTextLC)
      *
-     * @return string CSS-safe class name
+     * @return string 16-character (64-bit) lowercase hex token
      */
     public static function toClassName(string $string): string
     {
-        $length = mb_strlen($string, 'UTF-8');
-        $result = '';
-        for ($i = 0; $i < $length; $i++) {
-            $char = mb_substr($string, $i, 1, 'UTF-8');
-            // $char may be multi-byte; ord() only reads byte 0 (PHP 8.5 deprecates
-            // passing a multi-byte string). $char[0] makes that explicit and keeps
-            // the existing first-byte behavior.
-            // TODO(php8.5): consider mb_ord($char, 'UTF-8') for true codepoints —
-            // it would also escape codepoints 123-164 per this method's documented
-            // contract, which the first-byte approach currently lets through. That
-            // is a deliberate behavior change to CSS class names; defer until vetted.
-            $ord = ord($char[0]);
-            if (
-                ($ord < 48)
-                || ($ord > 57 && $ord < 65)
-                || ($ord > 90 && $ord < 97)
-                || ($ord > 122 && $ord < 165)
-            ) {
-                $result .= '¤' . self::toHex($char);
-            } else {
-                $result .= $char;
-            }
-        }
-        return $result;
+        return substr(hash('sha256', $string), 0, 16);
     }
 
     /**

src/frontend/js/modules/text/components/text_reader.ts

@@ -238,7 +238,7 @@ export function textReaderData(): TextReaderData {
       event.stopPropagation();
 
       // Get word data from element (use getAttribute for underscore attributes)
-      const hex = wordEl.getAttribute('data_hex') || wordEl.className.match(/TERM([0-9A-F]+)/)?.[1] || '';
+      const hex = wordEl.getAttribute('data_hex') || '';
       const position = parseInt(wordEl.getAttribute('data_order') || wordEl.getAttribute('data_pos') || '0', 10);
 
       if (!hex) return;

src/frontend/js/modules/text/pages/reading/text_keyboard.ts

@@ -273,8 +273,7 @@ export function handleTextKeydown(e: KeyboardEvent): boolean {
   const wid = getAttrElement(curr, 'data_wid');
   const ord = getAttrElement(curr, 'data_order');
   const stat = getAttrElement(curr, 'data_status');
-  const hex = getAttrElement(curr, 'data_hex') ||
-    curr.className.match(/TERM([a-f0-9]+)/)?.[1] || '';
+  const hex = getAttrElement(curr, 'data_hex') || '';
   const txt = curr.classList.contains('mwsty')
     ? getAttrElement(curr, 'data_text')
     : (curr.textContent || '');

src/frontend/js/modules/text/pages/reading/text_renderer.ts

@@ -111,9 +111,6 @@ function buildWordClasses(word: WordData, showAll: boolean): string {
   // Status class
   classes.push(`status${word.status}`);
 
-  // TERM class for hex lookup
-  classes.push(`TERM${word.hex}`);
-
   return classes.join(' ');
 }
 
@@ -429,7 +426,7 @@ export function updateWordStatusInDOM(
   newWordId: number | null = null,
   container: Element = document.body
 ): void {
-  const selector = `.TERM${hex}`;
+  const selector = `[data_hex="${hex}"]`;
   const elements = container.querySelectorAll<HTMLElement>(selector);
 
   elements.forEach(el => {
@@ -462,7 +459,7 @@ export function updateWordTranslationInDOM(
   romanization: string,
   container: Element = document.body
 ): void {
-  const selector = `.TERM${hex}`;
+  const selector = `[data_hex="${hex}"]`;
   const elements = container.querySelectorAll<HTMLElement>(selector);
 
   elements.forEach(el => {

src/frontend/js/modules/vocabulary/pages/expression_interactable.ts

@@ -18,6 +18,7 @@ import { getTextId } from '@modules/text/stores/text_config';
  */
 interface TermAttrs {
   class: string;
+  data_hex: string;
   data_trans: string;
   data_rom: string;
   data_code: number;

src/frontend/js/modules/vocabulary/pages/word_result_init.ts

@@ -222,7 +222,7 @@ function initHoverSaveResult(config: HoverSaveResultConfig): void {
   // Always generate tooltips (jQuery UI tooltips removed, now using native)
   const title = createWordTooltip(config.wordRaw, config.translation, '', String(config.status));
 
-  context.querySelectorAll<HTMLElement>(`.TERM${config.hex}`).forEach(el => {
+  context.querySelectorAll<HTMLElement>(`[data_hex="${config.hex}"]`).forEach(el => {
     el.classList.remove('status0');
     el.classList.add(`status${config.status}`, `word${config.wid}`);
     el.setAttribute('data_status', String(config.status));
@@ -248,7 +248,7 @@ function initAllWellKnownResult(config: AllWellKnownConfig): void {
       title = createWordTooltip(word.term, '*', '', String(word.status));
     }
 
-    context.querySelectorAll<HTMLElement>(`.TERM${word.hex}`).forEach(el => {
+    context.querySelectorAll<HTMLElement>(`[data_hex="${word.hex}"]`).forEach(el => {
       el.classList.remove('status0');
       el.classList.add(`status${word.status}`, `word${word.wid}`);
       el.setAttribute('data_status', String(word.status));

src/frontend/js/modules/vocabulary/services/word_dom_updates.ts

@@ -87,7 +87,7 @@ export function updateNewWordInDOM(params: WordUpdateParams): void {
   const context = getParentContext();
   const title = generateTooltip(text, translation, romanization, status);
 
-  context.querySelectorAll<HTMLElement>(`.TERM${hex}`).forEach(el => {
+  context.querySelectorAll<HTMLElement>(`[data_hex="${hex}"]`).forEach(el => {
     el.classList.remove('status0');
     el.classList.add(`word${wid}`, `status${status}`);
     el.setAttribute('data_trans', translation);
@@ -189,7 +189,7 @@ export function markWordWellKnownInDOM(wid: number, hex: string, term: string):
 
   const title = createWordTooltip(term, '*', '', '99');
 
-  frameL.querySelectorAll<HTMLElement>(`.TERM${hex}`).forEach(el => {
+  frameL.querySelectorAll<HTMLElement>(`[data_hex="${hex}"]`).forEach(el => {
     el.classList.remove('status0');
     el.classList.add('status99', `word${wid}`);
     el.setAttribute('data_status', '99');
@@ -211,7 +211,7 @@ export function markWordIgnoredInDOM(wid: number, hex: string, term: string): vo
 
   const title = createWordTooltip(term, '*', '', '98');
 
-  frameL.querySelectorAll<HTMLElement>(`.TERM${hex}`).forEach(el => {
+  frameL.querySelectorAll<HTMLElement>(`[data_hex="${hex}"]`).forEach(el => {
     el.classList.remove('status0');
     el.classList.add('status98', `word${wid}`);
     el.setAttribute('data_status', '98');
@@ -306,7 +306,7 @@ export interface BulkWordUpdateParams {
 export function updateBulkWordInDOM(term: BulkWordUpdateParams, useTooltip: boolean): void {
   const context = getParentContext();
 
-  context.querySelectorAll<HTMLElement>(`.TERM${term.hex}`).forEach(el => {
+  context.querySelectorAll<HTMLElement>(`[data_hex="${term.hex}"]`).forEach(el => {
     el.classList.remove('status0');
     el.classList.add(`status${term.WoStatus}`, `word${term.WoID}`);
     el.setAttribute('data_wid', String(term.WoID));
@@ -345,7 +345,7 @@ export function updateHoverSaveInDOM(
   const context = getParentContext();
   const title = createWordTooltip(wordRaw, translation, '', String(status));
 
-  context.querySelectorAll<HTMLElement>(`.TERM${hex}`).forEach(el => {
+  context.querySelectorAll<HTMLElement>(`[data_hex="${hex}"]`).forEach(el => {
     el.classList.remove('status0');
     el.classList.add(`status${status}`, `word${wid}`);
     el.setAttribute('data_status', String(status));

tests/backend/Core/IntegrationTest.php

@@ -227,16 +227,26 @@ public function testStrToHex(): void
 
     public function testStrToClassName(): void
     {
-        $this->assertEquals('hello', StringUtils::toClassName('hello'));
-        $this->assertEquals('test123', StringUtils::toClassName('test123'));
+        // The identity token is a 16-char (64-bit) lowercase-hex prefix of sha256.
+        $hello = StringUtils::toClassName('hello');
+        $this->assertSame(substr(hash('sha256', 'hello'), 0, 16), $hello);
+        $this->assertMatchesRegularExpression('/^[0-9a-f]{16}$/', $hello);
 
-        // Space (ASCII 32) is outside allowed range, converted to ¤20
-        $this->assertEquals('hello¤20world', StringUtils::toClassName('hello world'));
+        // Deterministic: the same input always yields the same token.
+        $this->assertSame($hello, StringUtils::toClassName('hello'));
 
-        // Non-ASCII should be converted to hex with ¤ prefix
-        $result = StringUtils::toClassName('hello 世界');
-        $this->assertStringStartsWith('hello', $result);
-        $this->assertStringContainsString('¤', $result);
+        // Distinct inputs yield distinct tokens.
+        $this->assertNotSame($hello, StringUtils::toClassName('test123'));
+
+        // Spaces and non-ASCII stay pure hex — selector-safe, no escaping needed.
+        $this->assertMatchesRegularExpression(
+            '/^[0-9a-f]{16}$/',
+            StringUtils::toClassName('hello world')
+        );
+        $this->assertMatchesRegularExpression(
+            '/^[0-9a-f]{16}$/',
+            StringUtils::toClassName('hello 世界')
+        );
     }
 
     public function testReplTabNl(): void