fix: correct _UnicodeTokenizer for CJK segmentation and stemming

The previous tokenizer had two defects:

- Its char-level fallback was unreachable: it split non-ASCII text on
  whitespace first, and scripts without spaces (Chinese, Japanese, Thai)
  yield a single token, so the `list(text)` fallback never ran. Two
  different CJK strings sharing characters scored 0.0 instead of getting
  partial credit.
- Passing a custom `tokenizer=` makes rouge-score ignore `use_stemmer`,
  so English stemming was silently dropped (e.g. "running" no longer
  matched "run").

Now ASCII-majority text is delegated to rouge-score's DefaultTokenizer
(preserving Porter stemming and existing behavior exactly), and non-ASCII
text keeps Latin/digit runs as words while splitting remaining word
characters individually so partial overlap is scored.

Verified: Thai exact=1.0, CJK exact=1.0, CJK partial(你好世界 vs 你好朋友)=0.5,
English stemming(running fast vs run fast)=1.0, Latin sanity matches default.

Author: tcconnally

src/google/adk/evaluation/final_response_match_v1.py

@@ -96,24 +96,29 @@ def _get_eval_status(score: float, threshold: float):
 class _UnicodeTokenizer:
   """Tokenizer that handles Unicode text with word-boundary awareness.
 
-  The default RougeScorer tokenizer splits on whitespace, which works for
-  ASCII and Latin-script text but produces zero tokens for text in scripts
-  without word boundaries (Chinese, Japanese, Thai, etc.).
-
-  For ASCII-majority text this tokenizer uses Unicode-aware word-character
-  matching (``\\w+`` in re). For non-ASCII text it falls back to whitespace
-  splitting, then character-level tokenization.
+  The default RougeScorer tokenizer strips characters outside ``[a-z0-9]``, so
+  text in scripts without Latin word boundaries (Chinese, Japanese, Thai, etc.)
+  produces zero tokens and scores 0.0 even on an exact match.
+
+  ASCII-majority text is delegated to rouge-score's ``DefaultTokenizer`` so the
+  existing behavior -- including Porter stemming -- is preserved exactly. For
+  non-ASCII text, Latin/digit runs are kept as words and each remaining word
+  character (e.g. a CJK ideograph) becomes its own token, so partial overlap is
+  scored instead of collapsing into a single opaque token.
   """
 
+  def __init__(self, use_stemmer: bool = True):
+    self._default = rouge_scorer.tokenizers.DefaultTokenizer(use_stemmer)
+
   def tokenize(self, text: str) -> list[str]:
     """Tokenizes text using Unicode-aware word boundaries."""
+    text = text.lower()
+    if not text:
+      return []
     ascii_chars = sum(1 for c in text if ord(c) < 128)
     if ascii_chars > len(text) * 0.5:
-      return re.findall(r"\w+", text.lower())
-    tokens = text.lower().split()
-    if tokens:
-      return tokens
-    return list(text.lower())
+      return self._default.tokenize(text)
+    return re.findall(r"[a-z0-9]+|\w", text, re.UNICODE)
 
 
 def _calculate_rouge_1_scores(candidate: str, reference: str):