Skip to content

fix(tokens): dedup defaultCountTokens, correct stale comments - #113

Open
ranxianglei wants to merge 1 commit into
masterfrom
2026-08-11_tokenizer-dedup
Open

fix(tokens): dedup defaultCountTokens, correct stale comments#113
ranxianglei wants to merge 1 commit into
masterfrom
2026-08-11_tokenizer-dedup

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

  • src/search-index.ts: replace the locally-duplicated defaultCountTokens regex with a delegation to the kernel's imported defaultCountTokens.
  • src/commands.ts: correct two stale comments that called the counter "chars/4" — it is actually CJK-aware (1:1 for CJK chars, chars/4 for non-CJK).

Why

search-index.ts had a verbatim copy of the kernel's defaultCountTokens regex ([\u4e00-\u9fff\u3040-\u30ff\uac00-\ud7af] counted 1:1, remainder chars/4). Identical to the kernel today, but a drift bomb: if the kernel heuristic ever changes, this copy diverges silently and the search index's token estimates fall out of sync with the rest of the pipeline. Delegating to the imported function removes the sync burden.

commands.ts comments described the counter as "chars/4" and "chars/4-vs-real-tokenizer drift". Misleading — defaultCountTokens counts CJK characters 1:1 (not 4:1), so Chinese/Japanese/Korean text is NOT underestimated 4×. Corrected to name defaultCountTokens and its real behavior, so readers don't assume CJK content is severely undercounted.

Context

Found during the acp-kernel tokenizer-consistency review (kernel PR #56 fix(recommend): use ctx.countTokens). billion-context-pi's per-message counting is already consistent with the kernel (all paths use defaultCountTokens); this PR closes a drift risk and fixes misleading docs.

Out of scope

  • acp-kernel version bump (pinned acp-kernel@0.0.17): deferred until kernel PR fix(delegate): replace status tool with blocking wait #56 merges and publishes, then a separate release branch bumps it (pi CI npm ci requires the published version).
  • Dual-currency design (tokenCount from real provider usage vs per-message heuristic): intentional — the footer/nudge anchor on real usage. Not touched.

Verification

  • tsc --noEmit: clean
  • node --import tsx --test tests/*.test.ts: 153/153 pass
  • scripts/ci/check-pr.sh: passed (branch name + version)

search-index.ts duplicated the kernel's defaultCountTokens regex locally
- identical today, but a drift bomb if the kernel heuristic ever changes.
Delegate to the imported defaultCountTokens instead.

commands.ts comments called the counter "chars/4" but it is CJK-aware
(1:1 for CJK, chars/4 for non-CJK). Correct the wording so readers don't
assume Chinese text is underestimated 4:1.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant