fix: use tiktoken for accurate CJK token counting in document chunking

[Sanjay-Kirti]

Summary

Fixes #1354

Replace text.split() tokenizer in TextToEntries with tiktoken cl100k_base encoding to fix markdown/org chunking for CJK (Chinese, Japanese, Korean) languages.

Problem

TextToEntries.tokenizer() uses text.split() which splits on whitespace. CJK languages don't use spaces between words, so a 744-token Chinese document was counted as ~152 tokens. This caused MarkdownToEntries (and OrgToEntries) to save entire CJK documents as single entries instead of chunking by headings severely degrading search quality for CJK users.

Changes

• src/khoj/processor/content/text_to_entries.py: Replace text.split() with tiktoken.get_encoding("cl100k_base").encode() in TextToEntries.tokenizer(). Encoder is lazy initialized and cached as a class attribute.
• tests/test_markdown_to_entries.py: Add 3 CJK specific test cases:
  • Chinese markdown is chunked by headings (core bug fix)
  • Small CJK content stays as single entry (regression test)
  • Mixed CJK/English markdown is properly chunked

Token Count Improvements

Concrete examples of how tiktoken correctly counts tokens compared to the old whitespace-split method:

Text	text.split() (old)	tiktoken (new)	Result
Chinese sentence (65 chars)	1 token ❌	69 tokens ✅	Accurate counting
Full CJK markdown example	11 tokens ❌	212 tokens ✅	Properly chunks now
Japanese sentence	1 token ❌	55 tokens ✅	Accurate counting
English sentence (18 words)	18 tokens ✅	18 tokens ✅	Identical behavior

Why this is safe

All 3 call sites only use len(TextToEntries.tokenizer(...)) they never inspect individual tokens. The return type change from List[str] → List[int] has no impact. tiktoken is already a project dependency (used in helpers.py for chat token counting).

Test Results

Ran the full test suite locally with Postgres/pgvector enabled:

• All 12/12 tests in test_markdown_to_entries.py pass (including the 3 new CJK ones)
• No regressions for English markdown processing

[debanjum]

Thanks for submitting a PR to fix this! Left some comments, can merge the changes once they're resolved and CI passes

[Sanjay-Kirti]

@debanjum Thanks for the review, I've addressed all the feedback and just pushed the updates:

1. Switched from cl100k_base to the more modern o200k_base in ext_to_entries.py for better Indic language support.
2. changed the CJK tests in test_markdown_to_entries.py per your suggestion:

• Removed the unnecessary small content test.
• Merged the mixed CJK/English test into test_cjk_markdown_is_chunked_by_headings. The single test now validates chunking with 2 Chinese notes followed by 1 English note.
  All tests are passing locally. Let me know if anything else is needed or if you want me to work on another issue.
  Would love to contribute.