FIX: Chunk large inputs to prevent tiktoken-rs stack overflow - #9

FIX: Chunk large inputs to prevent tiktoken-rs stack overflow#9

xfalcox merged 1 commit into

maindiscourse/discourse_ai-tokenizers:mainfrom

fix/tiktoken-stack-overflow-chunkingdiscourse/discourse_ai-tokenizers:fix/tiktoken-stack-overflow-chunkingCopy head branch name to clipboard