Discussion: Context Management Primitives and Token Estimation

[luckenco]

This is a discussion issue to determine the library's role in helping users manage context windows.

While strict state management (history truncation/summarization) likely belongs in user-land, users currently lack the primitives to implement this effectively.

Proposed Solution:
Expose utility functions for token estimation.

// Example usage for user-implemented logic
if rsai::utils::estimate_tokens(&history)? > 8000 {
    // user logic to prune messages
}

Open Questions:

1. Should we integrate heavy dependencies like tiktoken-rs?
2. Or should we provide a lightweight "heuristic" estimator (e.g. char count / 4) to keep the library slim?
3. How should we handle tokenizer differences across providers (Gemini vs OpenAI)?