⚡ Bolt: Optimize OllamaClient with persistent session reuse - #24
⚡ Bolt: Optimize OllamaClient with persistent session reuse#24Rohith-Shimori wants to merge 1 commit into
Conversation
- Initialize `requests.Session` in `OllamaClient.__init__` with keep-alive - Reuse session for `generate`, `generate_with_system`, `chat`, and streaming methods - Update streaming methods to use `with` context manager for robust resource cleanup - Removes connection overhead for consecutive LLM calls - Reduces latency for high-frequency interactions Co-authored-by: Rohith-Shimori <228351330+Rohith-Shimori@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
1 participant
⚡ Bolt: Optimize OllamaClient with persistent session reuse
💡 What:
Refactored
OllamaClientto use a persistentrequests.Sessioninstead of creating new connections for every request.Updated all methods (
generate,chat,stream, etc.) to use this shared session.Implemented proper resource cleanup for streaming responses using context managers.
🎯 Why:
Creating a new TCP connection (and SSL handshake if applicable, though usually HTTP for Ollama) for every request adds unnecessary overhead, especially for local inference where latency should be minimal.
requests.Sessionautomatically handles connection pooling and keep-alive, significantly reducing the time-to-first-token for subsequent requests.📊 Impact:
🔬 Measurement:
session.postis called.PR created automatically by Jules for task 1749510183679477248 started by @Rohith-Shimori