Nice project and video!
As an LLM newbie I might be being too optimistic trying to run this with the Llama 2 quantized model llama-2-7b-chat.Q4_K_M.gguf on a cpu with only 8GB RAM nominal. The Chainlit page loads and after entering a question after a while it appears to timeout with the message "Could not reach the server". Should I increase the session_timeout parameter in config.toml or run some of the code in async mode?
Nice project and video!
As an LLM newbie I might be being too optimistic trying to run this with the Llama 2 quantized model llama-2-7b-chat.Q4_K_M.gguf on a cpu with only 8GB RAM nominal. The Chainlit page loads and after entering a question after a while it appears to timeout with the message "Could not reach the server". Should I increase the session_timeout parameter in config.toml or run some of the code in async mode?