Local Chatbot

A local, private AI chatbot built with LangChain + LangGraph, running against a self-hosted Ollama server. No cloud LLM APIs: the chat model and embeddings run on my own hardware. The only outbound calls are web searches (Tavily) and optional LangSmith tracing.

It remembers you across conversations, answers questions from your own PDFs, searches the web when it needs to, and summarizes long chats so the context window never blows up.

Demo

▶ Watch the 2-minute demo

A local model answering from a PDF with page-level citations, remembering a name across a new conversation, and surviving a full restart without losing history.

Features

• Persistent conversations. Every thread is checkpointed to SQLite, so you can close the app, come back, and pick up exactly where you left off. Switch between threads from the sidebar.
• Long-term memory. The model has a remember tool backed by a LangGraph Store. Tell it your name or preferences once and it knows them in every future conversation, in every thread.
• Local RAG. Upload PDFs from the UI (or CLI). They get chunked, embedded with nomic-embed-text, and stored in a local Chroma database. Relevant excerpts are retrieved on every turn and cited in answers.
• Tools. Calculator, Tavily web search, and the memory tool, routed through a standard LangGraph ToolNode.
• Rolling summarization. Once a conversation grows past 8 messages, a summarizer node compresses the older history into a running summary and prunes the raw messages with RemoveMessage. The context stays small without losing the plot.
• Streamlit frontend. Streaming token-by-token responses, thread switching, and PDF ingestion in one simple UI. A terminal client is included too.

Architecture

The agent is a single LangGraph StateGraph whose state is the message history plus a running summary string:

graph TD
    START([START]) --> chatbot
    chatbot -->|tool calls| tools[ToolNode<br/>calculator · web search · remember]
    tools --> chatbot
    chatbot -->|"> 8 messages"| summarize
    chatbot -->|done| END([END])
    summarize --> END

Loading

On every turn, the chatbot node builds a system prompt from three sources before calling the model:

1. Known facts about the user, fetched from the long-term memory store
2. Document excerpts, retrieved from Chroma for the latest user message
3. The running summary of earlier, pruned conversation

Design notes

• Always-retrieve instead of a retrieval tool. Document retrieval was originally exposed as a search_documents tool, but the local gpt-oss:20b model routed to it unreliably. It kept picking web search instead, and once returned a confident, cited answer from the wrong website entirely. RAG context is now retrieved unconditionally for every user message and injected into the system prompt. Slightly more work per turn, far more reliable answers.
• Two layers of persistence. The SQLite checkpointer stores per-thread conversation state. The SQLite store holds cross-thread long-term memories. Deleting a thread never deletes what the bot knows about you.
• Staged development. Each capability (core graph, then persistence, then long-term memory, RAG, summarization, and finally the UI) was built and verified one at a time, tested through the terminal client and LangGraph Studio before moving on.

Requirements

• Python 3.12 and uv
• An Ollama server (local or remote) with two models pulled:

  ollama pull gpt-oss:20b        # chat model
  ollama pull nomic-embed-text   # embeddings for RAG

• A Tavily API key (free tier works) for the web-search tool

Setup

git clone https://github.com/ShauryaKumar09/Chatbot.git
cd Chatbot
uv sync
cp .env.example .env   # then fill in your values

.env variables:

Variable	Required	Purpose
OLLAMA_BASE_URL	yes	URL of your Ollama server, e.g. http://localhost:11434
TAVILY_API_KEY	yes	Web search tool
LANGSMITH_TRACING / LANGSMITH_API_KEY / LANGSMITH_PROJECT	optional	LangSmith tracing for debugging graph runs

Running it

Streamlit UI (recommended):

uv run streamlit run app.py

Terminal chat (shows tool calls and summarization as they happen):

uv run python scripts/chat.py [thread_id]

Ingest a PDF from the command line:

uv run python scripts/ingest.py path/to/file.pdf

LangGraph Studio (visual graph debugger):

uv run langgraph dev

Project structure

├── app.py                  # Streamlit UI: streaming chat, threads, PDF upload
├── langgraph.json          # LangGraph Studio config
├── scripts/
│   ├── chat.py             # Terminal client
│   └── ingest.py           # CLI PDF ingestion
├── src/chatbot/
│   ├── config.py           # Model + env config (single source of truth)
│   ├── graph.py            # The StateGraph: chatbot, tools, summarize nodes
│   ├── memory.py           # SQLite checkpointer + long-term memory store
│   ├── rag.py              # Chroma vector store: ingest + retrieve
│   └── tools.py            # calculator, Tavily web search, remember
└── data/
    ├── checkpoints/        # SQLite DBs (gitignored)
    └── uploads/            # Ingested PDFs (gitignored)

How the pieces fit

Concern	Mechanism
Per-thread chat history	SqliteSaver checkpointer, keyed by thread_id
Cross-thread user facts	SqliteStore + remember tool with InjectedStore
Document knowledge	Chroma (persistent, local) + OllamaEmbeddings
Context-window control	summarize node + RemoveMessage pruning after 8 messages
Streaming UI	graph.stream(..., stream_mode="messages") filtered to the chatbot node