TXST RAG Assistant

TXST RAG Assistant is a student-facing question-answering application grounded in public Texas State University website content. It combines a controlled web crawler, Gemini embeddings and generation, a persistent ChromaDB vector store, a FastAPI backend, and a responsive chat interface.

The assistant is designed to answer from retrieved TXST sources instead of guessing. When the available website context is insufficient, it returns:

I don't know based on the provided TXST website context.

Note

This is an independent student project and is not an official Texas State University service.

Demo

The chat experience gives students a simple place to ask about TXST resources, start from suggested questions, and see whether the official-source index is available.

Ask a question and inspect the evidence

Each supported answer is paired with the public TXST pages used to produce it. Students can review source titles, URLs, retrieved excerpts, and the number of contexts checked before following a source for more detail.

The interface also lets students copy an answer, retry the same question, or continue with another question.

View more grounded-answer examples

Campus recreation

The assistant summarizes indoor, outdoor, and competitive recreation options and exposes the supporting Campus Recreation pages.

Bobcat Shuttle

For transportation questions, the assistant returns the available schedule information and links students back to the official route pages.

Features

• Crawls an explicit allowlist of public TXST websites while respecting robots.txt, crawl delays, depth limits, and page limits.
• Rejects authenticated, private, account, payment, portal, and unsafe external destinations.
• Cleans HTML pages, splits their text into overlapping chunks, and stores source metadata with each chunk.
• Uses Gemini embeddings for semantic retrieval and Gemini generation for context-grounded answers.
• Persists vectors locally in ChromaDB.
• Skips unchanged content during repeated ingestion and checkpoints each completed source page.
• Returns source links and retrieved-context counts with each answer.
• Provides a responsive chat interface with connection status, suggested questions, conversation history, and error handling.
• Keeps crawl, ingestion, and vector-reset operations out of the public frontend proxy.

Architecture

Approved TXST websites
        |
        v
Safe bounded crawler -> cleaned page JSON -> text chunks
                                                |
                                                v
                                      Gemini embeddings
                                                |
                                                v
                                      Persistent ChromaDB
                                                |
Student question -> FastAPI retrieval -> grounded Gemini answer
        ^
        |
Student chat UI -> same-origin frontend proxy

Technology

Backend

• Python 3.11+
• FastAPI and Uvicorn
• Google Gemini API
• ChromaDB
• Beautiful Soup, lxml, and HTTPX
• Pytest

Frontend

• TypeScript
• React and Next.js
• Vinext, Vite, and Cloudflare Workers
• ESLint and Node.js test runner

Project structure

txstAssistant/
├── backend/
│   ├── app/
│   │   ├── crawler/       # URL safety, robots policy, fetching, and cleaning
│   │   └── rag/           # chunking, embeddings, retrieval, and generation
│   ├── data/              # ignored crawl output and crawl state
│   ├── chroma_db/         # ignored local vector database
│   ├── scripts/           # crawl, ingestion, and query commands
│   └── tests/
├── frontend/
│   ├── app/               # student chat experience
│   ├── worker/            # public FastAPI proxy
│   └── tests/
└── README.md

Local setup

1. Configure and start the backend

cd backend
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

Create a Gemini API key and place it only in backend/.env:

GEMINI_API_KEY=your_key_here

Never add a real key to .env.example or commit backend/.env.

Build the local knowledge base:

python scripts/crawl_and_ingest.py

Start FastAPI:

uvicorn app.main:app --reload

The API will be available at http://localhost:8000, with interactive documentation at http://localhost:8000/docs.

2. Configure and start the frontend

In another terminal:

cd frontend
npm install
cp .env.example .env.local
npm run dev

Open http://localhost:3000. The frontend defaults to a backend running at http://localhost:8000.

API overview

Method	Route	Purpose
GET	/health	Report backend, Gemini configuration, and vector-store health
POST	/ask	Retrieve TXST context and answer a student question
POST	/crawl	Run a bounded crawl of configured public sites
POST	/ingest	Ingest previously crawled pages
POST	/crawl-and-ingest	Run both maintenance stages
POST	/reset-vector-store	Clear the configured Chroma collection

Only /health and /ask are exposed by the student-facing frontend. Protect the maintenance routes before making the backend publicly accessible.

Verification

Run backend tests from backend/:

.venv/bin/pytest -q

Run frontend checks from frontend/:

npm test
npm run lint

The tests run offline and do not call TXST websites or Gemini.

Deployment notes

• Provide GEMINI_API_KEY through the backend host's secret manager.
• Attach persistent storage for ChromaDB and the crawled-page data.
• Run the backend with a production server bound to the platform's assigned host and port.
• Set the hosted frontend's TXST_API_BASE_URL to the public HTTPS FastAPI URL.
• Protect maintenance endpoints and add suitable request-rate controls before public launch.

More detailed service-specific instructions are available in backend/README.md and frontend/README.md.