Intelligent Unstructured Document Understanding

Multi-Modal Semantic Integration for Intelligent Unstructured Document Understanding — a full-stack GraphRAG system that turns complex PDFs (text, tables, charts) into an explorable knowledge graph and answers natural-language questions with grounded, streamed, cited responses.

A React web app talks to a FastAPI backend that wraps a hybrid GraphRAG engine: documents are parsed with Docling, charts are read by a vision model, content is embedded into ChromaDB and structured into a NetworkX knowledge graph, and answers are generated by retrieving across both stores.

---

Features

• Multi-modal ingestion — text, tables, and charts/figures (described by a vision-language model), parsed with Docling.
• Hybrid GraphRAG retrieval — combines vector search (ChromaDB) with knowledge-graph context (NetworkX) for distributed, relational reasoning instead of nearest-chunk RAG.
• Streaming answers (SSE) — responses and live pipeline status stream token-by-token to the UI via Server-Sent Events.
• Adversarial robustness — answer_query_robust gracefully handles unanswerable / out-of-scope / ambiguous questions instead of hallucinating.
• Real-time / progressive ingestion — stream_ingest indexes a document chunk-by-chunk so it becomes queryable early.
• Semantic query cache — fuzzy-matched (≥ 0.88 similarity) LRU cache short-circuits repeat or near-duplicate questions.
• Polished React frontend — Vite + React 19 + Tailwind + GSAP animated UI.

---

Architecture

┌──────────────┐   upload PDF        ┌─────────────────────────┐
│ React (Vite) │ ──POST /api/ingest─▶│  FastAPI (main.py)      │
│  frontend/   │                     │                         │
│              │ ──POST /api/chat───▶│  • SemanticCache (LRU)  │
│              │ ◀──SSE: status,─────│  • SSE streaming        │
│              │     answer, done    │                         │
└──────────────┘                     └───────────┬─────────────┘
                                                 │ injestion.py
                                                 ▼
            ┌────────────────────────────────────────────────────────┐
            │  Docling parse → NVIDIA vision (charts) → chunking      │
            │        │                                                │
            │        ├──▶ NVIDIA nv-embed → ChromaDB (vectors)        │
            │        └──▶ LLM entity/relation extraction → NetworkX   │
            │                                  (knowledge graph)      │
            │                                                         │
            │  Query: retrieve + NVIDIA rerank + 2-hop subgraph       │
            │         → Nemotron-Super-49B → grounded, cited answer   │
            └────────────────────────────────────────────────────────┘

---

Tech Stack

Layer	Technology
Frontend	React 19, Vite, Tailwind CSS, GSAP, lucide-react
API	FastAPI, Server-Sent Events (SSE), Uvicorn
Document parsing	Docling (layout + tables + figures)
Vector store	ChromaDB
Knowledge graph	NetworkX (DiGraph)
LLM (reasoning)	nvidia/llama-3.3-nemotron-super-49b-v1 (NVIDIA NIM)
LLM (graph extraction)	meta/llama-3.1-8b-instruct (NVIDIA NIM)
Vision (charts)	meta/llama-3.2-90b-vision-instruct (NVIDIA NIM)
Embeddings	nvidia/nv-embed-v1 (NVIDIA NIM)
Re-ranking	nvidia/rerank-qa-mistral-4b (NVIDIA NIM)
Graph viz	streamlit-agraph

All language/vision/embedding inference runs through NVIDIA NIM APIs — a single NVIDIA_API_KEY covers every model.

---

Project Structure

.
├── main.py            # FastAPI server: /api/ingest, /api/chat (SSE), /api/health, /api/index
├── injestion.py       # GraphRAG engine: parsing, vision, chunking, Chroma + NetworkX,
│                       #   retrieval, rerank, answer_query_robust, stream_ingest
├── requirements.txt   # Python deps
├── backend/uploads/   # uploaded PDFs + extracted chart images
├── frontend/          # React + Vite single-page app
│   ├── src/App.jsx
│   └── package.json
├── PRD.md / workflow.md / execution_plan.md   # design docs
└── README.md

---

Getting Started

Prerequisites

• Python 3.11+
• Node.js 18+ (for the frontend)
• A free NVIDIA NIM API key — get one at https://build.nvidia.com (sign in → pick any model → Get API Key).

1. Backend (FastAPI + GraphRAG engine)

# from the repo root
python -m venv .venv
source .venv/bin/activate            # Windows: .venv\Scripts\activate
pip install -r requirements.txt

# add your key
echo "NVIDIA_API_KEY=nvapi-xxxxxxxx" > .env

# run the API (http://localhost:8000)
uvicorn main:app --reload --port 8000
# or: python main.py

2. Frontend (React + Vite)

cd frontend
npm install
npm run dev                          # http://localhost:5173

If the frontend needs the API URL, set it in frontend/.env (e.g. VITE_API_BASE=http://localhost:8000). CORS is open on the backend by default.

3. Use it

Open the frontend, upload a PDF, wait for ingestion, then ask questions — answers stream in with status updates.

---

API Reference

Base URL: http://localhost:8000

Method	Endpoint	Description
GET	/api/health	Liveness check
GET	/api/index	Whether a document index is loaded + stats
POST	/api/ingest	Upload a PDF (multipart/form-data, field file); builds the hybrid index
POST	/api/chat	Ask a question (JSON { "query": "..." }); streams the answer via SSE

/api/chat SSE events

The endpoint returns text/event-stream. Event types:

• status — pipeline progress (request_received, cache_hit / cache_miss, starting_generation, …)
• answer — { "text": "<chunk>" } streamed answer fragments
• done — { "cached": bool, "metadata": { … } } final payload (sources, graph context, etc.)
• error — { "message": "…" } on failure

A semantic cache (LRU, 128 entries, 0.88 similarity) replays cached answers for repeat/near-duplicate queries.

---

How It Works

1. Ingest — Docling extracts text, tables, and chart images. Charts are described by the vision model. Content is chunked, embedded into ChromaDB, and structured into a NetworkX knowledge graph (entities + relations extracted by the LLM, with deterministic entity resolution). stream_ingest makes the document queryable progressively.
2. Retrieve — the query is first expanded via HyDE (a hypothetical answer is generated and embedded), top chunks are pulled from ChromaDB and re-ranked; a 2-hop subgraph around the relevant entities adds relational context.
3. Answer — answer_query_robust feeds the fused context to Nemotron-Super-49B, returns a grounded answer with inline [S#] citations, and refuses gracefully when the documents don't support an answer.

---

Configuration

Where	Setting	Purpose
.env	NVIDIA_API_KEY	NVIDIA NIM auth (required)
injestion.py	*_MODEL constants	swap NVIDIA models (also overrideable via NVIDIA_REASONING_MODEL, NVIDIA_GRAPH_MODEL, etc. env vars)
injestion.py	DEFAULT_CHUNK_*, DEFAULT_CONCURRENCY_LIMIT	chunking & API concurrency
main.py	CACHE_MAX_ENTRIES, CACHE_SIMILARITY_THRESHOLD	semantic cache behaviour

---

Notes & Limitations

• Requires internet + an NVIDIA API key (free-tier credits + rate limits apply).
• Designed for digital, English PDFs (no scanned/handwritten OCR).
• The NetworkX graph is in-memory per session; re-ingest after a restart.

---

References

• Lewis et al. (2020), Retrieval-Augmented Generation (arXiv:2005.11401)
• Edge et al. (2024), From Local to Global: A Graph RAG Approach (arXiv:2404.16130)