Multimodal Graph RAG System

An enterprise-grade Retrieval-Augmented Generation (RAG) system that supports text, image, audio, and video ingestion, builds a searchable knowledge graph, and enables hybrid search using keyword and vector retrieval.

Architecture

• Framework: LangGraph/LangChain for orchestration
• Graph Database: Neo4j for entity and relationship storage
• Vector Database: Weaviate for semantic search
• Embeddings: sentence-transformers/all-mpnet-base-v2 (override via EMBEDDING_MODEL; swap to heavier Tencent KaLM model when infra allows)
• Reranking: BAAI bge-reranker-v2-m3
• Evaluation: RAGAS + DeepEval

Project Structure

multimodal-graph-rag/
├── src/
│   ├── ingestion/          # Data ingestion pipelines
│   ├── processing/          # Modality-specific processors
│   ├── graph/              # Neo4j graph operations
│   ├── vector/             # Weaviate vector operations
│   ├── retrieval/          # Hybrid retrieval logic
│   ├── agents/             # LangGraph agent workflows
│   └── evaluation/         # Evaluation framework
├── tests/                  # Unit and integration tests
├── docs/                   # Documentation
├── data/                   # Sample data
└── ui/                     # Streamlit/Gradio interface

Quick Start

See docs/setup.md for detailed setup instructions.

Basic Setup

1. Install dependencies:

pip install -r requirements.txt

2. Set up environment variables: create .env with the cloud endpoints and keys (see docs/setup.md for a template). At minimum you need NEO4J_URI, NEO4J_USER, NEO4J_PASSWORD, WEAVIATE_URL, and WEAVIATE_API_KEY.

3. Choose database deployment:

   • Cloud (recommended): use the shared Neo4j AuraDB + Weaviate Cloud Service by keeping the .env values above.
   • Local: run docker-compose up -d to start both services on your machine.

4. Test / Ingest text documents:

# Smoke test with synthetic sample
python scripts/test_text_pipeline.py

# Ingest SEC 10-Q dataset (20 PDFs)
python scripts/test_text_pipeline.py --docs-dir data/text/data/v1/docs

# Run interactive RAG queries and print contexts
python scripts/run_rag_queries.py --limit 3

# Test image ingestion
python scripts/test_image_pipeline.py

# Test audio ingestion (demo tone or pass --audio-dir/--audio-file)
python scripts/test_audio_pipeline.py --demo

5. Run evaluation suite (text):

python -m src.evaluation.run_evals
python scripts/run_deepeval_text_eval.py --limit 5  # DeepEval on PDFs

• Image QA DeepEval: python scripts/run_deepeval_image_eval.py --limit 5 (defaults to data/image/spdocvqa_images/qna_image_samples.csv)

Evaluation-First Approach

This system follows an evaluation-first methodology:

• Define success criteria before implementation
• Test each modality independently
• Continuous evaluation throughout development
• Document all metrics and failure modes

See docs/eval.md for detailed evaluation framework.

Structured Chunking

• PDFs are processed with a structure-aware pipeline (based on unstructured) that emits parent/child chunks, table-only chunks, and per-page context for citations.
• Chunk metadata includes hierarchy (chunk_level, parent_id), section paths, and page numbers so the hybrid retriever can navigate parent-child relationships.
• Text files fall back to recursive character chunking with consistent metadata.

Evaluation Scripts

• scripts/run_rag_queries.py — multi-modality CLI. Use --modality text|image|audio|video, optional --csv, and --deepeval to automatically launch DeepEval. Example:

  python scripts/run_rag_queries.py --modality image --limit 3 --deepeval

• scripts/run_deepeval_text_eval.py — DeepEval for SEC PDFs (defaults to data/text/data/v1/qna_data_mini.csv).
• scripts/run_deepeval_image_eval.py — DeepEval for DocVQA image snippets (defaults to data/image/spdocvqa_images/qna_image_samples.csv).
• scripts/test_audio_pipeline.py — smoke-test audio ingestion with --demo (synthetic WAV) or point to real audio via --audio-dir.

License

MIT