DeepInvestigate

An AI-powered investigative knowledge graph platform that ingests multi-modal source documents, transforms them into temporal knowledge graphs, and enables AI agents to produce citable, wiki-style reports.

Overview

DeepInvestigate combines document ingestion, knowledge graph construction, and AI-powered synthesis to create a comprehensive investigative research platform. Every fact in the knowledge graph maintains direct provenance links to source documents, ensuring citability and data integrity.

Architecture

• Frontend: React Native (mobile-first interface)
• Backend API: Python FastAPI
• Document Processing: docling library
• Relational Database: PostgreSQL (source metadata & extraction artifacts)
• Object Storage: MinIO (S3-compatible storage for processed documents)
• Knowledge Graph: Graphiti framework with Neo4j backend
• Caching: Redis (hot cache for extraction artifacts)
• AI Framework: LangChain/LlamaIndex
• LLM Provider: OpenAI GPT series

Current Status

Phase 1: Foundational Backend & Ingestion - In Progress

✅ Project structure and configuration ✅ Docker infrastructure setup ✅ Database schema design 🚧 Python backend implementation ⏳ Document ingestion pipeline ⏳ API endpoints

Prerequisites

• Docker and Docker Compose (required for infrastructure)
• Python 3.11+ (for backend development)
• Node.js 18+ (for frontend development - Phase 3)
• Git

Quick Start

1. Clone the Repository

git clone <repository-url>
cd DeepInvestigate

2. Environment Configuration

Copy the example environment file and configure your settings:

cp .env.example .env

Edit .env to set your configuration (most defaults work for local development).

3. Start Infrastructure Services

Start PostgreSQL, MinIO, Neo4j, and Redis using Docker Compose:

docker-compose up -d

This will start:

• PostgreSQL on localhost:5432 (database automatically initialized with schema)
• MinIO on localhost:9000 (API) and localhost:9001 (Web Console)
• Neo4j on localhost:7474 (HTTP) and localhost:7687 (Bolt)
• Redis on localhost:6379

4. Verify Services

Check that all services are running:

docker-compose ps

All services should show as "healthy" or "running".

Access Web Interfaces:

• MinIO Console: http://localhost:9001 (minioadmin / minioadmin)
• Neo4j Browser: http://localhost:7474 (neo4j / changeme)

5. Set Up Python Backend

Create a virtual environment and install dependencies:

cd backend
python -m venv venv

# On Linux/Mac:
source venv/bin/activate

# On Windows:
venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

6. Run the Backend API

# From the backend directory with venv activated
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

The API will be available at:

• API: http://localhost:8000
• Interactive Docs: http://localhost:8000/docs
• OpenAPI Schema: http://localhost:8000/openapi.json

Development Workflow

Starting Services

# Start infrastructure
docker-compose up -d

# Activate Python environment
cd backend
source venv/bin/activate  # or venv\Scripts\activate on Windows

# Run API server
uvicorn app.main:app --reload

Stopping Services

# Stop API server: Ctrl+C

# Stop infrastructure
docker-compose down

# Stop infrastructure and remove volumes (CAUTION: deletes all data)
docker-compose down -v

Viewing Logs

# All services
docker-compose logs -f

# Specific service
docker-compose logs -f postgres
docker-compose logs -f minio
docker-compose logs -f neo4j
docker-compose logs -f redis

API Endpoints (Phase 1)

Source Management

• POST /api/v1/sources/upload - Upload a new source document

  • Calculates SHA-256 hash for deduplication
  • Processes document with docling
  • Stores in MinIO and metadata in PostgreSQL

• GET /api/v1/sources - List all uploaded sources

  • Supports filtering and pagination
  • Returns source metadata with upload info

Database Schema

PostgreSQL Tables

• users: User accounts and authentication
• sources: Source document metadata (hash, storage path, upload info)
• extraction_artifacts: Versioned LLM extraction results (Phase 2)
• source_ratings: User ratings (1-5 stars) for sources
• source_comments: User commentary on sources

Neo4j Graph Schema

Node Types:

• Episode (source document)
• Person, Organization, Location
• Event, Technology, Symptom, Project

Relationship Properties:

• Graphiti bi-temporal model: created_at, expired_at, valid_at, invalid_at

Project Structure

DeepInvestigate/
├── backend/
│   ├── app/
│   │   ├── api/v1/          # API endpoints
│   │   ├── core/            # Configuration and utilities
│   │   ├── models/          # Database models
│   │   └── services/        # Business logic services
│   ├── tests/               # Backend tests
│   └── requirements.txt     # Python dependencies
├── frontend/                # React Native app (Phase 3)
├── infrastructure/
│   └── docker/              # Docker initialization scripts
├── docker-compose.yml       # Infrastructure services
├── .env.example             # Environment template
└── README.md

Implementation Phases

✅ Phase 1: Foundational Backend & Ingestion (Current)

• Docker infrastructure setup
• FastAPI backend with upload/list endpoints
• Document hashing and deduplication
• docling integration for document processing

Phase 2: Graph Construction & Artifact Storage

• Knowledge graph ontology definition
• Graph Architect Agent for entity extraction
• Three-tier caching (Redis → PostgreSQL → LLM)
• Graphiti integration with Neo4j

Phase 3: Frontend UI & Initial Agent Interaction

• React Native mobile application
• Source library with search and filters
• Basic investigative agent interface

Phase 4: Advanced Synthesis & Report Generation

• STORM-inspired synthesis agent
• Query decomposition and outline generation
• Citation mechanism with provenance links

Phase 5: Curation, Collaboration & Production Hardening

• Human-in-the-loop curation tools
• Knowledge graph promotion (staging → canonical)
• Production deployment and scaling

Key Design Principles

1. Integrated Provenance: Every fact links to its source document (Episode node)
2. Federated Knowledge Spaces: Multiple graphs (canonical_kg, staging_kg, hypothesis_kg)
3. Versioned Extractions: Cache LLM results by content hash + extraction version
4. Entity Normalization: Critical for graph quality and deduplication

Error Tracking & Monitoring

DeepInvestigate integrates with Sentry.io for comprehensive error tracking and performance monitoring.

Setting up Sentry (Optional)

1. Create a free account at https://sentry.io
2. Create a new Python project for the backend
3. Copy your DSN from the project settings
4. Update backend/.env:

   ENABLE_SENTRY=true
   SENTRY_DSN=https://[your-key]@o[org-id].ingest.sentry.io/[project-id]

For detailed setup instructions, see backend/SENTRY_SETUP_GUIDE.md

Note: Sentry is optional for local development but recommended for staging and production environments.

Troubleshooting

Services won't start

# Check Docker is running
docker --version

# View service logs
docker-compose logs

# Restart services
docker-compose restart

Database connection errors

# Verify PostgreSQL is healthy
docker-compose ps postgres

# Check database logs
docker-compose logs postgres

# Manually test connection
docker exec -it deepinvestigate-postgres psql -U deepinvestigate -d deepinvestigate

MinIO bucket not created

The application will automatically create the documents bucket on first use. If manual creation is needed:

1. Open MinIO Console: http://localhost:9001
2. Login with credentials from .env
3. Create bucket named documents

Contributing

This project is in active development. See CLAUDE.md for detailed architectural guidance and development instructions.

License

[License TBD]