Document AI Platform

An intelligent knowledge platform that analyzes any text-based dataset (documents, codebases, logs, notes, etc.) and maintains contextual memory. The system intelligently parses, stores, and retrieves information while providing insightful answers and generating comprehensive documentation.

Features

Core Capabilities

• Multi-Format Parsing: PDF, Text, Markdown, HTML, Code (40+ languages)
• GPU-Accelerated Processing: CUDA/MPS support for fast embeddings
• Intelligent Chunking: Content-aware chunking that preserves document structure
• Vector Storage: ChromaDB for semantic search and retrieval
• Deduplication: Automatic detection of duplicate documents

Query & Analysis

• LLM-Powered Insights: Synthesizes answers from multiple sources using Ollama
• Contextual Retrieval: Retrieves surrounding context for better answers
• Dataset Awareness: Understands the overall structure of your knowledge base

Documentation Generation

• Insightful Wiki Pages: Generates comprehensive topic pages with cross-references
• Related Work Linking: Automatically finds and links related topics
• Source Summaries: Creates summaries for each source document
• Purpose-Driven: Custom prompts to tailor wiki for specific audiences and purposes
• HTTP Serving: Serve generated docs with built-in HTTP server

Visualization

• Relationship Graphs: Interactive D3.js visualizations
• Topic Maps: Topic-centric graph views
• Timeline Views: Temporal document analysis

Installation

Prerequisites

• Python 3.10+
• Docker (for Ollama LLM)
• NVIDIA GPU + CUDA (optional, for faster processing)

Setup

# Clone the repository
git clone https://github.com/yourusername/document-ai.git
cd document-ai

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Install in development mode
pip install -e .

# For GPU support (NVIDIA), install CUDA PyTorch:
pip install torch --index-url https://download.pytorch.org/whl/cu118

GPU Support

The platform automatically detects and uses GPU if available:

• NVIDIA CUDA: Install PyTorch with CUDA (see above)
• Apple MPS: Works automatically on M1/M2 Macs
• CPU Fallback: Works on any system (slower)

Quick Start

1. Initialize a Project

docai init

2. Add Files

Add files to ./file_store:

• PDFs, text files, markdown
• Code files (Python, JavaScript, etc.)
• HTML files, configuration files

3. Analyze Files

# Analyze all files recursively
docai analyze -r

# Force re-analysis
docai analyze -r --force

4. Query the Knowledge Base

# Get insightful answers synthesized from your data
docai query "What techniques are used for browser fingerprinting?"

# Query with more context
docai query "Compare different anti-fingerprinting approaches" --top-k 20

5. Generate Documentation

# Generate topic page
docai document "Browser Fingerprinting"

# Export all documentation
docai export

# Export with custom purpose/audience prompt
docai export --prompt "This wiki is for web scraping professionals to understand industry challenges and solutions"

# Fast export (skip some LLM calls)
docai export --fast

6. Serve Documentation

# Serve docs on localhost:8080
docai serve

# Custom port
docai serve --port 3000

7. Visualize Relationships

# Generate interactive relationship graph
docai visualize --type relationships

# Generate topic map
docai visualize --type topics --format html

CLI Commands

Command	Description
docai init [PATH]	Initialize a new project
docai analyze [PATH] -r	Analyze files recursively
docai query <QUESTION>	Ask questions and get synthesized answers
docai document <TOPIC>	Generate documentation for a topic
docai export [--prompt]	Export all documentation (with optional purpose prompt)
docai serve	Serve documentation via HTTP
docai scrape <URL>	Scrape a blog/website and download posts
docai status	Show processing status and dataset insights
docai config	View/manage configuration
docai watch	Watch for file changes
docai visualize	Generate visualizations

Configuration

Configuration is stored in docai.yaml:

project:
  name: "my-project"
  file_store: "./file_store"
  output_dir: "./docs"

llm:
  model: "llama3.2:3b"
  temperature: 0.3
  max_tokens: 2048

chunking:
  size: 1000
  overlap: 200
  preserve_sections: true
  content_aware: true

retrieval:
  top_k: 10
  similarity_threshold: 0.3 # Lower = more results

chromadb:
  persist_directory: "./.docai/chroma"
  collection_name: "documents"
  embedding_model: "sentence-transformers/all-MiniLM-L6-v2"

LLM Setup (Ollama via Docker)

For synthesized answers, run Ollama in Docker:

Option 1: Docker Compose (Recommended)

# Start Ollama
docker compose up -d

# Pull the model
docker exec ollama ollama pull llama3.2:3b

For GPU support, uncomment the deploy section in docker-compose.yml.

Option 2: Docker Run

# With GPU
docker run -d --gpus all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

# Without GPU
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

# Pull model
docker exec ollama ollama pull llama3.2:3b

Quick Commands

docker stop ollama      # Stop
docker start ollama     # Start
docker logs ollama      # View logs
docker rm -f ollama     # Remove

The platform connects to http://localhost:11434 automatically. Falls back to context-only responses if unavailable.

Supported File Types

Documents

• PDF (.pdf) - Academic papers, reports, documentation
• Text (.txt) - Plain text files
• Markdown (.md, .markdown) - Documentation, notes
• HTML (.html, .htm) - Web pages, exported docs

Code (40+ Languages)

• Python (.py)
• JavaScript/TypeScript (.js, .ts, .jsx, .tsx)
• Go (.go)
• Rust (.rs)
• Java (.java)
• C/C++ (.c, .cpp, .h, .hpp)
• And many more...

Data Files

• JSON (.json)
• YAML (.yaml, .yml)
• XML (.xml)
• CSV (.csv)
• TOML (.toml)

Architecture

docai/
├── agents/           # Document analysis & generation agents
│   ├── analyzer.py   # Document analysis with topic extraction
│   └── generator.py  # Insightful documentation generator
├── parsers/          # File parsers
│   ├── pdf.py        # PDF parser with structure detection
│   ├── text.py       # Text/Markdown parser
│   ├── html.py       # HTML parser + web scraper
│   └── code.py       # Code parser (multi-language)
├── memory/           # ChromaDB integration
│   ├── store.py      # GPU-accelerated storage with dedup
│   └── retrieval.py  # Contextual retrieval with synthesis
├── workflows/        # LangGraph workflows
│   └── documentation.py
├── llm/              # LLM integration
│   ├── ollama.py     # Ollama client
│   └── prompts.py    # Insight-focused prompts
├── utils/            # Utilities
│   ├── chunking.py   # Content-aware chunking
│   └── visualization.py
├── cli.py            # CLI with serve command
└── config.py         # Configuration management

Development

Run Tests

pytest tests/ -v

# With coverage
pytest tests/ --cov=docai --cov-report=html

Code Style

# Format code
black docai/ tests/

# Lint
ruff check docai/

# Type check
mypy docai/

Performance Tips

1. Use GPU: Install CUDA-enabled PyTorch for 10x faster embeddings
2. Batch Processing: Process files in batches with docai analyze -r
3. Lower Threshold: Set similarity_threshold: 0.3 for better recall
4. Skip Duplicates: Deduplication is automatic - no need to reprocess

Roadmap

• Phase 1: Foundation (Multi-format parsing, ChromaDB, CLI)
• Phase 2: Core Analysis (LangGraph workflow, Ollama integration)
• Phase 3: Documentation Generation (Wiki pages, cross-references)
• Phase 4: Enhancement (GPU support, HTTP serve, deduplication)
• Phase 5: Advanced Features (Multi-user, AWS deployment)

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run tests
5. Submit a pull request

License

MIT License - see LICENSE file for details.