🔭 SAAO arXiv Paper Curator

A comprehensive Retrieval-Augmented Generation (RAG) system designed to ingest, process, and query astrophysics research papers from arXiv. Built to support the South African Astronomical Observatory (SAAO) research community with intelligent paper discovery and question-answering capabilities.

✨ Features

Automated Paper Ingestion: Fetch papers from arXiv using a configurable list of paper IDs
PDF Processing: Deep parsing with Docling for structured text extraction
Hybrid Search: Combined BM25 + vector search using OpenSearch for optimal retrieval
RAG Question Answering: Ask natural language questions and receive contextual answers with citations
Streaming Responses: Real-time streaming answers for better user experience
Workflow Orchestration: Apache Airflow DAGs for automated ingestion pipelines
Observability: Full tracing with Langfuse for debugging and analytics
Caching: Redis-based query caching for improved performance
Modern UI: Gradio-powered web interface for interactive exploration

🏗 Architecture Overview

┌─────────────────────────────────────────────────────────────────────────────┐
│                              SAAO arXiv RAG System                          │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐  │
│  │   Gradio    │    │  FastAPI    │    │   Airflow   │    │  Langfuse   │  │
│  │   (UI)      │───▶│   (API)     │◀───│   (DAGs)    │    │ (Tracing)   │  │
│  │  Port 7860  │    │  Port 8000  │    │  Port 8080  │    │  Port 3001  │  │
│  └─────────────┘    └──────┬──────┘    └──────┬──────┘    └─────────────┘  │
│                            │                  │                             │
│  ┌─────────────────────────┼──────────────────┼────────────────────────┐   │
│  │                         ▼                  ▼                         │   │
│  │  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐              │   │
│  │  │  OpenSearch │    │  PostgreSQL │    │    MinIO    │              │   │
│  │  │  (Vectors)  │    │    (DB)     │    │  (Storage)  │              │   │
│  │  │  Port 9200  │    │  Port 5432  │    │  Port 9000  │              │   │
│  │  └─────────────┘    └─────────────┘    └─────────────┘              │   │
│  │                                                                      │   │
│  │  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐              │   │
│  │  │   Ollama    │    │    Redis    │    │ ClickHouse  │              │   │
│  │  │   (LLM)     │    │   (Cache)   │    │ (Analytics) │              │   │
│  │  │  Port 11434 │    │  Port 6379  │    │             │              │   │
│  │  └─────────────┘    └─────────────┘    └─────────────┘              │   │
│  │                         Data Layer                                   │   │
│  └──────────────────────────────────────────────────────────────────────┘   │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

📋 Prerequisites

Before installing the application, ensure you have the following tools installed on your system.

Required Software

Tool	Version	Purpose
Docker	24.0+	Container runtime
Docker Compose	2.20+	Multi-container orchestration
Python	3.12+	Runtime environment
uv	0.4+	Fast Python package manager
Make	3.80+	Build automation

Installing Prerequisites

1. Docker Desktop (macOS/Windows)

Download and install Docker Desktop.

# Verify installation
docker --version
docker compose version

Important: For this application, configure Docker Desktop with at least:

Memory: 12GB+ (required for PDF processing and LLM inference)
Swap: 4GB+
Disk: 50GB+

2. uv (Python Package Manager)

Install uv - a blazing-fast Python package manager:

# macOS/Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows (PowerShell)
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

# Verify installation
uv --version

3. Make (Build Tool)

# macOS (with Xcode Command Line Tools)
xcode-select --install

# Ubuntu/Debian
sudo apt-get install build-essential

# Windows (using Chocolatey)
choco install make

🚀 Installation

Step 1: Clone the Repository

git clone https://github.com/Danselem/saao-arxiv-paper.git
cd saao-arxiv-paper

Step 2: Initialize the Python Environment

# Create virtual environment and install dependencies
make init
make install

Step 3: Configure Environment Variables

# Copy the example environment file
make env
# or manually:
cp .env.example .env

Edit .env to configure your settings. Key variables include:

# Database
POSTGRES_DATABASE_URL=postgresql+psycopg2://rag_user:rag_password@localhost:5432/rag_db

# OpenSearch
OPENSEARCH__HOST=http://localhost:9200

# Ollama (LLM)
OLLAMA__HOST=http://localhost:11434
OLLAMA__MODEL=llama3.2:1b

# MinIO (Object Storage)
MINIO__ENDPOINT=http://localhost:9000
MINIO__ACCESS_KEY=minioadmin
MINIO__SECRET_KEY=minioadmin

# Redis (Cache)
REDIS__HOST=localhost
REDIS__PORT=6379

# Langfuse (Observability) - Optional
LANGFUSE__ENABLED=true
LANGFUSE__PUBLIC_KEY=your_public_key
LANGFUSE__SECRET_KEY=your_secret_key

Step 4: Start Services

# Start all services
make start

# Or start specific service groups (see Service Deployment Profiles below)
make up-core    # Core infrastructure only
make up-api     # API + Core dependencies
make ingest     # Full ingestion pipeline
make rag-app    # Complete RAG application with observability

Step 5: Pull Required LLM Models

make pull-models

This downloads the required Ollama models:

nomic-embed-text - For generating embeddings
llama3.2:1b - For RAG answer generation

⚡ Quick Start

After installation, get up and running in minutes:

# 1. Start the full stack
make rag-app

# 2. Pull LLM models (first time only)
make pull-models

# 3. Add paper IDs to airflow/papers.txt
echo "2401.12345" >> airflow/papers.txt

# 4. Open Airflow UI and trigger the ingestion DAG
open http://localhost:8080
# Login: admin / admin
# Trigger: fetch_saao_papers DAG

# 5. Launch the Gradio UI
make app

# 6. Open the application
open http://localhost:7860

📦 Service Deployment Profiles

The application supports flexible deployment with different service combinations:

Command	Services Started	Use Case
`make up-core`	PostgreSQL, Redis, MinIO	Development/Testing
`make up-api`	Core + OpenSearch + API	API Development
`make up-pipeline`	Core + OpenSearch + Ollama + Dashboards	Indexing Pipeline
`make ingest`	Pipeline + Airflow	Full Ingestion Workflow
`make rag-app`	All services including Langfuse	Production RAG App
`make start`	Everything	Full Stack

🌐 Microservices & Ports

Service	Container Name	Port	Description
FastAPI	`rag-api`	`8000`	REST API for RAG queries and search
Gradio UI	Local	`7860`	Web interface for interactive queries
Airflow	`rag-airflow`	`8080`	Workflow orchestration and DAG management
OpenSearch	`rag-opensearch`	`9200`	Vector search and BM25 indexing
OpenSearch Dashboards	`rag-dashboards`	`5601`	Search analytics and visualization
PostgreSQL	`rag-postgres`	`5432`	Paper metadata and application state
Redis	`rag-redis`	`6379`	Query caching and rate limiting
MinIO	`rag-minio`	`9000` / `9001`	PDF storage (API / Console)
Ollama	`rag-ollama`	`11434`	Local LLM inference (embeddings + generation)
Langfuse Web	`rag-langfuse-web`	`3001`	Observability and tracing dashboard
ClickHouse	`rag-clickhouse`	`8123`	Analytics database for Langfuse

📚 Usage Guide

Adding Papers for Ingestion

Add arXiv paper IDs to airflow/papers.txt:
```
2401.12345
2312.67890
2311.54321
```
Trigger the ingestion pipeline through Airflow UI:
- Navigate to http://localhost:8080
- Login with admin / admin
- Enable and trigger the fetch_saao_papers DAG

Querying via API

# Health check
curl http://localhost:8000/api/v1/health

# Search papers
curl -X POST http://localhost:8000/api/v1/search \
  -H "Content-Type: application/json" \
  -d '{"query": "exoplanet detection methods", "size": 5}'

# Ask a question (RAG)
curl -X POST http://localhost:8000/api/v1/ask \
  -H "Content-Type: application/json" \
  -d '{"query": "What are the main techniques for detecting exoplanets?", "top_k": 3}'

Using the Gradio Interface

make app
# Open http://localhost:7860 in your browser

Features:

Query Input: Enter natural language questions
Category Filters: Filter by arXiv categories (astro-ph.GA, astro-ph.SR, etc.)
Model Selection: Choose between different LLM models
Hybrid Search Toggle: Enable/disable vector + BM25 search
Source Links: Direct links to source papers

🛠 Development

Code Quality

# Format code
make format

# Lint and fix issues
make lint

# Run tests
make test

# Run tests with coverage
make test-cov

Useful Commands

# View service logs
make logs

# Check service health
make health

# View running containers
make status

# Stop all services
make stop

# Clean up (remove volumes)
make clean

Project Structure

saao-arxiv-paper/
├── airflow/                 # Airflow DAGs and configuration
│   ├── dags/               # DAG definitions
│   │   └── arxiv_ingestion/ # Paper ingestion pipeline
│   ├── papers.txt          # List of arXiv IDs to ingest
│   └── Dockerfile          # Airflow container
├── src/                    # Main application source
│   ├── config.py           # Centralized configuration
│   ├── main.py             # FastAPI application entry
│   ├── gradio_app.py       # Gradio UI
│   ├── routers/            # API endpoints
│   ├── services/           # Business logic
│   │   ├── arxiv/          # arXiv API client
│   │   ├── embeddings/     # Ollama embeddings
│   │   ├── indexing/       # Hybrid indexer
│   │   ├── opensearch/     # Search client
│   │   ├── pdf_parser/     # Docling parser
│   │   └── ollama/         # LLM client
│   ├── models/             # SQLAlchemy models
│   ├── schemas/            # Pydantic schemas
│   └── repositories/       # Data access layer
├── compose.yaml            # Docker Compose configuration
├── Dockerfile              # API container
├── Makefile                # Build automation
├── pyproject.toml          # Python project configuration
└── requirements.txt        # Python dependencies

⚙️ Configuration

Environment Variables Reference

Variable	Default	Description
`POSTGRES_DATABASE_URL`	`postgresql+psycopg2://rag_user:rag_password@localhost:5432/rag_db`	PostgreSQL connection string
`OPENSEARCH__HOST`	`http://localhost:9200`	OpenSearch endpoint
`OLLAMA__HOST`	`http://localhost:11434`	Ollama API endpoint
`OLLAMA__MODEL`	`llama3.2:1b`	Default LLM model
`EMBEDDING_MODEL`	`nomic-embed-text`	Embedding model name
`MINIO__ENDPOINT`	`http://localhost:9000`	MinIO endpoint
`MINIO__ACCESS_KEY`	`minioadmin`	MinIO access key
`MINIO__SECRET_KEY`	`minioadmin`	MinIO secret key
`MINIO__BUCKET_NAME`	`arxiv-pdfs`	PDF storage bucket
`REDIS__HOST`	`localhost`	Redis host
`REDIS__PORT`	`6379`	Redis port
`CHUNKING__CHUNK_SIZE`	`600`	Words per chunk
`CHUNKING__OVERLAP_SIZE`	`100`	Overlap between chunks

🔧 Troubleshooting

Common Issues

1. Ollama Connection Errors

Cannot connect to Ollama service: All connection attempts failed

Solution: Ensure Ollama is running and the OLLAMA__HOST is correctly configured.

# Check if Ollama is running
docker ps | grep ollama

# Restart Ollama
docker compose restart ollama

# Verify connectivity
curl http://localhost:11434/api/version

2. Out of Memory (OOM) During PDF Processing

Task exited with return code -9

Solution: Increase Docker memory allocation to at least 12GB in Docker Desktop settings.

3. OpenSearch Index Not Found

index_not_found_exception

Solution: Run the indexing pipeline to create the index:

# Trigger indexing through Airflow or directly
make ingest

4. Model Not Found

model 'nomic-embed-text' not found

Solution: Pull the required models:

make pull-models

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

👤 Author

Daniel Egbo

GitHub: @Danselem

🙏 Acknowledgments

Docling - PDF document processing
Ollama - Local LLM inference
OpenSearch - Search and vector database
Langfuse - LLM observability
FastAPI - Modern Python web framework
Gradio - ML web interfaces

Made with ❤️ for the Astrophysics Research Community

Name		Name	Last commit message	Last commit date
airflow		airflow
src		src
tests		tests
.env.example		.env.example
.env.test		.env.test
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
compose.yaml		compose.yaml
gradio_launcher.py		gradio_launcher.py
pyproject.toml		pyproject.toml
requirements-dev.txt		requirements-dev.txt
requirements.txt		requirements.txt
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation