Research Harvester

An arXiv research-paper pipeline: harvest → store → index → search → ask → digest. It fetches papers from arXiv, stores the PDFs in MinIO, indexes embeddings in Qdrant, keeps metadata in PostgreSQL, and exposes a FastAPI service for semantic search, LLM-powered Q&A over the corpus (Gemini), and a weekly email digest.

Status: early stage — the core pipeline (harvest → store → index → search → ask) works.

Architecture

arXiv ──harvest──> PostgreSQL (metadata)
                   MinIO (PDF blobs)
                   Qdrant (embeddings) ──search/ask──> FastAPI ──> you / weekly email digest

Infrastructure (Postgres, MinIO, Qdrant) runs via Docker Compose; the FastAPI app runs locally.

API

Method	Route	Purpose
GET	/api/health	Liveness + DB check
POST	/api/harvest	Fetch papers from arXiv and store/index them
GET	/api/search	Semantic search over indexed papers
GET	/api/papers/{paper_id}/pdf	Fetch a stored PDF from MinIO
GET	/api/ask?query=...	LLM Q&A grounded in the indexed corpus (Gemini)
POST	/api/email	Send a digest email

Run locally

# 1. Start infrastructure (Postgres + MinIO + Qdrant)
docker compose up -d

# 2. Configure
cp .env.example .env        # fill in GEMINI_API_KEY + email vars if you want Q&A / digest

# 3. Install + run the API
pip install -r requirements.txt
uvicorn app.main:app --reload
# docs at http://localhost:8000/docs

Optional: python scheduler.py runs the weekly harvest + digest job.

Stack

FastAPI · PostgreSQL (SQLAlchemy) · MinIO · Qdrant · sentence-transformers · Gemini · Docker Compose