LLM Auto RAG Survey

LLM Auto RAG Survey is a literature discovery pipeline for organizing arXiv papers around large language models, retrieval-augmented generation, agents, evaluation, multimodal models, and related research directions.

The system uses the Kaggle arXiv metadata snapshot as its source corpus, Ollama qwen3-embedding:8b for local semantic embedding, FAISS for vector retrieval, and Gemini 3 Pro for taxonomy-level classification. It stops at structured classification and Markdown reporting; it does not generate a finished literature review.

The configured Gemini model is gemini-3-pro-preview. Google lists Gemini 3 Pro Preview with a 1M-token input context window and up to 64k output tokens, which is large enough for survey-scale classification prompts built from about 1,000 paper-level title/abstract records. Full-PDF survey writing should still be handled by a human workflow or a chunked reading pipeline.

Project Goals

• Build a reproducible arXiv metadata processing pipeline.
• Use local embedding to retrieve LLM/RAG-related papers at scale.
• Use local semantic embeddings and vector indexing for transparent candidate retrieval.
• Use Gemini 3 Pro only for classification into research topics.
• Produce a Markdown report that supports human reading, selection, and writing.

Pipeline

1. step1_download_arxiv_dataset.py prepares the Kaggle arXiv metadata snapshot.
2. step2_prepare_corpus.py filters and normalizes LLM/RAG-related arXiv records.
3. step3_embed_corpus.py embeds paper titles and abstracts with Ollama qwen3-embedding:8b.
4. step4_build_faiss_index.py builds a FAISS index over normalized embeddings.
5. step5_retrieve_candidates.py retrieves candidate papers for configured LLM/RAG queries.
6. step6_deduplicate_candidates.py merges duplicate papers across query results.
7. step7_classify_with_gemini.py classifies candidates with gemini-3-pro-preview.
8. step8_generate_markdown_report.py writes a Markdown classification report.

step9_generate_survey_draft.py is intentionally manual-only. It is not called by main.py, because the Step 8 report should be reviewed, edited, and verified by a human before draft generation.

Technical Stack

• Python
• Kaggle API
• Ollama
• Qwen3 Embedding 8B
• FAISS
• Gemini API
• HDF5 / JSONL

Model Capacity

• Embedding model: qwen3-embedding:8b through Ollama.
• Classification model: gemini-3-pro-preview.
• Context budget: Gemini 3 Pro Preview provides a 1M-token input context and up to 64k output tokens, based on the Gemini 3 Developer Guide.
• Intended use: load and classify a survey-scale candidate set, such as roughly 1,000 arXiv title/abstract records, then output a structured taxonomy report.
• Cost control: .env ships with placeholder keys and the pipeline does not call Gemini unless a real GEMINI_API_KEY is provided.
• Reference: https://ai.google.dev/gemini-api/docs/gemini-3

Project Structure

.
├── main.py
├── requirements.txt
├── .env.example
├── queries/
│   └── llm_rag_queries.txt
├── src/
│   ├── config.py
│   ├── step1_download_arxiv_dataset.py
│   ├── step2_prepare_corpus.py
│   ├── step3_embed_corpus.py
│   ├── step4_build_faiss_index.py
│   ├── step5_retrieve_candidates.py
│   ├── step6_deduplicate_candidates.py
│   ├── step7_classify_with_gemini.py
│   ├── step8_generate_markdown_report.py
│   └── step9_generate_survey_draft.py
├── data/
│   ├── raw/
│   ├── processed/
│   ├── embeddings/
│   ├── index/
│   └── results/
└── reports/

Configuration

Create .env from .env.example and fill in local credentials only when the corresponding step should be run.

KAGGLE_USERNAME=enter_your_kaggle_username
KAGGLE_KEY=enter_your_kaggle_api_key
GEMINI_API_KEY=enter_your_gemini_api_key
GEMINI_MODEL=gemini-3-pro-preview
OLLAMA_EMBEDDING_MODEL=qwen3-embedding:8b

All data paths are relative paths. The repository is configured to ignore local credentials, downloaded data, embedding stores, FAISS indexes, and generated reports.

Usage

Install dependencies:

pip install -r requirements.txt

Pull the local embedding model:

ollama pull qwen3-embedding:8b

Run the full pipeline:

python main.py

Individual steps can also be run directly:

python -m src.step2_prepare_corpus
python -m src.step3_embed_corpus
python -m src.step4_build_faiss_index

Data Source

The pipeline expects the Kaggle arXiv metadata snapshot:

Cornell-University/arxiv

The raw metadata file should be located at:

data/raw/arxiv-metadata-oai-snapshot.json

Automatic Kaggle download is disabled by default:

ENABLE_KAGGLE_DOWNLOAD=false

Set it to true only when the dataset should be downloaded through the Kaggle API.

Output

The final report is generated at:

reports/auto_rag_survey_report.md

The report contains category distributions, relevance counts, representative papers, retrieval scores, arXiv links, and short classification evidence. It is intended as a structured reading aid, not as a completed academic survey.

Manual Final Writing Step

The main pipeline stops after Step 8. To generate a survey draft, first review and edit:

reports/auto_rag_survey_report.md

Save the human-edited version as:

reports/auto_rag_survey_report_verified.md

Then run Step 9 manually:

python -m src.step9_generate_survey_draft

Step 9 uses gemini-3-pro-preview to produce:

reports/survey_draft.md

The prompt is designed for academic drafting: it preserves arXiv identifiers, avoids invented citations or experimental claims, marks weak evidence cautiously, and produces a draft intended for human revision. Gemini 3 Pro Preview's 1M input context and 64k output budget make this final step suitable for a large manually curated survey report containing roughly 1,000 paper-level records.

Current Status

• Project structure is complete.
• The pipeline is implemented as eight explicit steps.
• Runtime credentials and generated artifacts are excluded from version control.
• The report layer is limited to classification and metadata organization.

Notes

Before using the generated results in academic writing, paper metadata, citations, claims, and conclusions should be manually verified against the original arXiv pages and papers.