RAG-Based Article Generator

A Retrieval Augmented Generation (RAG) system that generates newspaper-style articles using different knowledge sources and the Google Gemini API.

Project Overview

This project implements a RAG (Retrieval Augmented Generation) system that:

1. Retrieves information from either Wikipedia or user-uploaded text files
2. Processes and embeds the text using HuggingFace embeddings
3. Stores the embeddings in a Chroma vector database
4. Performs similarity search to find relevant context for a user query
5. Generates a newspaper-style article using the Google Gemini API

The project includes two main implementations:

• Wikipedia-based RAG: Fetches information from Wikipedia based on user queries
• File-based RAG: Uses content from user-uploaded text files as the knowledge base

Both implementations are accessible through Streamlit web applications.

Files and Components

Core Files

• requirements.txt: Lists all Python dependencies required for the project
• rag.ipynb: Jupyter notebook demonstrating the RAG implementation with Wikipedia
• fetch.py: Utility script to download Wikipedia content about eco-friendly topics

RAG Implementation Files

• rag_with_wiki.py: Implementation of RAG using Wikipedia as the knowledge source
• rag_with_uploaded.py: Implementation of RAG using user-uploaded files as the knowledge source

Web Application Files

• app_wiki.py: Streamlit web application for the Wikipedia-based RAG
• app_upload.py: Streamlit web application for the file-based RAG

Installation

1. Clone this repository
2. Install the required dependencies:

pip install -r requirements.txt

3. Set up your environment variables:
   • Create a .env file in the project root
   • Add your Google Gemini API key: GEMINI_API_KEY=your_api_key_here

Usage

Wikipedia-Based Article Generator

Run the Wikipedia-based web application:

streamlit run app_wiki.py

1. Enter a topic or query in the text input field
2. Click "Generate Article"
3. The application will:
   • Fetch relevant information from Wikipedia
   • Process and embed the text
   • Generate a newspaper-style article using the Gemini API

File-Based Article Generator

Run the file-based web application:

streamlit run app_upload.py

1. Upload one or more text files to provide context
2. Enter a topic or query in the text input field
3. Click "Generate Article"
4. The application will:
   • Process and embed the content from your uploaded files
   • Generate a newspaper-style article using the Gemini API

Fetching Wikipedia Content

To download Wikipedia content about eco-friendly topics:

python fetch.py

This will create a directory called ecofriendly_activities containing text files with Wikipedia content on various eco-friendly topics.

Technical Details

RAG Architecture

The project implements a standard RAG architecture:

1. Retrieval:

   • Wikipedia API or user-uploaded files provide the knowledge base
   • Text is embedded using HuggingFace's "all-MiniLM-L6-v2" model
   • Embeddings are stored in a Chroma vector database

2. Augmentation:

   • User queries are used to perform similarity search in the vector database
   • The most relevant documents are retrieved and combined to form the context

3. Generation:

   • The context and query are sent to the Google Gemini API
   • A prompt instructs the model to generate a newspaper-style article
   • The generated article is returned to the user

Dependencies

Key dependencies include:

• langchain: Framework for building applications with LLMs
• langchain_huggingface: HuggingFace integration for LangChain
• langchain_community: Community components for LangChain
• chromadb: Vector database for storing embeddings
• streamlit: Web application framework
• wikipedia: Python library for accessing Wikipedia content
• dotenv: For loading environment variables
• requests: For making HTTP requests to the Gemini API

Future Improvements

Potential enhancements for this project:

• Add support for more file formats (PDF, DOCX, etc.)
• Implement caching to improve performance
• Add more customization options for the generated articles
• Implement error handling and retry mechanisms for API calls
• Add authentication for the web applications
• Expand to other LLM providers beyond Google Gemini