Skip to content

README.md

Latest commit

 

History

History
134 lines (105 loc) · 3.28 KB

README.md

File metadata and controls

134 lines (105 loc) · 3.28 KB
Raw

Study Case Vidavox ETL Pipeline

A comprehensive ETL (Extract, Transform, Load) pipeline for processing research papers, extracting text and charts, and providing a searchable interface.

🌟 Features

  • Text Extraction: Uses PaddleOCR for accurate text extraction from PDFs
  • Chart Detection: OpenCV-based chart detection and analysis
  • NLP Processing: SpaCy integration for entity recognition and text analysis
  • Vector Search: FAISS implementation for semantic search capabilities
  • REST API: FastAPI-powered API for data access
  • Interactive UI: Streamlit-based frontend for visualization

🛠️ Technology Stack

  • OCR Engine: PaddleOCR
  • Image Processing: OpenCV
  • NLP: SpaCy, Sentence Transformers
  • Vector Search: FAISS
  • Database: PostgreSQL
  • API: FastAPI
  • Frontend: Streamlit
  • Language: Python 3.8+

📋 Prerequisites

  • Python 3.8 or higher
  • PostgreSQL 12 or higher
  • Virtual environment (recommended)

🚀 Installation

  1. Clone the repository:
mkdir research-paper-etl
cd research-paper-etl
git clone https://github.com/siabang35/PaddleOCRForPDF.git
  1. Create and activate a virtual environment:
python -m venv env
source env/bin/activate #On Linux:
env\Scripts\activate #On Windows:
  1. Install dependencies:
pip install -r requirements.txt
python -m spacy download en_core_web_sm
  1. Set up environment variables: Create a .env file with your database credentials:
DATABASE_URL=your_supabasedb
  1. Initialize the database:
psql -U postgres -f database/db.sql

🏃‍♂️ Running the Pipeline

  1. Extract text and charts:
python pipeline/extract.py
  1. Transform and process data:
python pipeline/transform.py
  1. Load data into database:
python pipeline/load.py

🌐 Starting the Services

  1. Start the API server:
uvicorn backend.api.main:app --reload --port 8000
  1. Launch the frontend:
streamlit run frontend/app.py

📁 Project Structure

research-paper-etl/
├── api/                 # FastAPI application
│   ├── main.py         # API endpoints
│   ├── models.py       # Pydantic models
│   └── db.py           # Database operations
├── pipeline/           # ETL pipeline components
│   ├── extract.py      # Text and chart extraction
│   ├── transform.py    # Data processing
│   └── load.py         # Database loading     
    └── logger.py
    └── config.py
├── frontend/           # Streamlit frontend
│   └── app.py         # UI implementation
├── data/              # Data storage
│   ├── input/         # Input PDFs
│   └── processed/     # Processed data
├── database/          # Database scripts
│   └── db.sql       # Database initialization
└── requirements.txt   # Python dependencies

🔍 API Endpoints

  • GET /: API health check
  • GET /documents/{doc_id}: Retrieve document by ID
  • POST /search: Search documents

👥 Contributing

  1. Fork the repository
  2. Create a feature branch
  3. Commit your changes
  4. Push to the branch
  5. Create a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.