# MasterScraper

 A generic Django-based scraper for collecting X posts via Nitter and RSS/Google Alerts feeds, organized by categories (e.g., WELLNESS, TECH) and keywords (e.g., #wellbeing, tesla). Stores X data in `RawData` and feed data in `FeedData` within a PostgreSQL database (`dataengine`). Includes a Django admin interface for managing configurations and triggering scraping.

 ## Features
 - Scrapes X posts for configurable categories and keywords via Nitter.
 - Scrapes RSS and Google Alerts feeds for keyword-matched content.
 - Supports hashtag, term, user modes for X; keyword matching for RSS/Alerts.
 - Stores X data in `RawData` (text, user_handle, stats_likes, etc.).
 - Stores feed data in `FeedData` (title, text, source_url, source_type).
 - Commands: `check_db`, `load_config`, `print_config`, `scrape_x`, `scrape_rss`, `scrape_alerts`.
 - Django admin interface to view/edit configurations and trigger scraping.
 - Foolproof design with retries, validation, and logging.

 ## Prerequisites
 - **Python**: 3.8 or higher
 - **PostgreSQL**: 12 or higher, with database `dataengine`
 - **Virtualenv**: Recommended for isolated environment
 - **Dependencies**: Listed in `requirements.txt`

 ## Installation

 1. **Clone the Repository** (if applicable):
    ```bash
    git clone <repository-url>
    cd MasterScraper