Verify Scraper

A Django web application for scraping reviews, pricing data, and news content from web sources.

Project Overview

This project includes:

• A Django app named scraper
• REST API endpoints for scraping reviews, pricing, and news
• A homepage UI for selecting scraping methods
• Environment-based database configuration with SQLite fallback for local development

Features

• POST /scrape/reviews - expected payload: query, source
• POST /scrape/pricing - expected payload: url
• POST /scrape/news - expected payload: query
• Simple frontend served from scraper/templates/scraper/home_page.html

Note: current scraper logic in scraper/handlers.py returns placeholder sample responses.

Requirements

• Python 3.11+ (project uses Python 3.14 in the current virtual environment)
• Django
• Django REST Framework
• psycopg2-binary (for PostgreSQL support)
• python-decouple
• BeautifulSoup4 and lxml (for scraping logic)

Setup

1. Create and activate the virtual environment:

cd C:\Users\ZENOID\Desktop\Home\home\self_made.project\veritlyscraper
.env\Scripts\Activate.ps1

2. Install dependencies:

pip install -r requirements.txt

3. Copy or create .env in the project root.

4. Configure environment variables in .env.

Example .env

DB_ENGINE=sqlite3
SECRET_KEY=your-secret-key
DB_NAME=veritly
DB_USER=postgres
DB_PASSWORD=your-db-password
DB_HOST=localhost
DB_PORT=5432

Database Configuration

The project supports both SQLite and PostgreSQL via DB_ENGINE.

• sqlite3 uses the local file db.sqlite3
• postgres uses the PostgreSQL settings from .env

Local Development

For local development, use:

DB_ENGINE=sqlite3

Migrations

Run the Django migrations:

python manage.py migrate

Run the App

Start the development server:

python manage.py runserver

Open http://127.0.0.1:8000/ in the browser.

API Endpoints

Reviews Scrape

POST /scrape/reviews

Body:

{
  "query": "Acme Corp",
  "source": "Trustpilot"
}

Pricing Scrape

POST /scrape/pricing

Body:

{
  "url": "https://example.com/product"
}

News Scrape

POST /scrape/news

Body:

{
  "query": "technology"
}

Project Structure

• veritly/ - Django project settings and configurations
• scraper/ - main Django app with views, serializers, templates, and scraper handlers
• env/ - Python virtual environment
• db.sqlite3 - default SQLite database file

Notes

• The frontend currently uses static assets in scraper/static/
• Update scraper/handlers.py to implement real scraping behavior
• Keep .env out of version control (already ignored by .gitignore)