🔍 Web Scraper Dashboard

A modern, real-time web scraping dashboard for extracting data from Google Maps and YouTube. Features a beautiful React frontend with live progress updates, analytics charts, and export functionality.

✨ Features

Core Functionality

📊 Real-Time Dashboard - Watch data arrive live as it's scraped via WebSocket
📈 Analytics Charts - Visual insights with Recharts
🌙 Dark/Light Mode - Easy theme toggle with persistence
📤 Export Options - Excel, CSV, and JSON export
🔄 Multiple Concurrent Sessions - Run multiple scrapers simultaneously

Scrapers

Scraper	Modes	Data Points
🗺️ Google Maps	Search	Name, address, phone, website, rating, reviews, social links
🎥 YouTube	Search, Trending, Suggestions, Deep Metadata	Title, views, likes, duration, uploader, tags, categories, description

UI/UX

🎨 Premium Design - Glass morphism, gradients, smooth animations
📱 Fully Responsive - Works on desktop, tablet, and mobile
♿ Accessible - ARIA labels, keyboard navigation support
🔔 Real-Time Notifications - Toast notifications for all events
⚡ Fast Performance - Optimized React rendering, virtual scrolling ready

🏗️ Architecture

web-scraper/
├── backend/                    # Node.js + Express + Socket.io
│   ├── server.js              # Main server with graceful shutdown
│   ├── scrapers/              # Async generator scrapers
│   │   ├── google.js          # Google Maps scraper
│   │   └── youtube.js         # YouTube scraper (4 modes)
│   ├── controllers/           # API handlers
│   │   ├── googleController.js
│   │   └── youtubeController.js
│   ├── routes/                # API endpoints
│   │   └── scrapers.js
│   └── utils/                 # Utilities
│       ├── sessionManager.js  # Session state management
│       ├── browserManager.js  # Puppeteer browser tracking
│       ├── logger.js          # Winston logging
│       └── errorHandler.js    # Error classification
│
├── frontend/                   # React + Vite + Tailwind
│   └── src/
│       ├── components/        # React components
│       │   ├── ScraperSelector.jsx
│       │   ├── GoogleForm.jsx
│       │   ├── YouTubeForm.jsx
│       │   ├── ProgressBar.jsx
│       │   ├── ResultsTable.jsx
│       │   ├── Charts.jsx
│       │   ├── ExportButtons.jsx
│       │   ├── ActiveSessions.jsx
│       │   └── ThemeToggle.jsx
│       ├── context/           # Global state
│       │   └── ScraperContext.jsx
│       └── styles/            # Tailwind CSS + custom animations
│
├── scrapers/                   # Legacy CLI scrapers
└── output/                     # Exported data

🔧 Requirements

Node.js v18+ (with ES modules support)
npm v9+
yt-dlp (for YouTube metadata extraction)

Install yt-dlp:

# macOS
brew install yt-dlp

# Linux
sudo curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp
sudo chmod a+rx /usr/local/bin/yt-dlp

# Windows
choco install yt-dlp

🚀 Quick Start

1. Clone and Install

# Clone the repository
git clone <repo-url>
cd web-scraper

# Install backend dependencies
cd backend
npm install

# Install frontend dependencies
cd ../frontend
npm install

2. Configure Environment

Backend (backend/.env):

NODE_ENV=development
PORT=3000
FRONTEND_URL=http://localhost:5173

Frontend (frontend/.env):

VITE_API_URL=http://localhost:3000

3. Start the Servers

Terminal 1 - Backend:

cd backend
npm start

Server runs on http://localhost:3000

Terminal 2 - Frontend:

cd frontend
npm run dev

Dashboard opens on http://localhost:5173

4. Stop the Servers

Press Ctrl+C in the terminal. The server will:

Cancel all running scraping sessions
Close all Puppeteer browser instances
Close Socket.io connections
Exit gracefully within 10 seconds

📖 Usage

Web Dashboard

Open http://localhost:5173
Select a scraper (Google Maps or YouTube)
Configure options:
- Google Maps: Search term, headless mode, max results
- YouTube: Mode, search term/country, content filter
Start scraping and watch real-time results
View analytics charts automatically update
Export to Excel/CSV/JSON when complete

Google Maps Scraper

Configuration:

Search Term (required) - e.g., "Restaurants in New York"
Headless Mode - Run browser without UI (default: true)
Max Results - Limit number of results (optional)

Data Extracted:

{
  name: "Business Name",
  address: "123 Main St",
  phone: "+1 234 567 8900",
  website: "https://example.com",
  rating: "4.5",
  reviews: "123",
  fb: "facebook.com/page",
  insta: "instagram.com/page",
  linkedln: "linkedin.com/company",
  url: "maps.google.com/..."
}

YouTube Scraper

Modes:

Search - Search videos by keyword
- Input: Search term
- Results: Video list with views, duration, channel
Trending - Get trending videos by country
- Input: Country code, content filter (all/videos/shorts)
- Results: Trending videos with metadata
Suggestions - Get autocomplete suggestions
- Input: Query term
- Results: Suggestion list
Deep Metadata - Get full video metadata
- Input: Video URL
- Results: Complete metadata including tags, categories, description

Data Extracted:

{
  title: "Video Title",
  url: "https://youtube.com/watch?v=...",
  views: 1234567,
  likes: 45678,
  duration: 627,           // seconds
  upload_date: "20240115",
  tags: ["tag1", "tag2"],
  categories: ["Entertainment"],
  description: "Video description...",
  uploader: "Channel Name",
  thumbnail: "https://..."
}

📡 API Reference

Scraping Endpoints

POST /api/scrape/google
Body: { searchTerm, headless?, maxResults? }

POST /api/scrape/youtube
Body: { mode, searchTerm?, countryCode?, filterType?, headless? }

Session Management

GET    /api/scrape/active           # List all active sessions
GET    /api/scrape/status/:id       # Get session status
DELETE /api/scrape/:id              # Cancel a session

Export

GET /api/scrape/:sessionId/export?format=excel|csv|json

WebSocket Events

Client → Server:

socket.emit('scrape:start', { scraperType, config })
socket.emit('scrape:cancel', { sessionId })
socket.emit('session:join', { sessionId })

Server → Client:

socket.on('session:started', { sessionId, scraperType, startTime })
socket.on('scrape:progress', { sessionId, progress })
socket.on('scrape:data', { sessionId, data })
socket.on('scrape:batch', { sessionId, data })
socket.on('session:complete', { sessionId, totalItems, duration })
socket.on('session:error', { sessionId, error })
socket.on('session:cancelled', { sessionId })
socket.on('sessions:active', { sessions })

🛠️ Tech Stack

Backend

Technology	Purpose
Express.js	Web framework
Socket.io	Real-time WebSocket communication
Puppeteer	Headless browser automation
yt-search	YouTube search API
yt-dlp	YouTube metadata extraction
Winston	Logging
UUID	Session ID generation
ExcelJS	Excel export

Frontend

Technology	Purpose
React 18	UI framework
Vite	Build tool and dev server
Tailwind CSS	Styling
Recharts	Data visualization
Socket.io-client	WebSocket client
axios	HTTP requests
react-hot-toast	Notifications

⚙️ Configuration

Browser Options

Puppeteer launch options (in scrapers):

{
  headless: true,                    // Run without UI
  args: [
    '--no-sandbox',                  // Required for Linux
    '--disable-setuid-sandbox',      // Security
    '--disable-dev-shm-usage'        // Memory optimization
  ]
}

Session Limits

Max items per session: 10,000
Session retention: 24 hours
Auto-cleanup: Every hour

🔒 Environment Variables

Variable	Default	Description
`NODE_ENV`	development	Environment mode
`PORT`	3000	Backend server port
`FRONTEND_URL`	http://localhost:5173	CORS origin
`VITE_API_URL`	http://localhost:3000	Backend API URL

🐛 Troubleshooting

Backend won't start

# Check if port 3000 is in use
lsof -i :3000
# Kill the process or change PORT in .env

Puppeteer errors

# Install required dependencies
# Ubuntu/Debian
sudo apt-get install -y \
  chromium-browser \
  fonts-liberation \
  fonts-noto-color-emoji \
  fonts-ipafont-gothic \
  fonts-wqy-zenhei \
  fonts-thai-tlwg \
  fonts-kacst \
  fonts-freefont-ttf

# macOS
# No additional dependencies needed

YouTube metadata missing

# Verify yt-dlp installation
yt-dlp --version

# Update yt-dlp
yt-dlp --update

Ctrl+C not stopping server

The graceful shutdown waits up to 10 seconds for browsers to close
Force quit: Press Ctrl+C twice or Ctrl+\

Frontend not connecting to backend

Check VITE_API_URL in frontend .env
Check FRONTEND_URL in backend .env
Ensure backend is running on correct port

📊 License

This project is intended for educational and research purposes only.

Terms of Service Compliance

Ensure you comply with the Terms of Service of:

Usage Guidelines

Respect rate limits - Add delays between requests
Don't overload servers - Use reasonable concurrent sessions
Attribution - Credit data sources appropriately
Legal compliance - Follow local laws and regulations

👨🏻‍💻 Developer

Bilal Ahmed

Email: reply2bilal.ahmed@gmail.com
GitHub: @bilalahmed

🙏 Acknowledgments

yt-search - YouTube search library
Puppeteer - Headless Chrome automation
Vite - Next-gen frontend tooling
Tailwind CSS - Utility-first CSS framework

Name		Name	Last commit message	Last commit date
Latest commit History 14 Commits
backend		backend
frontend		frontend
.gitignore		.gitignore
README.md		README.md

Folders and files

Latest commit

History

Repository files navigation

🔍 Web Scraper Dashboard

✨ Features

Core Functionality

Scrapers

UI/UX

🏗️ Architecture

🔧 Requirements

🚀 Quick Start

1. Clone and Install

2. Configure Environment

3. Start the Servers

4. Stop the Servers

📖 Usage

Web Dashboard

Google Maps Scraper

YouTube Scraper

📡 API Reference

Scraping Endpoints

Session Management

Export

WebSocket Events

🛠️ Tech Stack

Backend

Frontend

⚙️ Configuration

Browser Options

Session Limits

🔒 Environment Variables

🐛 Troubleshooting

Backend won't start

Puppeteer errors

YouTube metadata missing

Ctrl+C not stopping server

Frontend not connecting to backend

📊 License

Terms of Service Compliance

Usage Guidelines

👨🏻‍💻 Developer

🙏 Acknowledgments

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Packages