⚡ WEBPULSE AI

Autonomous Self-Healing Web Scraping Orchestrator
Built for Into the Scrape-Verse | WeMakeDevs in collaboration with Bright Data

🎨 UI Design Evolution & Dashboard Showcase

WEBPULSE features a state-of-the-art Obsidian Navy (#0B0E17) glassmorphic interface designed for high-density telemetry, real-time node monitoring, and seamless dataset exploration.

📸 Before vs After Progress Showcase

Key Improvements in the Redesign:

Unified Obsidian Navy Design System: Switched from generic dark zinc to an ultra-premium #0B0E17 obsidian canvas with #141724 glassmorphic cards and #23283A borders.

5-Card Glassmorphism Accordion Stack Deck: Interactive header-hover card deck displaying all active scraper nodes with smooth slide animations and zero text overlap.

Pixel-Perfect Alignment & Compact Layout: 8% optimized space efficiency across all secondary screens with 100% flush card bottom border alignment on the main dashboard.

Single-Title Header Bar: Streamlined global navigation headers across all 10 section pages.

🏆 Hackathon

Built for the Into the Scrape-Verse | WeMakeDevs hackathon in collaboration with Bright Data.

Challenge: Build a self-healing web scraper that can detect website changes, repair its extraction logic, and continue producing structured data.

WebPulse AI addresses this challenge by integrating Bright Data Scraper Studio with backend orchestration, monitoring, self-healing, structured data processing, and an interactive dashboard.

✨ Key Features & Technical Highlights

🤖 1. Self-Healing Integration Workflow

Structural Drift Monitoring: Integrates with Bright Data Scraper Studio to detect DOM structural updates and locator mutations across target web endpoints.
Automated Selector Re-Mapping: Captures self-healing diagnostic feedback and selector repairs directly through Bright Data's self-healing workflow.
Version Mutation Audit Log: Tracks configuration version mutations (v1 → v2) with confidence scores, suggested repair actions, and execution audit trails.

🌐 2. Bright Data CLI & Scraper Studio Integration

Bright Data CLI: brightdata/cli — Official Bright Data CLI for scraping, searching, and extracting structured web data directly from the terminal.
Asynchronous CLI Subprocess Orchestration: The Node.js backend uses the Bright Data CLI to execute Scraper Studio collectors asynchronously, capture execution output, monitor scraper status, and process structured results for the dashboard.
Resilient Polling & Proxy Health: Tracks execution timeouts, bandwidth quotas, and proxy unblocking status across all active scraper runs.

📊 3. Interactive Telemetry & Health Analytics

Live Health Distribution: Real-time donut chart and velocity analytics powered by Recharts.
Monitored Node Stack: Interactive glassmorphic accordion deck for instant status checking (Healthy, Repaired, Failed).

💾 4. Multi-Format Payload Exporter

Data Explorer Studio: Search, filter, and inspect raw extracted JSON records in real time.
One-Click Export: Download clean payload datasets instantly in CSV, JSON, or NDJSON formats.

🚧 Feature Implementation Status

The majority of WebPulse AI is fully functional, dynamically connected to our Express backend, local database persistence layer, and live Bright Data scraping workflows. To maximize technical depth within hackathon time constraints, engineering effort was focused on autonomous scraping, telemetry, and self-healing, while non-core account-level UI features were kept static/demo-only.

CORE FUNCTIONALITY → Dynamic & Working

🕷️ Scraping: Live Bright Data CLI subprocess orchestration, collector creation, raw payload capture, and manual run overrides.
🔄 Monitoring: Real-time collector status tracking, health score indexing, and scheduled sync queue management.
🤖 Self-Healing: Automated DOM structural drift detection, fuzzy layout selector re-mapping, version mutation tracking (v1 → v2), and healing audit logs.
📊 Data: Live payload inspection, search filtering, and multi-format data export (CSV, JSON, NDJSON).
📈 Analytics: Interactive health distribution donut charts, scrape run velocity metrics, and 5-card monitored node stack deck.
🔌 Backend Integration: Node.js + Express REST API, persistent JSON database with atomic file locking, and automated seeder scripts.

Note: All core features listed above are fully implemented, operational, and query real backend/database state rather than relying on hardcoded UI mockups.

NON-CORE UI → Static

🔐 Login / Authentication: Static session toggle and login modal placeholder.
👤 Account / Profile: Demo user header avatar and profile metadata.
💳 Quota / Usage Overview: Static proxy bandwidth usage meter.
⚙️ Some Account Settings: Static UI tabs for team organization settings and subscription tier toggles.

Scope Rationale: These non-core features are currently static UI/demo components and their displayed values do not dynamically mutate or integrate with external authentication or billing providers. This was an intentional design choice for the hackathon to dedicate 100% of development bandwidth to Bright Data scraper integration, self-healing resilience, monitoring analytics, data export tools, and backend API engineering.

📄 Structured Output from Scraper Studio

WebPulse standardizes raw data output into valid, machine-readable JSON schemas. Below is a real sample payload extracted from the Books to Scrape collector:

[
  {
    "title": "A Light in the Attic",
    "price": {
      "value": 51.77,
      "currency": "GBP",
      "symbol": "£"
    },
    "availability": "In stock (22 available)",
    "rating": "Three",
    "upc": "a897fe39b1053632",
    "number_of_reviews": 0,
    "image_url": "https://books.toscrape.com/media/cache/fe/72/fe72f0532301ec28892ae79a629a293c.jpg",
    "product_page_url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "input": {
      "url": "https://books.toscrape.com/"
    }
  }
]

Auto-Generated Dynamic JSON Schema

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "BooksScraperExtractedSchema",
  "description": "Auto-generated schema from current extracted payload fields.",
  "type": "object",
  "properties": {
    "items": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "title": { "type": "string" },
          "price": { "type": "object" },
          "availability": { "type": "string" },
          "rating": { "type": "string" },
          "upc": { "type": "string" },
          "product_page_url": { "type": "string" }
        },
        "required": ["title", "price", "availability", "product_page_url"]
      }
    }
  }
}

🏗️ Architecture & Project Structure

WebScraper_WeMakeDevsHackathon/
├── Before after dashboard UI.png   # Dashboard UI Evolution Showcase
├── frontend/                       # React 18 + Vite + Tailwind CSS v4 + Recharts
│   ├── src/
│   │   ├── components/             # Reusable UI components (Header, Sidebar, Cards)
│   │   ├── context/                # Global ScraperContext & State Management
│   │   ├── pages/                  # Dashboard, Scrapers, ScraperDetails, DataExplorer,
│   │   │                           # Monitoring, Schedule, Alerts, RepairLogs, Settings, ApiKeys
│   │   └── mock/                   # System fallback datasets
├── backend/                        # Express + TypeScript Server API
│   ├── data/                       # Persistent JSON DB (scrapers, runs, baselines, healingEvents)
│   ├── src/
│   │   ├── controllers/            # Scraper & telemetry route handlers
│   │   ├── services/               # Bright Data CLI & AI Self-Healing logic
│   │   ├── utils/                  # File locking DB utilities
│   │   └── tests/                  # DB Seeder (seedScrapers.ts)

🛠️ Quick Start & Setup Instructions

Prerequisites

Node.js: v18.0.0 or higher
npm: v9.0.0 or higher

1. Clone the Repository

git clone https://github.com/karina-purswani/WebScraper_WeMakeDevsHackathon.git
cd WebScraper_WeMakeDevsHackathon

2. Install Dependencies

Install dependencies for both frontend and backend in one command:

npm run install:all

3. Seed Initial Telemetry & Active Scraper Nodes

Populate the local persistence layer with active scrapers (Books Scraper, Amazon Tech Deals, Quotes Catalog, Hacker News, GitHub Trending):

npx tsx backend/src/tests/seedScrapers.ts

4. Run the Application locally

Start both the Express backend API and Vite frontend server concurrently:

# Start backend API (Port 5000)
npm run dev:backend

# Start frontend development server (Port 5173)
npm run dev:frontend

Open http://localhost:5173 in your browser to launch WebPulse AI!

🧪 Verification & Build Commands

To build the project for production verification:

# Test frontend production build
cd frontend
npm run build

⚠️ Deployment Note: The New Scraper feature uses the Bright Data CLI-based backend. Due to the hackathon's deployment environment/rules, this CLI-based scraping functionality is not available in the online deployed version. The feature works in the local development environment. The deployed version is provided to demonstrate the rest of the platform and its self-healing workflow.

🤝 Hackathon Credits & Acknowledgments

Hackathon: Into the Scrape-Verse | WeMakeDevs
Organizer: WeMakeDevs
Technology Partner: Bright Data
Built By: Anusha Randive & Karina Purswani
Scraping Platform: Bright Data Scraper Studio
CLI: Bright Data CLI integrated through the Node.js backend

WebPulse AI — Autonomous, Resilient, & Beautiful Web Scraping Telemetry.

Name		Name	Last commit message	Last commit date
backend		backend
database		database
frontend		frontend
intelligence		intelligence
scraper		scraper
tests		tests
.gitignore		.gitignore
Before after dashboard UI.png		Before after dashboard UI.png
README.md		README.md
package-lock.json		package-lock.json
package.json		package.json

Navigation Menu

Repository files navigation

⚡ WEBPULSE AI

🎨 UI Design Evolution & Dashboard Showcase

📸 Before vs After Progress Showcase

🏆 Hackathon

✨ Key Features & Technical Highlights

🤖 1. Self-Healing Integration Workflow

🌐 2. Bright Data CLI & Scraper Studio Integration

📊 3. Interactive Telemetry & Health Analytics

💾 4. Multi-Format Payload Exporter

🚧 Feature Implementation Status

CORE FUNCTIONALITY → Dynamic & Working

NON-CORE UI → Static

📄 Structured Output from Scraper Studio

Auto-Generated Dynamic JSON Schema

🏗️ Architecture & Project Structure

🛠️ Quick Start & Setup Instructions

Prerequisites

1. Clone the Repository

2. Install Dependencies

3. Seed Initial Telemetry & Active Scraper Nodes

4. Run the Application locally

🧪 Verification & Build Commands

🤝 Hackathon Credits & Acknowledgments

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

⚡ WEBPULSE AI

🎨 UI Design Evolution & Dashboard Showcase

📸 Before vs After Progress Showcase

🏆 Hackathon

✨ Key Features & Technical Highlights

🤖 1. Self-Healing Integration Workflow

🌐 2. Bright Data CLI & Scraper Studio Integration

📊 3. Interactive Telemetry & Health Analytics

💾 4. Multi-Format Payload Exporter

🚧 Feature Implementation Status

CORE FUNCTIONALITY → Dynamic & Working

NON-CORE UI → Static

📄 Structured Output from Scraper Studio

Auto-Generated Dynamic JSON Schema

🏗️ Architecture & Project Structure

🛠️ Quick Start & Setup Instructions

Prerequisites

1. Clone the Repository

2. Install Dependencies

3. Seed Initial Telemetry & Active Scraper Nodes

4. Run the Application locally

🧪 Verification & Build Commands

🤝 Hackathon Credits & Acknowledgments

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages