🚀 LeadGen Leads - AI-Powered Lead Generation Platform

Project Showdown: Archive LeadGen

Overview

LeadGen Leads is a comprehensive, full-stack AI-powered lead generation platform designed to help businesses automatically discover, extract, and manage potential customer leads from various online sources. The platform combines advanced web scraping capabilities, AI-driven data extraction, and a robust job queue system to efficiently process large volumes of lead generation requests.

Status: Archived on 14 June 2026 Duration: May 2026 - June 2026 Built by: Harsh Verma Reason: Strategic shift toward KorexBase AI Infrastructure

📋 Key Features

• 🎯 AI-Powered Lead Extraction: Intelligent data extraction and filtering
• ⚡ High-Performance Processing: Async job processing with BullMQ and Redis
• 📊 User Dashboard: Comprehensive dashboard for lead management
• 📧 Multi-Channel Integration: Email campaigns via Mailjet and Zepto Mail
• 💳 Payment Processing: Subscription management with Lemon Squeezy
• 🌐 Advanced Web Scraping: Browser automation with Playwright and HTML parsing
• 🔐 Secure Authentication: JWT-based auth with role-based access control
• 📈 Real-Time Job Tracking: Monitor lead generation jobs in real-time
• 📁 Excel Exports: Export leads in multiple formats
• 🏥 Health Monitoring: System health checks and performance metrics
• 📝 Comprehensive Logging: Structured logging with Winston

🏗️ Architecture

This is a full-stack application with clearly separated frontend and backend:

Frontend

• Framework: React 18 + Vite
• Location: /frontend
• Purpose: User dashboard for lead management, subscription plans, and job tracking
• Features: Real-time updates, responsive design, integrated payment checkout

Backend

• Framework: Node.js + Express.js
• Location: /backend
• Purpose: API server, lead generation engine, job queue management
• Features: RESTful API, WebSocket support, background workers, payment webhooks

🏄 User Dashboard

The comprehensive dashboard allows users to:

• View and manage extracted leads
• Track job status in real-time
• Filter and search leads
• Export data to Excel
• Monitor subscription usage

💰 Pricing Plans

Flexible subscription plans for different business needs:

• Starter Plan
• Professional Plan
• Enterprise Plan

For detailed information about each component:

• Backend README - Complete backend documentation including:

• Tech stack and dependencies

• Project structure and architecture

• API endpoints reference

• Installation and setup guide

• Docker deployment instructions

• Background job processing

• Security and performance optimization

• Frontend README - Frontend documentation including:

• Development setup

• Running locally

• Production build process

• Environment configuration

🛠️ Tech Stack

Frontend

• React 18
• Vite
• TypeScript
• Tailwind CSS
• Axios
• Zustand (State Management)

Backend

• Node.js (ES Modules)
• Express.js v5.2.1
• MongoDB v9.6.2
• Redis v5.12.1
• BullMQ v5.76.10
• Playwright v1.60.0
• Cheerio v1.2.0
• JWT Authentication
• Winston Logging

Infrastructure

• Docker & Docker Compose
• MongoDB Container
• Redis Container
• Ngrok (Development tunneling)

🚀 Quick Start

Prerequisites

• Node.js v18 or higher
• npm or yarn
• MongoDB instance
• Redis instance
• Docker & Docker Compose (optional)

Lead Extraction Form

Easy-to-use form to submit lead generation parameters with advanced filtering options.

Backend Setup

# Navigate to backend
cd backend

# Install dependencies
npm install

# Configure environment
cp env.example .env
# Edit .env with your credentials

# Start development server
npm run dev

# Start background worker (in another terminal)
npm run worker:leads

Frontend Setup

# Navigate to frontend
cd frontend

# Install dependencies
npm install

# Configure environment
cp .env.example .env.local
# Edit .env.local with your API key

# Start development server
npm run dev

Docker Deployment

# From project root, start all services
docker-compose up -d

# Services will be available at:
# Frontend: http://localhost:3000
# Backend API: http://localhost:2000
# MongoDB: localhost:27017
# Redis: localhost:6379

📖 API Overview

Authentication

• POST /app/v1/auth/register - Register new user
• POST /app/v1/auth/login - User login
• POST /app/v1/auth/refresh-token - Refresh JWT token
• POST /app/v1/auth/logout - User logout

Lead Generation

• POST /app/v1/generate-leads - Create lead generation job
• GET /app/v1/jobs/:jobId - Get job status
• GET /app/v1/jobs - Get running jobs

Dashboard

• GET /app/v1/dashboard/leads - Get extracted leads
• GET /app/v1/dashboard/leads/:jobId - Get leads by job
• GET /app/v1/dashboard/jobs - Get job history
• GET /app/v1/dashboard/stats - Get dashboard statistics

Payment

• GET /app/v1/payment/plans - Get available plans
• POST /app/v1/payment/checkout - Create payment session
• GET /app/v1/payment/subscription - Get user subscription

Admin

• GET /admin/settings - Get system settings
• PUT /admin/settings - Update system settings
• GET /admin/users - Get all users

System

• GET /health - System health check
• POST /webhooks/lemon-squeezy - Payment webhook handler

🏢 Project Structure

LeadGen_Leads_GEN/
├── frontend/ # React + Vite application
│ ├── src/
│ │ ├── components/ # React components
│ │ ├── pages/ # Page components
│ │ ├── services/ # API services
│ │ ├── store/ # State management
│ │ └── styles/ # CSS and styling
│ ├── index.html
│ ├── vite.config.ts
│ ├── tsconfig.json
│ └── package.json
│
├── backend/ # Node.js + Express application
│ ├── src/
│ │ ├── app.js # Express app
│ │ ├── server.js # Server entry point
│ │ ├── config/ # Configuration
│ │ ├── controllers/ # Route controllers
│ │ ├── models/ # Mongoose schemas
│ │ ├── routes/ # API routes
│ │ ├── middlewares/ # Express middlewares
│ │ ├── services/ # Business logic
│ │ ├── jobs/ # Background jobs
│ │ ├── webhooks/ # Webhook handlers
│ │ └── utils/ # Utility functions
│ ├── test/ # Test files
│ ├── Dockerfile
│ ├── docker-compose.yml
│ ├── package.json
│ └── env.example
│
├── assests/ # Project assets
├── Results Demo/ # Demo results
└── ARCHIVE.md # Archive information

📊 Lead Generation Pipeline

1. Request Submission - User submits lead generation parameters
2. Job Queuing - Request queued in BullMQ
3. Web Scraping - Playwright and Cheerio extract data from web
4. AI Processing - Intelligent data extraction and cleaning
5. Data Storage - Results saved to MongoDB
6. Notifications - User notified via email upon completion
7. Dashboard Access - Leads available in user dashboard

Extracted Leads

View, manage, and export all extracted leads with detailed information and filtering options.

🔐 Security Features

• JWT Authentication - Secure token-based authentication
• Role-Based Access Control - Different permission levels for users
• CORS Protection - Configured CORS headers
• Input Validation - Request validation on all endpoints
• Environment Secrets - Sensitive data stored in .env
• Raw Body Verification - Webhook signature verification
• Password Security - Bcrypt hashing for passwords

📊 Database Models

Key Mongoose schemas in MongoDB:

• User - User accounts and authentication data
• Lead - Extracted lead information
• Job - Lead generation job records
• Subscription - User subscription details
• AdminSettings - System configuration

🔍 Monitoring & Health

• Health Endpoint - GET /health returns system status
• System Metrics - CPU, memory, disk usage tracking
• Request Logging - Morgan HTTP request logging
• Error Tracking - Winston error logs
• Job Monitoring - BullMQ queue dashboard

📈 Performance Optimization

• Redis Caching - Cache frequently accessed data
• Async Job Queue - BullMQ for background processing
• Connection Pooling - MongoDB connection optimization
• Concurrency Control - p-limit for parallel operations
• Response Compression - Gzip compression enabled

🐛 Troubleshooting

MongoDB Connection Issues

• Check MONGO_URI_PRIMARY in .env
• Ensure MongoDB server is running
• Verify network connectivity

Redis Connection Issues

• Check REDIS_HOST and REDIS_PORT
• Ensure Redis server is running
• Check firewall rules

Email Sending Issues

• Verify Mailjet/Zepto Mail credentials
• Check email templates
• Review email service logs

Job Processing Issues

• Verify worker process is running: npm run worker:leads
• Check BullMQ queue dashboard
• Review Redis for stuck jobs

📝 Environment Configuration

Both frontend and backend require environment files. See:

• backend/env.example - Backend environment template
• frontend/.env.example - Frontend environment template

Excel Exports

Export your extracted leads directly to Excel format for further analysis and integration with other tools.

🤝 Contributing

1. Create a feature branch
2. Make your changes
3. Commit with descriptive messages
4. Push to the branch
5. Create a Pull Request

📄 License

ISC License - see LICENSE file for details

👤 Author

Harsh Verma

📞 Support

For issues, feature requests, and documentation, please refer to:

• Backend README for API and backend-specific issues
• Frontend README for UI and frontend-specific issues
• ARCHIVE.md for project archival information

🗓️ Project Timeline

• Started: May 2026
• Completed: June 2026
• Archived: 14 June 2026

📚 Additional Resources

• Backend README - Comprehensive backend documentation
• Frontend README - Frontend setup and development guide
• ARCHIVE.md - Project archival information
• Results Demo - Demo results and screenshots

---

Last Updated: June 2026

For more detailed information about specific components, see the individual README files in the backend and frontend directories.