ProspectAI — Autonomous B2B Lead Generation & Qualification Agent

🤖 AI-powered lead generation platform that autonomously identifies, enriches, and scores B2B prospects using intelligent scraping, NLP analysis, and real-time data enrichment.

🎯 What is ProspectAI?

ProspectAI is an autonomous, ethical B2B lead generation agent that identifies high-quality prospects by analyzing publicly available company data. Using advanced NLP, web scraping, and AI scoring, it delivers qualified leads ready for outreach across:

Cybersecurity Services (pen testing, vulnerability audits, security consulting)
AI Integration & Automation (workflow optimization, business process automation)
Website Performance & Security (audits, optimization, compliance)
AI Readiness Consulting (digital transformation, enterprise AI deployment)

Key Differentiators

✅ 100% Ethical & Compliant — Only public data, no PII collection, GDPR/CCPA ready
✅ AI-Powered Scoring — Multi-factor lead ranking based on industry, tech stack, pain points
✅ Real-Time Enrichment — Tech stack detection, employee insights, company financials
✅ Multi-Source Intelligence — LinkedIn, Crunchbase, G2, SEC EDGAR, company websites
✅ Production-Ready APIs — FastAPI server for enterprise integration
✅ Interactive Dashboard — Streamlit UI for campaign management & monitoring
✅ Automated Email Outreach — Template-based personalized campaigns with tracking
✅ Auto-Updater — Keep lead data fresh with scheduled background refreshes

🚀 Quick Start

Prerequisites

Python 3.11+
PostgreSQL or SQLite
(Optional) Redis for caching
(Optional) Node.js 20+ for advanced browser automation

Installation

# Clone repository
git clone https://github.com/yourusername/prospectai.git
cd prospectai

# Install dependencies
pip install -r requirements.txt

# Configure environment
cp .env.example .env
# Edit .env with your API keys and database URL

Run the Agent

# Interactive CLI mode
python main.py cli

# Start FastAPI server (localhost:8000)
python main.py api

# Launch Streamlit dashboard (localhost:8501)
python main.py dashboard

# Run with Docker
docker-compose up

First Lead Analysis

python main.py cli
# Follow prompts to define ICP (Ideal Customer Profile)
# Agent will begin prospect research and scoring

📊 Core Features

1. Multi-Source Web Scraping 🔍

LinkedIn Company Research — Public company profile analysis
Google Search Intelligence — News, mentions, industry rankings
Website Crawling — Content extraction, tech detection, business intelligence
Tech Stack Detection — BuiltWith integration for technology inventory
Security Insights — Shodan integration for public exposure scoring
Business Directory Enrichment — Crunchbase, AngelList, Companies House data

2. Advanced NLP & Intelligence Layer 🧠

Pain Point Extraction — Identify company challenges from web content analysis
Intent Classification — Detect buying signals and readiness indicators
Industry Segmentation — Classify prospects by vertical and opportunity size
Sentiment Analysis — Understand market positioning and company mood
Keyword Extraction — Surface key business terms and pain points
Competitive Intelligence — Analyze competitor mentions and market positioning

3. AI-Powered Lead Scoring ⭐

Multi-Factor Scoring — Company size, growth rate, tech adoption, hiring velocity
ICP Matching Algorithm — Custom Ideal Customer Profile alignment
Risk/Opportunity Assessment — Identify quick wins vs. strategic accounts
Rank Tier Classification — 🔴 Hot, 🟡 Warm, 🔵 Cold lead categorization
Customizable Weights — Adjust scoring based on your business model
Trend Detection — Identify growth signals, expansion, or distress indicators

4. Email Campaign Management 📧

Template-Based Generation — Automatically craft personalized outreach
Lead Tracking & Analytics — Open rates, click rates, reply tracking
A/B Testing Framework — Test subject lines and email variations
LLM-Powered Personalization — AI-generated contextual email bodies
Multi-Channel Support — Gmail, SendGrid, Mailgun, AWS SES
Campaign Scheduling — Batch sending with rate limiting
Unsubscribe Management — Automatic compliance with email regulations

5. Auto-Updater & Data Refresh 🔄

Scheduled Background Sync — Keep prospect data fresh automatically
Incremental Updates — Only refresh changed data
Lead Re-Scoring — Continuous ranking updates
Keyword Monitoring — Track new company keywords and updates
Hot Lead Priority Refresh — Prioritize tier-1 prospects for updates
Configurable Intervals — Customize refresh schedules per tier

6. API & Integration 🔌

RESTful API — All features accessible via HTTP endpoints
Batch Processing — Import/export 1000s of leads at scale
Webhook Support — Real-time lead notifications
CRM Sync — Salesforce, HubSpot integration ready
Custom Integrations — Python SDK for programmatic access
Rate Limiting — Built-in request throttling and queuing

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         API Layer (FastAPI)                      │
│         /analyze  /batch-import  /campaigns  /leads              │
└─────────────────────────────────────────────────────────────────┘
                                 ↓
┌─────────────────────────────────────────────────────────────────┐
│                     ProspectAI Agent (Core)                      │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Scraper Layer                                           │   │
│  │ ├─ Playwright (JS-rendered pages)                       │   │
│  │ ├─ BeautifulSoup (HTML parsing)                         │   │
│  │ ├─ Scrapy (Large-scale crawling)                        │   │
│  │ └─ httpx/aiohttp (Async requests)                       │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                 ↓                                │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Data Enrichment                                         │   │
│  │ ├─ Shodan (Security intelligence)                       │   │
│  │ ├─ BuiltWith (Tech stack detection)                     │   │
│  │ ├─ Crunchbase API (Company insights)                    │   │
│  │ └─ Custom extractors                                    │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                 ↓                                │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ NLP Pipeline                                            │   │
│  │ ├─ Pain Point Extraction                                │   │
│  │ ├─ Intent Classification                                │   │
│  │ ├─ Sentiment Analysis                                   │   │
│  │ └─ Keyword Extraction                                   │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                 ↓                                │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ LLM Router & Intelligence                               │   │
│  │ ├─ OpenAI GPT-4/3.5                                     │   │
│  │ ├─ Anthropic Claude                                     │   │
│  │ ├─ Ollama (Local, privacy-first)                        │   │
│  │ └─ Fallback chains                                      │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                 ↓                                │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Lead Scoring Engine                                     │   │
│  │ └─ Multi-factor algorithm + custom weights              │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘
                                 ↓
┌─────────────────────────────────────────────────────────────────┐
│                      Storage Layer                               │
│  ┌─────────────────┐  ┌──────────────┐  ┌─────────────────┐   │
│  │  PostgreSQL     │  │    Redis     │  │    Pinecone     │   │
│  │  (Lead DB)      │  │  (Cache)     │  │ (Vector Search) │   │
│  └─────────────────┘  └──────────────┘  └─────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

📦 Tech Stack

Category	Technologies
Runtime	Python 3.11+, Node.js 20+ (optional)
Web Scraping	Playwright, BeautifulSoup4, Scrapy, httpx, aiohttp
API Framework	FastAPI, Uvicorn, Pydantic
Dashboard UI	Streamlit, Pandas, Plotly
NLP & ML	spaCy, transformers, scikit-learn, torch
LLM Integration	LangChain, OpenAI, Anthropic, Ollama
Data Storage	SQLAlchemy, PostgreSQL, Redis, Pinecone
Task Scheduling	APScheduler (background jobs)
Email	SMTP, SendGrid, Mailgun, AWS SES
Data Enrichment	Shodan, BuiltWith, Crunchbase, Hunter.io
Async Runtime	asyncio, aiohttp, asyncpg
Containerization	Docker, Docker Compose

💼 Use Cases

SaaS & Agency Sales Teams

Identify SMBs & mid-market companies for targeted outreach. Build high-quality prospect lists at scale. Reduce lead research time from days to minutes.

B2B Service Providers

Cybersecurity firms targeting vulnerable tech stacks
Automation consultancies finding process optimization opportunities
AI integration specialists identifying AI-ready companies
Website optimization agencies finding underperforming sites

Enterprise Account Development

Automated territory planning
ICP-based prospect sourcing
Competitive intelligence gathering
Account-based marketing (ABM) campaigns

Market Research & Analytics

Industry trend analysis
Competitor mapping & benchmarking
Technology adoption patterns
Market size estimation

Recruitment & Talent Acquisition

Identify high-growth companies for recruiting outreach
Find companies in expansion mode
Track hiring velocity trends

🔒 Ethical & Compliance

ProspectAI is built with ethics and privacy at its core:

✅ Zero PII Collection — Only aggregated, public company data
✅ GDPR/CCPA Compliant — Respects international privacy regulations
✅ Transparent Scraping — Honors robots.txt, respects rate limits, identifies as bot
✅ Public Data Only — LinkedIn public profiles, company websites, news, business directories
✅ No Account Takeover — Never uses credentials, no authentication circumvention
✅ Audit Trail — Full logging of data sources and collection timestamps
✅ Configurable Rate Limiting — Prevent server overload, respect TOS

🔧 Configuration

Environment Variables (.env)

# Application
DEBUG=False
APP_VERSION=2.0.0

# Database
DATABASE_URL=postgresql+asyncpg://user:password@localhost:5432/prospectai
# Or use SQLite for development:
# DATABASE_URL=sqlite:///./prospectai.db

# Redis Cache
REDIS_URL=redis://localhost:6379/0

# LLM Configuration
LLM_PROVIDER=auto  # auto, openai, anthropic, ollama, none
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...

# Local LLM (Ollama)
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_DEFAULT_MODEL=llama3.2:3b
OLLAMA_QUALITY_MODEL=llama3.1:8b

# Data Enrichment APIs
SHODAN_API_KEY=...
BUILTWITH_API_KEY=...
CRUNCHBASE_API_KEY=...
HUNTER_API_KEY=...
CLEARBIT_API_KEY=...

# Email Configuration
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=your-email@gmail.com
SMTP_PASSWORD=your-app-password
SENDGRID_API_KEY=sg-...
MAILGUN_API_KEY=...
MAILGUN_DOMAIN=mg.yourdomain.com
AWS_SES_ACCESS_KEY=...
AWS_SES_SECRET_KEY=...
AWS_SES_REGION=us-east-1
FROM_EMAIL=leads@yourcompany.com
FROM_NAME=Your Company

# Email Limits & Tracking
EMAIL_DAILY_SEND_LIMIT=100
EMAIL_RATE_LIMIT_PER_MINUTE=10
EMAIL_TRACKING_ENABLED=True

# Scraping Configuration
MAX_CONCURRENT_SCRAPES=5
REQUEST_DELAY_SECONDS=2.0
RATE_LIMIT_PER_DOMAIN=30

# Auto-Updater
UPDATER_ENABLED=True
REFRESH_INTERVAL_HOURS=24
KEYWORD_SYNC_INTERVAL_HOURS=168
AUTO_REFRESH_HOT=True
AUTO_REFRESH_WARM=True
AUTO_REFRESH_COLD=False

# Vector Search (Optional)
PINECONE_API_KEY=...
PINECONE_ENVIRONMENT=us-west-2

# Monitoring (Optional)
SENTRY_DSN=https://...

📚 API Documentation

Start API Server

python main.py api

Visit: http://localhost:8000/docs for interactive Swagger UI

Key Endpoints

Analyze Companies

POST /analyze
Content-Type: application/json

{
  "companies": [
    {
      "name": "TechCorp Inc",
      "website": "https://techcorp.com",
      "industry": "fintech"
    }
  ],
  "icp": {
    "industries": ["fintech", "healthcare"],
    "geographies": ["US", "UK"],
    "company_sizes": ["mid-market", "enterprise"],
    "pain_points": ["cybersecurity", "automation"]
  }
}

Create Email Campaign

POST /campaigns
{
  "name": "Q2 Cybersecurity Outreach",
  "lead_tier": "hot",
  "template_id": "template-123",
  "subject_line": "Security Assessment for {{company_name}}",
  "scheduled_at": "2026-05-15T09:00:00Z"
}

Get Lead Details

GET /leads/{lead_id}

Batch Import Leads

POST /batch-import
Content-Type: multipart/form-data
file: companies.csv

See API Docs for full reference.

🖥️ Dashboard Guide

Access Dashboard

python main.py dashboard
# Or: streamlit run streamlit_app.py
# Visit: http://localhost:8501

Dashboard Sections

1. Analyze Lead — Single company research & scoring
2. Batch Import — Upload CSV of companies for bulk processing
3. Leads Table — Browse, filter, and manage all leads
4. ICP Config — Define your Ideal Customer Profile
5. Email Campaigns — Create, schedule, and monitor outreach
6. Email Templates — Manage template library
7. Email Logs — Track open rates, clicks, replies
8. Auto-Updater — Monitor background refresh jobs
9. Keywords — Manage tracked keywords and sync schedules

📧 Email Campaigns

Create a Campaign

from src.email_outreach.campaign_manager import CampaignManager

manager = CampaignManager()

campaign = await manager.create_campaign(
    name="Q2 Outreach",
    leads=[lead1, lead2, lead3],
    template_name="security-audit",
    schedule_time="2026-05-15T09:00:00Z"
)

# Schedule for sending
await manager.schedule_campaign(campaign.id)

Personalization Variables

Templates support dynamic personalization:

{{company_name}} — Company name
{{company_industry}} — Industry classification
{{contact_name}} — Contact person
{{pain_point}} — Identified challenge
{{tech_stack}} — Detected technologies
{{opportunity}} — Custom opportunity text

Track Results

stats = await manager.get_campaign_stats(campaign_id)
print(f"Opens: {stats.opens}")
print(f"Clicks: {stats.clicks}")
print(f"Replies: {stats.replies}")

🔄 Auto-Updater & Data Refresh

ProspectAI keeps your lead database fresh automatically.

How It Works

Hot Leads — Updated daily (🔴 priority)
Warm Leads — Updated weekly (🟡 regular)
Cold Leads — Updated monthly (optional, 🔵 low priority)
Keyword Monitoring — Continuous tracking of new mentions

Configuration

UPDATER_ENABLED=True
REFRESH_INTERVAL_HOURS=24
AUTO_REFRESH_HOT=True
AUTO_REFRESH_WARM=True
AUTO_REFRESH_COLD=False

Monitor Updates in Dashboard

Go to Auto-Updater section to:

View refresh schedule
Monitor running jobs
Check last update timestamps
Manually trigger refreshes
View update history

🚀 Deployment

Local Development

# Install dependencies
pip install -r requirements.txt

# Run all services
python main.py cli
python main.py api &
streamlit run streamlit_app.py

Docker Deployment

# Build and run with Docker Compose
docker-compose up

# Access services:
# API:       http://localhost:8000
# Dashboard: http://localhost:8501

Cloud Deployment (AWS/GCP/Azure)

Build Docker image

docker build -t prospectai:latest .

Push to registry

aws ecr get-login-password | docker login --username AWS --password-stdin YOUR_ECR
docker tag prospectai:latest YOUR_ECR/prospectai:latest
docker push YOUR_ECR/prospectai:latest

Deploy to ECS/GKE/AKS with environment variables configured

🧪 Testing

Run the test suite:

# All tests
pytest tests/

# Specific test file
pytest tests/test_nlp.py -v

# With coverage
pytest --cov=src tests/

Test Categories

test_email_outreach.py — Email campaign logic
test_integration.py — End-to-end workflows
test_llm_integration.py — LLM provider integration
test_nlp.py — NLP pipeline accuracy
test_scoring.py — Lead scoring algorithm
test_updater.py — Background refresh jobs

📈 Performance & Scalability

Metric	Value
Lead Generation Speed	50-100 companies/hour (rate-limited for compliance)
NLP Accuracy	85%+ precision on pain point extraction
Average API Response	<500ms for single lead analysis
Batch Processing	1000+ leads in <5 minutes
Uptime Target	99.9% with distributed architecture
Concurrent Users	100+ simultaneous dashboard users
Daily Capacity	10,000+ leads refreshed per day

Optimization Tips

Use Redis caching to reduce DB queries
Enable batch scraping for large datasets
Configure rate limits per domain
Use Ollama for local LLM inference (privacy + speed)
Leverage Pinecone for vector search on large datasets

🤝 Contributing

We welcome contributions! Areas for improvement:

Additional data source integrations (Glassdoor, AngelList, etc.)
Enhanced NLP models for pain point extraction
Distributed scraping worker pools
CRM integrations (Salesforce, HubSpot, Pipedrive)
Advanced reporting and BI features
Mobile app for dashboard
More LLM provider support

Development Setup

git clone https://github.com/yourusername/prospectai.git
cd prospectai
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt
# Make changes, test, submit PR

📝 License

MIT License — See LICENSE file for details

🆘 Support & Community

Issues & Bugs — GitHub Issues
Discussions — GitHub Discussions
Documentation — Full Docs
Email Support — support@prospectai.io

⭐ Show Your Support

If ProspectAI helps your team, consider starring the repo! 🌟

Made with ❤️ for B2B Sales & Lead Generation Teams

Version 2.0.0 | Last Updated: May 2026

Name		Name	Last commit message	Last commit date
agents		agents
src		src
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
B2B_LeadGen_Agent_SystemPrompt.md		B2B_LeadGen_Agent_SystemPrompt.md
Dockerfile		Dockerfile
README.md		README.md
docker-compose.yml		docker-compose.yml
main.py		main.py
requirements.txt		requirements.txt
streamlit_app.py		streamlit_app.py

Latest commit

History

Folders and files

Repository files navigation