Skip to content

INDEX.md

Latest commit

 

History

History
398 lines (327 loc) · 10.6 KB

INDEX.md

File metadata and controls

398 lines (327 loc) · 10.6 KB
Raw

Agrolead - Complete Delivery Package

Welcome to Agrolead - A production-ready Python application for discovering and scoring European agricultural importers and wholesalers.

📚 Documentation Map

Start here based on what you need:

🚀 Getting Started (15 minutes)

  1. QUICKSTART.md - 5-minute setup guide

    • Installation steps
    • First crawl example
    • Common tasks

  2. README.md - Complete feature guide

    • Feature overview
    • Installation options
    • Usage examples
    • Configuration reference

🔧 Development & Extension

  1. IMPLEMENTATION.md - Architecture guide

    • Component overview
    • Adding custom adapters
    • Performance tuning
    • Testing guidelines

  2. CONTRIBUTING.md - Developer guide

    • Code style guidelines
    • Pull request process
    • Testing requirements
    • Example: Adding a new adapter

🚢 Production Deployment

  1. DEPLOYMENT.md - Production guide
    • Docker Compose setup
    • Linux manual installation
    • Systemd service configuration
    • Database optimization
    • Monitoring and backups
    • Scaling strategies

❓ Troubleshooting

  1. FAQ.md - Frequently asked questions
    • Installation issues
    • Configuration help
    • Usage questions
    • Performance optimization
    • Error troubleshooting

📋 Reference

  1. CHANGELOG.md - Version history

    • What's new in v1.0.0
    • Planned features
    • Breaking changes

  2. PROJECT_SUMMARY.md - Project overview

    • Statistics and metrics
    • File structure
    • Technology stack

📦 What's Included

Core Application

✅ 9 Python modules
✅ Pydantic data models
✅ SQLAlchemy ORM layer
✅ Async crawler infrastructure
✅ Website enrichment system
✅ Lead scoring engine
✅ Export functionality
✅ CLI interface
✅ Configuration system

Testing & Quality

✅ 24+ unit tests
✅ Type hints throughout
✅ Pre-commit hooks
✅ Black formatting
✅ flake8 linting
✅ mypy type checking

DevOps & Deployment

✅ Docker containerization
✅ Docker Compose stack
✅ Makefile with 30+ commands
✅ GitHub Actions CI/CD
✅ Systemd service templates
✅ Backup scripts

Documentation

✅ 50+ pages of guides
✅ Inline code documentation
✅ Example configurations
✅ Troubleshooting guide
✅ Deployment guide
✅ Contributing guide

🎯 Quick Navigation by Use Case

"I want to try it right now"

→ QUICKSTART.md (5 minutes)

"I need to install and use it"

→ README.md + QUICKSTART.md

"I want to deploy to production"

→ DEPLOYMENT.md

"I want to extend/customize it"

→ IMPLEMENTATION.md

"I want to contribute code"

→ CONTRIBUTING.md

"I have questions"

→ FAQ.md

"I need to understand architecture"

→ IMPLEMENTATION.md

"I want to see what changed"

→ CHANGELOG.md

📂 Project Structure

agrolead/
├── 📖 QUICKSTART.md              ← Start here!
├── 📖 README.md                  ← Feature guide
├── 📖 IMPLEMENTATION.md          ← Architecture
├── 📖 DEPLOYMENT.md              ← Production setup
├── 📖 CONTRIBUTING.md            ← Developer guide
├── 📖 FAQ.md                     ← Troubleshooting
├── 📖 CHANGELOG.md               ← Version history
├── 📖 PROJECT_SUMMARY.md         ← Overview
│
├── 🐍 agrolead/                  ← Main package
│   ├── models/
│   │   └── company.py           ← Pydantic models
│   ├── config/
│   │   └── settings.py          ← Configuration
│   ├── crawler/
│   │   ├── base_adapter.py      ← Base classes
│   │   └── sources/
│   │       └── adapters.py      ← Directory adapters
│   ├── database/
│   │   └── models.py            ← Database layer
│   ├── enrichment/
│   │   └── enricher.py          ← Enrichment system
│   ├── scoring/
│   │   └── scorer.py            ← Scoring engine
│   ├── export/
│   │   └── exporter.py          ← Export formats
│   ├── cli/
│   │   └── main.py              ← CLI interface
│   ├── app/
│   │   └── orchestrator.py      ← Main orchestrator
│   └── utils/
│       └── helpers.py           ← Utilities
│
├── 🧪 tests/                     ← Unit tests
│   ├── test_models.py
│   └── test_enrichment.py
│
├── 📦 Requirements & Config
│   ├── requirements.txt          ← Dependencies
│   ├── setup.py                  ← Installation config
│   ├── pyproject.toml            ← Modern Python config
│   ├── .env.example              ← Config template
│   ├── .pre-commit-config.yaml   ← Pre-commit hooks
│   └── Makefile                  ← Development tasks
│
├── 🐳 Docker & Deployment
│   ├── Dockerfile                ← Container image
│   ├── docker-compose.yml        ← Local & prod setup
│   └── .github/workflows/
│       └── ci-cd.yml             ← GitHub Actions
│
├── 📄 License & Legal
│   ├── LICENSE                   ← MIT License
│   ├── .gitignore                ← Git rules
│   └── DELIVERY_MANIFEST.md      ← What's included

🚀 Quick Start (Choose One)

Option 1: Local Development (Simplest)

cd agrolead
pip install -r requirements.txt
playwright install chromium
cp .env.example .env
python -m agrolead.cli.main init
python -m agrolead.cli.main search "tomato importer" --country FR

Time: 5-10 minutes

Option 2: Docker (Recommended for Production)

docker-compose up -d
docker-compose exec agrolead python -m agrolead.cli.main init
docker-compose exec agrolead python -m agrolead.cli.main search "tomato"

Time: 5 minutes (includes PostgreSQL)

Option 3: Makefile (Easiest Commands)

make install
make init
make search QUERY="tomato importer"
make crawl
make score
make export

Time: 5 minutes

⚙️ Key Configuration

All settings in .env file:

# Database choice
DB_TYPE=sqlite                    # or postgresql

# Crawling speed
CRAWLER_MAX_CONCURRENT_TASKS=5   # 5-10 for faster
CRAWLER_RATE_LIMIT_DELAY=1.0    # Seconds between requests

# Data quality
SCORING_MINIMUM_SCORE=20         # Min score to include
ENRICHMENT_ENABLED=true          # Extract website data

# Export options
EXPORT_FORMATS=csv,excel,json   # What to export

🎓 Learning Path

  1. Beginner: Run through QUICKSTART.md examples
  2. Intermediate: Explore README.md features and configuration
  3. Advanced: Read IMPLEMENTATION.md for architecture
  4. Expert: Study source code and extend with custom adapters

📊 What You Can Do

✅ Search B2B directories (Europages, Kompass) ✅ Extract emails, phones, social media ✅ Score leads 0-100 scale ✅ Export to CSV, Excel, JSON ✅ Store in SQLite or PostgreSQL ✅ Schedule automatic crawling ✅ Add custom directory sources ✅ Customize scoring weights ✅ Deploy to production

🔍 Features Highlights

Crawling

  • Multiple B2B directories
  • JavaScript site support
  • Concurrent crawling (10+ sites at once)
  • Automatic retries
  • URL caching

Enrichment

  • Email extraction & validation
  • Phone normalization (E.164)
  • Language detection
  • Social media discovery
  • Certification detection

Scoring

  • Multi-factor scoring (0-100)
  • Keyword matching
  • Configurable weights
  • Quality thresholds

Export

  • CRM-ready CSV
  • Formatted Excel
  • Structured JSON
  • SQLite database

💻 Technology Stack

  • Language: Python 3.12+
  • Web: Playwright, httpx, BeautifulSoup4
  • Data: Pydantic, SQLAlchemy, Pandas
  • CLI: Typer, Rich
  • Database: SQLite, PostgreSQL
  • DevOps: Docker, Docker Compose
  • CI/CD: GitHub Actions

📈 Performance

  • Speed: 10+ concurrent crawl tasks
  • Throughput: 100-1000 companies/day
  • Scalability: Handles 100k+ companies
  • Rate Limiting: Respectful crawling (configurable)
  • Caching: Smart URL cache prevents duplicates

🔒 Security & Ethics

✅ Respects robots.txt ✅ Configurable rate limiting ✅ No LinkedIn scraping ✅ No authentication bypass ✅ Input validation ✅ Type-safe code ✅ Environment variables for secrets

📞 Support

Need Help? Read This
Installation QUICKSTART.md
Configuration README.md
Architecture IMPLEMENTATION.md
Deployment DEPLOYMENT.md
Troubleshooting FAQ.md
Contributing CONTRIBUTING.md
Errors FAQ.md Troubleshooting section

🎯 Next Steps

  1. Read: QUICKSTART.md (5 min)
  2. Install: Follow setup instructions (5 min)
  3. Try: Run first search/crawl (5 min)
  4. Configure: Customize .env for your needs
  5. Deploy: Follow DEPLOYMENT.md for production
  6. Extend: Add custom adapters via IMPLEMENTATION.md

📊 Project Statistics

  • Total Files: 40+
  • Lines of Code: 3,166 (Python)
  • Documentation: 50+ pages
  • Test Coverage: 24+ test cases
  • Modules: 9 core + utilities
  • Deployment Options: Docker, systemd, manual
  • Supported Directories: 3 (extensible)

🎁 What You Get

✅ Production-ready application
✅ Complete source code
✅ Comprehensive documentation
✅ Docker containerization
✅ CI/CD pipeline
✅ Unit tests
✅ Configuration templates
✅ Deployment guides
✅ Troubleshooting help
✅ Extension examples

📝 License

MIT License - Free for commercial use, modification, and distribution. See LICENSE file.

🌱 Made For

Agricultural businesses discovering European suppliers:

  • Fresh produce importers
  • Food distributors
  • Wholesale companies
  • Foodservice suppliers
  • Retail sourcing teams
  • Food manufacturers

🚀 Ready to Start?

→ Go to QUICKSTART.md for 5-minute setup

Or jump to specific guide:

🌱 Happy lead hunting!

For agricultural businesses discovering European suppliers. Made with ❤️ and Python 3.12+ by m223rx