Skip to content

FAQ.md

Latest commit

 

History

History
433 lines (341 loc) · 10.8 KB

FAQ.md

File metadata and controls

433 lines (341 loc) · 10.8 KB
Raw

Agrolead FAQ (Frequently Asked Questions)

Installation & Setup

Q: What are the minimum system requirements?

A:

  • Python 3.10+ (3.12 recommended)
  • 2GB RAM minimum, 4GB+ recommended
  • 500MB disk space for initial installation
  • Internet connection for crawling

Q: How do I install Agrolead?

A:

cd agrolead
pip install -r requirements.txt
playwright install chromium
cp .env.example .env
python -m agrolead.cli.main init

See QUICKSTART.md for detailed steps.

Q: Can I use Python 3.9 or older?

A: No, Agrolead requires Python 3.10+. Type hints and async features used require this version.

Q: What about on Windows?

A: Yes, Agrolead works on Windows. Use:

python -m venv venv
venv\Scripts\activate
pip install -r requirements.txt
playwright install chromium

Q: I get "playwright not found" error

A: Run playwright install chromium to download browser binaries.

Configuration

Q: Where do I set configuration options?

A: Edit the .env file in the project root. Copy .env.example as a template.

Q: What's the difference between SQLite and PostgreSQL?

A:

  • SQLite: Single file, good for development/testing, limited concurrent access
  • PostgreSQL: Network database, good for production, handles concurrent access

Q: Can I use a remote PostgreSQL database?

A: Yes. Set DB_POSTGRESQL_URL environment variable:

DB_POSTGRESQL_URL=postgresql://user:password@remote.server.com:5432/agrolead

Q: How do I change the rate limit?

A: Edit .env:

CRAWLER_RATE_LIMIT_DELAY=1.0  # Seconds between requests

Lower values = faster crawling but may get blocked.

Usage

Q: How do I search for companies?

A:

python -m agrolead.cli.main search "tomato importer" --country FR --limit 50

Q: What countries are supported?

A: France (FR), Germany (DE), Italy (IT), Spain (ES), Netherlands (NL), Belgium (BE), United Kingdom (UK), Portugal (PT), Poland (PL)

Q: Can I crawl multiple directories at once?

A: Not in one command, but you can run multiple commands:

python -m agrolead.cli.main crawl --directory europages
python -m agrolead.cli.main crawl --directory kompass

Q: What export formats are available?

A: CSV, Excel, JSON, and SQLite database.

Q: Can I export only high-quality leads?

A: The export automatically filters by minimum score (default 20). Adjust in .env:

SCORING_MINIMUM_SCORE=50  # Only export scores >= 50

Q: How long does crawling take?

A: Depends on:

  • Number of companies (typically 1 URL per company)
  • Rate limiting (default 1 second/request)
  • Website response times
  • Network speed

Example: 100 companies × 1 sec = ~2-3 minutes with crawling + enrichment.

Q: Can I resume a stopped crawl?

A: Not automatically. You'd need to:

  1. Identify which companies were already crawled
  2. Create a new list without those
  3. Run crawl again on the new list

Or enable deduplication to automatically skip existing companies.

Crawling & Scraping

Q: Am I respecting robots.txt?

A: Agrolead follows robots.txt guidelines and implements:

  • Rate limiting between requests
  • Proper user-agent headers
  • Caching of visited URLs
  • Respectful crawling practices

Q: My crawler is getting blocked

A: Try these solutions:

  1. Increase rate limit: CRAWLER_RATE_LIMIT_DELAY=3.0
  2. Reduce concurrent tasks: CRAWLER_MAX_CONCURRENT_TASKS=2
  3. Wait before retrying (different IP)
  4. Use proxy support if configured

Q: Can I crawl LinkedIn?

A: No, and we don't recommend it. LinkedIn's Terms of Service explicitly prohibit automated crawling. Agrolead focuses on public B2B directories.

Q: Why are some companies not found?

A: Reasons include:

  • Not listed in that directory
  • Directory doesn't include that region
  • Company is too small or inactive
  • Directory has access restrictions

Q: Can I add custom directories?

A: Yes! See IMPLEMENTATION.md for extending with custom adapters.

Data & Scoring

Q: Why is a company's score low?

A: Scoring considers:

  • No public email: -10 points
  • No contact page: -10 points
  • Not tagged as importer: -25 points
  • No tomato/produce keywords: -20 points

To view detailed scoring: Check the final_score field.

Q: Can I change scoring weights?

A: Yes, in .env:

SCORING_SCORE_IMPORTER=40
SCORING_SCORE_FRESH_PRODUCE=30
SCORING_SCORE_TOMATO_MENTION=20

Q: Why are duplicate companies being marked?

A: Duplicates are detected by:

  • Same website
  • Same company name (fuzzy match)
  • Same primary email

To prevent: Ensure input data is clean.

Q: How do I handle duplicates?

A:

# View duplicates
python -c "
from agrolead.database.models import create_database, get_session, CompanyModel
from agrolead.config.settings import settings

session = get_session(create_database(settings.get_database_url()))
dups = session.query(CompanyModel).filter(CompanyModel.is_duplicate == True).all()
print(f'Found {len(dups)} duplicates')
"

# Exclude in export
EXPORT_INCLUDE_DUPLICATES=false

Database

Q: How do I back up my database?

A:

SQLite:

cp data/agrolead.db data/agrolead-backup-$(date +%Y%m%d).db

PostgreSQL:

pg_dump -U agrolead agrolead | gzip > agrolead-backup-$(date +%Y%m%d).sql.gz

Q: My database is too large

A: Options:

  1. Reduce EXPORT_MAX_ROWS_PER_FILE
  2. Delete old companies:
session.query(CompanyModel).filter(
    CompanyModel.date_crawled < old_date
).delete()
  1. Archive to PostgreSQL (larger capacity)

Q: How do I view data in the database?

A:

SQLite:

sqlite3 data/agrolead.db
SELECT * FROM companies LIMIT 10;

PostgreSQL:

psql -U agrolead -d agrolead
SELECT * FROM companies LIMIT 10;

Enrichment

Q: Why aren't emails being found?

A: Reasons:

  • Emails in images (not text)
  • Behind contact forms
  • JavaScript-generated content
  • Obfuscated (encoded)

Solution: Manually check website or use contact form.

Q: Why are phone numbers not normalized?

A: May occur if:

  • Invalid format
  • Missing country code
  • Incorrect country detection

Check phone_extractor.py for supported formats.

Q: Can I enrich faster?

A: Options:

  1. Reduce pages visited: ENRICHMENT_MAX_PAGES_TO_VISIT=3
  2. Disable enrichment: ENRICHMENT_ENABLED=false
  3. Increase concurrency: CRAWLER_MAX_CONCURRENT_TASKS=10

Docker & Deployment

Q: How do I run Agrolead in Docker?

A:

docker-compose up -d
docker-compose exec agrolead python -m agrolead.cli.main init

Q: How do I access the PostgreSQL database from Docker?

A:

docker-compose exec postgres psql -U agrolead -d agrolead

Q: My Docker container keeps stopping

A: Check logs:

docker-compose logs agrolead

Common issues:

  • Database not initialized: Run init command
  • Out of memory: Reduce concurrency
  • Port already in use: Change port in docker-compose.yml

Q: Can I deploy to cloud (AWS, Azure, etc.)?

A: Yes! See DEPLOYMENT.md for guides.

Performance

Q: How can I speed up crawling?

A:

  1. Increase concurrent tasks: CRAWLER_MAX_CONCURRENT_TASKS=10
  2. Decrease rate limit: CRAWLER_RATE_LIMIT_DELAY=0.5
  3. Disable enrichment: ENRICHMENT_ENABLED=false
  4. Use lighter parsing: Switch from Playwright to HTTPAdapter

Q: High memory usage

A: Solutions:

  1. Reduce concurrent tasks
  2. Process in smaller batches
  3. Increase swap space
  4. Add more RAM

Q: Database queries are slow

A: Solutions:

  1. Add indexes:
CREATE INDEX idx_score ON companies(final_score);
CREATE INDEX idx_country ON companies(country);
  1. Archive old data
  2. Upgrade to PostgreSQL
  3. Add more RAM

Errors & Troubleshooting

Q: "ModuleNotFoundError: No module named 'agrolead'"

A: Install in development mode:

pip install -e .

Q: "Playwright browser not found"

A: Install browsers:

playwright install chromium

Q: "Database is locked" (SQLite)

A: Solutions:

  1. Close other connections
  2. Wait a few seconds
  3. Switch to PostgreSQL
  4. Reduce concurrent tasks

Q: "Connection refused" (PostgreSQL)

A: Check:

  1. PostgreSQL is running: systemctl status postgresql
  2. Connection string is correct
  3. User has permissions
  4. Firewall allows connection

Q: ImportError for Playwright

A: Reinstall:

pip install --force-reinstall playwright==1.40.0
playwright install chromium

Q: Timeout errors

A: Increase timeouts:

CRAWLER_REQUEST_TIMEOUT=60
CRAWLER_PAGE_TIMEOUT=120

Q: "No such file or directory: .env"

A: Create it:

cp .env.example .env

Testing

Q: How do I run tests?

A:

make test              # Run all tests
make coverage          # With coverage report
make test-fast         # Stop on first failure

Q: How do I write tests?

A: See CONTRIBUTING.md and tests/ examples.

Q: What's the test coverage?

A: Run: make coverage and check htmlcov/index.html

Security

Q: Is Agrolead secure?

A: Yes:

  • ✅ No credentials in code
  • ✅ Uses environment variables for secrets
  • ✅ Input validation on all user inputs
  • ✅ No SQL injection (uses SQLAlchemy ORM)
  • ✅ Regular dependency updates

Q: Should I use this in production?

A: Yes, but follow security practices:

  1. Use strong database passwords
  2. Enable HTTPS for any web interfaces
  3. Keep dependencies updated
  4. Regular backups
  5. Monitor logs
  6. Use firewall rules

Q: How do I report security issues?

A: Email security@agrolead.io (or open private issue on GitHub).

Licensing

Q: What license is AgroLead under?

A: MIT License - Free for commercial use, modification, distribution.

Q: Can I use this commercially?

A: Yes! MIT license allows commercial use. See LICENSE.

Support & Community

Q: Where can I get help?

A:

  1. Check this FAQ
  2. Read README.md and QUICKSTART.md
  3. Search GitHub issues
  4. Open a new issue
  5. Check IMPLEMENTATION.md

Q: How do I request a feature?

A: Open a GitHub issue with the enhancement label.

Q: How do I report a bug?

A: Open a GitHub issue with detailed:

  • Steps to reproduce
  • Expected vs actual behavior
  • Error messages and logs
  • System information

Q: Can I contribute?

A: Yes! See CONTRIBUTING.md.

Q: Is there a roadmap?

A: See CHANGELOG.md for planned features.

Getting More Help

  • Documentation: Check README.md and guides
  • Examples: See tests/ directory
  • Issues: Search GitHub issues
  • Community: Contribute to improve

Still have questions? Open an issue on GitHub or check our documentation!

🌱 Happy lead hunting!