A high-performance, asynchronous web scraper built with Python (aiohttp and parsel) to extract structured eBay listing data via ScraperAPI while bypassing anti-bot measures.
- Asynchronous Networking (
aiohttp): Uses Python's nativeasyncioevent loop to handle concurrent non-blocking HTTP requests cleanly on Windows environments without Twisted engine overhead. - Lightweight DOM Parsing (
parsel): Utilizes Scrapy's core parsing engine (Parsel) for extraction via CSS selectors and XPath logic. - Proxy & Anti-Bot Bypass: Integrated with ScraperAPI for automated IP rotation and header handling.
- Dual-Pipeline Design: Includes both an asynchronous primary driver (
ebay_async.py) and a synchronous fallback script (ebay_sync.py) for rapid testing and debugging. - Regex ID Normalization: Custom regex logic extracts canonical eBay Item IDs from both standard product listing URLs (
/itm/) and catalog URLs (/p/).
- Python 3.10+
- ScraperAPI Key
- Clone the repository:
git clone [https://github.com/YOUR_USERNAME/ebay-scraper.git](https://github.com/YOUR_USERNAME/ebay-scraper.git)
cd ebay-scraper- Set up a virtual environment:
python -m venv venv
# On Windows PowerShell:
.\venv\Scripts\Activate.ps1
# On Windows CMD:
.\venv\Scripts\activate.bat- Install dependencies:
pip install aiohttp parsel requestsPrimary Async Scraper (aiohttp + parsel) Runs the non-blocking asynchronous pipeline:
python ebay_async.pyFallback Sync Scraper (requests + parsel) Runs the single-threaded synchronous script for quick selector debugging:
python ebay_sync.pyScraped listings are exported to data/SAMPLE_UK_PS5_10_Items.csv encoded in utf-8-sig for Excel compatibility:
| Column | Description |
|---|---|
ebay_item_number |
Unique numerical eBay Listing ID extracted via Regex |
title |
Cleaned item listing title |
condition |
Item state (e.g., New, Refurbished, Used) |
price |
Displayed listing price |
url |
Canonical product page URL |