Rubbernecker

A web scraping engine built with Python and SeleniumBase that crawls web pages, stores raw HTML, and parses structured data. Supports configurable page actions and depth-based crawling.

Installation

Prerequisites

Python 3.12+

Google Chrome

macOS:

brew install --cask google-chrome

Fedora/RHEL (including WSL 2):

sudo dnf install -y fedora-workstation-repositories
sudo dnf config-manager setopt google-chrome.enabled=1
sudo dnf install -y google-chrome-stable

Ubuntu/Debian:

wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" | sudo tee /etc/apt/sources.list.d/google-chrome.list
sudo apt update
sudo apt install -y google-chrome-stable

Setup

make install

Or manually:

uv sync

Quick Start

See QUICKSTART.md for a step-by-step tutorial.

Commands

Command	Description
crawl	Scrape websites and save raw HTML to Avro files
fetch	Download assets from a list of URLs
parse	Extract structured data from crawled HTML
sitemap	Discover page URLs from sitemaps or robots.txt

Documentation

• Action Scripts — Automate page interactions during crawls
• Output Formats — Avro schemas for all command outputs
• Development — Testing, linting, and build commands

License

Apache-2.0