Scraping Pipeline

A polite, resumable web scraper that builds its own dataset from books.toscrape.com — with robots.txt compliance, rate limiting, exponential-backoff retries, incremental re-scraping, and 32 tests that run without touching the network.

pip install -r requirements.txt
python -m src.pipeline              # runs against saved fixtures, safe anywhere
python -m src.pipeline --live       # actually crawl books.toscrape.com
pytest                             # 32 tests, offline, under a second

This is the third in a series on the same theme — turning raw, messy inputs into trustworthy data. online-retail-eda cleaned a given file; retail-dimensional-model modelled it into a warehouse; this one produces a dataset from scratch, which is the step most portfolios skip by downloading a ready-made CSV.

---

What makes it a pipeline and not a script

A script downloads everything every time and falls over on the first surprise. The engineering is in what surrounds the requests.get:

Concern	How it is handled	Where
Don't be a nuisance	robots.txt is fetched and obeyed per host; a disallowed URL raises rather than being silently skipped	fetcher.py
Don't hammer the server	a minimum delay is enforced by the fetcher, so no calling code can forget to throttle	fetcher.py
Survive transient failures	5xx and network errors retry with exponential backoff; 4xx never retries, because a 404 won't fix itself	fetcher.py
Identify honestly	a descriptive User-Agent with a contact URL	config.py
Don't refetch what you have	fetched pages are cached on disk by URL hash	fetcher.py
Do the minimum next time	per-URL content hashes classify each book as new / changed / unchanged	state.py
Never lose 999 records to one bad page	every field extractor returns None on absence and records the gap	parser.py
Stop at the real end	pagination follows the site's own "next" links, not a guessed page-N.html	parser.py

The incremental promise, demonstrated

$ python -m src.pipeline --live --max-pages 3
crawl summary: {'books_found': 60, 'scraped_new': 60, 'skipped_unchanged': 0}

$ python -m src.pipeline --live --max-pages 3      # immediately again
crawl summary: {'books_found': 60, 'scraped_new': 0, 'skipped_unchanged': 60}

The second run does no writes and, on a warm cache, no network requests. A test asserts exactly this (test_second_crawl_skips_unchanged).

---

Resilience is the headline feature

The one fixture that matters most is the broken one. parser.py treats every field as optional and records what went missing, so a page with no price table and no rating produces a partial record, not an exception that kills the run:

book = parse_detail(malformed_html, url)
book.title            # 'Book With Missing Fields'  — what survived
book.rating           # None                        — honestly absent
book.missing_fields   # ['rating', 'availability']  — and reported

Those missing_fields flow into the scrape report as field-completeness percentages, so data quality is measured on every run rather than assumed.

---

Output: two formats for two audiences

• SQLite (data/books.db) — for querying with SQL. The write is an idempotent upsert keyed on each book's UPC, so re-running updates rows in place instead of duplicating them.
• Parquet (data/books.parquet) — for analysis in pandas / Polars / DuckDB: columnar, typed, compressed.

Rows that arrive without a UPC (a badly broken page) are kept in Parquet but excluded from SQLite's keyed table — and counted in the report, so nothing vanishes silently.

SELECT category, count(*) AS books, round(avg(price), 2) AS avg_price
FROM books GROUP BY category ORDER BY books DESC;

---

Why it runs offline

Every test, and the default pipeline run, use saved HTML fixtures in tests/fixtures/ — real page structures captured from the target. This is not a shortcut; it is the correct way to test a scraper:

• tests are fast and deterministic, and pass whether or not the site is up;
• CI needs no network, so the badge means something;
• a new contributor sees the whole pipeline work with one command, with zero risk of hammering the live site while they learn the code.

--live is the explicit switch that turns on real requests. The network is behind a deliberate flag, never the default.

---

Repository layout

src/
  config.py     target URL, politeness settings, retry policy — all in one place
  fetcher.py    robots.txt, rate limiting, backoff retries, disk cache
  parser.py     resilient HTML -> typed Book records, with provenance
  state.py      incremental new/changed/unchanged classification
  storage.py    idempotent SQLite upsert + Parquet output
  crawler.py    orchestration: walk pages, follow details, apply state
  pipeline.py   CLI (fixture vs --live) and the quality report
tests/
  fixtures/     real page structures, including one deliberately malformed
  test_parser.py    field extraction and the no-crash guarantee
  test_pipeline.py  retries, robots, state, storage, end-to-end crawl

---

Running it

python -m src.pipeline                      # fixtures, safe
python -m src.pipeline --live               # full crawl of ~1000 books
python -m src.pipeline --live --max-pages 5 # first 5 catalogue pages
python -m src.pipeline --live --force       # re-scrape even unchanged books
pytest                                      # 32 tests

A live full crawl of all ~1000 books takes a few minutes at the default half-second delay — deliberately gentle, because the point is to be a good citizen, not to be fast.

---

Ethics and limitations

• This targets a sandbox built for scraping. books.toscrape.com exists to be crawled. The same code pointed at a real site would need its rate limit revisited, its robots.txt genuinely respected (it is), and its terms of service checked — none of which the code can decide for you.
• The parser is coupled to this site's markup. That coupling is contained in parser.py and pinned by fixtures, so when the site changes, exactly one file and its tests need updating — which is the point of separating fetch, parse, and store.
• No JavaScript rendering. This site is server-rendered HTML. A JS-dependent target would need a headless browser (Playwright), which is a different fetch layer behind the same parser/state/storage interface.
• State is a single JSON file. Fine for thousands of URLs; a large crawl would move it to SQLite alongside the data.

Data source

books.toscrape.com — a demo bookstore published by scrapinghub / Zyte specifically as a legal, stable target for practising web scraping.