PendingDeleteDomainsCrawler

A crawler for scraping domain metrics from PendingDelete.Domains — a platform that curates expiring domains with high traffic, strong authority, quality backlinks, and valuable keyword rankings.

Background

PendingDelete.Domains aggregates expiring domain data including:

• Global Rank — Alexa/similar traffic rank
• Monthly Visits — Estimated monthly visitor volume
• Domain Authority (DA) — Authority score of the expiring domain
• Backlink Count — Number of referring domains/links
• Keyword Rankings — Organic keywords the domain currently ranks for

While some domains carry high direct traffic, many others are valuable primarily for their backlink profiles and existing keyword rankings — making them attractive for SEO practitioners, domain investors, and content site builders. Full daily domain lists are only disclosed to paying subscribers; this crawler retrieves publicly accessible data without requiring a subscription or login.

How It Works

1. Discover today's domains via sitemap — The script fetches https://pendingdelete.domains/sitemap.xml to extract the domain report URLs published that day (e.g. https://pendingdelete.domains/domains/minicruzer.com).

2. Scrape each domain report page — For each URL discovered, the script visits the page and parses the domain metrics: global rank, monthly visits, domain authority, backlink count, keyword rankings, etc.

3. Googlebot user-agent required — Requests must include a User-Agent string containing googlebot to receive the full page content. Without this, the server returns restricted or empty results.

User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Usage

# Install dependencies
npm install   # or: pip install -r requirements.txt

# Run the crawler
node index.js   # or: python crawler.py

Output is saved to a CSV / JSON file in the output/ directory, containing one row per domain with all scraped metrics.

Output Fields

Field	Description
domain	Domain name
global_rank	Global traffic rank
monthly_visits	Estimated monthly visits
domain_authority	Domain authority score
backlinks	Number of backlinks
keywords	Number of ranking keywords
crawled_at	Timestamp of the crawl

Notes

• This tool is intended for research and personal use. Respect the site's robots.txt and terms of service.
• Rate-limit your requests to avoid overloading the server.
• Domain data refreshes daily; re-run each day to capture the latest listings.

License

MIT