This package get, fetch, crawl, sitemap pages recursively and fetch all links in between <loc> tag.
First-party Google Search Console and technical SEO engine. Automate Googlebot indexing, audit Core Web Vitals, and capture Google AI Overviews in pure Ruby with zero dependencies.
Python tool to extract URLs from XML sitemaps (including nested sitemap indexes) and automatically submit them to Google Indexing API for faster indexing. Supports bulk processing, rate limiting, and detailed progress reporting.
Fast Python web crawler for RAG and AI ingestion. Extracts clean Markdown from any site for LLMs and vector stores.
Collect links through the sitemap.xml or robots.txt
GoSitemap2Md is a Golang program that generates a sitemap URL in Markdown format and stores the URLs in a urls.json file for easy adding of new URLs. This tool simplifies the process of generating and maintaining a sitemap for your website.
internal links extraction tool
Automated Performance Auditing & Surgical Refactoring for High-Traffic Platforms
MCP server for XML sitemap discovery, parsing and URL extraction for AI agents
XML sitemap extractor CLI and GitHub Action with nested index, gzip, API discovery, and URL output
a python script that crawls website sitemap in a very quick way with multi threading and extract, write SEO based data to CSV file
🌐 Automate URL extraction from XML sitemaps and submit to Google Indexing API for faster indexing and improved SEO performance.
To associate your repository with the sitemap-crawler topic, visit your repo's landing page and select "manage topics."