site-crawler

Open-source SEO monitoring. GSC + Site Crawler + Core Web Vitals in one dashboard.

Generate clean, AI-ready llms.txt files for your website or docs. Supports crawling, sitemaps, static builds, and framework-aware adapters (Next.js, Vite, Nuxt, Astro, Remix). Includes Markdown/MDX docs mode and robots.txt generator for LLM and search crawlers.

Free Chrome extension for comprehensive Shopify SEO analysis. 50+ checks, real Core Web Vitals, site crawler, Google rank tracker, competitor comparison, internal link map, and PDF reports. No API keys, no signup, no cost.

SnapShot — Web Capture API by WOLVAREX Capture websites as screenshots, recordings, and more through a simple, powerful API.

Omnipotent Master SEO & Security Auditor (Cyber Fusion Engine v4.0)

Crawl an entire website and surface every broken link — self-contained PHP + cURL, dark-themed HTML dashboard + CSV export.

grandstage catalog data scraper

Crawlcat public REST API: OpenAPI contract, multi-language code examples, and docs for site SEO audits — crawl sites, export Copy for AI markdown, and integrate with Cursor, Claude, or ChatGPT via MCP.

Site crawler used in Ethora platform as an option to import your specific business data into your AI agent chat bot.