bingbot

🔨 List all IP ranges from: Google (Cloud & GoogleBot), Bing (Bingbot), Amazon (AWS), Microsoft, Oracle (Cloud), GitHub, Facebook (Meta), OpenAI (GPTBot) and other with daily updates.

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

A crawler that crawls search engine! 😎 Usable for collecting site with dorks and wildcards. Also provides output in web interface with more than 3 API endpoints!

Continuously-updated public IP ranges for major cloud providers, SaaS services, and bots — one txt file per provider, refreshed every 4 hours via GitHub Actions. Drop-in for firewalls, allowlists, and geo blocks.

A Bash script that fetches and parses JSON-based IP range data for trusted search engine bots (Googlebot, Bingbot, and others), ideal for use with ModSecurity and other web application firewalls and web servers.

Turn raw server/CDN access logs into verified-crawler analytics: parse many log formats, verify Googlebot/Bingbot by IP (not user-agent), enrich with DuckDB, and analyze crawl budget, crawl waste, and crawl traps with SQL.

Interactive crawler IP intelligence dashboard for search, AI, and user-triggered fetchers.

Sitenize googlebot, yandexbot, bingbot gibi kullanıcılar geldiğinde göstebileceğiniz veya göstermek istemediğiniz şeyleri ayarlayabileceğiniz basit bir php kodudur.

Fetch and inspect any URL as Googlebot, Bingbot or a real browser, directly from your Umbraco site. Built for SEO debugging, diagnosing hacked/cloaked pages and inspecting redirects.