It's unofficial, and Google ignores it, but it might be polite to honor it if a site mentions us by name! https://en.wikipedia.org/wiki/Robots.txt#Crawl-delay_directive
It's unofficial, and Google ignores it, but it might be polite to honor it if a site mentions us by name!
https://en.wikipedia.org/wiki/Robots.txt#Crawl-delay_directive