Request: Rate Limit

[pBrambi]

I love your CLI tool! I am truly missing just one thing - rate liming.

I use --concurrency 1 and even than, on a larger website, I get rate limited (our websites) usually after 10-15 seconds.

Some of them are on shared hosting so I have no way to whitelist/ignore my IP.

**It would be lovely to have an option to set "pause" inbetween the requests (in ms).

Ex.: --pause 500ms**

Thank you for this wonderful tool!

[JustinBeckwith]

So this is a really tough one - ultimately a lot of websites are going to be very aggressive about this kind of thing. One of the ways I try to get around this is properly supporting 429 responses. For the websites where you're seeing this problem - do you know what status code they're returning? What's the behavior like when you get rate limited? An example site would be incredibly helpful!

[pBrambi]

Hello,

as for my shared hosting provider, it returns:

403 Forbidden

They use FailToBan I think, which rate limits me for about 60 seconds if requests in a set timeframe exceed some set ammount of requests.

After 60 seconds about, I get 200 Ok again right away - and the timeframe starts again.

It does not return 429 Too many requests before - it goes from 200 Ok to straight 403 Forbidden.

[gratach]

My Nginx server setup returns code 0 when the rate limit is exceeded.

I used a script to create a demo website.

When I run linkinator -r https://linkinator-test.debablo.de the tool detects more than 600 broken links even though the corresponding pages are online.

I think it would be good to have a --pause or --crawl-delay command line option. Maybe a --follow-robots-txt option that makes sure that the crawl delay from the robots.txt is used would also be nice to have. Especially when your website links a lot of third-party pages, this could prevent your IP address from being blacklisted.