Crawling slows to a crawl (pun intended)

[LeoI07]

Last year, I attempted to use Heritrix to archive specific parts of a single website, using beans to reject URIs not in its domain. I ran it on an Intel Core i7-4770HQ, giving Java 2 GiB of memory, until it ran out of disk space. At first I had one job running, and then toward the end I started a second one. The first job reached 1.2 million URLs, 106 GiB, downloaded over the course of 75 days (with another 6.3 million URLs queued), while the second reached 200,000 URLs, 29 GiB, downloaded over the course of 24 days (with another 1.4 million URLs queued).
At the start it was crawling rapidly, but it progressively got slower and slower, and by the end, both jobs (IIRC) were going at around 11 seconds per URL. While I think having two jobs running at once contributed to the slowdown, the sheer sluggishness didn't make sense, as if I tried accessing the same website I was crawling, (IIRC) even from the same computer I was running Heritrix on, it loaded at a normal speed, so I don't think I had been rate limited. I'd like to make another attempt to archive this website, so does anyone know why it was going so slow before, and if there's a way to fix it?

[ato]

It's possible the site's responses are slowing down the deeper the crawl gets. This can happen for example if there's a search page the crawler is paging deeply into, that they've implemented by retrieving all the results and discarding up to the current page.

Heritrix's default politeness settings are intentionally very conservative. The default delayFactor is 5.0 so if a response takes 2.2 seconds then Heritrix will wait 11 seconds before sending the next request. For single site crawls I normally use delayFactor = 2.0 and minDelayMs = 500.

Others things to check:

• is there a Crawl-delay directive in the site's robots.txt file? - this can be overriden with respectCrawlDelayUpToSeconds
• response time of individual requests (the number after + in 7th column of crawl.log) - is the delay you're seeing this multiplied by delayFactor?
• size of the responses (3rd column of crawl.log) - is it slow because large files are being downloaded?
• CPU usage (top) and disk IO usage (iostat) - is it spending a lot of time in garbage collection? slow disk reading/writing crawl state?

[LeoI07]

Both delayFactor and minDelayMs were set to zero.

After examining what URIs were fetched, it appears as though the vast majority of pages fetched weren't even valid pages! It apparently interpreted JavaScript object property names as relative paths and then crawled them (apparently this issue). It also fetched some outlinks despite those being excluded from the crawl via MatchesListRegexDecideRule beans. I believe that excluding these would be the most effective way to speed up the crawl. Most of the property names could be excluded from the crawl via MatchesListRegexDecideRule beans, but based on the outlinks, I expect that they would still be fetched. Is there a way to exclude URIs that match a regular expression not just from being crawled, but from being fetched at all?