Add a new error and delay management layer

[piggyatbaqaqi]

I have 5 pull requests that implement a series of improvements I've found helpful for ingesting large numbers of files. A lot of it is about being a better netizen by following guidance from the web sites we're downloading from.