How fast is the download module?

[dynabler]

I was wondering, how fast does the download module "download"? Currently, I'm using download on 2 separate sites, while a third flyscrape is running on 1 request per second. It looks like the download module is unlimited. If it's unlimited, could a throttle be added to config?

[philippta]

The download module is unlimited and has no throttling. It is capable of downloading 5 files/images/etc in parallel.

I'm not to sure if I understand the benefit of throttling. Do you mind explaining how this benefits you?

[dynabler]

I'm not to sure if I understand the benefit of throttling. Do you mind explaining how this benefits you?

I can imagine running Flyscrape in the cloud and download speed wouldn't be a problem.

I can also imagine people scraping for hobby or once-in-a-while endeavor while using residential internet. In a resident, you're not alone and everybody likes to use the same internet connection without lagging and buffering videos etc. (I say resident because people live in it, be it home, college dorm, apartment shared internet etc.)

I was thinking of the download module as some sort of download manager. All download managers have throttle options, (example jDownloader). Something like the Flyscrape rate config.

And just another way of hiding the fact that a scraper is downloading, which is quite hard when a log shows 10K images downloaded with an average of 20Mbps over a span of a day.