Limit running tasks per batch

[WilliamDEdwards]

I use the Batch API to batch requests per customer.

While I can control the concurrency of Celery workers with WORKER_CONCURRENCY, is there a way to limit the concurrency of tasks per batch? Currently, there are situations in which Internet.nl does tens of requests per second to a specific customer (with a few hundred domains, all in the same batch).

[bwbroersma]

Since version v1.11.0 there are more requests than initially expected, these are greatly reduces in the future release v1.11.2, see:

• Reduce (START)TLS requests #2031

There are efforts to reduce these requests even more, hopefully that would remove the need to limit this.

[WilliamDEdwards]

@bwbroersma

Does "more requests than initially expected" also apply to web scans? I started a batch scan for 599 domains, and saw over 14,804 requests in the span of 20 minutes.

[WKobes]

@WilliamDEdwards both web and email had increased requests. Latest version 1.11.2 should decrease them significantly (yet still more than in 1.10.x)

[bwbroersma]

@WilliamDEdwards:
See #2130 second table column 'test v1.b6a.nl per IPv' for the amount of Client Hello's (TLS setups) per IPv4/IPv6, so if both are supported it's double this count. Also the more efficient cipher testing in v1.11.2 is equal to the supported ciphers in TLSv1.2 + 4, for this test domain the supported TLSv1.2 ciphers are 6.