match wait time between SIGTERM to 1/2 a batch.

[petersilva]

The other half of #1531 is that we consult the metrics available for all the flows being stopped. We add a meanBatch metric, to the msgRate metrics to understand how long it should take for given flow to finish stopping. So it doesn't SIGKILL until a time that is >2.5 times the average time to process an entire batch has gone by.

It turned out much easier than the method described in the issue discussion. Instead of having to use new statefiles, I was able to use existing metrics files. Using the batch setting turned out to be bad, because often flow have very small batches, so the maximum given by the setting is far too long. Added a meanBatch to the flow metrics to understand how big the average batch is. Use that new metric with the existing msgRate to say that:

• if stop is requested, it should not take more than the 2 1/2 times the average duration of processing a batch of messages.

Often this is a very short period of time, and it will result in stop running faster.
When dealing with very large file transfers, or large batches of files being transferred, this change will make it wait a more appropriate amount of time.

[github-actions]

Test Results

245 tests   243 ✅  1m 34s ⏱️
  1 suites    1 💤
  1 files      1 ❌

For more details on these failures, see this check.

Results for commit 753ee43.

♻️ This comment has been updated with latest results.

[petersilva]

hey guys? This is still here. Nothing wrong with it, afaik. It just changes the delay between SIGTERMs and SIGKILLS. It should help a lot in the case of big batches, because it should wait about 2.5 batches between sending signals and thus be far more likely to terminate cleanly.

[reidsunderland]

This sounds promising but we haven't had a chance to test it yet. We'll test on ddsr-dev