Add ability to dynamically adjust workload as it's running

[rustyrazorblade]

I've been in several situations now where I've wanted to change the rate limit of a running job without stopping it entirely. Sometimes it would be nice to just reduce the throughput by 10% or change the read rate.

When starting a job, add an option to open up a control port. My current thinking is to use gRPC to listen on that control port, and add a client mode to easy-cass-stress that would connect to that control port and accept commands to adjust the workload.

[rustyrazorblade]

As much as I hate to say this it might make sense to do this over JMX.

[rustyrazorblade]

Thinking about this further, I think it makes sense to add the jobserver in #42, and expose rest endpoints via ktor to do any sort of dynamic work like this.

[Atreus-Technologies]

Often when benchmarking the objective is to progressively increase the rate for a given pX latency target until the latency target is reached (maybe reached more than once to ensure transient issues aren’t triggering an early stop).

Is this something you’d consider supporting?

[rustyrazorblade]

It's already a feature - check out --maxrlat and --maxwlat, which set the max p99 latencies. There's a rate limiter optimizer that will try to dial in the max throughput given those parameters.

This issue is for when a user wants to manually intervene and have more control over the throughput, or potentially through automation with external coordination.

[Atreus-Technologies]

Amazing, thanks!