Up to now, we have been making use of the SynchronousComputeService as our workhorse for executing alchemical Transformations at scale. This service executes a ProtocolDAG one ProtocolUnit at a time, currently in-process, and by virtue of its simplicity it is also very robust.
However, a downside to this service is that it can leave a lot of potential throughput on the table. For example, if the service is provisioned with a GPU, running one ProtocolUnit at a time may not come anywhere close to saturating it. Running more than one ProtocolUnit in parallel on the GPU may yield substantially more throughput if there is capacity for it.
We propose creating an AsynchronousComputeService that is capable of performing one or Tasks/ProtocolDAGs concurrently, with multiple ProtocolUnits executed in parallel. Because there isn't a reliable way to predict how much a given ProtocolUnit may saturate the compute service's resources (CPU, memory, GPU load, GPU memory), we also propose that this service perform reactive scheduling:
- if its resources aren't yet saturated, it will attempt to run an additional
ProtocolUnit in parallel with those already running.
- if its resources are oversaturated, it will kill the most recently started
ProtocolUnit it is running in parallel, and not attempt to run it again until conditions change.
Up to now, we have been making use of the
SynchronousComputeServiceas our workhorse for executing alchemicalTransformations at scale. This service executes aProtocolDAGoneProtocolUnitat a time, currently in-process, and by virtue of its simplicity it is also very robust.However, a downside to this service is that it can leave a lot of potential throughput on the table. For example, if the service is provisioned with a GPU, running one
ProtocolUnitat a time may not come anywhere close to saturating it. Running more than oneProtocolUnitin parallel on the GPU may yield substantially more throughput if there is capacity for it.We propose creating an
AsynchronousComputeServicethat is capable of performing one orTasks/ProtocolDAGs concurrently, with multipleProtocolUnits executed in parallel. Because there isn't a reliable way to predict how much a givenProtocolUnitmay saturate the compute service's resources (CPU, memory, GPU load, GPU memory), we also propose that this service perform reactive scheduling:ProtocolUnitin parallel with those already running.ProtocolUnitit is running in parallel, and not attempt to run it again until conditions change.