Currently, there exist no enforced limit on the number of Tasks a user can create for a given Transformation, meaning it is very easy to (accidentally or otherwise) create very large (thousands or more) Tasks on a single Transformation. There is also no enforced limit on the number of such Tasks that can have ACTIONSrelationships to a givenTaskHub`, which can choke the compute services due to the bottleneck of the claiming code (reduced by #121, but not able to scale infinitely).
To address this, we should create server-side configurable limits to:
- the number of
waiting Tasks allowed on a Transformation at a time
- the number of
ACTIONS relationships on a TaskHub at a time
User-client side, this will mean that when a user calls AlchemiscaleClient.create_tasks(...), they will get back ScopedKeys for Tasks created until no more can be created for that Transformation; if a Task can't be created due to the limit, this will be reflected in the list of ScopedKeys returned, with a None given in its place.
Likewise, when a user calls AlchemiscaleClient.action_tasks(...), the method will action as many of the Tasks as it can up till the limit, then a None will be given for those Tasks that were not actioned.
Currently, there exist no enforced limit on the number of
Tasks a user can create for a givenTransformation, meaning it is very easy to (accidentally or otherwise) create very large (thousands or more)Tasks on a singleTransformation. There is also no enforced limit on the number of suchTasks that can haveACTIONSrelationships to a givenTaskHub`, which can choke the compute services due to the bottleneck of the claiming code (reduced by #121, but not able to scale infinitely).To address this, we should create server-side configurable limits to:
waitingTasks allowed on aTransformationat a timeACTIONSrelationships on aTaskHubat a timeUser-client side, this will mean that when a user calls
AlchemiscaleClient.create_tasks(...), they will get backScopedKeys forTasks created until no more can be created for thatTransformation; if aTaskcan't be created due to the limit, this will be reflected in the list ofScopedKeys returned, with aNonegiven in its place.Likewise, when a user calls
AlchemiscaleClient.action_tasks(...), the method will action as many of theTasks as it can up till the limit, then aNonewill be given for thoseTasks that were not actioned.