Task Batch Scheduling

Task Batch Scheduling

TES currently uses Azure Batch to download/upload the files and run the executors defined in the submitted tasks. This document briefly describes this process including the configurable settings affecting this behavior. Currently only one executor is allowed per task. The task's resources determine the parameters of the Azure batch pool and task used to run the executor in a compute node provisioned by Azure batch.

Azure Batch

TES uses Azure Batch in one of two modes: shared autoscaled pools (the default, new in 4.0) and legacy autopool jobs (the only option before 4.0). TES always uses one batch job per pool and as a result that job's lifetime is always tied 1:1 to that pool. Each of these modes are configured in TES as described below:

Configuration	Shared autoscaled	Legacy autopool
Enabled by	BatchScheduling:UseLegacyAutopools = false (the default)	BatchScheduling:UseLegacyAutopools = true
Maximum pool lifetime	BatchScheduling:PoolRotationForcedDays (deployment default is 7)	n/a
Batch Pool/Job Id “prefix”	BatchScheduling:Prefix	n/a
Autopool job cleanup	n/a	BatchScheduling:DisableJobCleanup (default: false)
Note on "prefix": This is used to enable sharing of Azure batch accounts by multiple TES deployments with scheduling enabled (which is the default) and it needs to be different for each deployment that shares a batch account. If you will never share the batch account, you can simply accept the randomly created prefix by the deployer, or simply specify something like "TES1".

Shared autoscaled pools

This is the new default method for TES to use Azure Batch. TES separates each element of each task’s configuration into those provided by batch pools and batch tasks. It then checks to see if a pool with that configuration has already been created by that TES deployment. If no pool is found, a pool (with accompanying job) is created per that specification. A batch task is then created in that pool’s job. The batch account keeps track of how many tasks are assigned to each pool’s job(s), and it will automatically scale (up to quota and availability limits) to try to run all tasks in parallel if possible (checked every 5 minutes). When no tasks are either queued or running, the pool will scale down to zero compute nodes automatically. Note that most of the time, empty pools carry no billable costs. TES monitors each pool for various errors in order to more efficiently bring each pool back to stable operating condition as quickly as possible. Those conditions are reported back on pending/running TES tasks as appropriate, as well as always logged. To allow pool definitions to be refreshed (including the docker-host VM images that run on the compute nodes) by Azure Batch to ensure that security patches are applied, etc., pools lifetimes are capped. They will stop accepting tasks once this age has been reached and will then be deleted as soon as they have no remaining assigned tasks. If a new task is created in TES that would have used this pool, a new pool will be created that will run this new task (along with any other with matching criteria). This means that both pools may be running tasks in parallel for a time.

Legacy autopool jobs

This is the simplest method of using Azure Batch and is the method used in CoA versions before v4. The batch specifications for the TES task are determined and divided between the pool and the task as in the autoscaled mode, but are both presented together to Azure batch, which then creates an Azure batch job, pool and task, adds one compute node to that pool, runs the batch task, and then removes and recycles all the consumed batch resources. This method was discovered to have various issues when running at scale but may be a better fit for very low volume usage patterns.

Azure Batch Compute Node configuration

The compute nodes are configured according to the following items:

Compute node parameter	TES task	Configuration
Preemptibility	resources.preemptible	BatchScheduling:UsePreemptibleVmsOnly
Executor image	executors.image	n/a
Task "resources"	resources, including all current backend_parameters	n/a
"Global" managed identity	n/a	BatchNodes:GlobalManagedIdentity
Global start task	n/a	BatchNodes:GlobalStartTask
Public IP address	n/a	BatchNodes:DisablePublicIpAddress
Private network subnet	n/a	BatchNodes:SubnetId

Azure Batch Quotas and Availability

In both modes, Azure batch and TES will wait without errors whenever either quota or availability constraints prevent all tasks from being assigned a compute node. In autoscale mode, if batch pool and/or job quota were to be reached, one of the existing pools that’s empty with its associated job will be deleted to make room for the newly needed pool/job. If none are empty, the task(s) will remain queued and TES will keep checking for availability and for pools/jobs that can be removed automatically.

Logging

To be written