Using Dask
You may need to install dask and dask-jobqueue following
dask_jobqueue allows you to run your dask computations on a cluster, through a job scheduler such as SLURM. All you need to do is to add some code in the beginning to setup the basic job information such as the number of cores and memory, and then call cluster.scale(num_jobs) to submit the job.
from dask.distributed import Client
from dask_jobqueue import SLURMCluster
# Using total 96 cores (filling a single stellar nodes), with 3 threads per process.
# interface = "ib0" tells `SLURMCluster` that the network is through infiniband.
cluster = SLURMCluster(cores=96, memory='740 GiB', processes=32, interface="ib0", walltime='00:30:00')
# cluster.scale() submits a SLURM job.
# You can view the job script using cluster.job_script()
cluster.scale(1) # Submit for a single node; increase the number to use multiple nodes.
client = Client(cluster)
# Now, your computations below are done using the requested node!See Example notebook for basic usage.
SLURMCluster is designed in such a way that a single SLURM (or PBS, etc) job can only allocate a single node. However, this does not mean you can only run single-node computations. For example, if you setup SLURMCluster(cores=96) to fill an entire node (per job) and then launch two jobs via cluster.scale(2), those different jobs will coordinate with each other such that your calculations uses all 96x2 = 192 cores in the two nodes.
Sometimes, the SLURM cluster is configured in a way that it prioritize a large single job rather than many small jobs. In this case, you may want to use SLURMRunner which allows to submit multi-node job. An example script using SLURMRunner can be found in this link. See also Dask-Jobqueue documentation for more details.
In general, you would want to use SLURMCluster in an interactive jupyter notebook session, whereas you may find SLURMRunner more useful for submitting batch jobs.
For more information, see Dask-Jobqueue documentation.
Dask dashboard is extremely useful for monitoring the currently running processes.
The screenshot below shows an example Dashboard diagnostics associated with Example notebook. Each row represents a thread (32 threads in this example), executing various tasks (colored rectangles) from left to right. Note that all 32 threads run in parallel.
See Example notebook for how to use the dask dashboard.