Advanced Usage
More advanced techniques for submitting jobs, e.g. GPU, array and MATLAB tasks and full fsl_sub usage information
If your task comprises a complicated pipeline of interconnected tasks there are several options for splitting into dependent tasks or parallelisation of independent portions across many cluster nodes. Information on these techniques and other advance options is in this section.
How to submit pipeline stages such that they wait for their predecessor to complete
If you have a multi-stage task to run, you can submit the jobs all at once, specifying that later stages must wait for the previous task to complete. This is achieved by providing the '-j' (or --jobhold) option with the job id of the task to wait for. For example:
jid=$(fsl_sub -T 1 -R2 ./my_first_stage) fsl_sub -T 24 -R 16 -j $jid ./my_second_stage
Note the $() surrounding the first fsl_sub command, this captures the output of a command and stores the text in the variable 'jid'. This is then passed as the job id to wait for before running 'my_second_stage'.
It is also possible to submit array holds with the --array_hold command which takes the job id of the predecessor array task. This can only be used when both the first and subsequent job are both array tasks of the same size (same number of sub-tasks) and each sub-task in the second array depends only on the equivalent sub-task in the first array.
How to request a GPU for your job
Whilst GPU tasks can simply be submitted to the short.gq or long.gq queues fsl_sub also provides helper options which can automatically select a GPU queue and select the appropriate CUDA toolkit for you.
- -c|--coprocessor <coprocessor name>: This selects the coprocessor with the given name (see fsl_sub --help for details of available coprocessors)
- --coprocessor_multi <number>: This allows you to request multiple GPUs. On the FMRIB cluster you can select no more than two GPUs. You will automatically be given a two-slot openmp parallel environment
- --coprocessor_class <class>: This would allow you to select which GPU hardware model you require, see fsl_sub --help for details
- --coprocessor_class_strict: If a class is requested you will normally be allocated a card at least as capable as the model requested. By adding this option you ensure that you only get the GPU model you asked for
- --coprocessor_toolkit <toolkit version>: This allows you to select the API toolkit your sofware needs. This will automatically make available the requested CUDA libraries where these haven't been compiled into the software
- cuda selects GPUs capable of high-performance double-precision workloads and would normally be used for queued tasks such as Eddy and BedpostX.
- cuda_all selects all GPUs.
- cuda_ml selects GPUs more suited to machine learning tasks, the typically have very poor double-precision performance, instead being optimised for single, half and quarter precision workloads - use these for tasks involving ML inference and development, although training may still be more optimal on the general purpose GPUs depending on the task this involves, ask the developer of the software for advice on this.
INTERACTIVE JOBS (INCLUDING GPU/MACHINE LEARNING TASKS)
Where your program requires interaction we offer an interactive queue which can be used to get a terminal session on one of the cluster nodes.
To request a terminal session for GPU tasks, issue the following command on a resomp head node:
srun -p gpu_short --pty bash
There may be a delay whilst the system finds a suitable host. Once one becomes available, if this is the first time you have logged into a particular node you may be asked to accept the host key. Enter `yes` to accept this host key and then you will be presented with a terminal session. Your job will be subject to the same limits as a batch job and if you expect your session to need to be interupted then you should start screen or tmux on rescomp before using qlogin.
srun -p gpu_short --gres gpu:quadro-rtx8000:1 --pty bash
The RTX8000 nodes are deployed in pairs, so if your software is multi-GPU aware you can request two GPUs to double your available GPU compute power and GPU memory. You can do this by increasing the number of GPUs requested with:
srun -p gpu_short --gres gpu:quadro-rtx8000:2 --pty bash
The pairs of cards share a high-speed interconnect, so although slower than local GPU memory, the performance of this distant GPU memory is significantly higher than a traditional multi-GPU setup.
How to request a multi-threaded slot and how to ensure your software only uses the CPU cores it has been allocated
Running multi-threaded programs can cause significant problems with cluster scheduling software if the clustering software is not made aware of the multiple threads (your job is allocated one slot but actually consumes many more, often ALL the CPUs, overloading the machine).
We support the running of shared memory multi-threaded software only (e.g. OpenMP, multi-threaded MKL, OpenBLAS etc) and we attempt to limit the threads used automatically.
Where software would normally use all available cores you should invesigate how to limit these threads (see the MATLAB guide for example).
To submit an OpenMP job, use the -s (or --parallelenv) option to fsl_sub. For example:
fsl_sub -s 2 <command or script>
2 being the number of threads you wish to allocate to your jobs.
The task running on the queue will be able to determine how many slots it has by querying the environment variable pointed to by FSLSUB_NSLOTS. For example in BASH the number of slots is equal to ${!FSLSUB_NSLOTS}.
In Python you would be able to get this figure with the following code:
import os slots = os.environ[os.environ['FSLSUB_NSLOTS']]
To be able to provide these threads the cluster software needs to reserve slots on compute nodes, this may lead to significant wait times whilst sufficient slots become available on a single device.
How to submit non-interactive MATLAB scripts to the queues
See our MATLAB job submission page.
Other potentially useful submission options or techniques
Capturing job submission information
fsl_sub can store the commands used to submit the job if you provide the option --keep_jobscript. When provided, post submission you will find a file in the current folder (assuming you have write permissions there) a script called wrapper-<jobid>.sh. This exact submission may be repeated by using:
fsl_sub -F wrapper-<jobid>.sh
The script contents is described below:
| #!/bin/bash | Run the script in BASH |
| #SBATCH OPTION | SLURM options |
| #SBATCH OPTION | |
| module load <module name> | Load a Shell Module |
| # Built by fsl_sub v.2.3.0 and fsl_sub_plugin_sge v.1.3.0 | Version of fsl_sub and plugin that submitted the job |
| # Command line: <command line> | Command line that invoked fsl_sub |
| # Submission time (H:M:S DD/MM/YYYY) <date/time> | Date and time that the job was submitted |
| <command> |
| <command> |
PASSING ENVIRONMENT VARIABLES TO QUEUED JOBS
It is not possible to inherit all the environment variables from the shell that submits a job, so fsl_sub allows you to specify environment variables that should be transferred to the job. This can also be useful if you are scheduling many similar tasks and need to specify a different value for an environment variable for each run, for example SUBJECTS_DIR which FreeSurfer uses to specify where your data sets reside. The --export option is used for this purpose.
SKIPPING COMMAND VALIDATION
By default fsl_sub will check the command given (or the commands in the lines in an array task file) can be found and are executable. If this causes issues, often because a particular program is only available on the compute nodes, not the submission host, then you can disable this check with -n (--novalidation).
Requesting a specific resource
Some resources may have a limited quantity available for use, e.g. software licenses or RAM. fsl_sub has the ability to request these resources from the cluster (the --coprocessor options do this to automatically to request the appropriate number of GPUs). The option -r (--resource) allows you to pass a resource string directly through to the Grid Engine software. If you need to do this you will be advised by the computing help team or software documentation the exact string to pass.
Environment variables that can be set to control fsl_sub submitted tasks
See our fsl_sub environment variables page.
How to change fsl_sub's configuration for all jobs you run
See our fsl_sub configuration page.