Using Midway¶
Now that you are able to log in to Midway (Connecting to RCC Resources), upload and access your files on the cluster (Data Storage and Data Transfer), and load software tools using the module system (Software), you are ready for the next step: scheduling access to the RCC compute cluster to perform computations. This is the topic of this section of the RCC User Guide.
Overview¶
The Midway compute cluster is a shared resource used by the entire University of Chicago community. Sharing computational resources creates unique challenges:
- Jobs must be scheduled in a way that is fair to all users.
- Consumption of resources needs to be recorded.
- Access to resources needs to be controlled.
The Midway compute cluster uses a scheduler to manage requests for access to compute resources. These requests are called jobs. In particular, we use the Slurm resource manager to schedule jobs as well as interactive access to compute nodes.
Here, we give the essential information you need to know to start computing on Midway. For more detailed information on running specialized compute jobs, see Running Jobs on Midway.
Service Units and Allocations¶
We measure computing resources using “service units” (SUs). One
service unit is defined as use of 1 compute core for 1 hour. SUs are
the “currency” used to pay for computing resources at RCC. When you
submit a job that requests the use of N cores for M hours, N
* M SUs will be deducted from your account.
An “allocation” is a quantity of computing time (SUs) and storage resources that are granted to a group of users, usually a lab managed by a principal investigator (PI). Without an allocation, you cannot schedule and run jobs on the RCC compute cluster. For more information about SU allocations, see RCC Allocations.
Note
For all jobs on Midway2 nodes, multiply all requests by 1.5 to
determine the number of SUs required; that is, a job on Midway2
that requests the use of N cores for M hours will use
1.5 * N * M SUs.
Checking your account balance¶
The accounts tool can be used to check account balances. After logging into Midway, simply type:
$ accounts balance
If you are a member of multiple groups, this will display the allocations and usage for all your groups. The accounts balance command has a number of options for summarizing allocation usage. For information on these options, type
$ accounts balance --help
To see an overall summary of your usage, simply enter:
$ accounts usage
You can also get a more detailed breakdown of your usage by job using
the --byjob option:
$ accounts usage --byjob
For more commands available in the accounts tool, type
$ accounts --help
Types of Compute Nodes¶
The Midway compute cluster is made up of compute nodes with a variety architectures and configurations. A partition is a collection of compute nodes that all have the same, or similar, architecture and configuration. Currently, Midway has the following partitions:
| Cluster | Partition | Compute cores (CPUs) | Memory | Other configuration details |
|---|---|---|---|---|
| midway1 | westmere | 12 x Intel X5675 3.07 GHz | 24 GB | |
| sandyb | 16 x Intel E5-2670 2.6GHz | 32 GB | ||
| bigmem | 16 x Intel E5-2670 2.6GHz | 256 GB | ||
| 32 x Intel E7-8837 2.67GHz | 1 TB | |||
| gpu | 16 x Intel E5-2670 2.6GHz | 32 GB | 2 x Nvidia M2090 or K20 GPU | |
| 20 x Intel E5-2680v2 2.8GHz | 64 GB | 2 x Nvidia K40 GPU | ||
| mic | 16 x Intel E5-2670 2.6GHz | 32 GB | 2 x Intel Xeon Phi 5100 coprocessor | |
| amd | 64 x AMD Opteron 6386 SE | 256 GB | ||
| ivyb | 20 x Intel E5-2680v2 2.8GHz | 64 GB | ||
| midway2 | broadwl | 28 x Intel E5-2680v4 2.4GHz | 64 GB | |
| bigmem2 | 28 x Intel E5-2680v4 @ 2.4 GHz | 512 GB | ||
| gpu2 | 28 x Intel E5-2680v4 @ 2.4 GHz | 64 GB | 4 x Nvidia K80 GPU |
You can also retrieve a summary of the partitions on Midway using the sinfo command:
$ sinfo -s
In the sinfo summary, the right-most number in the “NODES” column gives the total number of nodes in each partition. This summary also lists partitions that are reserved for use by certain labs.
Interactive Jobs¶
After submitting an “interactive job” on Midway, the Slurm job scheduler will connect you to a compute node, and will load up an interactive shell environment for you to use on that compute node. This interactive session will persist until you disconnect from the compute node, or until you reach the maximum requested time.
sinteractive¶
The command sinteractive is the recommended Slurm command for
requesting an interactive session. As soon as the requested resources
become available, sinteractive will do the following:
- Log in to the node.
- Change into the directory you were working in.
- Set up X11 forwarding for displaying graphics.
- Transfer your current shell environment, including any modules you have previously loaded.
To get started (with the default interactive settings), simply enter
sinteractive in the command line:
$ sinteractive
By default, an interactive session times out after 2 hours. If you
would like more than 2 hours, be sure to include a --time=HH:MM:SS
flag to specify the necessary amount of time. For example, to request
an interactive session for 6 hours, run the following command:
$ sinteractive --time=06:00:00
There are many additional options for the sinteractive command,
including options to select the number of nodes, the number of cores
per node, the amount of memory, and so on. For example, to request
exclusive use of two compute nodes equipped with the Infiniband
interconnect on the Midway1 sandyb partition for 8 hours, enter
the following:
$ sinteractive --exclusive --partition=sandyb --nodes=2 --constraint=ib --time=08:00:00
For more details about these and other useful options, read below
about the sbatch command, and see Running Jobs on Midway. Note that
all options available in the sbatch command are also available for
the sinteractive command.
Note
It is possible to request an interactive session on a Midway2
partition (e.g., broadwl) from a Midway1 login node
(midway1.rcc.uchicago.edu), and vice versa. However, since
access to files and software modules is different on Midway1 and
Midway2, it is Best Practice to request Midway1 partitions
from Midway1 login nodes, and Midway2 partitions from Midway2
login nodes.
Batch Jobs¶
The sbatch command is the command most commonly used by RCC users
to request computing resources on the Midway cluster. Rather than
specify all the options in the command line, users typically write an
“sbatch script” that contains all the commands and parameters
neccessary to run the program on the cluster.
In an sbatch script, all Slurm parameters are declared with #SBATCH,
followed by additional definitions.
Here is an example of an sbatch script:
#!/bin/bash
#SBATCH --job-name=example_sbatch
#SBATCH --output=example_sbatch.out
#SBATCH --error=example_sbatch.err
#SBATCH --time=00:05:00
#SBATCH --partition=broadwl
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=16
#SBATCH --mem-per-cpu=2000
module load openmpi
mpirun ./hello-mpi
Here is an explanation of what each of these options does:
In this example, we have requested 4 compute nodes with 16 CPUs
each. Therefore, we have requested a total of 64 CPUs for running our
program. The last two lines of the script load the openMPI module and
launch the MPI-based executable that we have called hello-mpi (see
MPI Jobs).
Note
If the --partition option is not specified, the request will
automatically be directed to the sandyb partition on
Midway1. Since this is the default partition, it can be more
heavily requested than other partitions. Therefore, it is often
beneficial to override the default and select a different
partition.
Continuing the example above, suppose that this script is saved in the
current directory into a file called example.sbatch. This script
is submitted to the cluster using the following command:
$ sbatch ./example.sbatch
Many other options are available for submitting jobs using the sbatch command. For more specialized computational needs, see Running Jobs on Midway. Additionally, for a complete list of the available options, see the Official SBATCH Documentation.
Note
When writing your sbatch script, make sure that it accesses only
files that are available on the partition you are using. For
example, if you plan to use the Midway2 Broadwell nodes
(--partition=broadwl), then any files stored on
/scratch/midway will be unavailable to your sbatch job; use
/scratch/midway2 instead. Also note that none of the Midway2
partitions have local file systems, so you should not attempt to
save files to /local/scratch. For more details on which files
are accesible from Midway1 and Midway2, see Data Storage.
Managing Jobs¶
The Slurm job scheduler provides several command-line tools for checking on the status of your jobs and for managing them. For a complete list of Slurm commands, see the Slurm man pages. Here are a few commands that you may find particularly useful:
- squeue: find out the status of jobs submitted by you and other users.
- sacct: retrieve job history and statistics about past jobs.
- scancel: cancel jobs you have submitted.
In the next couple sections we explain how to use squeue to find out the status of your submitted jobs, and scancel to cancel jobs in the queue.
Checking your jobs¶
Use the squeue command to check on the status of your jobs, and
other jobs running on Midway. The simplest invocation lists all jobs
that are currently running or waiting in the job queue (“pending”),
along with details about each job such as the job id and the number of
nodes requested:
$ squeue
Any job with 0:00 under the TIME column is a job that is still
waiting in the queue.
To view only the jobs that you have submitted, use the --user flag
$ squeue --user=<username>
This command has many other useful options for querying the status of
the queue and getting information about individual jobs. For example,
to get information about all jobs that are waiting to run on the
ivyb partition, enter:
$ squeue --state=PENDING --partition=ivyb
Alternatively, to get information about all your jobs that are
running on the ivyb partition, type:
$ squeue --state=RUNNING --partition=ivyb
The last column of the output tells us which nodes are allocated for
each job. For example, if it shows midway572 for one of the jobs
under your name, you may type ssh midway572 to log in to that
compute node and inspect the progress of your computation locally.
For more information, consult the command-line help by typing squeue
--help, or visit the official online documentation.
Canceling your jobs¶
To cancel a job you have submitted, use the scancel command. This
requires you to specify the id of the job you wish to cancel. For
example, to cancel a job with id 8885128, do the following:
$ scancel 8885128
If you are unsure what is the id of the job you would like to cancel,
see the JOBID column from running squeue --user=<username>.
To cancel all jobs you have submitted that are either running or waiting in the queue, enter the following:
$ scancel --user=<username>
Job Limits¶
To distribute computational resources fairly to all Midway users, the RCC sets limits on the amount of computing resources that may be requested by a single user at any given time.
The maximum run-time for an individual job is 36 hours. This applies to all batch and interactive jobs submitted to nodes in the general-access partitions (sandyb, westmere, bigmem, broadwl, and so on). Groups participating in the cluster parternership program may customize resources limits for their partitions.
Additional information on limits, such as the maximum number of CPUs
that can be requested by a user at any one time, or the number of jobs
that can be submitted concurrently on a given partition, can be found
by entering the command accounts qos on any login or compute node
on midway. Observe that these limits are often different depending on
the partition.
Usage limits may change, so accounts qos will always give the most
up-to-date information.
If your research requires a temporary exception to a particular limit, you may apply for a special allocation. Special allocations are evaluated on an individual basis and may or may not be granted.