.. index::
      pair: Tutorial; Index

.. _GPU_computing:

=====================
GPU Computing Jobs
=====================

If you have an application capable of utilizing GPUs, you could submit a job to RCC's ``gpu`` or ``gpu2`` partitions. The ``gpu`` partition is on Midway1 and ``gpu2`` is on Midway2. The two partitions include 15 GPU nodes in total (9 nodes on Midway1 and 6 nodes on Midway2) with the following specification:

+---------------------+---------------------------+--------------------+-----------------------+
| Partition Name      | CPU Cores/GPUs per Node   | Number of Nodes    | Memory per Node       |
+=====================+===========================+====================+=======================+
| gpu                 | 16x E5-2670 @ 2.6 GHz     | 3                  | 32 GB                 |
|                     | 2x Nvidia M2090           |                    |                       |
+---------------------+---------------------------+--------------------+-----------------------+
| gpu                 | 16x E5-2670 @ 2.6 GHz     | 2                  | 32 GB                 |
|                     | 2x Nvidia K20m            |                    |                       |
+---------------------+---------------------------+--------------------+-----------------------+
| gpu                 | 20x E5-2680v2 @ 2.8 GHz   | 4                  | 64 GB                 |
|                     | 2x Nvidia K40m            |                    |                       |
+---------------------+---------------------------+--------------------+-----------------------+
| gpu2                | 28x E5-2680v4 @ 2.4 GHz   | 6                  | 128 GB                |
|                     | 4x Nvidia K80             |                    |                       |
+---------------------+---------------------------+--------------------+-----------------------+


For information on compiling CUDA and OpenACC code on midway, see :ref:`GPU_compiling`. 
  
Running GPU code on Midway
--------------------------

When submitting jobs to the GPU nodes, you must use one of the following ``#SBATCH`` options:

.. code:: bash

	# to run on Midway1 GPU nodes
	#SBATCH --partition=gpu
	#SBATCH --gres=gpu:<N>

OR

.. code:: bash

	# to run on Midway2 GPU nodes
	#SBATCH --partition=gpu2
	#SBATCH --gres=gpu:<N>


The flag :option:`--gres=gpu:N` is used to request :option:`N` GPU devices on each of the nodes on which your job will run.  Valid numbers for N in the ``gpu`` partition is either 1 or 2 and in the ``gpu2`` partition is 1 to 4.  If you are requesting all GPUs in a node, we also suggest including the :option:`#SBATCH --exclusive` option in your submission script to prevent other jobs from being placed on that node.

Additionally, in the ``gpu`` partition, you can specify which type of GPU device your job runs on by including a :option:`--constraint` option in your job submission script.  To ensure your job runs on a M2090, K20, or K40 device, include one of the following lines in your submission script:

.. code:: bash

  #SBATCH --constraint=m2090
  or
  #SBATCH --constraint=k20m
  or
  #SBATCH --constraint=k40m

An example GPU-enabled job script for a CUDA program is given below :download:`gpu.sbatch`:

.. literalinclude:: gpu.sbatch
  :language: bash