.. _tut-bigmem: 

.. index::
      pair: Tutorial; Index

.. _bigmem:

=================
Large Memory Jobs
=================

In the case that you require more than 64 GB of memory for your job, a node in the ``bigmem`` or ``bigmem2`` partitions can be 
used. These two partitions include eight nodes (2 nodes on Midway1 and 6 nodes on Midway2) with the following specifications:

+-------------------+---------------------------+--------------------+-----------------------+
| Partition Name    | CPU Cores per Node        | Number of Nodes    | Memory per Node       |
+===================+===========================+====================+=======================+
| bigmem            | 16x E5-2670 @ 2.6 GHz     | 1                  | 256 GB                |
+-------------------+---------------------------+--------------------+-----------------------+
| bigmem            | 32x E7-8837 @ 2.66 GHz    | 1                  | 1024 GB               |
+-------------------+---------------------------+--------------------+-----------------------+
| bigmem2           | 28x E5-2680v4 @ 2.4 GHz   | 6                  | 512 GB                |
+-------------------+---------------------------+--------------------+-----------------------+


In the case where you are not sure how much memory your job was using, you can learn this (once your job finished) by using the ``maxrss`` and ``maxvmsize`` format options with ``sacct``, like so, with ``<jobid>`` being the ID number of your job:

.. code:: bash

	sacct -j <jobid> --format=jobid,jobname,partition,account,alloccpus,state,cputime,maxrss,maxvmsize

The last two columns of this output will be the maximum RAM and virtual memory size of all tasks in the job.

Running Large Memory Jobs on Midway
-----------------------------------

When submitting jobs to the big memory nodes, you must include one of the following ``#SBATCH`` options in your job submission script:

.. code:: bash

	# to run on Midway1
	#SBATCH --partition=bigmem

Or

.. code:: bash

	# to run on Midway2
	#SBATCH --partition=bigmem2

The above options will ensure that your job runs on a node in either the ``bigmem`` or ``bigmem2`` partition. 

.. note::
   If you want to use the ``bigmem`` partition, please make sure that you have compiled your code on one of the Midway1 login nodes and if you want to use the ``bigmem2`` parition, please make sure that you have compiled your code on one of the Midway2 login nodes. Not doing so may result in getting your job crashed.


When running your job in the ``bigmem`` partition, you may decide to run it on the 256 GB node. In this case, you must also include the following ``#SBATCH`` option:

.. code:: bash

	#SBATCH --constraint=256G

Similarly, if you need to ensure your job runs on the 1 TB node, you must include the following ``#SBATCH`` option:

.. code:: bash

	#SBATCH --constraint=1024G

If you have used ``maxrss`` from above, or if you simply have a good idea of how much memory your job will need, you can use the options :option:`--cpus-per-task` and :option:`--mem-per-cpu` to request the amount of memory you need. However, if your job requires multiple CPUs, you will need to divide the total amount of memory you need by the number of CPUs you are requesting. For example, if you want to request 8 CPU cores and 128 GB of memory (128 GB / 8 cores = 16 GB/core), you would use the following ``#SBATCH`` options:

.. code:: bash

	#SBATCH --ntasks=1
	#SBATCH --cpus-per-task=8
	#SBATCH --mem-per-cpu=16000

Running Large Memory Jobs Interactively
---------------------------------------

When using the ``bigmem`` or ``bigmem2`` partitions interactively, the same ``#SBATCH`` options from above can be used. For example, to access the 256 GB node while requesting 1 CPU core and 128 GB of memory, you would use the following command:

.. code:: bash

	sinteractive --partition=bigmem --constraint=256G --ntasks=1 --cpus-per-task=1 --mem-per-cpu=128000

Similarly, to access a node from the ``bigmem2`` partition while requesting 8 CPU cores and 128 GB of memory, you would use the following command:

.. code:: bash

	sinteractive --partition=bigmem2 --ntasks=1 --cpus-per-task=8 --mem-per-cpu=16000