.. index:: pair: RCC; User Guide .. _data-storage: =============================================================== Data Storage =============================================================== RCC provides a high-performance GPFS shared file system which is used for users' home directories, shared project spaces, and high-throughput scratch space. Most compute nodes also have local disk storage that can be used for temporary scratch space if necessary. .. contents:: :local: Quotas ========== The amount of data that can be stored in home directories, project directories, and shared scratch directories is controlled by quota. RCC enforces hard and soft limits on quotas. A soft quota can be exceeded for a short period of time called a grace period. The hard quota cannot be exceeded under any circumstances. Additional storage is available through the `Cluster Partnership Program`_, a `Research I Allocation`_, `Research II Allocation`_ or, in certain circumstances, a `Special Allocation`_. Checking available space ------------------------ To check your current quotas use the ``quota`` command. Typical output may look like this .. image:: img/quota.png :width: 75 % Descriptions of the fields: .. describe:: Filesystem This is the file system or file set where this quota is valid. .. describe:: type This is the type of quota. This can be *USR* for a user quota, *GRP* for a group quota, or *FILESET* for a file set quota. File set quotas can be considered a directory quota. *USR* and *GRP* quotas can exist within a *FILESET* quota to further limit a user or group quota inside a file set. .. describe:: blocks This is the amount of disk space consumed in the specified location. .. describe:: quota This is the *soft quota* associated with the specified location. It is possible for usage to exceed the soft quota for the grace period or up to the hard limit. .. describe:: limit This is the *hard quota* associated with the specified location. When your usage exceeds this limit, you will be unable to write to that filesystem. .. describe:: grace This is the grace period, or the amount of time remaining that the soft quota can be exceeded. The value *none* means that the quota is not exceeded. After a soft quota has been exceeded for longer than the grace period, it will no longer be possible to create new files. .. describe:: files This is the number of files currently counted in the quota. .. describe:: quota (files) This is the *soft quota* associated with the number of files in the specified location. It is possible for usage to exceed the soft quota for the grace period or up to the hard limit. .. describe:: limit (files) This is the *hard quota* associated the number of files in the specified location. When your usage exceeds this limit, you will be unable to write to that filesystem. You may not have more than 20M files in any file system. .. describe:: grace (files) This is the grace period, or the amount of time remaining that the soft quota can be exceeded. The value *none* means that the quota is not exceeded. Persistent Storage ==================== Persistent storage areas are appropriate for long term storage. The home directory has both file system :ref:`snapshots` and :ref:`tape_backup` for data protection. The two locations for persistent storage are the home and project directories. Home Directories ----------------- Every RCC user has a home directory located at :file:`/home/`. This directory is accessible from all RCC compute systems and is generally used for storing frequently used items such as source code, binaries, and scripts. By default, a home directory is only accessible by its owner (mode ``0700``) and is suitable for storing files which do not need to be shared with others. Project Directories --------------------- All RCC PI Groups are allocated a Project Directory located at :file:`/project/` or :file:`/project2/` where ** is the CNetID of your RCC PI account holder. These directories are accessible by all members of the PI Group and are generally used for storing files which need to be shared by members of the group. Additional storage in project directories is available through the `Cluster Partnership Program`_, a `Research I Allocation`_ or `Research II Allocation`_ or, in certain circumstances, a `Special Allocation`_. The default permissions for files and directories created in a project directory allow group read/write with the group sticky bit set (mode ``2770``). The group ownership is set to the PI group. Scratch Space =================== Shared Scratch Space ------------------------ High performance shared scratch space is available in :file:`$HOME/midway-scratch`. This is a symlink to :file:`/scratch/midway/{CNetID}`. This scratch space is intended to be used for reading or writing data required by jobs running on the cluster. **Scratch space is neither snapshotted nor backed up**. .. note:: It is the responsibility of the user to ensure any important data in scratch space is moved to persistent storage. Scratch space is meant to be used for temporary, short-term storage only. The default permissions for scratch space allow access only by its owner (mode ``0700``). The standard quota for the high performance scratch directory is 5 TB with a 100GB soft limit. The grace period that the soft limit may be exceeded is 30 days for shared scratch space. Local Scratch Space ------------------------ Most Midway compute nodes have a local hard disk available for scratch space for situations where that would be more appropriate. It is available in :file:`/scratch/local`. Users should create a sub-directory in this location and use that directory for scratch space. All files in :file:`/scratch/local` are deleted when the node is rebooted. File System Permissions ========================= Let's summarize the default file system permissions: +-------------------------------+--------------------------------------------------+ | Directory | Permissions | +===============================+==================================================+ | :file:`$HOME` | ``0700`` -- Accessible only to the owner | +-------------------------------+--------------------------------------------------+ | :file:`$HOME/scratch-midway` | ``0700`` -- Accessible only to the owner | +-------------------------------+--------------------------------------------------+ | :file:`/project/{project}` | ``2770`` -- Read/write for the project group | +-------------------------------+--------------------------------------------------+ | :file:`/project2/{project}` | ``2770`` -- Read/write for the project group | +-------------------------------+--------------------------------------------------+ The default umask is ``002``. When new files or directories are created, the umask influences the default permissions of those files and directories. With the umask set to ``002`` all files and directories will be group readable and writable by default. In your home directory, the group ownership will be set to your personal group, which is the same as your CNetID, so you will still be the only user that can access your files and directories. In the project directories, the group sticky bit causes the group ownership to be the same as the directory. This means files created in a project directory will be readable and writable by the project group, which is typically what is wanted in those directories. Here is an example of what this means in practice:: $ ls -ld $HOME /project/rcc drwx------ 108 wettstein wettstein 32768 2013-01-15 10:51 /home/wettstein drwxrws--- 24 root rcc-staff 32768 2013-01-15 10:48 /project/rcc $ touch $HOME/newfile /project/rcc/newfile $ ls -l /project/rcc/newfile $HOME/newfile -rw-rw-r-- 1 wettstein wettstein 0 2013-01-15 10:48 /home/wettstein/newfile -rw-rw-r-- 1 wettstein rcc-staff 0 2013-01-15 10:48 /project/rcc/newfile Both files are readable and writable by the group owner due to the default umask, but the group owner differs due to the sticky bit being set on :file:`/project/rcc`. .. note:: This applies only to newly created files and directories. If files or directories are copied from elsewhere, the ownership and permission may not work like this. Contact RCC help if you need assistance with setting filesystem permissions. .. _data_recovery: Data Recovery and Backups =========================== .. _snapshots: Snapshots --------- Automated snapshots of home directories are available in case of accidental file deletion or other problems. Currently snapshots are available for these time periods: * 4 hourly snapshots * 7 daily snapshots * 4 weekly snapshots Snapshots are in these directories: * :file:`/snapshots/home/{SNAPSHOT}/home/{CNetID}` -- Home snapshots The {SNAPSHOT} refers to the time of the backup, e.g. daily-2012-10-04.06h15 or hourly-2012-10-09.11h00. To view the available snapshots, use the command ``ls /snapshots/home`` To restore a file from a snapshot, simply copy the file to where you want it with either ``cp`` or ``rsync``.