Data Storage¶
RCC provides a high-performance GPFS shared file system which is used for users’ home directories, shared project spaces, and high-throughput scratch space. Most compute nodes also have local disk storage that can be used for temporary scratch space if necessary.
Quotas¶
The amount of data that can be stored in home directories, project directories, and shared scratch directories is controlled by quota. RCC enforces hard and soft limits on quotas. A soft quota can be exceeded for a short period of time called a grace period. The hard quota cannot be exceeded under any circumstances.
Additional storage is available through the Cluster Partnership Program, a Research I Allocation, Research II Allocation or, in certain circumstances, a Special Allocation.
Checking available space¶
To check your current quotas use the quota command. Typical output may look like this
Descriptions of the fields:
-
Filesystem This is the file system or file set where this quota is valid.
-
type This is the type of quota. This can be USR for a user quota, GRP for a group quota, or FILESET for a file set quota. File set quotas can be considered a directory quota. USR and GRP quotas can exist within a FILESET quota to further limit a user or group quota inside a file set.
-
blocks This is the amount of disk space consumed in the specified location.
-
quota This is the soft quota associated with the specified location. It is possible for usage to exceed the soft quota for the grace period or up to the hard limit.
-
limit This is the hard quota associated with the specified location. When your usage exceeds this limit, you will be unable to write to that filesystem.
-
grace This is the grace period, or the amount of time remaining that the soft quota can be exceeded. The value none means that the quota is not exceeded. After a soft quota has been exceeded for longer than the grace period, it will no longer be possible to create new files.
-
files This is the number of files currently counted in the quota.
-
quota (files) This is the soft quota associated with the number of files in the specified location. It is possible for usage to exceed the soft quota for the grace period or up to the hard limit.
-
limit (files) This is the hard quota associated the number of files in the specified location. When your usage exceeds this limit, you will be unable to write to that filesystem. You may not have more than 20M files in any file system.
-
grace (files) This is the grace period, or the amount of time remaining that the soft quota can be exceeded. The value none means that the quota is not exceeded.
Persistent Storage¶
Persistent storage areas are appropriate for long term storage. The home directory has both file system Snapshots and tape_backup for data protection. The two locations for persistent storage are the home and project directories.
Home Directories¶
Every RCC user has a home directory located at /home/<CNetID>. This directory
is accessible from all RCC compute systems and is generally used for storing
frequently used items such as source code, binaries, and scripts. By default, a
home directory is only accessible by its owner (mode 0700) and is suitable
for storing files which do not need to be shared with others.
Project Directories¶
All RCC PI Groups are allocated a Project Directory located at /project/<CNetID> or /project2/<CNetID> where <CNetID> is the CNetID of your RCC PI account holder. These directories are accessible by all members of the PI Group and are generally used for storing
files which need to be shared by members of the group. Additional storage in project directories is available through the Cluster Partnership Program,
a Research I Allocation or Research II Allocation or, in certain circumstances, a Special Allocation.
The default permissions for files
and directories created in a project directory allow group read/write with the
group sticky bit set (mode 2770). The group ownership is set to the PI group.
Scratch Space¶
Local Scratch Space¶
Most Midway compute nodes have a local hard disk available for scratch space for
situations where that would be more appropriate. It is available in
/scratch/local. Users should create a sub-directory in this location and use that directory for
scratch space. All files in /scratch/local are deleted when the node is
rebooted.
File System Permissions¶
Let’s summarize the default file system permissions:
| Directory | Permissions |
|---|---|
$HOME |
0700 – Accessible only to the owner |
$HOME/scratch-midway |
0700 – Accessible only to the owner |
/project/project |
2770 – Read/write for the project group |
/project2/project |
2770 – Read/write for the project group |
The default umask is 002. When new files or directories are created, the umask
influences the default permissions of those files and directories. With the
umask set to 002 all files and directories will be group readable and
writable by default. In your home directory, the group ownership will be set
to your personal group, which is the same as your CNetID, so you will still
be the only user that can access your files and directories. In the project
directories, the group sticky bit causes the group ownership to be the same
as the directory. This means files created in a project directory will be
readable and writable by the project group, which is typically what is wanted
in those directories.
Here is an example of what this means in practice:
$ ls -ld $HOME /project/rcc
drwx------ 108 wettstein wettstein 32768 2013-01-15 10:51 /home/wettstein
drwxrws--- 24 root rcc-staff 32768 2013-01-15 10:48 /project/rcc
$ touch $HOME/newfile /project/rcc/newfile
$ ls -l /project/rcc/newfile $HOME/newfile
-rw-rw-r-- 1 wettstein wettstein 0 2013-01-15 10:48 /home/wettstein/newfile
-rw-rw-r-- 1 wettstein rcc-staff 0 2013-01-15 10:48 /project/rcc/newfile
Both files are readable and writable by the group owner due to the default
umask, but the group owner differs due to the sticky bit being set on
/project/rcc.
Note
This applies only to newly created files and directories. If files or directories are copied from elsewhere, the ownership and permission may not work like this. Contact RCC help if you need assistance with setting filesystem permissions.
Data Recovery and Backups¶
Snapshots¶
Automated snapshots of home directories are available in case of accidental file deletion or other problems. Currently snapshots are available for these time periods:
- 4 hourly snapshots
- 7 daily snapshots
- 4 weekly snapshots
Snapshots are in these directories:
/snapshots/home/SNAPSHOT/home/CNetID– Home snapshots
The {SNAPSHOT} refers to the time of the backup, e.g.
daily-2012-10-04.06h15 or hourly-2012-10-09.11h00. To view the available snapshots, use the command ls /snapshots/home
To restore a file from a snapshot, simply copy the file to where you want it with either cp or rsync.