Storage

Storage Area	Path	Description
Home	`/home`	Automated creation on first login. Mirrored for redundancy.
Work	`/work`	Large data or shared project files. Admins create the directory on request.
Temp	`$TMPDIR`	Resides in RAM (tmpfs); data counts toward job memory limit. Not shared between jobs. Read about the details below.
Scratch	`/scratch`	Fast LUSTRE network storage shared across nodes. Everyone can write into `/scratch/tmp` Recommended directory to use is `/scratch/tmp/$USER`
Local Scratch	`/scratch_local`	Local disk on specific nodes; requires manual cleanup. Read about the details below.

Space and quotas on the cluster

Available space, volumes and quotas are described below.

Default permissions

Your home directory /home/<CASlogin> can be accessed only by you.
By default, newly created files and directories in any storage area can be read (and accessed) by everyone and written to only by you. This includes user directories created on /work.
To make your directory on /work/<CASlogin> accessible only to you type chmod 700 /work/<CASlogin> on the command line.
Detailed instructions for changing the permissions are explained below.

Do not do this

Run a large number of jobs at the same time which access a large volume of data from the same storage volume. This overloads the filesystem data servers and slows down the response time for everyone on the cluster. /scratch is more resilient to this than the other two.
Using the flush function, opening and closing files, or otherwise force synchronisation many times per second. This overloads the filesystem servers.
Create a large number of directories (millions) in a single directory. The filesystem is not optimized for a large number of small files.
Move large files between the remotely connected clusters as part of your job.

If you need to do one of these things and don't know how to do it efficiently, please do not hesitate to contact the cluster support.

LUSTRE File System

Chimera utilizes the LUSTRE parallel file system for high-performance data access. The home volume of central storage in Troja is visible from all clusters as /troja/home. The intention is to enable transfer of small data or scripts (e.g. input data) between the clusters while allowing to launch jobs at any of the remotely connected clusters. It is not intended for transfer of large files.

Working with permissions for files and directories

Permissions for a file or a directory (folder) are characterized by the level of access for the owner,group and other type of user. They're abbreviated by the letters u, g, o. Owner is the owner of the file/directory, group is group of the file (usually the group of the owner with only the owner in it) and other is everyone else on the cluster.

There are three types of actions you can perform: read, write, execute. They're abbreviated by the letters r, w, x. Each type of action (permission) is set separately for each user type so there are in total 9 characters (3x3) characterizing the permission level for each file or directory.

Checking the details of your files and directories

To see the permissions for a file path/filename use the command ls -l path/filename.
To see the permissions for a directory path/to/directory add the option d and use ls -ld path/to/directory.
To check the total size of the directory use du -h path/to/directory.

Example

We can look at the permissions for "bin" in the current directory:

$ ls -ld bin
drwxr-xr-x.  2 eliaj2am eliaj2am       4096 May 26  2022  bin

First, we focus on the string: drwxr-xr-x.

The first symbol (here d) represents what type of file it is: - for a regular file, d for directories, l for symlinks and more. In this example "bin" is a directory.
The following 9 characters (here rwxr-xr-x) specify the permissions for the user, group and other types of users, in this order. In this example the user's permissions are rwx, the group's permissions are r-x and the other's permissions are r-x.
If the corresponding action is not allowed the letter is replaced with -. In this example the "group" and "other" types of users don't have the write permissions for the directory but they can see its contents (to open the directory the x permission must be set).

The string 4096 May 26 2022 shows the size of "bin" and the date of the last change. The size 4096 bytes only shows the size of the record of the "existence" of the directory in the filesystem (it is the same for all directores), not the total size of its contents (that can be obtained by du -h bin).

Changing the permissions

You can use the chmod command to change the permissions for directories or files.
To change the permissions for a directory (or a file) so no one else except you can access it use chmod o-rwx directory/or/file. This will remove [r]ead,[w]rite and e[x]ecute permissions from [o]ther types of users. Similarly, to restore the permissions to the previous state, use + in the chmod command: chmod o+rwx directory/or/file.
Execute permission for a directory are needed to cd to that directory.

Alternatively, you can set the permissions using a bitmask - usually 3 numbers (for owner, group, other), each with an integer value between 0 and 7. You can get the number by summing the permissions value for read 4, write 2 and execute 1 as in the following table:

Permissions	---	--x	-w-	-wx	r--	r-x	rw-	rwx
Value	0	1	2	3	4	5	6	7

For example, to change the permissions for a directory to be accessible only for the owner use chmod 700 my_file_or_folder from the command line. Note that using this method you always have to specify the permissions for all types of users.

A fine-grained control over the permissions, down to the list of specific users, is available through ACL (Access-control list).

Storage spaces, quotas, network speed

Volume	Purpose	Default quota (limit)	Total storage space	Storage type	Network connection speed(*)	Notes
`/home` or `/troja/home`	Source codes, binaries, scripts, essential/small calculation results	10(15) GB 0.5 (1) M inodes(**) Request more if needed	73T	HDD	Up to 200 Gb/s	Backup
`/work`	Production runs, large input/output files	40 (50) TB Request to gain access	463T	HDD	Up to 200 Gb/s	No backup
`/scratch`	Temporary files for the duration of a running calculation in `tmp` directory. Clean-up after use!	Unlimited	121T	NVMe (SSD)	Up to 200 Gb/s	Content of /scratch/tmp can be deleted by admin at any time
`/archive`	Long-term and large data storage	Unlimited Request to gain access	419T	HDD	Up to 10 Gb/s	No backup

(*) depends on the compute node you are accessing volume from
(**) roughly equal to number of files and directories

Paths

The home directory is now automatically created during the first login. The path to the home directory is /home/<CASlogin> or $HOME. You can ask the administrators to create a personal directory for big data or a directory for shared project files on /work volume. The /home and /work volumes are on the local disk array for each cluster.

To facilitate data sharing between clusters in the metacenter, the /home and /work directories are shared with other clusters using the tunneled NFS protocol and are available on fixed paths according to the cluster location, e.g. the /home directory of the Chimera cluster located in Troja has the path /troja/home on all clusters (including the Chimera cluster).

Use these shared directories with caution, as they could overwhelm the network connection for that location. Copy your frequently used data to the local cluster directories (/home or /work) where the computation will take place.

Temporary directories on chimera cluster

There are multiple choices for temporary files (typically needed for the duration of your job) for you jobs: - $TMPDIR - It is set up automatically by slurm on tmpfs (in RAM). Files there are cleaned up after job is finished. Data there are counted to the memory limit of the job. - You can use this for setup of your job - i.e. copy / compile binaries and run your job there and copy results to persistent place after finishing - this automatically cleans your environment and is easy solution for cleaning preempted jobs that should be restarted from the start - Fast temporary space if you need save small amount of data during running task

/scratch - This is a fast network-attached storage shared across all nodes. Only nodes with Infiniband take full advantage of the speed. If your temporary files are large and do not fit memory ($TMPDIR) alongside your running task, please create your own directory in /scratch/tmp and cleanup after yourself.
- You should use this, if your temporary data are large - i.e. you would have to request too much memory or it will not fit into memory with your task.
- You can use it as a temporary storage between multiple jobs, but if you want to use this for longer term storage, plese contact administrator.
/scratch_local - This option is not recommended. Few nodes have also local scratch. It is on nodes with local_scratch feature. You can also request these by specifying minimal amount of space for local scratch using sbatch option --tmp=<size>[units]. It is mounted to /scratch_local - The directory always exist, but is mounted only on nodes with local_scratch feature. You have to clean up after yourself and be aware that slurm does not check how much of the scratch is already in use, only maximum capacity specified in configuration so the amount of space requested using --tmp option is not guaranteed.
- You should not use these, unless you know what you are doing. Only some older nodes have local scratch and its usage have been mostly moved to /scratch.