Storage

Storage Area Path Description
Home /home/<CASlogin> Automated creation on first login. Mirrored for redundancy.
Work /work Large data or shared project files (request from admin).
Chimera Temp $TMPDIR Resides in RAM (tmpfs); data counts toward job memory limit.
Scratch /scratch Fast LUSTRE network storage shared across nodes.
Local Scratch /scratch_local Local disk on specific nodes; requires manual cleanup.

LUSTRE File System: Chimera utilizes the LUSTRE parallel file system for high-performance data access. The central storage in Troja is visible from all clusters as /troja/home. The intention is to enable transfer of small data or scripts (e.g. input data) between the clusters while allowing to launch jobs at any of the remotely connected clusters. It is not intended for transfer of large files.

Storage spaces, quotas, network speed

Volume Purpose Default quota (limit) Total storage space Storage type Network connection speed(*) Notes
/home or /troja/home Source codes, binaries, scripts, essential/small calculation results 10(15) GB
0.5 (1) M inodes(**)
Request more if needed		 73T HDD Up to 200 Gb/s Backup
/work Production runs, large input/output files 40 (50) TB
Request to gain access		 463T HDD Up to 200 Gb/s
/scratch Temporary files for the duration of a running calculation in tmp directory. Clean-up after use! Unlimited 121T NVMe (SSD) Up to 200 Gb/s Content of /scratch/tmp can be deleted by admin at any time
/archive Long-term and large data storage Unlimited
Request to gain access		 419T HDD Up to 10 Gb/s
  • (*) depends on the compute node you are accessing volume from
  • (**) roughly equal to number of files and directories

Paths

The home directory is now automatically created during the first login. The path to the home directory is /home/<CASlogin>. You can ask the administrators to create a personal directory for big data or a directory for shared project files in /work directory. The /home and /work directories are on the local disk array for each cluster.

To facilitate data sharing between clusters in the metacenter, the /home and /work directories are shared with other clusters using the tunneled NFS protocol and are available on fixed paths according to the cluster location, e.g. the /home directory of the Chimera cluster located in Troja has the path /troja/home on all clusters (including the Chimera cluster).

Use these shared directories with caution, as they could overwhelm the network connection for that location.

Copy your frequently used data to the local cluster directories (/home or /work) where the computation will take place.

Temporary directories on chimera cluster

There are multiple choices for temporary files (typically needed for the duration of your job) for you jobs: - $TMPDIR - It is set up automatically by slurm on tmpfs (in RAM). Files there are cleaned up after job is finished. Data there are counted to the memory limit of the job. - You can use this for setup of your job - i.e. copy / compile binaries and run your job there and copy results to persistent place after finishing - this automatically cleans your environment and is easy solution for cleaning preempted jobs that should be restarted from the start - Fast temporary space if you need save small amount of data during running task

  • /scratch - This is a fast network-attached storage shared across all nodes. Only nodes with Infiniband take full advantage of the speed. If your temporary files are large and do not fit memory ($TMPDIR) alongside your running task, please create your own directory in /scratch/tmp and cleanup after yourself.

    • You should use this, if your temporary data are large - i.e. you would have to request too much memory or it will not fit into memory with your task.
    • You can use it as a temporary storage between multiple jobs, but if you want to use this for longer term storage, plese contact administrator.

  • /scratch_local - This option is not recommended. Few nodes have also local scratch. It is on nodes with local_scratch feature. You can also request these by specifying minimal amount of space for local scratch using sbatch option --tmp=<size>[units]. It is mounted to /scratch_local - The directory always exist, but is mounted only on nodes with local_scratch feature. You have to clean up after yourself and be aware that slurm does not check how much of the scratch is already in use, only maximum capacity specified in configuration so the amount of space requested using --tmp option is not guaranteed.

    • You should not use these, unless you know what you are doing. Only some older nodes have local scratch and its usage have been mostly moved to /scratch.