Software & Environment

Fair Share

Chimera cluster employs the Fair Share feature of SLURM. This means that users who used the cluster less in last weeks have higher priority than users that used cluster more. For now, this only account for allocated CPUs. The half-life of the fairshare is now set to 14 days - after 14 day the job contribution to "Fair share" is halved. Overall priority calculation also depends on other factors - size of job, how long it is in queue and more. Details can be found here. For detailed parameter on cluster priority settings please check scontrol show config | grep Priority.

Details on preemption

Preemption is set up on partitions indicated by preemption set to REQUEUE or GANG,SUSPEND (ffa-preempt, ucjf, auuk). Job running in lower priority partitions with preemption set up can be preempted by jobs on partitions with higher priority (with Oversubs set to FORCE:1) if there are no free resources left for the new job.

  • If Preemption is set to REQUEUE Slurm sends signal (SIGCONT unless specified differently in sbatch,...) to low priority job and kills it one minute later (if it not ends itself). If job is killed this way it will be requeued.
  • If Preemption is set to GANG,SUSPEND the low priority job is suspended and resumed after the high priority job ends.
  • Details about preemtpion can be found here.

Default resources

Default resources for all partitions are now set to:

  • 1 node
  • 1 cpu
  • 1 GB RAM/CPU

Time is set between 30 minutes and one day depending on the partition, please check scontrol -o show part | cut -d' ' -f1,8 for details on front-end of cluster you want to run jobs at.

Monitoring Cluster State

  • Web browser: Real-time status of the cluster can be monitored here.
  • freenodes: A command-line tool on Chimera providing node and partition-oriented load statistics. It was developed for the use on cluster Snehurka - mathematical section of MFF UK. For details visit the help page.
  • smypartition: Displays your available accounts and partitions.
  • sview: A graphical tool for real-time load monitoring; requires SSH X11 forwarding (ssh -X). Sometimes this tool can consume a lot of memory on the headnode so please don't use it unless you know what you're doing.

Modules

Software is managed via LMod modules. * module avail: List available software. * module load <name>: Load software into the environment.

How to use MPI

  • Intel MPI: Load via module load oneapi/mpi. Use mpiicc or mpiifort for compilation. For details please see Intel documentaton.

  • OpenMPI:

    • openmpi-5.0.8/ib: Default for InfiniBand nodes (request with -C ib).
    • openmpi-5.0.8/tcp: Required if any allocated node lacks InfiniBand.

    • These modules will correctly setup srun options and environment variables for runtime. It should not matter which one you are using at compile time, unless the building of software is running MPI code. But the result should be independent on ib / tcp. There is also the package openmpi-5.0.8-base which sets up only paths to OpenMPI environment.

    • You can also use an older module openmpi-4.1.5. It is set up similarly: default for InfiniBand nodes (openmpi-4.1.5/ib), nodes without InfiniBand openmpi-4.1.5/tcp and openmpi-4.1.5-base.

Specialized Software

  • Intel Parallel Studio XE 2019
  • Intel oneAPI
  • Mathematica: Version 13.3; use module load Mathematica.
  • Molpro: Version 2012.1; requires a license token. Please contact one of K.Houfek, Z.Masin, J.Eliasek to obtain one. Once you have it the license token should be placed into ~/.molpro/token. Alternatively, run Molpro like this molpro molpro.inp -k "CONTENT_OF_THE_LICENSE_TOKEN" \.
  • UKRmol+: Atomic and molecular scattering codes.
  • DMTCP: Distributed MultiThreaded Checkpointing for job recovery.

Mathematica: details on how to use

  • only for MFF and CERGE
  • is available on chimera cluster by using module load Mathematica

  • for interactive GUI environment you should:

    • allocate resources you need - ie. salloc -n1; you can specify other things, like partition - the result sould look something like:

      $salloc -n1 -p ffa-preempt
salloc: Granted job allocation 9324110
salloc: Nodes ucjf-asusb2 are ready for job
$

    • leave this open (in this case we got node ucjf-asusb2)

    • login to cluster with X11 forwarding ssh -Y -l username hpc.troja.mff.cuni.cz
    • login to allocated node with X11 forwarding ssh -Y NODE (in our example the NODE is ucjf-asusb2)
    • run mathematica - module load Mathematica; mathematica

Advanced features and use cases

Checkpointing

Checkpointing is a technique in which the running program processes the signal sent to it by the scheduling system to perform certain actions, such as storing the program state to disk, before the program is killed by the scheduler and requeued, as is done in the preempt partitions. This allows to run, effectively, infinite-running jobs on the cluster. An example of such a program is given in the Tutorials section.

  • The DMTCP system is available on the cluster. It performs an automatic checkpointing of any program (except for MPI applications) without the need to modify the source code. An example of use is in the directory user_checkpointing_demo/low_priority_job_with_dmtcp/ of this repository. It is done with source scheckpoint - there are some options available - see source scheckpoint -h for details

Containerization - Charliecloud (gpulab nodes)

Charliecloud provides user-defined software stacks (UDSS) for HPC. It allows you to run nearly any software stack (like TensorFlow) on the cluster even it is not system-wide installed and available. All informations about Charliecloud can be found on its Charliecloud documentation page.

  • Parlab, Gpulab nodes run Charliecloud - version 0.38
Basic workflow

This workflow is valid for Charliecloud version 0.34.

1. Get or create Docker image

We don't use Docker anymore, as it poses significant security hole. Docker was replaced by Podman. Podman is installed on all worker nodes and the command podman is aliased by docker. You can use scripts using docker command without changing them.

You can either pull already prepared Docker image (e.g. for TensorFlow) or you may create your own one from Dockerfile. You will make this step only once for the given UDSS. Of course, you must restart the whole workflow, if there is a new version of the UDSS.

If you are pulling a Docker image, use

ch-image pull dockertag

If you are building your own Docker image from Dockerfile, use

ch-image build -t dockertag .

2. Create tar/directory image from Docker image

You must convert prepared Docker image to a directory structure. You will make this step only once for the given UDSS.

Use command

ch-convert -i ch-image -o dir dockertag imgdir

which exports Docker image with dockertag tag to a directory imgdir, which will be used for executing the container.

3. Import CUDA libraries

This step is required only for UDSS with CUDA requirement (like TensorFlow with GPU support). If your UDSS does not require CUDA, skip this step. You will make this step only once for the given UDSS, but you should repeat this step, when new host drivers/CUDA are installed. It works only with directory structure. All commands for this step must be run on volta[01-05] or ampere[01-02] nodes.

Execute on gpulab

srun -p gpu-short --gpus=1 ch-fromhost --nvidia imgdir

which copies some necessary CUDA files from the host to your image directory structure.

4. Execute created UDSS by SLURM

This step is executed many times as necessary on any node of parlab and gpulab clusters.

For an interactive job run

srun <slurm params> ch-run <charlie options> imgdir <my img command>

You will probably use SLURM batch job mode more often, as the length of computation is usually several hours or days. In this case use

ch-run <charlie options> imgdir <my img command>

in your shell script passed to the sbatch SLURM command.

Beware, that by default, your home is not bound to the image. You may bind home or any additional directories by using options --bind=/some/dir (which will appear as /some/dir in your UDSS environment) or by --bind=/source/dir:/dest/dir. All Charliecloud images have free mountpoints /mnt/[0-9].

Advanced techniques, troubleshooting, and notes

Builders

Charliecloud brings a term builder. A builder is anything capable of producing a Linux filesystem tree from either a prepared set of container images or from some container description.

Currently Charliecloud supports several builders. We have enabled only two of them: Docker and ch-image.

ch-image command/builder builds an image in unprivileged mode from Dockerfile. This command doesn't need Docker, it can be run anywhere on workers.

When using ch-image command, you have to use parameter -i ch-image in step 2.

CUDA import errors

It can happen, executing ch-fromhost from the 3rd step will produce some errors, which looks something like

/sbin/ldconfig.real: Can't stat /usr/local/nvidia/lib: No such file or directory
/sbin/ldconfig.real: Can't stat /usr/local/nvidia/lib64: No such file or directory
/sbin/ldconfig.real: Can't stat /usr/local/lib/x86_64-linux-gnu: No such file or directory

Ignore safely these errors, they do no harm to you.

This warning/notice was contributed by Vít Kabele.

CUDA is not working inside your UDSS

If you suspect that CUDA is not working inside your container, run the nvidia-smi command from the container command line. If nvidia-smi prints the CUDA version correctly, then CUDA is functional. However, if it prints "ERR", CUDA does not work.

In this case, follow the checklist: - Did you correctly import the CUDA libraries? See step 3 of the basic workflow. - Is libcuda.so (or libcuda.so.1) loadable? Check LD_LIBRARY_PATH environment variable inside your container. If not set to the CUDA library directory, set it to the correct path, e.g. export LD_LIBRARY_PATH=/usr/local/cuda/lib64. Be careful, if the variable is already set to some additional paths.

Containerization - Apptainer

Apptainer (formerly Singularity) simplifies the creation and execution of containers, ensuring software components are encapsulated for portability and reproducibility. Please see Apptainer documentation for details.

If you want to run containers with --fakeroot option on the cluster, please contact the administrator to be added to /etc/subuid and /etc/subgid on headnode.

  • Chimera runs apptainer - version 1.2.2-1.el9