Tutorials & examples

Detailed tutorials including example programs and SLURM scripts for various use cases can be found in the MFF GitLab repository.

HPC and RSE workshop (23rd June 2026)

The full recording can be found here and the slides are available here.

  • The purpose of the infrastructre and the people behind it (0:00 - 12:27)
  • The new data center Troja and its parameters (12:28 - 30:45)
  • Chimera cluster, JupyterHUB and how to use it (30:46 - 53:42)
  • Future infrastructure (53:46 - 1:05:45)
  • Discussion: cluster (1:05:46 - 1:17:04)
  • Research Software Engineering services (1:17:10 - 1:25:10)
  • Intro to AI Agents and natural language control of the HPC cluster (1:25:11 - 1:32:07)
  • ChimeraMCP server: how to connect your agent with Chimera (1:32:08 - 1:41:42)
  • LangFlow: how to design your own AI agent (1:41:43 - 1:48:07)
  • Discussion: AI (1:48:08 - end)

Introductory training (3rd June 2022)

The recording of the full session can be found here and the slides are here (the slides don't contain all the material covered in the hands-on session).

  • Part 1 covers: architecture, hardware, accounting, queues, limits, use of cluster for teaching (accounts for students, reservation of computing resource, software containers), use of the GitLab repository to access documentation and report issues, available software.
  • Part 2 (starting at 1:03:04) is a hands-on sessions which covers: accessing the cluster from Linux and Windows, introduction to SLURM (checking your account, cluster status, job monitoring), running and controlling jobs: serial, parallel (MPI and OpenMP), job preemption (i.e. job priority), interactive sessions, running jobs on a different cluster (for that please see also the correction starting at 2:05:44).

SLURM Crash Course

All informations about SLURM can be found on its SLURM documentation page or on SLURM tutorials page. Anyway, we have provided a short description of SLURM and of our clusters.

Terminology

A cluster is a bunch of nodes. Nodes are grouped together to partitions. Partitions may overlap, ie. one node can be in more partitions. Feature is a string describing a feature of a node, e.g. avx2 for a node capable of executing AVX2 instructions. Each node has two sets of features: current features and available features. Usually, they are same. But in some cases, the node is capable of changing current features set on demand.

SLURM manages resources and schedules jobs according to available resources. The most important resources are CPUs and memory, but it is able to manage other generic resources (GRES), like GPUs, etc. Moreover, the time is resource as well.

A user is identified by his/her login. Account is a billing entity (well, we won't charge you for using our clusters). Each user must have assigned an account. Moreover, user can be assigned to more accounts and use them depending on what he/she is doing. Accounts can only be allowed access to certain partitions.

A user can launch a job. Job has a state, has some reserved and assigned resources, and returns an error code after completion. Job consists from steps (usually 1 step). Each step is executed by a bunch of tasks (usually 1 task), where resources are equally assigned to the tasks of a step. Jobs are inserted to a scheduling queue, where you can find them.

Partition has a priority. A job submitted to a partition with higher priority can preempt an another job submitted to a partition with lower priority. Our clusters use only two kind of preemption:

  • SUSPEND - the preempted job is suspended and releases CPUs. Unfortunately, it does not release memory or GPUs.
  • REQUEUE - the preempted job is killed and requeued to the scheduling queue, waiting for available resources. All resources including GPUs are released. Jobs submitted to the partition should use some form of checkpointing, otherwise all work can be lost, when the job is killed during preemption.

Important commands

There are many commands (see SLURM man pages or SLURM command summary ). The most important commands are:

  • srun - reserves and assigns resources and runs an interactive job, use it only for short or debug jobs
  • sbatch - reserves and assigns resources and submit a batch as a job, use it for long jobs (>1 hour)
  • salloc - only reserves and does not assign resources, use srun or sbatch for assigning resources subsequently
  • sinfo - get info about cluster state
  • scancel - cancels a job
  • squeue - view info about scheduling queue
  • sbcast - distributes a file to the nodes allocated to a job
  • sstat - display info about a job
  • seff - displays statistics of resources for finnished job

On Chimera there is also script smypartition - display your accounts and partitions you can use (ALL means any account can be used)

Job submission commands (srun, sbatch, salloc) have a common set of important options:

Shrt Long Description
-A --acount= Charge resources used by the job to the specified account. If not specified, user's default account is charged
-B --extra-node-info= Select only nodes with at least specified number of sockets, cores per socket, and threads per core. This option does not specify the resource allocation, its just constraint.
-C --constraint= Select only nodes with matching features
-c --cpus-per-task= Number of CPUs per 1 task
-e --error= Standard error stream is redirected to the specified file
-G --gpus= Specifies the number of GPUs required for the job in the form [type:]count. It is a shortcut for --gres=gpu:type:count.
--gres= Specifies comma delimited list of GRES. Each entry on the list is in form name[[:type]:count]
-i --input= Standard input stream is redirected from the specified file
-J --job-name= Job name
-L --licenses= Specifies comma delimited list of licenses allocated to the job. Each entry on the list is in form name[:count]
-M --clusters= Clusters to issue commands to (comma separated list or all)
-m --distribution= Select distribution method for tasks and resources. For more info see documentation
--mem= Specify the real memory required per node, default unit is MB. To allocate all available memory on the node use --mem 0
--mem-per-cpu= Specify the memory required per allocated CPU
--mem-bind= Specify the memory binding. For more info see documentation
-N --nodes= A minimum of allocated nodes. Default is 1
-n --ntasks= Number of tasks. Default is 1
-o --output= Standard output stream is redirected to the specified file
-p --partition= Request a specific partition for the resource allocation. If not specified, default partition os chosen
-t --time= Set a limit on the total run time of a job. If not specified, default time for a selected partition is used. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours".
--threads-per-core= If set to 1, will turn off multithreading which is usefull for many applications; simmilar to --hint=nomultithread
-a --array= Specify indexes for job array; runs multiple of the same jobs (with the reqirements set for each job). Only difference is the environment variable SLURM_ARRAY_JOB_ID is set to one of the indexes

Useful examples

srun --pty bash

Stars interactive job (with bash as shell).

Interactive jobs do not work across clusters, please log into the correct frontnode to run interactive jobs.

salloc

Starts a shell in a minimal environment (resources). Useful for making small builds or debugging in a restricted environment.

salloc -N2 -n4

Starts a shell in a minimal environment with 4 tasks on 2 nodes from which you can run srun and see all outputs. Useful as an interactive debugging tool for MPI jobs.

sbatch -p gpu-long --gpus=1 mygpucode.sh

Starts a batch job which will have one NVIDIA V100 card available.

sinfo -o "%P %L %l"

Prints info about default and maximum job time for partitions in a cluster.

sinfo -o "%n %b %f"

Prints info about current and available feature set for nodes in a cluster.

sinfo -o "%.5P %.5a %.10l %.16F %.10p %N"

Prints partition summary with priority.

sinfo -Nel

Prints nodes list with resources for every partition.

srun -p mpi-homo-short -n 128 -N 8 --mem-per-cpu=2G mympijob

Starts a MPI job with 128 tasks/ranks spanning over 8 nodes in the homogeneous partition assigning 1 CPU and 2 GB RAM to each task.

Examples of script headers

These are examples of script header with Slurm options, that can be run by sbatch script.sh. You can use this format also with options mentioned above.

Standard single-node single-thread job

This is the simple example of single thread jobs. You need only specify how long it takes (otherwise it useses default value, depending on the partition) and memory consumption. If you run an array jobs, these requirements are for each job in the array.

#!/bin/bash
#SBATCH --partition=ffa           # partition you want to run job in
#SBATCH --time=00:15:00           # walltime for the job in format (days-)hours:minutes:seconds
#SBATCH --mem=24000               # memory resource
#SBATCH --job-name="job_name"     # change to your job name
#SBATCH --output=output.txt       # stdout and stderr output file
#SBATCH --mail-user=user@example.com # send email when job changes state to email address user@example.com


# some real work here
Standard multi-node job
#!/bin/bash
#SBATCH --partition=ffa           # partition you want to run job in
#SBATCH --time=00:15:00           # walltime for the job in format (days-)hours:minutes:seconds
#SBATCH --nodes=2                 # number of nodes (can be only 1)
#SBATCH --ntasks-per-node=4       # processes per node
#SBATCH --mem=24000               # memory resource per node
#SBATCH --job-name="job_name"     # change to your job name
#SBATCH --output=output.txt       # stdout and stderr output file
#SBATCH --mail-user=user@example.com # send email when job changes state to email address user@example.com
#SBATCH --exclusive               # Use whole node
#SBATCH --signal SIGTERM@60       # send signal SIGTERM 60s before ending job (preemption, time limit)


# some real work here

This will run job in ffa partition with length of 15 minutes on 2 nodes, 4 tasks per node with 24000MB of RAM on each node - and nothing else will be run on these nodes. Standard output and error will be redirected into output.txt. Slurm will send SIGTEM to the job 60s before walltime ends, or in case of gang preemption. And email will be sent to the user@example.com if job changes begins, ends, or fails. (this can be changed by --mail-type option.

Job-array: submitting many jobs at the same time
#!/bin/bash
#SBATCH --partition=ffa           # partition you want to run job in
#SBATCH -A project                # use account project instead of default
#SBATCH --time=12:00:00         # walltime for the job in format (days-)hours:minutes:seconds
#SBATCH --nodes=1                 # number of nodes
#SBATCH --ntasks-per-node=1       # processes per node
#SBATCH --cpus-per-task=8         # cpus per tasks
#SBATCH --mem-per-cpu=2G          # memory resource per cpu
#SBATCH --array 0-127             # duplicate the job 128 times - slurm sets SLURM_ARRAY_TASK_ID environment variable which corresponds to the index each job in the array
#SBATCH -d afterok:10:12?after:13 # run this job after jobs with ID 10 and 12 ends successfully and job 13 starts executing (this line is optional and only an example of course)
#SBATCH --job-name="job_name"     # change to your job name
#SBATCH --error=error\_%a.txt         # redirects stderr to error\_${SLURM_ARRAY_TASK_ID}.txt
#SBATCH --output=output\_%a.txt       # redirects stdout to output\_${SLURM_ARRAY_TASK_ID}.txt
#SBATCH --mail-user=user@example.com # send email when job changes state to email address user@example.com


# the environment variable $SLURM_ARRAY_JOB_ID is set to the ID of the first job in the array
# some real work here dependent on $SLURM_ARRAY_TASK_ID

Runs 128 jobs, every with 8 CPUs and 2GB of RAM per CPU. All is run in partition ffa and with account "project". Max time is 12 hours for every job. Jobs start after job 10 and 12 ends successfully and job 13 start to run. Here the standard error and standard output are redirected to separate files. Emails in this case are sent about array as a whole.

Recommendations

  • Prefer using sbatch for long jobs. Moreover sbatch allows setting of SLURM parameters in the executed shell-script, you don't need to write them always on the command-line. Moreover, jobs executed by sbatch can be requeued, when cancelled (e.g. for priority reasons).

  • Don't forget to request GPU using --gpus=1, when running on gpu-xxx partitions.

  • Do not use srun --pty bash followed by executing a long computation from the command-line. When the job finishes, you will block resources, until it either timeouts or you will exit the bash. Again, prefer using sbatch for long jobs.

  • Set mail-user and mail-type option in sbatch shell using #SBATCH --mail-user=mymail@isp.edu and #SBATCH --mail-type=END,FAIL. SLURM will send you an email, when the job finishes. There are many other mail-type variants.

Job resource usage

It is useful to check if your jobs are using assigned resources. It this section we will notice some linux and slurm tools to help you find how much resources your jobs are using and what may indicate there is a problem.

  • you can use time ./program (or /usr/bin/time -v ./program for more details) to get some statistics about used cpu time (and more) that the executable ./program used.
  • this is helpful to discover problems when your program spends too much time in system.
  • you can check running jobs by simply ssh to the node your job is running on and
  • use basic tools like top or htop - you can use option -u$USER to filter your processes.
  • you can use sstat job_id to get information about running jobs - you can sstat job_id | less -S for better viewing experience; some useful fields:
  • MaxRSS - maximum resident set size (memory footprint) of all tasks in job
    • if this is close to memory allocated for task in slurm, there might be problems - processes killed for trying to use more memory, caching problems, ...

  • MinCPU - Minimum (system + user) CPU time of all tasks in job.

    • if this is considerably smaller then job (wall time) * #CPUs, there is probably some problem (often waiting for I/O)

  • after the job finishes you can ask use tools like seff job_id to get resource usage; or you can use sacct -j job_id to get even more information about job

    • the seff command presents statistics in user friendly way
    • CPU efficiency around 50% is usually expected on nodes with SMT
    • if CPU efficiency is low (< 30%), there is probably some issue that can be fixed
    • if CPU efficiency is close to 100% and you are running on node with SMT this may indicate some issues with running too many threads and using tools like time would be helpful to check if there is a problem

If the CPU efficiency is low, there is usually some kind of problem - often it is stuck on I/O operations - if this happens when running larger batch of jobs, that read large data, please consider to run smaller number of the jobs simultaneously (consider using arrays with % operator) - this also often hinders other users access to storage

Examples of seff usage: - here is example of job with CPU Efficiency close to 100%

$ seff  1111
Job ID: 1111
Cluster: chimera
User/Group: user1/user1
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 4
CPU Utilized: 00:09:13
CPU Efficiency: 96.68% of 00:09:32 core-walltime
Job Wall-clock time: 00:02:23
Memory Utilized: 14.44 MB
Memory Efficiency: 0.35% of 4.00 GB
  • here is example of job with low CPU Efficiency; here the job was indeed limited by I/O operation:
$ seff  123 
Job ID: 123
Cluster: chimera
User/Group: user2/user2
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 2
CPU Utilized: 00:00:47
CPU Efficiency: 1.83% of 00:42:54 core-walltime
Job Wall-clock time: 00:21:27
Memory Utilized: 6.01 GB
Memory Efficiency: 42.92% of 14.00 GB