Computing Hardware & Partitions
If you are considering purchasing new hardware to integrate into the cluster, please contact the cluster administrator. Coordination is necessary to ensure the hardware complies with minimal requirements and to synchronize procurement efforts with other departments. The data center in Troja is available for hosting of servers only if they contribute to the shared computational resources immediately or by the end of an externally funded project (e.g. ERC, GAČR).
SLURM Essentials
- Terminology: A cluster consists of nodes, which are grouped into partitions. Users submit jobs (consisting of tasks) to a scheduling queue.
- GRES (Generic Resources): Used to request specialized hardware like GPUs. Use
--gpus=countor--gres=gpu:type:countin submission commands. - Fair Share Priority: Users who have used the cluster less in recent weeks receive higher priority. The priority half-life is 14 days.
- Preemption: Low-priority jobs in partitions like
ffa-preemptcan be interrupted by high-priority jobs.- REQUEUE: The job is killed and requeued; Slurm sends a signal one minute before termination.
- SUSPEND: The job is paused, releasing CPUs but not memory.
- Checkpointing: Essential for preemptable jobs. Use the
scheckpointcommand or the DMTCP system. - Important Commands:
sinfo(status),squeue(queue info),srun(interactive),sbatch(batch),scancel(terminate job),seff(efficiency statistics).
NOTE: Cores in next tables are per socket; also HT are per core - you can also use sinfo -N -O NodeList,SocketCoreThread,Memory:.10,Features:.50 | uniq to get information about sockets, cores, SMT, memory, and features.
Chimera Specification
Nodes with the "ib" feature are interconnected with HDR InfiniBand.
Chimera Nodes
| Node names | CPU | Sockets | Cores | HT | RAM | GRES | Additional info |
|---|---|---|---|---|---|---|---|
| deepth1[3-4] | AMD EPYC 7282 | 2 | 16 | 2 | 128 GB | IB | |
| utf-asus[1-3] | AMD EPYC 7302 | 2 | 16 | 2 | 256 GB | local_scratch, IB | |
| utf-hm1 | AMD EPYC 7763 | 2 | 64 | 2 | 2 TB | local_scratch, swap, IB | |
| node[0-3,6] | Intel(R) Xeon(R) E5-2630 v4 | 2 | 10 | 2 | 256 GB | local_scratch | |
| ucjf-asus1 | AMD EPYC 7532 | 2 | 32 | 2 | 512 GB | IB | |
| ucjf-asus1 | AMD EPYC 7532 | 2 | 32 | 2 | 512 GB | local_scratch, IB | |
| ucjf-asusb[1-2] | AMD EPYC 7352 | 2 | 24 | 2 | 256 GB | local_scratch, IB | |
| onos2 | Intel(R) Xeon(R) CPU E5-2670 v3 | 2 | 12 | 2 | 128 GB | ||
| onos[3-4] | Intel(R) Xeon(R) CPU E5-2670 v3 | 2 | 12 | 2 | 256 GB | ||
| c10t4[0-1] | Intel(R) Xeon(R) CPU E5-2640 v3 | 2 | 8 | 2 | 64 GB | ||
| c11t4[2-3] | Intel(R) Xeon(R) CPU E5-2640 v3 | 2 | 8 | 2 | 64 GB | ||
| c14t3[2-3] | Intel(R) Xeon(R) CPU E5-2640 v3 | 2 | 8 | 2 | 64 GB | ||
| c23t46 | Intel(R) Xeon(R) CPU E5-2630 v3 | 2 | 10 | 2 | 64 GB | ||
| c74t57 | AMD EPYC 7451 | 2 | 24 | 1 | 128 GB | gpu:quadro_m2000 vram:4G | NVIDIA Quadro M2000 4 GB |
| c71t55,c70t54,c72t56,c73t38 | AMD EPYC 7451 | 2 | 24 | 1 | 128 GB | ||
| c73t38 | AMD EPYC 7451 | 2 | 24 | 1 | 128 GB | ||
| c42t47 | Intel(R) Xeon(R) CPU E5-2650 v4 | 2 | 12 | 2 | 64 GB | ||
| c41t19 | Intel(R) Xeon(R) CPU E5-2650 v4 | 2 | 12 | 1 | 64 GB | ||
| c40t17,c43t48 | Intel(R) Xeon(R) CPU E5-2630 v4 | 2 | 10 | 1 | 64 GB | ||
| c27t15,c28t14 | Intel(R) Xeon(R) CPU E5-2630 v4 | 2 | 10 | 2 | 64 GB | ||
| c67t02 | Intel(R) Xeon(R) CPU E5-2630 v4 | 2 | 10 | 1 | 64 GB | gpu:quadro_m2000 vram:4G | NVIDIA Quadro M2000 4 GB |
| c64t03,c39t09 | Intel(R) Xeon(R) CPU E5-2630 v4 | 2 | 10 | 2 | 64 GB | ||
| c65t04,c66t05 | Intel(R) Xeon(R) CPU E5-2650 v4 | 2 | 12 | 2 | 64 GB | ||
| c77t06,c78t10,c75t34 | AMD EPYC 7532 | 2 | 32 | 1 | 256 GB | ||
| c93t33 | AMD EPYC 7543 | 2 | 32 | 1 | 512 GB | 16 numas | |
| c94t26 | AMD EPYC 7543 | 2 | 32 | 1 | 256 GB | 16 numas | |
| c08t08 | Intel(R) Xeon(R) CPU E5-2650 v2 | 2 | 8 | 1 | 64 GB | no AVX2, FMA | |
| mff-a2-[01-02] | AMD EPYC 7543 | 2 | 32 | 2 | 512 GB | gpu:T600:1 vram:4G | IB, NVIDIA T600 4 GB |
| *-a2-[03-18] | AMD EPYC 7543 | 2 | 32 | 2 | 512 GB | IB | |
| mff-a2-19 | AMD EPYC 7543 | 2 | 32 | 2 | 512 GB | gpu:L40:1 mps:100 vram:48G | IB, NVIDIA Tesla L40 48 GB |
| ucjf-a4-01 | AMD EPYC 9354 | 2 | 32 | 2 | 384 GB | gpu:L40:1 mps:100 vram:48G | IB, NVIDIA Tesla L40 48 GB |
| r[31-35] | Intel(R) Xeon(R) Gold 6140 CPU | 2 | 18 | 2 | 128 GB | IB(EDR) | |
| r[36-39,41-47] | Intel(R) Xeon(R) Gold 6240 CPU | 2 | 18 | 2 | 128 GB | IB(EDR) | |
| r40 | Intel(R) Xeon(R) Gold 6240 CPU | 4 | 18 | 2 | 512 GB | IB(EDR) | |
| r50 | AMD EPYC 9654 | 2 | 96 | 2 | 1.5 TB | gpu:MI210:2 vram:64G | IB(EDR), AMD MI210 |
| fuuk-1[2-4] | AMD EPYC 9654 | 1 | 96 | 2 | 1.5 TB | gpu:L40:4 vram:48G | IB(EDR), 4x NVIDIA TESLA L40s 48 GB |
Chimera node features
| Feature | description |
|---|---|
ib |
node has infiniband |
local_scratch |
nde has local scratch |
avx2, sse42, fma, |
CPU support this instruction set |
avx512_old |
CPUs supporting avx512f, avx512dq, avx512cd, avx512bw, avx512vl, avx512_vnni instruction sets (older Intel CPUs) |
avx512 |
supports avx512_old and avx512ifma, avx512_bf16, avx512vbmi, avx512_vbmi2, avx512_bitalg,avx512_vpopcntdq instruction sets (new AMD / Intel CPUs) |
#C |
Procesor has # CPU cores |
#T |
Core has # SMT threads |
#N |
Machine has # numa nodes |
#S |
Machine has # sockets |
Chimera Partitions
| Name | Nodes | Priority | Timelimit | DefaultTime | Preemption | Oversubs | Allowed accounts | Intended use |
|---|---|---|---|---|---|---|---|---|
| ffa | all without debug and subset of auuk, ucjf, mff-a2-[01-02] | 5 | 12 hours | 1 hour | no | FORCE:1 | any | free-for-all, any job |
| ffa-short | all without debug and subset of auuk, ucjf, mff-a2-[01-02] | 5 | 2 hours | 30 minutes | no | FORCE:1 | any | default, free-for-all, any job |
| ffa-long | mff-a2-19 | 5 | 07 days | 1 day | no | FORCE:1 | any | free-for-all, any, QoS only 1/2 of the node |
| ffa-preempt* | all without mff-a2-[01-02] | 5 | 36 hours | 1 hour | REQUEUE | NO | any | free-for-all, any job, with preemption |
| ffa-checkpoint* | all without subset of ucjf , mff-a2-[01-02] | 5 | 36 hours | 1 hour | REQUEUE | NO | any | free-for-all, jobs with checkpointing, with preemption, grace time 60s |
| auuk | subset of c[00-99]t[00-99], auuk-a2-[07-15,17-18] | 40 | 8 hours | 4 hours | NO | FORCE:1 | auuk | jobs for AUUK |
| auuk-preempt* | c[00-99]t[00-99], auuk-a2-[07-15,17-18] | 20 | 12 hours | 8 hours | REQUEUE | FORCE:1 | auuk | jobs for AUUK with preemption |
| utf | utf-* node[0-1,6], r4[1-7] | 40 | infinite | 1 day | no | FORCE:1 | utf | jobs for UTF |
| utf-pejcha | r-[31-40] | 40 | infinite | 1 day | no | FORCE:1 | utf-pejcha | jobs for Pejcha's group |
| fuuk | deepth1[3-4] | 40 | infinite | 1 day | no | FORCE:1 | fuuk | jobs for FUUK |
| fuuk-barvik | fuuk-1[2-4] | 40 | infinite | 1 day | no | FORCE:1 | fuuk-barvik | jobs for Barvik's group |
| ucjf | ucjf-* | 40 | infinite | 1 day | no | FORCE:1 | ucjf | jobs for UCJF |
| math | r[41-47,50] | 40 | infinite | 1 day | no | FORCE:1 | math | jobs for mathematical section |
| debug | onos[3-4] | 100 | 12 hour | 1 hour | no | FORCE:1 | any | for interactive jobs only, QoS debug (max CPUs:24 and memory:200GB) |
| checkpoint | onos[3-4], c14t3[2-3] | 50 | 10 minutes | 5 minutes | no | FORCE:1 | any | for starting jobs with checkpoints |
| gpu-ffa | mff-a2-19, ucjf-a4-01, fuuk-1[2-4] | 10 | 12 hours | 1 hour | no | FORCE:1 | any | only jobs that require gpu |
| edu | mff-a2-[01-02] | 20 | 3 hours | 2 hours | no | FORECE:1 | any | for education, only one job per user |
*preemptable partitions have lower fair share billing (TRESBillingWeights="CPU=") ; ffa-preempt 0.1; ffa-checkpoint 0.15; auuk-preempt 0.25
Parlab Specification
All nodes are interconnected by InfiniBand FDR (56 Gb/s) and 10 GbE. The front-end server is connected by 10 GbE to the external world. The latest version of OpenMPI is installed on all nodes.
Parlab Nodes
| Node names | CPU | Sockets | Cores | HT | RAM | GRES | Additional info |
|---|---|---|---|---|---|---|---|
| w[201-208] | Intel Xeon Gold 6130 | 2 | 16 | 2 | 128 GB |
Parlab Partitions
| Name | Nodes | Priority | Timelimit | Preemption | Allowed accounts | Intended use |
|---|---|---|---|---|---|---|
| ffa | all | 10 | 12 hours | REQUEUE | any | default, free-for-all, any job |
| mpi-homo-ffa | w[201-208] | 30 | 12 hours | REQUEUE | any | free-for-all, MPI jobs on homogeneous nodes |
| mpi-homo-long | w[201-208] | 100 | 7 days | REQUEUE | ksi, kdss, kdsstudent, nprg042s, nprg058s, nprg054s | debugging on newer CPUs, MPI debugging, long jobs |
| mpi-homo-short | w[201-208] | 150 | 2 hours | REQUEUE | ksi, kdss, kdsstudent, nprg042s, nprg058s, nprg054s | executing short jobs on 2-socket system, MPI jobs on homogeneous nodes |
Gpulab Specification
These nodes are integrated into Chimera with their own environment and 10 GbE interconnects.
Gpulab Nodes
| Node names | CPU | Sockets | Cores | HT | RAM | GRES | Additional info |
|---|---|---|---|---|---|---|---|
| volta01 | Intel Xeon Silver 4110 | 2 | 8 | 2 | 256 GB | gpu:V100:2, vram:16G | 1x NVIDIA V100 PCIe 16 GB, latest V100 compatible CUDA (12.9.1) |
| volta02 | Intel Xeon Silver 4110 | 2 | 8 | 2 | 192 GB | gpu:V100:2, vram:16G | 2x NVIDIA V100 PCIe 16 GB, latest V100 compatible CUDA (12.9.1) |
| volta03 | Intel Xeon Silver 4110 | 2 | 8 | 2 | 192 GB | gpu:V100:2, vram:16G | 2x NVIDIA V100 PCIe 16 GB, latest V100 compatible CUDA (12.9.1) |
| volta04 | Intel Xeon Gold 5218 | 2 | 16 | 2 | 384 GB | gpu:V100:1, vram:32G | 1x NVIDIA V100 SXM2 32 GB, latest V100 compatible CUDA (12.9.1) |
| volta05 | Intel Xeon Gold 5218 | 2 | 16 | 2 | 384 GB | gpu:V100:4, vram:32G | 4x NVIDIA V100 SXM2 32 GB, latest V100 compatible CUDA (12.9.1) |
| ampere01 | AMD EPYC 9554P | 1 | 64 | 2 | 384 GB | gpu:L40:4, vram:48G | 4x NVIDIA L40 PCIe 48 GB, latest CUDA |
| ampere02 | AMD EPYC 9374F | 2 | 32 | 2 | 768 GB | gpu:A100:2, vram:80G | 2x NVIDIA A100 PCIe 80 GB, latest CUDA |
| ampere03 | AMD EPYC 9554 | 1 | 64 | 2 | 384 GB | gpu:L40:1, vram:40G | 1x NVIDIA L40 PCIe 40 GB, latest CUDA |
| hopper01 | AMD EPYC 9554 | 2 | 48 | 2 | 1024 GB | gpu:H100:2, vram:80G | 2x NVIDIA H100 PCIe 80 GB, latest CUDA |
| bw01 | AMD EPYC 9454 | 1 | 48 | 1 | 768 GB | gpu:H100:2, vram:80G | 2x NVIDIA RTX PRO Blackwell Server Edition PCIe 96 GB, latest CUDA |
Gpulab Partitions
| Name | Nodes | Priority | Timelimit | Preemption | Allowed accounts | Intended use |
|---|---|---|---|---|---|---|
| ffa-gpulab | all | 10 | 12 hours | REQUEUE | any | default, free-for-all, any job |
| gpu-ffa-gpulab | volta[01-05],ampere[01-03],hopper[01] | 20 | 12 hours | REQUEUE | any | free-for-all, any GPU job |
| gpu-long | volta[01-05],ampere[01-02],hopper[01] | 100 | 7 days | REQUEUE | kdss, kdsstudent, ksviprj | long GPU jobs |
| gpu-short | volta[01-05],ampere[01-02],hopper[01] | 150 | 2 hours | REQUEUE | kdss, kdsstudent, ksviprj | debugging GPU tasks, executing short GPU jobs |
| gpu-long-ksi | volta[01-05],ampere[01-02],hopper[01] | 200 | 7 days | REQUEUE | ksi, ksiprj, ksibio | long GPU jobs for KSI department |
| gpu-short-ksi | volta[01-05],ampere[01-02],hopper[01] | 250 | 2 hours | REQUEUE | ksi, ksiprj, ksibio | debugging GPU tasks, executing short GPU jobs |
| gpu-short-teach | volta[01-05],ampere[01-02],hopper[01] | 300 | 2 hours | REQUEUE | nprg042s, nprg042t, nprg058s, nprg058t | parallel computing courses |
| gpu-bio | ampere[01-03],hopper[01] | 800 | 7 days | REQUEUE | ksibio | GPU jobs for Bioinformatics study program |