Savio scheduler configuration

Savio partitions

Partition Nodes Node Features Nodes shared? SU/core hour ratio
savio 160 savio exclusive 0.75
savio_bigmem 4 savio_bigmem or savio_m512 exclusive 1.67
savio2 163 savio2 or savio2_c24 or savio2_c28 exclusive 1.00
savio2_bigmem 36 savio2_bigmem or savio2_m128 exclusive 1.20
savio2_htc 20 savio2_htc shared 1.20
savio2_gpu 17 savio2_gpu shared 2.67 (5.12 / GPU)
savio2_1080ti 7 savio2_1080ti shared 1.67 (3.34 / GPU)
savio2_knl 28 savio2_knl exclusive 0.40
savio3 116 savio3 exclusive TBD
savio3_bigmem 16 savio3_bigmem exclusive TBD
savio3_xlmem 2 savio3_xlmem exclusive TBD
savio3_gpu 1 savio3_gpu shared TBD
savio3_2080ti 8 4rtx,8rtx shared TBD

Overview of QoS configurations for Savio

For details on specific Condo QoS configurations, see below.

QoS Accounts allowed QoS Limits Partitions
savio_normal FCA*, ICA 24 nodes max per job, 72:00:00 wallclock limit all**
savio_debug FCA*, ICA 4 nodes max per job, 4 nodes in total, 00:30:00 wallclock limit all**
savio_long FCA*, ICA 4 cores max per job, 24 cores in total,10 day wallclock limit savio2_htc
Condo QoS condos specific to each condo, see next section as purchased
savio_lowprio condos 24 nodes max per job, 72:00:00 wallclock limit all

(*) Including purchases of additional SUs for an FCA.

(**) Note that savio3 nodes (including the various bigmem, GPU, etc. nodes) are not yet available for use by FCAs or ICAs.

QoS configurations for Savio condos

Savio Condo QoS Configurations
Account QoS QoS Limit
co_acrb acrb_savio_normal 8 nodes max per group
co_aiolos aiolos_savio_normal 12 nodes max per group
24:00:00 wallclock limit
co_astro

astro_savio_debug


astro_savio_normal

4 nodes max per group
4 nodes max per job
00:30:00 wallclock limit


32 nodes max per group
16 nodes max per job

co_dlab dlab_savio_normal 4 nodes max per group
co_nuclear nuclear_savio_normal 24 nodes max per group
co_praxis praxis_savio_normal 4 nodes max per group
co_rosalind rosalind_savio_normal 8 nodes max per group
4 nodes max per job per user
Savio Bigmem Condo QoS Configurations

No condos in this partition.

Savio2 Condo QoS Configurations
Account QoS QoS Limit
co_biostat biostat_savio2_normal 20 nodes max per group
co_chemqmc chemqmc_savio2_normal 16 nodes max per group
co_dweisz dweisz_savio2_normal 8 nodes max per group
co_econ econ_savio2_normal 2 nodes max per group
co_hiawatha hiawatha_savio2_normal 40 nodes max per group
co_lihep lihep_savio2_normal 4 nodes max per group
co_mrirlab mrirlab_savio2_normal 4 nodes max per group
co_planets planets_savio2_normal 4 nodes max per group
co_stat stat_savio2_normal 2 nodes max per group
co_bachtrog bachtrog_savio2_normal 4 nodes max per group
co_noneq noneq_savio2_normal 8 nodes max per group
co_kranthi kranthi_savio2_normal 4 nodes max per group
Savio2 Bigmem Condo QoS Configurations
Account QoS QoS Limit
co_laika laika_bigmem2_normal 4 nodes max per group
co_dweisz dweisz_bigmem2_normal 4 nodes max per group
co_aiolos aiolos_bigmem2_normal 4 nodes max per group
24:00:00 wallclock limit
co_bachtrog bachtrog_bigmem2_normal 4 nodes max per group
co_msedcc msedcc_bigmem2_normal 8 nodes max per group
Savio2 HTC Condo QoS Configurations
Account QoS QoS Limit
co_rosalind rosalind_htc2_normal 8 nodes max per group
Savio2 GPU Condo QoS Configurations
Account QoS QoS Limit
co_acrb acrb_gpu2_normal 44 GPUs max per group
co_stat stat_gpu2_normal 8 GPUs max per group
Savio2 1080ti Condo QoS Configurations
Account QoS QoS Limit
co_acrb acrb_1080ti2_normal 12 GPUs max per group
co_mlab mlab_1080ti2_normal 16 GPUs max per group
Savio2 KNL Condo QoS Configurations
Account QoS QoS Limit
co_lsdi lsdi_knl2_normal 28 nodes max per group
5 running jobs max per user
20 total jobs max per user
Savio3 Condo QoS Configurations
Account QoS QoS Limit
co_chemqmc chemqmc_savio3_normal 4 nodes max per group
co_laika laika_savio3_normal 4 nodes max per group
co_noneq noneq_savio3_normal 8 nodes max per group
co_aiolos aiolos_savio3_normal 36 nodes max per group
24:00:00 wallclock limit
co_jupiter jupiter_savio3_normal 12 nodes max per group
co_aqmodel aqmodel_savio3_normal 4 nodes max per group
co_esmath esmath_savio3_normal 4 nodes max per group
co_biostat biostat_savio3_normal 8 nodes max per group
Savio3 Bigmem Condo QoS Configurations
Account QoS QoS Limit
co_genomicdata genomicdata_bigmem3_normal 4 nodes max per group
co_kslab kslab_bigmem3_normal 4 nodes max per group
co_moorjani moorjani_bigmem3_normal 4 nodes max per group
Savio3 Xlmem Condo QoS Configurations
Account QoS QoS Limit
co_genomicdata genomicdata_xlmem3_normal 1 nodes max per group
Savio3 2080ti Condo QoS Configurations
Account QoS QoS Limit
co_esmath esmath_2080ti3_normal 16 GPUs max per group
co_rail rail_2080ti3_normal 24 GPUs max per group

CGRL scheduler configuration

The clusters uses the SLURM scheduler to manage jobs. When submitting your jobs via sbatch or srun commands, use the following SLURM options:

  • The settings for a job in Vector (Note: you don't need to set the "account"): --partition=vector --qos=vector_batch
  • The settings for a job in Rosalind (Savio1): --partition=savio --account=co_rosalind --qos=rosalind_savio_normal
  • The settings for a job in Rosalind (Savio2 HTC): --partition=savio2_htc --account=co_rosalind --qos=rosalind_htc2_normal
  • NOTE: To check which QoS you are allowed to use, simply run "sacctmgr -p show associations user=$USER"

    Here are the details for each CGRL partition and associated QoS.
    Partition Account Nodes Node List Node Feature QoS QoS Limit
    vector   11 n00[00-03].vector0 vector,vector_c12,vector_m96 vector_batch 48 cores max per job    

    96 cores max per user

    n0004.vector0 vector,vector_c48,vector_m256
    n00[05-08].vector0 vector,vector_c16,vector_m128
    n00[09]-n00[10].vector0 vector,vector_c12,vector_m48
    savio co_rosalind 8 n0[000-095].savio1, n0[100-167].savio1 savio rosalind_savio_normal 8 nodes max per group
    savio2_htc co_rosalind 8 n0[000-011].savio2, n0[215-222].savio2 savio2_htc rosalind_htc2_normal 8 nodes max per group
Tags: hpc All Tags