[Hosting HPC in Azure] 2.3 -- Slurm

How to Set Up Slurm Database for Job Accounting?

Setting Up Slurm Database Service

1. In CycleCloud Wizard Advanced Settings Slurm Settings:

   • These parameters will be stored in /sched/<cluster-name>/slurmdbd.conf.

   Form	How to
   Database URL	Get private DNS in MySQL Flexible Server overview.
   Database name	Setting Up MySQL Flexible Server
   Database user	Setting Up MySQL Flexible Server
   Database password	Setting Up MySQL Flexible Server
   SSL Certificate URL	Select AzureCA.pem

2. Log in to the scheduler, where the Slurm service is running (see Scheduler). We can connect to the scheduler as long as it is launched, even though the CycleCloud wizard might show there is an error while setting up Slurm.

3. Copy the SSL certificate of the MySQL database (see Setting Up MySQL Flexible Server) to the scheduler.

4. Move the SSL certificate to /sched/<cluster_name> folder:

   sudo mv Cert.pem /sched/<cluster_name>/.
   cd /sched/<cluster_name>
   sudo cp Cert.pem AzureCA.pem # a soft link point to this file
   

5. Restart slurmdbd and slurmctld service:

   sudo systemctl restart slurmdbd
   sudo systemctl restart slurmctld
   

6. Check that we can use job accounting in Slurm:

   sacct    # print an empty table
   

Set Default Time

Azure CycleCloud puts Slurm partition (nodes) settings in /sched/<cluster-name>/azure.conf, and it is then imported it in slurm.conf. If we put another partition specification in slurm.conf, we will get errors.

• Add DefaultTime in azure.conf:

    PartitionName=hpc Nodes=neptune-hpc-[1-24] DefaultTime=01:00:00 Default=YES DefMemPerCPU=7782 MaxTime=INFINITE State=UP
    Nodename=neptune-hpc-[1-24] Feature=cloud STATE=CLOUD CPUs=16 ThreadsPerCore=1 RealMemory=124518
  

  DefaultTime sets the fallback runtime limit for any job submitted to a partition without an explicitly defined --time duration.

Enforce Slurm Job Accounting

• Add this in slurm.conf (in /sched/<cluster-name>/slurmd.conf):

    AccountingStorageEnforce=associations,limits,safe
  

  • associations: This will prevent users from running jobs if their association is not in the database.
  • limits: This will enforce limits set on associations and qos’.
  • safe: This will ensure a job will only be launched when using an association or qos that has a TRES-minutes limit set if the job will be able to run to completion. With the ‘safe’ option set, a job won’t be killed due to limits. Which is a job won’t be killed due to limits, even if the limits are changed after a job was started.

• Restart systemctld service.

    sudo systemctl restart slurmctld
  

Slurm Database

We first need to set a user to have admin level, then we can operate the slurm database. There are three levels, from top to bottom: cluster, account, user. The former includes the latter objects. association can be either one of them limited to the scope itself.

• Set user as admin:

    sudo -i
    sacctmgr add user Name=<user> AdminLevel=Admin Account=root
  

• Generally, Slurm database already has cluster created:

    sacctmgr list cluster
  

• Create an account:

    sacctmgr add account <account> Cluster=<cluster> Description="<description>"
    sacctmgr list account
  

• Create a user:

    sacctmgr add user <user> Account=<account>
  

  This gives user the permission to submit jobs, if we have enforce job accounting. We still need to set resource limits.

• List associations:

    sacctmgr list associations
  

Modify and Check User Resource Limits

• Set resource limits to user (there are tons of options, see slurm documents), for example, this assign 10 min cpu core time (2 cpu x 5 min ~ 10):

    sacctmgr modify user <user> set GrpTRESMins=cpu=10
  

• Reset to default:

    sacctmgr modify user <user> Account=<account> set GrpTRESMins=cpu=-1
  

• Get historical job runtime:

    sacct -u <user> --format=JobID,CPUTimeRAW,ElapsedRaw,NCPUS,State
  

  If it doesn’t show up anything, it may be you didn’t set the time range. Try:

  sacct -u <user> --starttime=2026-01-01
  

• Count the accumulated time:

    sacct -u test-user --format=CPUTimeRAW -n | awk '{sum += $1} END {print sum/120, "CPU-minutes"}'
  

  CPUTimeRAW is in second, and each job takes two rows. So we need to divide the sum by 60 for converting sec to min and divide it by 2.

FAQs

Unknown Server Host

• Details:

    error: mysql_real_connect failed: 2005 Unknown server host 
  

slurmdbd service cannot comprehend the private DNS you provide. Remember to add VNet in private DNS records.

Database settings not recommended values

• Details:

    error: Database settings not recommended values: innodb_buffer_pool_size innodb_lock_wait_timeout
  

Should change these parameters in MySQL Flexible Server.

Slurm

• Reference: https://slurm.schedmd.com/

Image from Slurm Documentation (https://slurm.schedmd.com/)

• slurmctld: Slurm manager on schedular

    sudo systemctl status slurmctld
  

• slurmd: slurm daemon on computing node
• slurmdbd: Slurm DataBase Daemon for record accounting information for multiple Slurm-managed clusters in a single database.

Configuration

• Show which slurm conf it is using:

    scontrol show config | grep SLURM_CONF
  

• slurm.conf for slurmctld daemon, and slurmdbd.conf for slurmdbd daemon:

    slurmctld -> (AccountingStorageType)-> slurmdbd -> (StorageType) -> database
  

• CycleCloud stores all the slurm configuration file in /sched/<cluster_name>. The files and changes will not be gone during restart and after termination.