Using Julia on Savio

We provide Julia for high-performance numerical computing. Julia provides an interactive computing environment (like Python, R and MATLAB) combined with a powerful optional typing system with functional multiple dispatch and a just-in-time compilation system.

Loading Julia and installing Julia packages

To access Julia, you need to load the Julia module.

Note that we do not provide many additional Julia packages. You'll need to add them to your projects in the usual fashion (using Pkg; Pkg.add("mypackage")), which will invoke installation if you (or we) have not installed the package before.

Threaded linear algebra

Julia uses OpenBLAS for fast, threaded versions of core linear algebra functions.

To some degree, you don't have to do anything -- any linear algebra that you run (either directly or that gets called by code that you run) will run in parallel on multiple threads.

Aligning the number of threads and cores

However, there are some serious shortcomings in how Julia determines the number of cores available.

Julia defaults to using half as many software threads as there are cores on the machine (this default seems to relate to assuming that hardware hyperthreading is enabled, which is not true on Savio). In doing so, Julia ignores the number of cores made available to your job via Slurm.

This means that Julia could use fewer than the number of cores available if you have requested a full node (or more than half the cores on a node). Alternatively Julia could use more threads than cores, if you are setting the number of cpus-per-task and ntasks via Slurm and the total number of cores requested is fewer than half the number of cores on the machine. In this case, your code will not use more cores than Slurm provides to the job (so your code will not impact other users), but there may be inefficiency from switching between threads, given there is more than one thread per core.

Furthermore, there are some circumstances under which you might want to set the number of threads yourself. These include:

• If you are running a hybrid parallelization in which individual threaded linear algebra operations are being run in parallel (e.g., using linear algebra nested within some other parallelized work, such as with the Distributed package. Here you probably want to set the number of threads equal to the number of cores available divided by the number of parallel workers running. Some (often smaller) problems may not run faster with more and more threads (or could possibly even run more slowly).

You can control the number of threads with using LinearAlgebra; BLAS.set_num_threads() inside Julia or by setting the OMP_NUM_THREADS in the shell before starting Julia.