Overview

MPI stands for Message Passing Interface. The MPI standard is defined by the Message Passing Interface Forum. The standard defines the interface for a set of functions that can be used to pass messages between processes on the same computer or on different computers. MPI can be used to program shared memory or distributed memory computers. There is a large number of implementations of MPI from various computer vendors and academic groups. MPI is supported on the Rivanna cluster.

MPI On Rivanna

MPI is a standard that describes the behavior of a library. It is intended to be used with compiled languages (C/C++/Fortran). Several implementations of this standard exist. Rivanna supports OpenMPI for all our compilers and IntelMPI for the Intel compiler. MPI can also be used with the interpreted languages R and Python through packages that link to an implementation; on Rivanna these languages use OpenMPI.

Selecting Compiler and Implementation

An MPI implementation must be built with a specific compiler. Consequently, only compilers for which MPI has been prepared can be used with it. All versions of the Intel compiler will have a corresponding IntelMPI. For OpenMPI run

module spider openmpi

This will respond with the versions of OpenMPI available. To see which version goes with which compiler, run

module spider openmpi/<version>

For example:

module spider 

Example output:

You will need to load all module(s) on any one of the lines below before the
"" module is available to load.
   gcc/system
   gcc/5.4.0
   gcc/7.1.0  

This shows that OpenMPI version is available for the gcc system (4.8.5), 5.4.0, and 7.1.0 compilers.

Once a choice of compiler and MPI implementation have been made, the modules must be loaded. First load the compiler, then the MPI. For instance, to use OpenMPI with gcc 7.1.0, run

module load gcc/7.1.0
module load openmpi

To load the Intel compiler version 18.0 and its IntelMPI version, run

module load intel/18.0
module load intelmpi/18.0

It is also possible to combine these into one line, as long as the compiler is specified first (this can result in errors if you not using the default compiler, however)

module load gcc openmpi

For a detailed description of building and running MPI codes on Rivanna, please see our HowTo.


**Available MPI library modules**
Module Category Description
intelmpi mpi IntelMPI from Intel.
mvapich2 mpi The MVAPICH2 software, based on MPI 3.1 standard, delivers the best performance, scalability and fault tolerance for high-end computing systems and servers using InfiniBand, Omni-Path, Ethernet/iWARP, and RoCE networking technologies.
openmpi mpi The Open MPI Project is an open source MPI-3 implementation.
platform-mpi mpi Platform MPI is an MPI-2 implementation from IBM.

Example SLURM Scripts

This example is a SLURM job command file to run a parallel (MPI) job using the OpenMPI implementation:

#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=16
#SBATCH --time=12:00:00
#SBATCH --output=output_filename
#SBATCH --partition=parallel
#SBATCH -A mygroup

module load gcc
module load openmpi

srun ./parallel_executable

In this example, the SLURM job file is requesting two nodes with sixteen tasks per node for a total of 32 processes. Both OpenMPI and IntelMPI are able to obtain the number of processes and the host list from SLURM, so these are not specified. In general, MPI jobs should use all of a node, but some codes cannot be distributed in that manner so we are showing a more general example here.

SLURM can also place the job freely if the directives specify only the number of tasks. In this case do not specify a node count. This is not generally recommended, however, as it can have a significant negative impact on performance.

#!/bin/bash
#SBATCH --ntasks=8
#SBATCH --time=12:00:00
#SBATCH --output=output_filename
#SBATCH --partition=parallel
#SBATCH -A mygroup

module load gcc
module load openmpi

srun ./parallel_executable

Example: MPI over an odd number of tasks

#!/bin/bash
#SBATCH --ntasks=97
#SBATCH --nodes=5
#SBATCH --ntasks-per-node=20
#SBATCH --time=12:00:00
#SBATCH --output=output_filename
#SBATCH --partition=parallel
#SBATCH -A mygroup

module load gcc
module load openmpi
srun ./parallel_executable

MPI with OpenMP

The following example runs a total of 32 MPI processes, 4 on each node, with each task using 5 cores for threading. The total number of cores utilized is thus 160.

#!/bin/bash
#SBATCH --ntasks=32
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=5
#SBATCH --time=12:00:00
#SBATCH --output=output_filename
#SBATCH --partition=parallel
#SBATCH -A mygroup

module load mvapich2/gcc
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
srun ./hybrid_executable