TensorFlow is an open source software library for high performance numerical computation. It has become a very popular tool for machine learning and in particular for the creation of deep neural networks. The latest TensorFlow versions are now provided as prebuilt Singularity containers on Rivanna. The basic concept of running Singularity containers on Rivanna is described here.
TensorFlow code is provided in two flavors, either with or without support of general purpose graphics processing units (GPUs). All TensorFlow container images provided on Rivanna require access to a GPU node. Access to GPU nodes is detailed in the sections below.
TensorFlow and Keras
Keras is a high-level neural networks application programming interface (API), written in Python and capable of running on top of TensorFlow, CNTK, or Theano. Since version 1.12.0, TensorFlow contains its own Keras API implementation as described on the TensorFlow website.
What is inside the TensorFlow containers?
Most containers provide documentation regarding the main applications that have been installed inside. This information can be queried with the
singularity run-help command.
module load singularity module load tensorflow/2.1.0-py37 singularity run-help $CONTAINERDIR/tensorflow-2.1.0-py37.sif
This container provides the Python 3.7.5 bindings for: * Tensorflow 2.1.0 with Keras implementation * Keras Visualization Toolkit 0.4 # tflearn 0.3.2 * scikit-learn 0.22.1 * Pandas 1.0.0 * OpenCV 184.108.40.206 * CUDA 10.1.243 * CuDNN 220.127.116.11
Local Copy of Container Image
The TensorFlow container images contain all the libraries and software packages required to run TensorFlow. In order to run SLURM or interactive jobs (ijobs), it is best practice to create a local copy of the container image in your own /home or /scratch directory. The container images can be copied like any other file on Rivanna.
A library of prepared TensorFlow containers prepared to run on Rivanna’s GPU nodes can be accessed through these commands:
module load singularity module avail tensorflow
Loading of any of these container modules produces an on-screen message with instructions on how to copy the TensorFlow container image file, which resides in
TensorFlow Jupyter Notebooks
Jupyter Notebooks can be used for interactive code development and execution of Python scripts and several other codes. A few TensorFlow kernels are available:
- 1.12.0 with Python 2.7
- 1.12.0 with Python 3.6
- 2.1.0 with Python 3.7
Accessing the JupyterLab Portal
- Open a web browser and go to URL: https://rivanna-portal.hpc.virginia.edu
- Use your
Netbadgecredentials to log in.
- On the top right of the menu bar of the Open OnDemand dashboard, click on “Interactive Apps”.
- In the drop-down box, click on
Requesting access to a GPU node
To start a JupyterLab session, fill out the resource request webform. To request access to a GPU, verify the correct selection for the following parameters:
- Under Rivanna Partition, choose “GPU”.
- Under Optional GPU Type, choose “NVIDIA K80”, “NVIDIA P100”, “NVIDIA V100”, “NVIDIA RTX20280”, or leave it as “default”.
Launchto start the session.
Review our Jupyer Lab documentation for more details..
Editing and Running the Notebook
Once the JupyterLab instance has started, you can edit and run your notebook as described here.
TensorFlow SLURM jobs
Singularity can make use of the local NVIDIA drivers installed on a host equipped with a GPU device. The SLURM script needs to include the
#SBATCH -p gpu and
#SBATCH --gres=gpu directives in order to request access to a GPU node and its GPU device. Please visit the Jobs Using a GPU section for details.
To run commands in an GPU-enabled container image, load the singularity module and add the
--nv flag when executing the singularity run or singularity exec commands. Before running the following commands it is assumed that a TensorFlow container image (e.g.
tensorflow-2.1.0-py37.sif) has been copied to your personal /scratch directory.
module load singularity singularity run --nv /scratch/$USER/tensorflow-2.1.0-py37.sif tf_example.py
In the container build script,
python is defined as the default command to be excuted and singularity passes the argument(s) after the image name, i.e.
tf_example.py, to the Python interpreter. So the above singularity command is equivalent to
singularity exec --nv /scratch/$USER/tensorflow-2.1.0-py37.sif python tf_example.py
The TensorFlow container images were built to include CUDA and cuDNN libraries that are required by TensorFlow. Since these libraries are provided within each container, we do not need to load the CUDA/cuDNN libraries available on the host.
Example SLURM Script:
#!/usr/bin/env bash #SBATCH -J tftest #SBATCH -o tftest-%A.out #SBATCH -e tftest-%A.err #SBATCH -p gpu #SBATCH --gres=gpu:1 #SBATCH -c 1 #SBATCH -t 00:01:00 #SBATCH -A mygroup module purge module load singularity # Assuming that the container has been copied to the user's /scratch directory containerdir=/scratch/$USER singularity run --nv $containerdir/tensorflow-2.1.0-py37.sif tf_example.py
TensorFlow Interactive Jobs (ijob)
Just as described for SLURM jobs, it is recommended to copy a TensorFlow container image (e.g. tensorflow-2.1.0-py37.sif) to your personal /scratch directory before starting an ijob.
Start an ijob. Note the addition of
-p gpu --gres=gpu to request access to a GPU node and its GPU device.
ijob -A mygroup -p gpu --gres=gpu -c 1
salloc: Pending job allocation 12345 salloc: job 12345 queued and waiting for resources salloc: job 12345 has been allocated resources salloc: Granted job allocation 12345
Now you can load the
singularity module and execute commands provided by the container. For example:
module purge module load singularity singularity run --nv /scratch/$USER/tensorflow-2.1.0-py37.sif tf_example.py
Interaction with the Host File System
The following user directories are overlayed onto each container by default on Rivanna:
Due to the overlay, these directories are by default the same inside and outside the container with the same read, write, and execute permissions. This means that file modifications in these directories (e.g. in
/home) via processes running inside the container are persistent even after the container instance exits. The
/project directories refer to leased storage locations that may not be available to all users.
Request a Desktop session under Interactive Apps via Open OnDemand. Fill out the form to submit the SLURM job. Launch the session and open a terminal in the desktop. Enter these commands (using
tensorflow/2.1.0-py37 as an example):
$ module load singularity tensorflow/2.1.0-py37 $ singularity shell --nv $CONTAINERDIR/tensorflow-2.1.0-py37.sif Singularity> python -m tensorboard.main --logdir=logdir
Open the resulting URL (of the form
http://localhost:xxxx/) in Firefox.
Can I install my own TensorFlow (that works on a GPU)?
Yes, but the TF/Python/CUDA versions have to be very specific. We recommend creating a conda environment. Using TensorFlow 1.14 as an example:
For your target TF version, look up the supported Python and CUDA versions here. In this example, we find that it is supported by:
- Python 2.7, 3.3-3.7
- CUDA 10.0
Check that the CUDA version is supported on Rivanna:
- Find the corresponding NVIDIA driver version here.
- Start an ijob on a GPU node and run
nvidia-smi. Look for the first line in the table. As of July 2021, our GPU nodes support up to CUDA 11.0.3.
NVIDIA-SMI 450.51.06 Driver Version: 450.51.06 CUDA Version: 11.0
- In this example 10.0 is less than 11.0.3, so the target version is supported.
Load the Anaconda module and create the environment with specific versions.
module load anaconda conda create -n tf1.14 tensorflow-gpu=1.14 cudatoolkit=10.0 python=3.7 -c anaconda -c conda-forge
If the versions are incompatible, either the installation will fail or TF will not be able to detect the GPU at runtime.