— Using Singularity to create portable applications

Red Hen makes use of HPC facilities in three locations: UCLA, Case and RRZE. To facilitate the exchange of pipelines and to minimize setup time, software that is to be run at multiple HPC centers and/or is difficult to install and/or has esoteric dependencies (e.g. old versions of libraries; libraries so large that they cannot be sensibly installed on HPC systems, etc.) can be installed within Singularity containers. The purpose of this document is to describe how to set up such a container.

Singularity images can be created on Singularity Hub or on a desktop computer; both methods are described below.

Related links

Singularity Hub build procedure

The preferred method of creating a Singularity image is to use the services provided by Singularity Hub. Here, images are built automatically, based on simple build recipes stored on your github account.

The advantage of using these recipes is that you can collaborate with others. People with different areas of expertise can rapidly contribute to a shared project. In addition, the images are built automatically on Singularity Hub and can be downloaded to your Linux server with a simple command.

It is possible to also run Singularity images on Mac and Windows laptops, but the procedure is still convoluted. Native support for OS X is in the planning stage.

Create a recipe

First create a recipe on Red Hen's github account at https://github.com/RedHenLab/singularity_containers. See this directory for examples; the process and naming conventions are described here.

The recipe will be built right away, typically completing within a few minutes; you can see the completed images here.

Download the image

To download an automatically built image to your user on a laptop, desktop, or high-performance computing cluster node, issue
$ singularity pull shub://maintainer/application
On the Case high-performance computing cluster, make sure you use Gallina, Red Hen's server inside the HPC, and always reachable from your home directory on the login nodes. The Singularity image should never be downloaded to a login node.
$ cd /mnt/rds/redhen/gallina/Singularity/
$ singularity pull shub://RedHenLab/singularity_containers:deepspeech2
Progress |===================================| 100.0% 
Done. Container is at: 
It will often be useful to keep your own version of the Singularity image; you can rename it as you download it:
    singularity pull --name deepspeech-mark.img shub://RedHenLab/singularity_containers:deepspeech2
That way revisions of the image won't interfere with files owned by others. 

On the Case HPC cluster, use the Gallina server (/mnt/rds/redhen/gallina/Singularity) for images -- we have massive amounts of storage available there, but a strictly limited quota in our home directories.

Run the image

To enter a shell within the Singularity container, use the -e switch to avoid importing your local environment and the -H switch to set the home directory to the current directory:
$ singularity shell -e -H `pwd` RedHenLab-singularity_containers-master-deepspeech2.simg
Singularity: Invoking an interactive shell within container...
On the Case HPC, we have a limited storage quota in home directories, but more or less unlimited space on gallina.

Run the image in an HPC job

1. Request a node with a GPU: 
$ srun -p gpu -C gpuk40 --gres=gpu:1 --pty bash
2. Load the Singularity software module (the version number may be important; the latest version should load by default):
$ module load singularity/2.5.1
3. Run the python demo, binding the current directory to /mnt:
$ cd /mnt/rds/redhen01/gallina/Singularity/DeepSpeech2
$ vim housing.py
$ singularity exec -B `pwd`:/mnt RedHenLab-singularity-deepspeech2.simg python /mnt/housing.py
The binding gives Singularity access to files on that mount point or file system.

Desktop build procedure  

1. Prerequisites

You will need a machine on which you have root rights and sufficient free space on the hard disk - 5 Gigabytes plus whatever your application needs should do for most tasks - and sufficient RAM to build the software (2 Gigs is not enough for Tensorflow; 6 Gigs is). This can be a virtual machine, for which a 20 Gig hard disk is recommended.
This document assumes you are running Ubuntu 16.04 LTS.

2. Installing Singularity

Follow the instructions for Linux or for Mac. If you have Windows, you will have to use a virtual machine with Linux installed. [Todo: provide virtual machine]
Copy-Paste for this step in Linux:
VERSION=2.5.1 wget https://github.com/singularityware/singularity/releases/download/$VERSION/singularity-$VERSION.tar.gz tar xvf singularity-$VERSION.tar.gz cd singularity-$VERSION ./configure --prefix=/usr/local make sudo make install

Alternatively, you can install the packaged version, in Debian/Raspbian called singularity-container.

3. Creating an image

In fact, the Quickstart Guide is just perfect, so there is no need to repeat it here. The only issue that you may encounter is that SquashFS-containers (the default format in Singularity 2.4 and above) cannot be created on NFS mounts (e.g. your home directory in the HPC system). Instead, you will have to build it on a physical disk (try /tmp).

The following command is outdated and is here only for documentation of our earlier processes.
sudo singularity create --size 10000 mycontainer.img
Creates an empty image of roughly 10 Gigabytes. Note that this is a sparse file, so it will not take up as much space initially, even if it looks like it:
redhen@server:~$ sudo singularity create --size 10000 singularity_containers/mycontainer.img
Creating a new image with a maximum size of 10000MiB...
Executing image create helper
Formatting image with ext3 file system
redhen@server:~$ ls -lh singularity_containers/mycontainer.img
-rwxr-xr-x 1 root root 9,8G Dez 22 15:18 singularity_containers/mycontainer.img
redhen@server:~$ du -h singularity_containers/mycontainer.img
289M    singularity_containers/mycontainer.img
Unfortunately, sparse files can inflate when they are being copied (use rsync, not scp) or when they are stored on NFS mounts. Thus, basically, you should expect your sparse file to end up at its full size eventually and thus avoid unneccessarily large containers.

4. Filling the image

To create a Debian or Ubuntu image, you have to install debootstrap:
sudo apt-get install debootstrap
Next you will have to create a Bootstrap definition. You can use the one attached to this page, which is based on an example found at the singularity webpage. The relevant parts read as follows; explanations are given in the comments:
BootStrap: debootstrap # The bootstrap command used
OSVersion: xenial # This is Ubuntu 16.04 LTS, codename Xenial Xerus
MirrorURL: http://us.archive.ubuntu.com/ubuntu/ # This is where the process can find the data needed. If you are in Europe, you can choose another mirror at https://launchpad.net/ubuntu/+archivemirrors or just use the archive provided by FAU: http://ftp.fau.de/ubuntu/

    echo "This is what happens when you run the container..." # Replace this with the default command including all necessary options, so that running the image will directly run the software for which you built the container.

# Everything in this section will be executed after bootstrapping the operating system is complete
    sed -i 's/$/ universe/' /etc/apt/sources.list # Some software is only available via the universe repository, so this will be added to apt's sources.list
    apt-get -y update # This makes sure the sources from the universe repository are loaded
    apt-get -y install nano git # Change this at your leisure. Neither of them is necessary.
   # Add all commands needed to install the software
While it is possible to install software interactively, it is generally recommended to do so via the definition file to make transparent and reproducible what was actually done to arrive at the image.

Example basic solidarity recipe, assuming you have root access:

sudo singularity build --sandbox abstract_pipeline_v1 abstract_pipeline_v1.def

This creates a directory abstract_pipeline_v1, which is the “container.” abstract_pipeline_v1.def is a file whose contents are:

BootStrap: debootstrap

OSVersion: xenial

#MirrorURL: http://ftp.fau.de/ubuntu/

# Use the following URL in the US:

MirrorURL: http://us.archive.ubuntu.com/ubuntu/



cd /localdata/abstracts/

./annotate_all.sh "$@"



echo "Hello from inside the container"

sed -i 's/$/ universe/' /etc/apt/sources.list

apt-get update

apt-get upgrade -y

apt-get install -y --force-yes git wget nano autoconf automake bzip2 python3-dev unzip python3-pip subversion python

apt-get -y autoremove 

mkdir /localdata

mkdir /localdata/abstracts

#cd /localdata/abstracts

#git clone WHATEVER




#make install

chmod a+w /.singularity.d/runscript

You can shell into the container as follows (-w means “writable):

sudo singularity shell -w abstract_pipeline_v1

To create the final portable unchangeable image (after training etc.):

sudo singularity build abstract_pipeline_v1_production.simg abstract_pipeline_v1

To run the image:

./abstract_pipeline_v1_production.simg FILE_TO_BE_ANNOTATED > OUTFILE.TXT