— How to install software on the Case HPC

Introduction

The high-performance computing cluster at Case Western Reserve University is used by Red Hen faculty and students for various processing pipelines.

Related resources

How to confirm installed binaries

To confirm that a particular utility is already installed, try 'which':

$ which wget

/usr/bin/wget

$ which automake

/usr/bin/automake

$ which autoconf

/usr/bin/autoconf

$ which unzip

/usr/bin/unzip

Most common unix utilities are already present.

If your starting point is a list of Debian packages and you don't know the name of the binary:

    • if you have access to cartago, and the package (say, zlib1g-dev) is installed there, issue
    • just list-files zlib1g-dev
    • and then on the Case HPC look for the binaries (executables) with 'which', the library files (libz.so) in /usr/lib64/ or the header (zlib.h) files in /usr/include/.
    • or try guessing the names by stripping off version information -- e.g., zlib1g looks like just zlib.

The 'which' command will find only binaries; if you need development files, see below.

How to find and activate modules

A large number of packages are available as modules. To list the common available modules:

$ module avail

For an exhaustive list,

$ module spider

To load a module,

$ module load python/2.7.10

To list loaded modules,

$ module list

Currently Loaded Modules:

1) intel/2015 2) openmpi/1.8.8 3) i/1.0.0 4) StdEnv 5) hdf5/1.8.15 6) python/2.7.10

A great deal of software is available either immediately or by loading a module.

How to determine if a package is present

If your starting point is a list of Debian packages, it may be helpful to do a Google search for the name of the Redhat rpm package. You may also want to know which version of Redhat you're installing for, so find the current version of the operating system installed on the Case HPC, issue

$ cat /etc/redhat-release

Red Hat Enterprise Linux Server release 6.8 (Santiago)

Once you know the name of the package, do a search on the Case HPC to see if it's already installed:

$ p=python-devel

$ rpm -qa | grep $p

python-devel-2.6.6-66.el6_8.x86_64

Alternatively, use yum to search for packages:

$ yum search zlib libatlas3

How to install new software

Please document your installation process here.

Installing Gentle

From Daniel Balagué via Sanjaya Gajurel, tweaked by Pete Broadwell:

    1. git clone https://github.com/lowerquality/gentle.git
    2. module load gcc/4.7.3 ; module load ffmpeg/2.8.2 ; module load python/2.7.10
  1. export PYTHONPATH="${PYTHONPATH}:~/myPython/virtualenv-1.9/ENV/lib/python2.7/site-packages/ (place in ~/.bashrc)
    1. modify the Linux section of the install_deps.sh script to look like this (remove the other lines):
    2. if [[ "$OSTYPE" == "linux-gnu" ]]; then
    3. pip install --install-option="--prefix=$HOME/<path_to_install>" .
    4. run the installation script install.sh

This worked in December 2016.

How to transfer files in bulk

If you need to transfer a large number of video files, we recommend this method:

cat current_queue.txt | parallel --no-notice -j 8 rsync -av --relative --stats --human-readable username@cartago.vrnewsscape.ucla.edu:/sweep/{} /target/directory/ > /log/directory/rsync_Nov_2016.log

The file current_queue.txt looks like this:

2016/2016-11/2016-11-30/2016-11-30_2300_US_WEWS_News_5_at_6pm.mp4

2016/2016-11/2016-11-30/2016-11-30_2300_US_KNBC_The_Ellen_DeGeneres_Show.mp4

2016/2016-11/2016-11-30/2016-11-30_2300_US_KABC_Eyewitness_News_3PM.mp4

2016/2016-11/2016-11-30/2016-11-30_2300_US_FOX-News_Special_Report_With_Bret_Baier.mp4

2016/2016-11/2016-11-30/2016-11-30_2300_US_CNN_Situation_Room.mp4

2016/2016-11/2016-11-30/2016-11-30_2200_US_CNN_Situation_Room.mp4

2016/2016-11/2016-11-30/2016-11-30_2100_US_KTTV-FOX_The_Dr_Oz_Show.mp4

2016/2016-11/2016-11-30/2016-11-30_2100_US_CNN_The_Lead_With_Jake_Tapper.mp4

2016/2016-11/2016-11-30/2016-11-30_2000_US_KNBC_Access_Hollywood_Live.mp4

2016/2016-11/2016-11-30/2016-11-30_2000_US_KCAL_KCAL_9_News_at_12PM.mp4