— How to install software on the Case HPC

Introduction

The high-performance computing cluster at Case Western Reserve University is used by Red Hen faculty and students for various processing pipelines.

Related resources

To confirm that a particular utility is already installed, try 'which':
$ which wget
/usr/bin/wget
$ which automake
/usr/bin/automake
$ which autoconf
/usr/bin/autoconf
$ which unzip
/usr/bin/unzip
Most common unix utilities are already present.
If your starting point is a list of Debian packages and you don't know the name of the binary:
  • if you have access to cartago, and the package (say, zlib1g-dev) is installed there, issue
               just list-files zlib1g-dev
    and then on the Case HPC look for the binaries (executables) with 'which', the library files (libz.so) in /usr/lib64/ or the header (zlib.h) files in /usr/include/.
  • or try guessing the names by stripping off version information -- e.g., zlib1g looks like just zlib.
The 'which' command will find only binaries; if you need development files, see below.

How to find and activate modules

A large number of packages are available as modules. To list the common available modules:
$ module avail
For an exhaustive list,
$ module spider
To load a module,
$ module load python/2.7.10
To list loaded modules,
$ module list
Currently Loaded Modules:
  1) intel/2015   2) openmpi/1.8.8   3) i/1.0.0   4) StdEnv   5) hdf5/1.8.15   6) python/2.7.10
A great deal of software is available either immediately or by loading a module.

How to determine if a package is present

If your starting point is a list of Debian packages, it may be helpful to do a Google search for the name of the Redhat rpm package. You may also want to know which version of Redhat you're installing for, so find the current version of the operating system installed on the Case HPC, issue
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.8 (Santiago)
Once you know the name of the package, do a search on the Case HPC to see if it's already installed:
$ p=python-devel
$ rpm -qa | grep $p
python-devel-2.6.6-66.el6_8.x86_64
Alternatively, use yum to search for packages:
$ yum search zlib libatlas3

How to install new software

Please document your installation process here.

Installing Gentle

From Daniel Balagué via Sanjaya Gajurel, tweaked by Pete Broadwell:
  1. git clone https://github.com/lowerquality/gentle.git
  2. module load gcc/4.7.3 ; module load ffmpeg/2.8.2 ; module load python/2.7.10
  3. export PYTHONPATH="${PYTHONPATH}:~/myPython/virtualenv-1.9/ENV/lib/python2.7/site-packages/ (place in ~/.bashrc)
  4. modify the Linux section of the install_deps.sh script to look like this (remove the other lines):

           if [[ "$OSTYPE" == "linux-gnu" ]]; then
                 pip install --install-option="--prefix=$HOME/<path_to_install>" .

  5. run the installation script install.sh
This worked in December 2016.

How to transfer files in bulk

If you need to transfer a large number of video files, we recommend this method:
   
cat current_queue.txt | parallel --no-notice -j 8 rsync -av --relative --stats --human-readable username@cartago.vrnewsscape.ucla.edu:/sweep/{} /target/directory/ > /log/directory/rsync_Nov_2016.log

The file current_queue.txt looks like this:

2016/2016-11/2016-11-30/2016-11-30_2300_US_WEWS_News_5_at_6pm.mp4
2016/2016-11/2016-11-30/2016-11-30_2300_US_KNBC_The_Ellen_DeGeneres_Show.mp4
2016/2016-11/2016-11-30/2016-11-30_2300_US_KABC_Eyewitness_News_3PM.mp4
2016/2016-11/2016-11-30/2016-11-30_2300_US_FOX-News_Special_Report_With_Bret_Baier.mp4
2016/2016-11/2016-11-30/2016-11-30_2300_US_CNN_Situation_Room.mp4
2016/2016-11/2016-11-30/2016-11-30_2200_US_CNN_Situation_Room.mp4
2016/2016-11/2016-11-30/2016-11-30_2100_US_KTTV-FOX_The_Dr_Oz_Show.mp4
2016/2016-11/2016-11-30/2016-11-30_2100_US_CNN_The_Lead_With_Jake_Tapper.mp4
2016/2016-11/2016-11-30/2016-11-30_2000_US_KNBC_Access_Hollywood_Live.mp4
2016/2016-11/2016-11-30/2016-11-30_2000_US_KCAL_KCAL_9_News_at_12PM.mp4