— How to set up a Red Hen capture station

If you are able to set up a capture station for television news, your language and region gets represented, and the Red Hen dataset improves for everyone.

The minimum requirement is that you have a good digital tv signal, either from an antenna or a cable feed, and a reliable connection to the Internet. A wireless connection may be adequate. (Of course, electrical power and a safe, controlled environment are also necessary.)

Related scrolls

Basic models

There are now 5 basic models for setting up a capture station, starting with the simplest and cheapest:
  1. A Raspberry Pi—see instructions below.
  2. An Intel NUC BOXNUC6CAYH—powerful enough to compress files, simplifying file transfers over slow networks. Cost €150 + RAM.
  3. A Mac with an Elgato Hybrid USB stick running EyeTV and our custom software. Red Hen has Mac capture stations in Cleveland, Denmark, Germany, and Spain. The Electronic Program Guide (EPG) is typically available through the DVB signal, but if it not, we use xmltv and pass a command-line argument to EyeTV (cf. Troll).
    (See picture below: A TV cable connects from the wallbox behind the drawers to a USB Elgato Hybrid plugged into redhen1. Fiber-optic Internet2 connects from the wallbox to a switcher in the bottom drawer, and an ethernet CAT6 cable connects the switcher to redhen1’s ethernet port. The usual macmini electrical cord provides power.)
  4. A Mac with an HDHomeRun for your region running our custom software (and optionally EyeTV)
  5. A Linux server with multiple capture cards or HDHomeRun tuners, controlled by our custom software.
Please contact Francis Steen or Mark Turner if you are interested.

Setting up a Raspberry Pi capture station

We've made it very cheap and simple to set up a Red Hen capture station on a Raspberry Pi. Here is what you do.   

Requirements

There are two simple requirements for a capture station: you must have access to a good television signal and a stable and reasonably fast Internet connection. 

Equipment

The following equipment is recommended:
  • Raspberry Pi 3B+ computer (/£30/€39) -- example
    • case -- example
    • power supply -- example. Be sure to get one for the country you are in.  Alternatively, use a USB charger (for your country) and a USB cable. Red Hen often powers the Raspberry Pi through the USB hub on the external hard drive.
  • 32GB microSD card (€14 / £9) -- example
  • USB flash drive; Red Hen writes logs to it so as to avoid excessive writing to the microSD card.
  • 1TB or more external hard drive with external power (€80-120) -- example. But be sure to get one with a power adapter for your country. As of 2017-06-22, Red Hen routinely uses a 5TB Seagate Expansion Drive. 8TB is also good.  The invaluable advantage of having a larger external hard drive is that capture can continue for a loooong time even if there are internet problems or processing problems, and then once connectivity is re-established, the Red Hen automatic scripts can do their job, or an actual person can access the station remotely and fix the problem, and then the Red Hen automatic scripts can finish the job. In the special case where data transfer via internet cannot be arranged, include a powered USB hub and connect external hard drives to it; in this way, mere USB-powered portable 1TB drives can be used; and these portable drives can be mailed or carried by some wandering Red Hen courier to a place where the data can be absorbed. Alternatively, the Seagate external drives often include a USB hub, which is quite useful.
  • The appropriate cables: an ethernet cable to connect the Raspberry Pi to the local area network (although it is possible for the Raspberry Pi to connect to the local area network via wifi), and another one to connect the HDhomerun, for example.
You will need two ethernet ports (or one ethernet port and an ethernet switch, so that you use two of the ethernet ports on the ethernet switch) and two ethernet cables, one to connect one ethernet port to the the tuner and one to connect the other ethernet port to the Raspberry Pi. To extend the life of your SD card, we also recommend getting a 16GB ultra fit USB stick (9 / £7).

In addition, you will need a tuner / capture device appropriate for the television standard and signal where you are:

Installing Raspbian

  • The first task is to install the Linux operating system, called Raspbian
  • If your laptop runs Windows, install the terminal program PuTTY
  • Download SDformatter to your laptop and format the microSD card (not needed if you use Etcher):
    • Insert the microSD card into the SD card adapter
    • Insert the adapter into your laptop
    • Run SDformatter to format your microSD card (see video for detailed instructions)
  • Download the Raspbian Stretch with Desktop image to your laptop and flash it to the microSD card:
  • Connect your laptop directly to the Raspberry Pi 3 with the first ethernet cable
  • Insert the SD card into the Raspberry Pi 3 and connect the unit to power
  • Open terminal on your laptop and type
             ping -c 1 raspberry.local
             ssh pi@raspberrypi.local
  • The password is raspberry
  • As soon as you connect, change the password for user pi by typing
              password <new password>
  • Plug in the separately powered external drive and attach it to the RPi via one of the four USB ports
  • Plug the tv cable, the second ethernet cable, and the power supply into the HDHomeRun
  • Initiate a tunnel so that Jacek can access your new RPi and install the Red Hen software

Installing Red Hen

  • Installing the Red Hen system itself is best left to Jacek; here are the main steps
  • Become user root on the RPi by issuing
                 su -
  • Create a single partition on the hard drive with gdisk
  • Format the hard drive with the xfs filesystem and mount it at /mnt/HD1
  • Copy the current version of the Red Hen software to /nest
  • Edit /nest/cfg/redhen.config -- include wifi credentials if available
  • Run /nest/cfg/install.sh (this takes a couple of hours, with occasional questions)
  • Verify that the installation was successful:
    • In /etc, check hosts, hosts.allow, hostname, mailname, fstab, and motd
    • Check the symlinks to /mnt/HD1 in user csa and /nest/tuner
    • Check that fail2ban is running
    • Verify users csa and pi have sudo rights with password only

Configure the capture device

  • Click on the globe icon next to the menu (a browser) and type in "what is my ip?"

    What is my IP?
Once you send us your IP address, we can access the system; if you're behind a firewall, we'll let you know how to set up a tunnel. We will complete the configuration with your help, ensuring that the Raspberry Pi can talk to the HDHR3 and start recording. Here are some of the steps:
  • Run find-tuners.sh and edit /tvspare/tuners/signal
  • Get the PID on the interesting channels, as described here, and add them to /tvspare/tuners/lineup
  • Add the local tv program grabber definition to the script /usr/local/bin/xmltv-download
  • Run the scheduler script /usr/local/bin/schedule on the chosen programs and update crontab
  • Add the new machine to the list of Red Hen capture stations at our HPC processing pipelines at UCLA and CWRU

Under the hood

If you'd like to learn more, see the gory details in Slick Capture and Portugal capture station

Transferring data

Where the internet connection is fast enough, Red Hen transfers data directly and no further action is needed. If the connection is too slow to handle the transfer of uncompressed files, the best solution is to add a local server to handle the compression task; compressed files are typically around 5% of the size of uncompressed files, and therefore transfer much more easily. 

If neither a fast internet connection nor a server to handle local compression is available, the fallback is to fill up a drive and swap it out for manual transport. 

Add the new drive by plugging it into power and into a spare USB port on the RPi. Check dmesg for the logical device it attaches to -- typically /dev/sdb, but it could switch places with the existing drive. Use gdisk to partition it into one linux partition. First press x and then z to remove previous partitions, then start gdisk again and add a new partition by pressing n and then w to write. Next, add the file system and label it:

  • mkfs.xfs /dev/sdb1
  • xfs_admin -L HD2 /dev/sdb1
  • add it to /etc/fstab, copying the HD1 line and changing to HD2
  • mount /mnt/HD2
  • chown csa:csa /mnt/HD2 (as root)
We can either copy the recorded files to the new drive or switch recording to the new drive. It's generally a bad idea to move video files from one drive to another unless you really have to; the files are so large that they place extra strain on the drives that can break them. However, sometimes this is still the best option, and we can use a command like this (can be given as a one-liner):

day 2017-02-02
for d in {478..0} ; do D=/mnt/HD2`pwd` ; mkdir -p $D 
 if [ ! -d $D ] ; then break ; fi
 for f in *mpg ; do rsync ${f%.*}.{mpg,txt,len} $D -av ; sleep 10 ; echo . ; done
 sleep 60 ; echo .. ; day + 1
done

-- where 2017-02-02 is the first day to copy, moving forward a day at a time to the present, 478 is the number of consecutive days to copy, and HD2 is the name of the drive we're transferring the data to. The job pauses ten seconds between video files and a minute between days to lower the strain on the drive. When the copy job is done, unmount and unplug the transfer drive.

If instead we replace the old drive with the new, we first need to create the directories we need, as user csa:

  • mkdir -p /mnt/HD2/tv /mnt/HD2/ts /mnt/HD2/xmltv /mnt/HD2/software /mnt/HD2/system/log /mnt/HD2/system/logs

We then need to change these symlinks -- first done by user root, but then chown to csa:csa:

  • ln -sf /mnt/HD2/tv /tv (root directory)
  • ln -sf /mnt/HD2/spool /mnt/spool
  • ln -sf /mnt/HD2/software /root/software

These links should be created by user csa:

  • ln -sf /mnt/HD2/tv /home/csa/tv
  • ln -sf /mnt/HD2/ts /home/csa/ts
  • ln -sf /mnt/HD2/system /home/csa/system
  • ln -sf /mnt/HD2/xmltv /home/csa/xmltv
  • ln -sf /mnt/HD2/xmltv /nest/xmltv
  • ln -sf /mnt/HD2/system/log /nest/log
  • ln -sf /mnt/HD2/system/logs /nest/tuners/logs

Once everything is switched over, the Red Hen software will record to the new drive. Unmount /mnt/HD1 and unplug it, and transport it to a location with a faster internet connection so that the data can be transferred. 

If we need to read the XFS-formatted transfer drive on OS X, use fuse-xfs -- it's read-only and slow, but it works.

To check that the capture drive remains available, we use this spell on root's crontab:

  # Attempt to remount the hard drive if it's failed
  59 * * * *      if [ ! -d /mnt/HD1/spool ] ; then shutdown -r now 'Hard drive detached' ; fi

If the capture drive has been moved to /mnt/HD2, make sure to change this cronjob too.