— How to set up a Red Hen capture station

If you are able to set up a capture station for television news, your language and region gets represented, and the Red Hen dataset improves for everyone.

The minimum requirement is that you have a good digital tv signal, either from an antenna or a cable feed, and a reliable connection to the Internet. A wireless connection may be adequate. (Of course, electrical power and a safe, controlled environment are also necessary.)

Related scrolls

Capture Station Manual

How to configure a satellite capture device
Brazil Capture Station (in production)
Changsha capture station (Raspberry Pi in production)
Germany capture stations (operational)
- Portugal Capture Station (Raspberry Pi in production)
Hong Kong Capture Station (pending)
The Ultimate Guide to Managing a Red Hen Capture Station

Basic models

There are three basic models for setting up a capture station, starting with the simplest and cheapest:

A Raspberry Pi—see instructions below.
An Intel NUC7CJYH (see Wikipedia) is powerful enough to compress files, simplifying file transfers over slow networks; €150
A rackmounted Linux for multiple tuners

Red Hen has standardized on HDHomeRun or JokerTV tuners. The entire capture process is controlled by our custom software.

Please contact Francis Steen or Mark Turner if you are interested.

Setting up a Raspberry Pi capture station

We've made it very cheap and simple to set up a Red Hen capture station on a Raspberry Pi. Here is what you do.

Requirements

There are two simple requirements for a capture station: you must have access to a good television signal and a stable and reasonably fast Internet connection.

Equipment

The following equipment is recommended:

Raspberry Pi 3B+ computer (/£30/€39) -- example
- case -- example
- power supply -- example. Be sure to get one for the country you are in. Alternatively, use a USB charger (for your country) and a USB cable. Red Hen often powers the Raspberry Pi through the USB hub on the external hard drive.
32GB microSD card (€14 / £9) -- example
USB flash drive; Red Hen writes logs to it so as to avoid excessive writing to the microSD card.
1TB or more external hard drive with external power (€80-120) -- example. But be sure to get one with a power adapter for your country. As of 2017-06-22, Red Hen routinely uses a 5TB Seagate Expansion Drive. 8TB is also good. The invaluable advantage of having a larger external hard drive is that capture can continue for a loooong time even if there are internet problems or processing problems, and then once connectivity is re-established, the Red Hen automatic scripts can do their job, or an actual person can access the station remotely and fix the problem, and then the Red Hen automatic scripts can finish the job. In the special case where data transfer via internet cannot be arranged, include a powered USB hub and connect external hard drives to it; in this way, mere USB-powered portable 1TB drives can be used; and these portable drives can be mailed or carried by some wandering Red Hen courier to a place where the data can be absorbed. Alternatively, the Seagate external drives often include a USB hub, which is quite useful.
The appropriate cables: an ethernet cable to connect the Raspberry Pi to the local area network (although it is possible for the Raspberry Pi to connect to the local area network via wifi), and another one to connect the HDhomerun, for example.

You will need two ethernet ports (or one ethernet port and an ethernet switch, so that you use two of the ethernet ports on the ethernet switch) and two ethernet cables, one to connect one ethernet port to the the tuner and one to connect the other ethernet port to the Raspberry Pi. To extend the life of your SD card, we also recommend getting a 16GB ultra fit USB stick (€9 / £7).

In addition, you will need a tuner / capture device appropriate for the television standard and signal where you are:

In Europe: HDHomeRun HDHR4-2DT (€122). Here is another example. But as always, get the tuner that works in the country you are in.
In Brazil: the new HDHomeRun model for Brazil, or PixelView Play TV USB SBTVD ($40)
In China: the JokerTV tuner ($100) -- it supports multiple standards and works in more or less every other country

For satellite capture: DVBSky-S960 V2 USB Box ($60)

Installing Raspbian

The first task is to install the Linux operating system, called Raspbian

If your laptop runs Windows, install the terminal program PuTTY
- Download SDformatter to your laptop and format the microSD card (not needed if you use Etcher):
- Insert the microSD card into the SD card adapter
- Insert the adapter into your laptop
  - Run SDformatter to format your microSD card (see video for detailed instructions)
- Download the Raspbian Stretch with Desktop image to your laptop and flash it to the microSD card:
  - On a Mac, use Etcher (better) or PiFiller to put the Raspbian image on it -- see details
  - On Windows, use Win32DiskImager
  - Enable headless login (full instructions on a separate page)
- Connect your laptop directly to the Raspberry Pi 3 with the first ethernet cable
- Insert the SD card into the Raspberry Pi 3 and connect the unit to power
- Open terminal on your laptop and type
- ping -c 1 raspberry.local
- ssh pi@raspberrypi.local
The password is raspberry
As soon as you connect, change the password for user pi by typing
password <new password>
Plug in the separately powered external drive and attach it to the RPi via one of the four USB ports
Plug the tv cable, the second ethernet cable, and the power supply into the HDHomeRun
Initiate a tunnel so that Jacek can access your new RPi and install the Red Hen software

Installing Red Hen

Installing the Red Hen system itself is best left to Jacek; here are the main steps
Become user root on the RPi by issuing
su -
Create a single partition on the hard drive with gdisk
Format the hard drive with the xfs filesystem and mount it at /mnt/HD1
Copy the current version of the Red Hen software to /nest
Edit /nest/cfg/redhen.config -- include wifi credentials if available
Run /nest/cfg/install.sh (this takes a couple of hours, with occasional questions)
Verify that the installation was successful:

Configure the capture device

See Capture Station Manual ; if you use a satellite signal, see How to configure a satellite capture device
Configure the HDHR4 by clicking on the HDHomeRun icon on the desktop and find your channels:

RPi desktop with HDHomeRun configuration

If you want automated scheduling, open FreeGuide and configure it for your location (optional; we now automate without Freeguide).

Identify the news programs to record and determine the teletext pages for subtitle captions.
Click on the globe icon next to the menu (a browser) and type in "what is my ip?"

Once you send us your IP address, we can access the system; if you're behind a firewall, we'll let you know how to set up a tunnel. We will complete the configuration with your help, ensuring that the Raspberry Pi can talk to the HDHR3 and start recording. Here are some of the steps:

Run find-tuners.sh and edit /tvspare/tuners/signal
Get the PID on the interesting channels, as described here, and add them to /tvspare/tuners/lineup
Add the local tv program grabber definition to the script /usr/local/bin/xmltv-download
Run the scheduler script /usr/local/bin/schedule on the chosen programs and update crontab
Add the new machine to the list of Red Hen capture stations at our HPC processing pipelines at UCLA and CWRU

Under the hood

If you'd like to learn more, see the gory details in Slick Capture and Portugal capture station.

Transferring data

Where the internet connection is fast enough, Red Hen transfers data directly and no further action is needed. If the connection is too slow to handle the transfer of uncompressed files, the best solution is to add a local server to handle the compression task; compressed files are typically around 5% of the size of uncompressed files, and therefore transfer much more easily.

If neither a fast internet connection nor a server to handle local compression is available, the fallback is to fill up a drive and swap it out for manual transport.

Add the new drive by plugging it into power and into a spare USB port on the RPi. Check dmesg for the logical device it attaches to -- typically /dev/sdb, but it could switch places with the existing drive. Use gdisk to partition it into one linux partition. First press x and then z to remove previous partitions, then start gdisk again and add a new partition by pressing n and then w to write. Next, add the file system and label it:

mkfs.xfs /dev/sdb1
xfs_admin -L HD2 /dev/sdb1
add it to /etc/fstab, copying the HD1 line and changing to HD2
mount /mnt/HD2
chown csa:csa /mnt/HD2 (as root)

We can either copy the recorded files to the new drive or switch recording to the new drive. It's generally a bad idea to move video files from one drive to another unless you really have to; the files are so large that they place extra strain on the drives that can break them. However, sometimes this is still the best option, and we can use a command like this (can be given as a one-liner):

day 2017-02-02

for d in {478..0} ; do D=/mnt/HD2`pwd` ; mkdir -p $D

 if [ ! -d $D ] ; then break ; fi

 for f in *mpg ; do rsync ${f%.*}.{mpg,txt,len} $D -av ; sleep 10 ; echo . ; done

 sleep 60 ; echo .. ; day + 1

done

-- where 2017-02-02 is the first day to copy, moving forward a day at a time to the present, 478 is the number of consecutive days to copy, and HD2 is the name of the drive we're transferring the data to. The job pauses ten seconds between video files and a minute between days to lower the strain on the drive. When the copy job is done, unmount and unplug the transfer drive.

If instead we replace the old drive with the new, we first need to create the directories we need, as user csa:

mkdir -p /mnt/HD2/tv /mnt/HD2/ts /mnt/HD2/xmltv /mnt/HD2/software /mnt/HD2/system/log /mnt/HD2/system/logs

We then need to change these symlinks -- first done by user root, but then chown to csa:csa:

ln -sf /mnt/HD2/tv /tv (root directory)
ln -sf /mnt/HD2/spool /mnt/spool
ln -sf /mnt/HD2/software /root/software

These links should be created by user csa:

ln -sf /mnt/HD2/tv /home/csa/tv
ln -sf /mnt/HD2/ts /home/csa/ts
ln -sf /mnt/HD2/system /home/csa/system
ln -sf /mnt/HD2/xmltv /home/csa/xmltv
ln -sf /mnt/HD2/xmltv /nest/xmltv
ln -sf /mnt/HD2/system/log /nest/log
ln -sf /mnt/HD2/system/logs /nest/tuners/logs

Once everything is switched over, the Red Hen software will record to the new drive. Unmount /mnt/HD1 and unplug it, and transport it to a location with a faster internet connection so that the data can be transferred.

If we need to read the XFS-formatted transfer drive on OS X, use fuse-xfs -- it's read-only and slow, but it works.

To check that the capture drive remains available, we use this spell on root's crontab:

# Attempt to remount the hard drive if it's failed

59 * * * * if [ ! -d /mnt/HD1/spool ] ; then shutdown -r now 'Hard drive detached' ; fi

If the capture drive has been moved to /mnt/HD2, make sure to change this cronjob too.

If the drive fails to mount, here are some tools:

smartctl -d usbsunplus -t short /dev/sda
smartctl -d usbsunplus -a /dev/sda
xfs_repair /dev/sda1
xfs_repair -L /dev/sda1

Setting up an Intel NUC

Intel NUC6CAYB and Intel NUC7CJYH specs -- Celeron amd64
Installing Debian on an Intel NUC7PJYH (the only difference is the CPU)
In tasksel, select SSH server; do not install a graphical user interface
To activate the NIC, install openresolv firmware-realtek
- Create the file /etc/systemd/network/dhcp.network with this content:
  - [Match] Name=en* [Network] DHCP=yes
- As user root, issue "systemctl restart systemd-networkd" -- the NIC is called enp2s0
To activate the wifi (Intel 8086:31dc), install wireless-tools firmware-iwlwifi
- modprobe -r iwlwifi ; modprobe iwlmvm
- dmesg should have something like this:
- iwlwifi 0000:00:0c.0: Detected Intel(R) Dual Band Wireless AC 9462, REV=0x318
- iwlwifi 0000:00:0c.0: enabling device (0000 -> 0002)
- iwlwifi 0000:00:0c.0: loaded firmware version 34.0.0 op_mode iwlmvm
- However, the firmware is not being loaded; we need to figure out why
- See How to configure wifi in Debian

Installing Red Hen

Copy /nest from ca:/home/svarog/system/nest/svarog/nest/ rather than from dola
Edit /nest/cfg/redhen.config -- include wifi credentials if available
Modify /nest/cfg/sources.list to use local Debian repositories
Use /nest/cfg/install-NUC.sh

The NUC can function as a capture machine (as in Croatia and Forschheim) or as a compression machine (as in France and India).

If it is used as a capture machine, see the section "Configure the capture device" above
if it is used as a text extraction and compression device

For compression NUCs, see kalpana's vishnu and corbac's corbeau (DVB text extraction). For capture NUCs, see sif and svarog.