Portugal Capture Station

For the executive version, see How to set up a Red Hen capture station.

Related scrolls and resources

• Raspberry Pi on github

  • Raspberry Pi blog with March 2016 Raspbian update (adds Bluetooth support)

  • Raspberry Pi 3 specs and benchmarks (includes FM radio)

• How to set up a Red Hen capture station

Parsing tools

NLP tools

• Tokenization and Parsing of Portuguese

• NLP on Brazilian Portuguese

Install Debian Linux on a Raspberry Pi 2B

By José Fonseca, Polytechnic Institute of Guarda, Portugal

Raspbian installation

• 1. Equipment: Mac OS X Yosemite, Raspberry Pi 2 B, HDHomeRun Dual (European Version), . . .

  2. Following the setup procedures described under —Slick capture

1. Steps in progress now:

   1. Install Raspbian Jessie into a 32GB SanDisk Ultra SD card

      1. Download the latest version of Raspbian. Currently Jessie from https://www.raspberrypi.org/downloads/raspbian/The file is in https://downloads.raspberrypi.org/raspbian_latest

      2. Unzip the file from a Mac OS X terminal window

      3. $ unzip 2015-09-24-raspbian-jessie.zip

      4. Install The Raspbian in the SD card based on http://wiki.openelec.tv/index.php/HOW-TO:Installing_OpenELEC/Writing_The_Disk_Image#tab=Mac_OSX

         1. Insert the SD card into the Mac

         2. Find which disk number is the SD card

         3. $ diskutil list

         4. Found that the SD card is disk4. Unmount the SD card disk

         5. $ diskutil unmountDisk /dev/disk4

         6. Zero out the partition map of the SD card

         7. $ sudo dd if=/dev/zero of=/dev/rdisk4 bs=1024 count=1

         8. Write the Raspbian Wheezy disk image to the SD card

         9. $ sudo dd if=2015-09-24-raspbian-jessie.img of=/dev/rdisk4 bs=4m

         10. Ensure the changes are synced to the SD card before removing it

         11. $ sync

         12. Remove the SD card from the Mac computer

      5. Configure the Raspbian

• • • 1. Connect the Raspberry Pi to a HDMI TV, with an Ethernet cable connected to the LAN and a USB keyboard in one UBS port

      2. Upon boot, configure the following options in the Raspberry Pi Software Configuration Tool (raspi-config) that shows automatically in the TV

         1. #1 Expand Filesystem

         2. #3 Enable Boot to Desktop/Scratch (Desktop Log in as user pi at the graphical desktop)

         3. #4 Internationalisation Options (to choose Portugal as the location and portuguese keyboard)

         4. #8 Advanced Options A2 Hostname (fenix)

         5. #8 Advanced Options A0 Update

         6. Reboot the Raspberry Pi

1. 1. 6. Connect to the Raspberry Pi remotely, using ssh

• • • 1. Since I changed the Hostname from the Raspberry Pi Software Configuration Tool I can access it using the Hostname using a terminal in my Mac

      2. The default login username is "pi" and the default password is "raspberry". The default password was changed immediately

      3. I have installed in my Mac the XQuartz X11 so I can execute graphical programs from the terminal window in the Mac

      4. For that I use the -X or the -Y option of the ssh

      5. $ ssh 193.137.162.96 -Y -l pi

      6. password: <secret_pass>

      7. Install vlc for the experiments

      8. $ sudo apt-get install vlc

1. 1. 7. After boot, mount the external drive that was configured according to (https://sites.google.com/site/distributedlittleredhen/home/the-cognitive-core-research-topics-in-red-hen/the-barnyard/-slick-capture):

$ sudo mount /dev/sda1 /mnt/HD1

Options for extending the life of the SD card

See discussion at Advanced Users. It may be worth moving the system logs to a USB drive to spare the SD card.

If system logs are moved to USB, we may be able to physically set the little tab on the SD card to a locked position to prevent any changes, as a security measure; see Protect your Raspberry Pi SD card with a read-only file system.

Backup and restore

Backup of the SD Card, according to (https://www.raspberrypi.org/documentation/linux/filesystem/backup.md):

Or with a better view of the process, using pipe viewer:

-- where 30908350464 is the size of the SD card. This is implemented in a community-developed backup script, called sdbackup.sh on fenix, which must be run as user root:

See also MicroSD backup and Preparing a custom image below.

Restore of the backup, according to (https://www.raspberrypi.org/documentation/linux/filesystem/backup.md):

Before we can automate the capture process, we need to configure the operating system and install the Red Hen capture scripts.

Operating System configuration

By Francis Steen

A Red Hen capture station uses a customized environment of Debian Linux.

Debian repositories

Create the following list of repositories with the command "sudo nano /etc/apt/sources.list":

# RPI Sources.list 2015-06-20

# To sign keys --

# KEY=07DC563D1F41B907

# gpg --keyserver pgp.mit.edu --recv-keys $KEY

# gpg --keyserver subkeys.pgp.net --recv-keys $KEY

# gpg --armor --export $KEY | apt-key add -

# Raspbian Stable

#deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi

deb http://mirror.ox.ac.uk/sites/archive.raspbian.org/archive/raspbian/ jessie main contrib non-free rpi

# Uncomment line below then 'apt-get update' to enable 'apt-get source'

deb-src http://archive.raspbian.org/raspbian/ jessie main contrib non-free rpi

# Raspbian Testing

#deb http://mirrordirector.raspbian.org/raspbian/ stretch main contrib non-free rpi

deb http://mirror.ox.ac.uk/sites/archive.raspbian.org/archive/raspbian/ stretch main contrib non-free rpi

deb-src http://archive.raspbian.org/raspbian/ stretch main contrib non-free rpi

# Debian Multimedia (http://www.deb-multimedia.org/) -- unstable used for handbrake-cli 2015-11-24

deb http://www.deb-multimedia.org jessie main non-free

#deb http://www.deb-multimedia.org stretch main non-free

#deb http://www.deb-multimedia.org sid main non-free

Since we need a few packages from testing (currently stretch), let's set pinning values in /etc/apt/preferences:

# RPI Preferences 2015-11-24

# Pinning values: see https://wiki.debian.org/AptPreferences

# apt-get install xmltv/unstable -- only the package, not the dependencies

# apt-get -t unstable install xmltv -- also the dependencies

# apt-cache policy -- inspect priorities as set here

# This sets the priorities correctly for stable (jessie), but not for testing and unstable.

# The order is critical -- FIXME!

# Unstable

Package: *

Pin: release a=unstable

Pin-Priority: -1

# Testing

Package: *

Pin: release a=testing

Pin-Priority: 10

# Deb-multimedia

Package: *

Pin: origin www.deb-multimedia.org

Pin-Priority: 100

# Debian

Package: *

Pin: origin http.debian.net

Pin-Priority: 500

# Stable

Package: *

Pin: release a=stable

Pin-Priority: 700

Package installation

• apt-get update

  • apt-get purge wolfram-engine penguinspuzzle scratch dillo squeak-vm squeak-plugins-scratch sonic-pi idle idle3 (more)

• apt-get purge bluej greenfoot minecraft-pi claws-mail (and their desktop files in /usr/share/raspi-ui-overrides/applications)

  • apt-get purge samba-common samba-libs vlc-plugin-samba

  • apt-get upgrade

  • apt-get install wajig alpine exim4 screen uuid dos2unix bc whois nmap fail2ban ffmpeg dvb-apps w-scan autossh

• ln -s /usr/bin/wajig /usr/bin/just ; sudo ln -s /usr/bin/alpine /usr/bin/pine

  • just install moreutils hexedit mp4v2-utils gpac project-x x11vnc tightvncserver smartmontools lynx kpartx

  • apt-get -t stretch install xmltv (for freeguide—pulls in a bunch of perl 5.20 packages)

  • apt-get -t stretch install handbrake-cli (from marillat)

  • just reconfigure locales

  • just reconfigure exim4-config

For the upgrade to Raspberry Pi 3B, add

• • just update

• just install piclone geany usb-modeswitch pi-bluetooth

• just install python-pigpio python3-pigpio

Install a package from source

We cannot use packages directly from Debian, since they are compiled for ARM v7 and Raspbian is compiled for ARM v6 (Raspberry Pi 2B and 3B support both). However, in some cases, a package may be available in Debian that is not (yet) available in Raspbian. In that case, we can install the package from the Debian source package, as follows.

First make sure that the external drive is mounted so that files can be executed on it. In /etc/fstab, add "exec" after "user":

LABEL=HD1 /mnt/HD1 xfs auto,nofail,rw,user,exec,relatime 0 0

Unmount when the drive is idle and remount.

Second, install any dependencies the package needs. Issue

apt-get build-dep dvblast

In the case of dvblast, also install these dependencies:

just install libbitstream-dev/stretch libev-dev/stretch

Third, activate the Debian repository in /etc/apt/sources.list, for instance

# Debian experimental

deb http://ftp.debian.org/debian experimental main contrib non-free

deb-src http://ftp.debian.org/debian experimental main contrib non-free

It may be useful to include not only deb-src but also deb, to see which packages are available with "just available", for instance "just available dvblast". When you have added the repository, comment out all the other repositories to prevent any confusion about which version to download. Run "just update" as user root.

Then switch to user csa and issue:

md ~/software/dvblast

cd ~/software/dvblast

apt-get source dvblast

This will download and uncompress the dvblast package. Walk into the uncompressed package and issue

dpkg-buildpackage -rfakeroot -us -uc

This will build a Debian package of the application.

Switch back to user root, install the new package and copy it to any other RPi that needs it. Make sure you also revert your changes to /etc/apt/sources.list and run "just update".

Here is an example build log for dvblast, generating dvblast_3.1-1_armhf.deb:

csa@aton:~/software/dvblast/dvblast-3.1 $ dpkg-buildpackage -rfakeroot -us -uc

dpkg-buildpackage: source package dvblast

dpkg-buildpackage: source version 3.1-1

dpkg-buildpackage: source distribution experimental

dpkg-buildpackage: source changed by Sebastian Ramacher <sramacher@debian.org>

dpkg-buildpackage: host architecture armhf

dpkg-source --before-build dvblast-3.1

fakeroot debian/rules clean

dh clean

dh_testdir

dh_auto_clean

make -j1 distclean

make[1]: Entering directory '/mnt/HD1/software/dvblast/dvblast-3.1'

CLEAN dvblast dvblastctl dvblast.o util.o dvb.o udp.o asi.o demux.o output.o en50221.o comm.o mrtg-cnt.o asi-deltacast.o util.o dvblastctl.o

rm -f dvblast dvblastctl dvblast.o util.o dvb.o udp.o asi.o demux.o output.o en50221.o comm.o mrtg-cnt.o asi-deltacast.o util.o dvblastctl.o

make[1]: Leaving directory '/mnt/HD1/software/dvblast/dvblast-3.1'

dh_autoreconf_clean

dh_clean

dpkg-source -b dvblast-3.1

dpkg-source: info: using source format `3.0 (quilt)'

dpkg-source: info: building dvblast using existing ./dvblast_3.1.orig.tar.bz2

dpkg-source: info: building dvblast in dvblast_3.1-1.debian.tar.xz

dpkg-source: info: building dvblast in dvblast_3.1-1.dsc

debian/rules build

dh build

dh_testdir

dh_update_autotools_config

dh_autoreconf

dh_auto_configure

dh_auto_build

make -j1

make[1]: Entering directory '/mnt/HD1/software/dvblast/dvblast-3.1'

CC dvblast.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c dvblast.c

CC util.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c util.c

CC dvb.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c dvb.c

CC udp.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c udp.c

CC asi.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c asi.c

CC demux.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c demux.c

CC output.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c output.c

CC en50221.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c en50221.c

CC comm.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c comm.c

CC mrtg-cnt.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c mrtg-cnt.c

CC asi-deltacast.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c asi-deltacast.c

LINK dvblast

cc -Wl,-z,relro -o dvblast dvblast.o util.o dvb.o udp.o asi.o demux.o output.o en50221.o comm.o mrtg-cnt.o asi-deltacast.o -lpthread -lev -lrt

CC dvblastctl.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c dvblastctl.c

LINK dvblastctl

cc -Wl,-z,relro -o dvblastctl util.o dvblastctl.o -lrt

make[1]: Leaving directory '/mnt/HD1/software/dvblast/dvblast-3.1'

dh_auto_test

fakeroot debian/rules binary

dh binary

dh_testroot

dh_prep

dh_auto_install

make -j1 install DESTDIR=/mnt/HD1/software/dvblast/dvblast-3.1/debian/dvblast AM_UPDATE_INFO_DIR=no

make[1]: Entering directory '/mnt/HD1/software/dvblast/dvblast-3.1'

INSTALL dvblast.1 -> /mnt/HD1/software/dvblast/dvblast-3.1/debian/dvblast/usr/share/man/man1

install -m 644 dvblast.1 "/mnt/HD1/software/dvblast/dvblast-3.1/debian/dvblast/usr/share/man/man1"

INSTALL dvblast dvblastctl dvblast_mmi.sh -> /mnt/HD1/software/dvblast/dvblast-3.1/debian/dvblast/usr/bin

install dvblast dvblastctl dvblast_mmi.sh "/mnt/HD1/software/dvblast/dvblast-3.1/debian/dvblast/usr/bin"

make[1]: Leaving directory '/mnt/HD1/software/dvblast/dvblast-3.1'

dh_installdocs

debian/rules override_dh_installchangelogs

make[1]: Entering directory '/mnt/HD1/software/dvblast/dvblast-3.1'

dh_installchangelogs NEWS

make[1]: Leaving directory '/mnt/HD1/software/dvblast/dvblast-3.1'

dh_installman

dh_perl

dh_link

dh_strip_nondeterminism

dh_compress

dh_fixperms

dh_strip

dh_makeshlibs

dh_shlibdeps

dh_installdeb

dh_gencontrol

dh_md5sums

dh_builddeb

dpkg-deb: building package `dvblast' in `../dvblast_3.1-1_armhf.deb'.

dpkg-genchanges >../dvblast_3.1-1_armhf.changes

dpkg-genchanges: including full source code in upload

dpkg-source --after-build dvblast-3.1

dpkg-buildpackage: full upload (original source is included)

Configure locales and mta

Install the appropriate locales, including en_US.UTF-8, en_US, and pt_PT.UTF-8. Set default locale to en_US.UTF-8.

For the exim4 mail transport agent (mta), define a fully qualified domain name, e.g., fenix.universia.pt, in /etc/mailname. Also add this line to /etc/hosts (fully qualified domain name, used by alpine):

193.137.xxx.xx fenix.universia.pt fenix

Configure the timezone

Configure the local timezone with tzselect, if that wasn't done on installation—Red Hen capture stations should use the local timezone, not UTC. See detailed instructions. Create local user csa:csa and add this to ~/.profile:

# Time zone (set with tzselect) TZ='Europe/Lisbon'; export TZ

Since the recording system may be unattended for long periods, also set the system time zone in the OS:

root@fenix:~# dpkg-reconfigure tzdata

Current default time zone: 'Europe/Lisbon'

Local time is now: Tue Mar 29 16:59:29 WEST 2016.

Universal Time is now: Tue Mar 29 15:59:29 UTC 2016.

Raspbian used to use the network time protocol ntp daemon directly -- queries atomic clocks on the web:

dpkg-reconfigure ntp

However, with Raspbian stretch (Debian 9), time keeping is handled by systemd, so remove the separate ntp package if present:

apt purge ntp

By default, the configuration file /etc/systemd/timesyncd.conf lists standard networked time servers. If you have local time servers, add them first:

NTP=ntp1.ku.de ntp2.ku.de ntp3.ku.de

Load the systemd components:

systemctl daemon-reload

systemctl restart systemd-timesyncd.service

systemctl status systemd-timesyncd.service

systemd-timesyncd.service - Network Time Synchronization

Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)

Drop-In: /lib/systemd/system/systemd-timesyncd.service.d

└─disable-with-time-daemon.conf

Active: active (running) since Fri 2018-07-06 17:28:15 CEST; 33s ago

Docs: man:systemd-timesyncd.service(8)

Main PID: 1395 (systemd-timesyn)

Status: "Synchronized to time server 141.78.3.236:123 (ntp1.ku.de)."

CGroup: /system.slice/systemd-timesyncd.service

└─1395 /lib/systemd/systemd-timesyncd

Jul 06 17:28:14 odin.ku.de systemd[1]: Starting Network Time Synchronization...

Jul 06 17:28:15 odin.ku.de systemd[1]: Started Network Time Synchronization.

Jul 06 17:28:15 odin.ku.de systemd-timesyncd[1395]: Synchronized to time server 141.78.3.236:123 (ntp1.ku.de).

This will correct the time even if it's off by more than a day. In the unlikely event you need to manually set the time, this is the procedure:

systemctl stop systemd-timesyncd

systemctl disable systemd-timesyncd

timedatectl set-time "2018-07-06 17:24:41"

systemctl enable systemd-timesyncd

systemctl daemon-reload

systemctl restart systemd-timesyncd

Verify:

csa@fenix ~ $ timedatectl

Warning: Ignoring the TZ variable. Reading the system's time zone setting only.

Local time: Tue 2016-03-29 17:00:29 WEST

Universal time: Tue 2016-03-29 16:00:29 UTC

RTC time: n/a

Time zone: Europe/Lisbon (WEST, +0100)

NTP enabled: no

NTP synchronized: yes

RTC in local TZ: no

DST active: yes

Last DST change: DST began at

Sun 2016-03-27 00:59:59 WET

Sun 2016-03-27 02:00:00 WEST

Next DST change: DST ends (the clock jumps one hour backwards) at

Sun 2016-10-30 01:59:59 WEST

Sun 2016-10-30 01:00:00 WET

Note that Raspberry Pis do not have a hardware clock, so each time it is rebooted, it needs to read the time from the Internet.

NTP uses port 123 to communicate with time servers (more). Where this port is blocked, and there is no local time server available, add this line to root's crontab to set the time at some interval, such as once an hour:

date -s "$(wget -qSO- --max-redirect=0 google.com 2>&1 | grep Date: | cut -d' ' -f5-8)Z"

Alternatively, use dvbdate from dvb-tools to set the time (untested). This is the preferred solution, since it's not dependent on a live Internet connection.

Finally, make sure you restart the cron daemon, so that it will use the new timezone:

service cron restart

If you forget, the crontabs of the recording schedule will use the previous timezone.

dvb-dvbt-ts

Download and compile dvb-dvbt-ts, a perl utility for repairing television transport stream files:

just install libpod-coverage-perl libtest-pod-coverage-perl dh-make

wget http://search.cpan.org/CPAN/authors/id/S/SD/SDPRICE/Linux-DVB-DVBT-TS-0.08.tar.gz

tar zxvf *gz

mv Linux-DVB-DVBT-TS-0.08 dvb-dvbt-ts-0.08

cd dvb-dvbt-ts-0.08/clib/libmpeg2

wget https://raw.githubusercontent.com/Distrotech/libmpeg2/master/libmpeg2/motion_comp_arm_s.S (missing file)

nano Subdir.mk -- add a line for compiling motion_comp_arm.o: $(libmpeg2)/motion_comp_mmx.o \ $(libmpeg2)/motion_comp_arm.o \

cd ../.. perl Makefile.PL

dh_make -s -f ../Linux-DVB-DVBT-TS-0.08.tar.gz -e your@e-mail debuild -i -us -uc -b

Install the resulting .deb file. Copy /usr/bin/dvbt-ts-repair to /usr/local/bin/dvbt-ts-repair-no-info and remove these three lines (they cause a hang):

my %info = Linux::DVB::DVBT::TS::info($filename, { 'debug' => $dbg_ts, }) ; printf "Video duration: %02d:%02d:%02d\n", $info{'duration'}{'hh'}, $info{'duration'}{'mm'}, $info{'duration'}{'ss'} ;

Note that the scripts in /usr/local/bin may be calling dvbt-ts-repair rather than dvbt-ts-repair-no-info; if so create a symlink from dvbt-ts-repair to dvbt-ts-repair-no-info.

Other configurations

Create the file /etc/redhen.config with these local parameter values:

# Red Hen configuration file location=IPG Portugal hostname=fenix.universia.pt ip_address=193.137.xxx.xx timezone=Europe/Lisbon e-mail=csa

In /etc/lightdm/lightdm.conf, set

autologin-user=csa

and add this:

touch ~/.Xresources

Because fenix filters the VNC ports, we will need to use VNC tunnelling. If the Raspberry Pi is located behind a firewall that filters all ports, we will need to use ssh tunneling; a good solution is autossh (see instructions):

autossh -M 20000 -f -N cartago -R 1234:localhost:22 -C

For instructions, see Securing VNC with SSH.

Install and configure fail2ban (for details, see Configure fail2ban):

• • cp /etc/fail2ban/fail2ban.conf /etc/fail2ban/fail2ban.local

  • cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

  • add ignoreip = 127.0.0.1/8 164.67.171.0/24 164.67.183.179

  • service fail2ban stop

  • service fail2ban start

Turn off password login (/etc/ssh/sshd_config) to secure the system. Verify with nmap that no unnecessary ports are open; if you run VNC, shut it down between use. Unless you need an ssh tunnel, do not allow any outbound ssh connections. Change /etc/motd to a welcome message, say

Welcome to Red Hen Lab's Raspberry Pi 2B recording station at the Instituto Politécnico da Guarda in Portugal

The HDHomerun device we are using with fenix is HDHR4-2DT, model hdhomerun4_dvbt. We've installed both the drivers provided by Silicon Dust, and I also installed the native Linux driver dvbhdhomerun. This involves building an out-of-tree kernel module, dvb_hdhomerun. I discovered we are running the 4.1.13-v7+ kernel, not sure why or how we got such a recent kernel. I couldn't find the source, and then found rpi-update, which is used by developers to test new kernels and firmware. Before I realized this was not intended for regular users, I ran rpi-update, which may have updated the firmware and reinstalled the same 4.1.13-v7+ kernel. I still couldn't find the source package, but then found the headers. Installing it prompted for the installation of gcc-4.7, which reassuringly installed from raspbian jessie. The packages then installed and built the new module:

just install cmake libhdhomerun-dev dkms dh-systemd module-assistant git clone https://github.com/h0tw1r3/dvbhdhomerun cd dvbhdhomerun && dpkg-buildpackage -b just install dvbhdhomerun-dkms_0.0.16+nmu3_all.deb just install dvbhdhomerun-utils_0.0.16+nmu3_armhf.deb

The dvbhomerun provides drivers for the HDHomerun devices that are fully integrated into the Linux digital television framework. This means we can use the usual Linux tools such as w_scan and gnutv -- tools we've used for years at the capture stations at UCLA. Importantly, we may also be able to use tvheadend, which provides a web-based scheduling tool that connects with the xmltv EPG. If the driver fails, we can remove the packages and maybe downgrade the kernel.

Configure the GUI

Raspbian comes with an LXDE menu; to customize, use the menu editor and see Adding LXDE start menu entries and desktop shortcuts. Menu entries are defined in .desktop file located in /usr/share/applications/usr/share/applications; they can be added and edited. For Raspberry Pi 3B (see updates), we cloned the fenix image and added the Bluetooth plugin for the lxpanel taskbar by right-clicking on the taskbar and following the menu selections. You can then click on the Bluetooth icon in the taskbar, make the RPi discoverable, scan for devices, and pair them. Once paired, the devices should automatically reattach on reboot.

If the devices do not reattach on reboot, or if you want access to the GUI from another computer for any reason, you can run VNC on the RPi by issuing

vncserver -geometry 800x600 -depth 24 :1

On your laptop, to create a safe connection, forward the VNC port for screen 1:

ssh fenix.local -nNT -L 5991:localhost:5901 -l csa

You can then connect to the Raspberry Pi desktop with this command on a Mac:

open vnc://fenix.local:5901

or run any VNC client.

Set up the HDHomeRun

By Jose Fonseca, Polytechnic Institute of Guarda, Portugal

1. Connect the HDHomeRun to my LAN Switch where I also have the Raspberry Pi connected

   1. In my network the HDHomeRun has the hostname: HDHR-123008C7

2. Connect the HDHomeRun to the cable TV

3. Install and configure HDHomeRun, based on —Slick capture

• 1. Install the HDHomeRun into the Raspberry Pi

     1. The software is in this page https://www.silicondust.com/support/linux/ (updated 2022-05-07 -- the links below have likely also changed)

     2. From a terminal window connected to the Raspberry Pi download the file http://download.silicondust.com/hdhomerun/libhdhomerun_20150615.tgz

     3. $ wget http://download.silicondust.com/hdhomerun/libhdhomerun_20150615.tgz

     4. Extract it

     5. $ tar -xvzf libhdhomerun_20150615.tgz

     6. From a terminal window connected to the Raspberry Pi download the file http://download.silicondust.com/hdhomerun/hdhomerun_config_gui_20150615.tgz

     7. $ wget http://download.silicondust.com/hdhomerun/hdhomerun_config_gui_20150615.tgz

     8. extract it

     9. $ tar -xvzf hdhomerun_config_gui_20150615.tgz

     10. Install the libgtk library needed to install the HDHomeRun $ sudo apt-get install libgtk2.0-dev

     11. cd hdhomerun_config_gui

     12. sudo ./configure

     13. sudo make

     14. sudo make install

     15. cd ..

  2. Configure the HDHomeRun

• • 1. Search for the HDHomeRun on the subnet

    2. $ hdhomerun_config discover

    3. Response: hdhomerun device 123008C7 found at 192.168.1.31

    4. Find the model of your HDHomeRun

    5. $ hdhomerun_config 123008C7 get /sys/model

    6. Response: hdhomerun4_dvbt

    7. Find the firmware version of your HDHomeRun

    8. $ hdhomerun_config 123008C7 get /sys/version

    9. Response: 20150505

    10. Set a variable for DEVICE and a variable for TUNER

    11. $ DEVICE=123008C7 TUNER=0

    12. Get the channelmap

    13. $ hdhomerun_config $DEVICE get /tuner0/channelmap

    14. Response: eu-bcast

• 3. List channels

• • 1. Scan tuner 0 for channels

    2. $ hdhomerun_config $DEVICE scan /tuner$TUNER scan0eu-cable.log

    3. Response:

       1. SCANNING: 858000000 (eu-bcast:69)

       2. LOCK: none (ss=0 snq=0 seq=0)

       3. SCANNING: 850000000 (eu-bcast:68)

       4. LOCK: none (ss=0 snq=0 seq=0)

       5. SCANNING: 842000000 (eu-bcast:67)

       6. LOCK: none (ss=0 snq=0 seq=0)

       7. SCANNING: 834000000 (eu-bcast:66)

       8. LOCK: none (ss=0 snq=0 seq=0)

       9. SCANNING: 826000000 (eu-bcast:65)

       10. LOCK: none (ss=0 snq=0 seq=0)

       11. SCANNING: 818000000 (eu-bcast:64)

       12. LOCK: none (ss=0 snq=0 seq=0)

       13. SCANNING: 810000000 (eu-bcast:63)

       14. LOCK: none (ss=0 snq=0 seq=0)

       15. SCANNING: 802000000 (eu-bcast:62)

       16. LOCK: none (ss=0 snq=0 seq=0)

       17. SCANNING: 794000000 (eu-bcast:61)

       18. LOCK: none (ss=0 snq=0 seq=0)

       19. SCANNING: 786000000 (eu-bcast:60)

       20. LOCK: none (ss=0 snq=0 seq=0)

       21. SCANNING: 778000000 (eu-bcast:59)

       22. LOCK: none (ss=0 snq=0 seq=0)

       23. SCANNING: 770000000 (eu-bcast:58)

       24. LOCK: none (ss=0 snq=0 seq=0)

       25. SCANNING: 762000000 (eu-bcast:57)

       26. LOCK: none (ss=0 snq=0 seq=0)

       27. SCANNING: 754000000 (eu-bcast:56)

       28. LOCK: t8qam64 (ss=79 snq=77 seq=100)

       29. TSID: 0x044D

       30. PROGRAM 1101: 1 RTP 1

       31. PROGRAM 1102: 2 RTP 2

       32. PROGRAM 1103: 3 SIC

       33. PROGRAM 1104: 4 TVI

       34. PROGRAM 1107: 5 ARTV

       35. PROGRAM 1111: 8 HD

       36. SCANNING: 746000000 (eu-bcast:55)

       37. LOCK: none (ss=0 snq=0 seq=0)

       38. SCANNING: 738000000 (eu-bcast:54)

       39. LOCK: none (ss=0 snq=0 seq=0)

       40. SCANNING: 730000000 (eu-bcast:53)

       41. LOCK: none (ss=0 snq=0 seq=0)

       42. SCANNING: 722000000 (eu-bcast:52)

       43. LOCK: none (ss=0 snq=0 seq=0)

       44. SCANNING: 714000000 (eu-bcast:51)

       45. LOCK: none (ss=0 snq=0 seq=0)

       46. SCANNING: 706000000 (eu-bcast:50)

       47. LOCK: none (ss=0 snq=0 seq=0)

       48. SCANNING: 698000000 (eu-bcast:49)

       49. LOCK: none (ss=0 snq=0 seq=0)

       50. SCANNING: 690000000 (eu-bcast:48)

       51. LOCK: t8qam64 (ss=66 snq=62 seq=100)

       52. TSID: 0x044D

       53. PROGRAM 1101: 1 RTP 1

       54. PROGRAM 1102: 2 RTP 2

       55. PROGRAM 1103: 3 SIC

       56. PROGRAM 1104: 4 TVI

       57. PROGRAM 1107: 5 ARTV

       58. PROGRAM 1111: 8 HD

       59. SCANNING: 682000000 (eu-bcast:47)

       60. LOCK: none (ss=0 snq=0 seq=0)

       61. SCANNING: 674000000 (eu-bcast:46)

       62. LOCK: none (ss=0 snq=0 seq=0)

       63. SCANNING: 666000000 (eu-bcast:45)

       64. LOCK: none (ss=0 snq=0 seq=0)

       65. SCANNING: 658000000 (eu-bcast:44)

       66. LOCK: none (ss=0 snq=0 seq=0)

       67. SCANNING: 650000000 (eu-bcast:43)

       68. LOCK: none (ss=0 snq=0 seq=0)

       69. SCANNING: 642000000 (eu-bcast:42)

       70. LOCK: none (ss=0 snq=0 seq=0)

       71. SCANNING: 634000000 (eu-bcast:41)

       72. LOCK: none (ss=0 snq=0 seq=0)

       73. SCANNING: 626000000 (eu-bcast:40)

       74. LOCK: none (ss=0 snq=0 seq=0)

       75. SCANNING: 618000000 (eu-bcast:39)

       76. LOCK: none (ss=0 snq=0 seq=0)

       77. SCANNING: 610000000 (eu-bcast:38)

       78. LOCK: none (ss=0 snq=0 seq=0)

       79. SCANNING: 602000000 (eu-bcast:37)

       80. LOCK: none (ss=0 snq=0 seq=0)

       81. SCANNING: 594000000 (eu-bcast:36)

       82. LOCK: none (ss=0 snq=0 seq=0)

       83. SCANNING: 586000000 (eu-bcast:35)

       84. LOCK: none (ss=0 snq=0 seq=0)

       85. SCANNING: 578000000 (eu-bcast:34)

       86. LOCK: none (ss=0 snq=0 seq=0)

       87. SCANNING: 570000000 (eu-bcast:33)

       88. LOCK: none (ss=0 snq=0 seq=0)

       89. SCANNING: 562000000 (eu-bcast:32)

       90. LOCK: none (ss=0 snq=0 seq=0)

       91. SCANNING: 554000000 (eu-bcast:31)

       92. LOCK: none (ss=0 snq=0 seq=0)

       93. SCANNING: 546000000 (eu-bcast:30)

       94. LOCK: none (ss=0 snq=0 seq=0)

       95. SCANNING: 538000000 (eu-bcast:29)

       96. LOCK: none (ss=0 snq=0 seq=0)

       97. SCANNING: 530000000 (eu-bcast:28)

       98. LOCK: none (ss=0 snq=0 seq=0)

       99. SCANNING: 522000000 (eu-bcast:27)

       100. LOCK: none (ss=0 snq=0 seq=0)

       101. SCANNING: 514000000 (eu-bcast:26)

       102. LOCK: none (ss=0 snq=0 seq=0)

       103. SCANNING: 506000000 (eu-bcast:25)

       104. LOCK: none (ss=0 snq=0 seq=0)

       105. SCANNING: 498000000 (eu-bcast:24)

       106. LOCK: none (ss=0 snq=0 seq=0)

       107. SCANNING: 490000000 (eu-bcast:23)

       108. LOCK: none (ss=0 snq=0 seq=0)

       109. SCANNING: 482000000 (eu-bcast:22)

       110. LOCK: none (ss=0 snq=0 seq=0)

       111. SCANNING: 474000000 (eu-bcast:21)

       112. LOCK: none (ss=0 snq=0 seq=0)

       113. SCANNING: 226500000 (eu-bcast:12)

       114. LOCK: none (ss=0 snq=0 seq=0)

       115. SCANNING: 219500000 (eu-bcast:11)

       116. LOCK: none (ss=0 snq=0 seq=0)

       117. SCANNING: 212500000 (eu-bcast:10)

       118. LOCK: none (ss=0 snq=0 seq=0)

       119. SCANNING: 205500000 (eu-bcast:9)

       120. LOCK: none (ss=0 snq=0 seq=0)

       121. SCANNING: 198500000 (eu-bcast:8)

       122. LOCK: none (ss=0 snq=0 seq=0)

       123. SCANNING: 191500000 (eu-bcast:7)

       124. LOCK: none (ss=0 snq=0 seq=0)

       125. SCANNING: 184500000 (eu-bcast:6)

       126. LOCK: none (ss=0 snq=0 seq=0)

       127. SCANNING: 177500000 (eu-bcast:5)

       128. LOCK: none (ss=0 snq=0 seq=0)

• 4. Capture Portuguese channels

• • 1. $ hdhomerun_config $DEVICE set /tuner$TUNER/channel auto:56

• 5. View the status:

• • 1. $ hdhomerun_config $DEVICE get /tuner$TUNER/status

• 6. Response:

     2. 1. ch=auto:56 lock=t8qam64 ss=80 snq=72 seq=100 bps=19854304 pps=0

        2. From the channels list we can see that the list of portuguese channels is:

           • PROGRAM 1101: 1 RTP 1

           • PROGRAM 1102: 2 RTP 2

           • PROGRAM 1103: 3 SIC

           • PROGRAM 1104: 4 TVI

        3. To capture, for example RTP1, we need to set the tuner to program 1101:

        4. $ hdhomerun_config $DEVICE set /tuner$TUNER/program 1101

• • • • 1. Then start the capture:

        2. $ hdhomerun_config $DEVICE save /tuner$TUNER TestCaptureRTP1.ts

        3. Response:

        4. (stop the recording with control-C)

        5. ...................................................................................................................................^C

        6. -- Video statistics --

        7. 27791 packets received, 0 overflow errors, 0 network errors, 0 transport errors, 0 sequence errors

• 6. 2. 5. The file generated may be transferred to a remote linux based computer by using scp from a terminal in that computer:

        6. $ scp pi@193.137.162.96:"/home/pi/TestCaptureRTP1.ts" "/remote_computer_folder_path/"

TV capture configuration

by Francis Steen

I've added the Red Hen scripts and directories, as follows:

• • chown -R csa:csa /mnt/HD1

  • mkdir /mnt/HD1/tv ; ln -s /mnt/HD1/tv /tv # Root of the repository tree

  • mkdir -p /mnt/HD1/tvspare/tuners ; ln -s /mnt/HD1/tvspare /tvspare # Used for tuner allocation and log

  • mkdir /mnt/HD1/ts ; ln -s /mnt/HD1/ts /home/csa/ts # Used for transport stream file repair

I built ccextractor-0.78 and copied these scripts to /local/user/bin:

• -rwxr-xr-x 1 csa csa 5828 Dec 25 08:58 cc-extract-bin.sh

  • lrwxrwxrwx 1 root csa 25 Dec 6 21:30 ccextractor -> /usr/bin/ccextractor-0.78

  • -rwxr-xr-x 1 csa csa 12270 Dec 25 13:30 cc-extract-teletext.sh

  • -rwxr-xr-x 1 csa csa 1333 Dec 4 22:54 channel

  • -rwxr-xr-x 1 csa csa 7643 Dec 25 13:43 channel_hdhr_2015-12-05.sh

  • -rwxr-xr-x 1 csa csa 27 Dec 5 19:16 channel_hdhr.name

  • -rwxr-xr-x 1 csa csa 9769 Dec 9 19:22 check-cc-single.sh

  • -rwxr-xr-x 1 csa csa 4446 Dec 9 11:37 check-mpg-single.sh

  • lrwxrwxrwx 1 root staff 19 Nov 25 14:01 compress -> ts2mp4-single-01.sh

  • -rwxr-xr-x 1 csa csa 1687 Dec 5 20:14 find-tuners.sh

  • lrwxrwxrwx 1 csa csa 15 Dec 7 08:36 schedule -> scheduler-03.sh

  • -rwxr-xr-x 1 csa csa 4585 Dec 18 12:49 scheduler-03.sh

  • -rwxr-xr-x 1 csa csa 6075 Dec 12 23:01 sdbackup.sh

  • -rwxr-xr-x 1 csa csa 3490 Dec 9 08:55 select-tuner-native-locking.sh

  • -rwxr-xr-x 1 csa csa 14972 Dec 9 11:44 ts2mp4-single-01.sh

  • -rwxr-xr-x 1 csa csa 1554 Dec 6 20:06 tunnel

  • -rwxr-xr-x 1 csa csa 3626 Dec 5 00:34 tunneld

  • lrwxrwxrwx 1 root staff 20 Nov 29 08:15 xmltv-download -> xmltv-download-01.sh

  • -rwxr-xr-x 1 csa csa 3790 Dec 9 11:35 xmltv-download-01.sh

  • -rwxr-xr-x 1 csa csa 466 Dec 5 06:05 zinfo

I then ran "find-tuners.sh", which generated /tvspare/tuners/signal, which I copied to /tvspare/tuners/current. This list of tuners is used to assign the two tuners dynamically to each new recording process. The system is designed to handle a pool of HDHR3s shared between several recording machines and is overkill for the RPi, but should work fine—and will scale as needed.

The recording system uses crontabs for scheduling. The syntax is as follows (see channel -h):

channel 1, 30min, "Program name", 1

where channel is the name of the script, 1 is the RTP-1 program 1101, "Program name" you fill in, and 1 is just a marker for our online schedule validator, which we may or may not end up activating.

Since /usr/local/bin/channel just links to the real script, you can do this to see the details:

bash -xv channel_hdhr_2015-11-06.sh 1, 2min, TEST, 1

Files are recorded to /mnt/spool. Ideally, this would be an external harddrive, to remove some of the load on the solid-state drive. It could even be a 7200rpm externally powered drive that would be fast enough to handle two simultaneous recordings, though this is not of first importance.

A crontab entry might look like this:

#JUNE 2015

# CSPAN1 061615 TUE Donald Trump Presidential Campaign Announcement

1 3 17 6 * channel 58, 60min, "US Presidential Politics", 3, "Donald Trump Presidential Campaign Announcement"

I created a crontab for user csa with a single-run test example.

When it ran, it also generated an e-mail receipt:

Date: Fri, 06 Nov 2015 20:19:57 +0000

From: Cron Daemon <root@fenix>

To: csa@fenix

Subject: Cron <csa@fenix> channel 1, 10min, "TEST", 1, "Optional comment"

Fenix initiating capture on Sexta 06 Novembro 2015 at 20:10:01 with tuner 123008C7-0, duration 595 seconds

2015-11-06_2010_PT_RTP-1_TEST

Fenix completed recording on Sexta 06 Novembro 2015 at 20:19:57

-rw-r--r-- 1 csa csa 203M Nov 6 20:19 /mnt/spool/2015-11-06_2010_PT_RTP-1_TEST.mpg

-rw-r--r-- 1 csa csa 49 Nov 6 20:19 /tv/2015/2015-11/2015-11-06/2015-11-06_2010_PT_RTP-1_TEST.t

You can see these e-mails in the alpine mail reader. So far, everything seems to be working, so if you could schedule some actual news programs, that would be great.

The file that ends in .t is a header file, used to construct the eventual teletext file, but I've not set that up yet.

You can move into the directory tree /tv that contains the recorded files by typing

day

for today, or

day 1

for yesterday, and so on going backwards in time. The directories are created automatically.

Television schedules

The list of Portuguese channels is:

PROGRAM 1101: 1 RTP 1

PROGRAM 1102: 2 RTP 2

PROGRAM 1103: 3 SIC

PROGRAM 1104: 4 TVI

The programs are all on eu-bcast:56. Add this information to the parameter file /tvspare/tuners/lineup, along with the network ID (from the xmltv downloads in /tvspare/xmltv), the country, the language, and the most-used teletext page number:

# Television networks at this location

CHANNEL=1 NWK=RTP-1 CH=56 PN=1101 ID=5.nos.pt COUNTRY=PT LAN=POR TP=885

CHANNEL=2 NWK=RTP-2 CH=56 PN=1102 ID=3.nos.pt COUNTRY=PT LAN=POR TP=884

CHANNEL=3 NWK=SIC CH=56 PN=1103 ID=7.nos.pt COUNTRY=PT LAN=POR TP=888

CHANNEL=4 NWK=TVI CH=56 PN=1104 ID=8.nos.pt COUNTRY=PT LAN=POR TP=888

The lineup file is used as a lookup table by the various scripts that need this information.

The three main channels (RTP, public and SIC and TVI, private channels) broadcast their evening news at 8pm. Viewers shares are highest for SIC, then TVI and the least seen is RTP (http://binaries.cdn.impresa.pt/dealer/2246924/AUDIENCIAS-Abril+20144205681231226447556.pdf)

Here's the link to the schedules:

SIC - http://sicnoticias.sapo.pt/programas/jornaldanoite/

TVI - http://www.tvi.iol.pt/programa/jornal-das-8/53c6b3903004dc006243d0cf

RTP - http://www.rtp.pt/programa/tv/p31394

There is a show that might be of interest. Every week they choose a news topic and invite politicians, experts and other relevant players to discuss it. Prós & Contras: http://www.rtp.pt/play/p1772/Pros-e-Contras

According to Wikipedia's list of networks with teletext, these three networks all have teletext numbers:

Sociedade Independente de Comunicação: teletext page 888 for live captions—online teletext

Televisão Independente: unknown teletext for live captions—online teletext

Rádio e Televisão de Portugal: unknown teletext for live captions—online teletext

Automating the capture schedule

After completing the operating system and tv capture configurations, the recording schedule can be automated through crontab, as in this example for user pi:

00 20 * * * channel 3, 60min, "Jornal da Noite", 1, "National evening news"

The full crontab looks like this:

#### CRONTAB FOR CSA ON FENIX ########################################

#

#### PATH AND MAIL INFORMATION #######################################

#

# Folders to search for program files

PATH=/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# Email output -- use a comma between addresses

# Copies should be sent to the local pi user

MAILTO=csa

#

#### CHANNELS ########################################################

# Defined in the channel script (currently channel_hdhr_2015-11-06)

# 1 ) NWK=RTP-1 CH=56 PN=1101 ;;

# 2 ) NWK=RTP-2 CH=56 PN=1102 ;;

# 3 ) NWK=SIC CH=56 PN=1103 ;;

# 4 ) NWK=TVI CH=56 PN=1104 ;;

#### AD HOC SCHEDULE #################################################

# November 2015

# RTP-1 Test recording

10 20 06 11 * channel 1, 10min, "TEST", 1, "Optional comment"

19 21 06 11 * channel 2, 10min, "TEST", 2, "Optional comment"

08 00 07 11 * channel 3, 10min, "TEST", 2, "Optional comment"

#### REGULAR SCHEDULE #################################################

# 20:00 SIC Main national evening news

00 20 * * * channel 3, 60min, "Jornal da Noite", 1, "National evening news"

#### CRONTAB SYNTAX ###############################################

#

# ------------- minute (00 - 59)

# | ----------- hour (00 - 23)# | | --------- day of month (01 - 31)

# | | | ------- month (01 - 12)

# | | | | ----- day of week (1 - 7) (Sunday=7)

# | | | |

# * * * * * command to be executed

#

###################################################################

The script xmltv-download.sh daily fetches the broadcast schedule for the networks we're interested in. In honor of the new recording location, we wrote a new script, scheduler.sh, which examines the downloaded schedule for shows we record and automatically generates the crontab recording schedule, see Automatic scheduling.

For the entire process to be automated, we also need the scripts that extract the text, compress the video, and copy the completed files to NewsScape, as described in the sections below.

Text extraction

The Portuguese television transport stream typically contains a timestamped transcript of the news show. We use CCExtractor to extract these subtitles. For how to retrieve and compile CCExtractor, see Brazil Capture Station.

The list of the teletext page of the transcription of the main news programs is the following (see RTP teletext):

• • RTP1 - 885

  • RTP2 - 884

  • SIC - some shows use 888

    • TVI - some shows use 888

Note that different types of programs use different teletext pages -- in the case of RTP:

• Página 887 - RTP1 programs

• Página 888 - RTP2 programs

• Página 889 - RTP Internacional programs

• Página 886 - RTP1 captions of live football games

• Página 884 - RTP2 automatic captions of news shows

  • Página 885 - RTP1 automatic captions of news shows

To look for the embedded teletext, try these commands:

FIL=2015-11-07_0008_PT_SIC_TEST

ccextractor -out=txt $FIL.mpg -o $FIL.txt

You may get teletext, or no text but a block of suggested teletext pages to check:

Notice: Teletext page with possible subtitles detected: 888

Notice: Teletext page with possible subtitles detected: 886

Notice: Teletext page with possible subtitles detected: 883

Notice: Teletext page with possible subtitles detected: 882

Notice: Teletext page with possible subtitles detected: 881

Notice: Teletext page with possible subtitles detected: 880

Notice: Teletext page with possible subtitles detected: 867

Notice: Teletext page with possible subtitles detected: 866

To check each of these, issue this command on one line:

for TP in 888 886 883 882 881 880 867 866 ;

do echo -e "\n\tTrying teletext page $TP:\n" ;

ccextractor -tpage $TP -out=txt $FIL.mpg -o $FIL.txt ; sleep 1

done

Once you do find text, add the teletext page you identified to the cc-extract-teletext.sh script and run it:

cc-extract-teletext 2015-11-07_0008_PT_SIC_TEST.mpg

Our first results for SIC teletext page 888 are excellent, at least for the telenovella A Regra do Jogo:

20151107000800.480|20151107000806.040|888|:) - Está aqui a tua parte. - Nem sei como te agradecer.

20151107000806.240|20151107000809.880|888|É fruto do teu trabalho. Mereceste-o.

20151107000810.360|20151107000815.720|888|:) - Sabes a Dona Zezela ? - Sei.

20151107000816.160|20151107000821.520|888|- Gostou do ajuste no vestido. - Eu alonguei o vestido.

20151107000821.920|20151107000823.800|888|Vai indicar-te às amigas.

20151107000824.720|20151107000828.920|888|O dinheiro vem numa hora tão boa. O Juca precisa...

20151107000829.320|20151107000833.120|888|:-/ Esse dinheiro é para ti. Para ficares bonita.

20151107000835.920|20151107000840.080|888|:) Tenho um vestido que te ficará muito bem.

20151107000842.200|20151107000847.480|888|:) - Está aqui. Olha. - Não é muito ousado ?

20151107000847.760|20151107000852.400|888|(Indira) Experimenta-o. Dá-te cor e vida.

20151107000854.840|20151107000856.160|888|Quero ver.

20151107000858.080|20151107000900.480|888|:) Eu sabia que ficaria maravilhoso.

20151107000901.120|20151107000905.600|888|:) - Só tenho de tirar a etiqueta. :) - Olha os meus ombros.

20151107000905.840|20151107000910.560|888|:) Estás linda ! Vai para casa. Não aceito devoluções.

20151107000910.800|20151107000914.320|888|:) - É uma prenda ? (Indira) - Vai para casa.

20151107000914.520|20151107000919.800|888|:) Veste-te bem, arranja um jeitoso que te mereça e valorize.

20151107000923.440|20151107000927.840|888|:) - Indira, obrigada. :) - De nada.

20151107000928.640|20151107000933.240|888|:) - Dá uma volta. (Domingas) :) - Que vergonha.

20151107000934.360|20151107000938.320|888|(# música ritmada - funk Dream Team do Passinho - De Ladin)

20151107000938.560|20151107000947.440|888|(#)

20151107000948.000|20151107000951.280|888|:-O Tina ! Acabou-se a comida !

20151107000951.760|20151107000957.880|888|(#)

20151107000958.080|20151107001001.160|888|:) - Comi tudo. (Rui) :-s - Tina...

20151107001001.360|20151107001006.680|888|:) Esquece isso. Vem cá. Olha para a luz.

20151107001007.160|20151107001011.440|888|:-& Que luz ? Há roupa espalhada e louça para lavar.

20151107001011.840|20151107001016.520|888|- Lava tu a loiça. :-s - Eu ?

20151107001016.840|20151107001019.520|888|- Desenvencilha-te ! :-s - Tina, tu...

20151107001021.600|20151107001024.680|888|:-s - O que está a acontecer ? - Decidi ouvir-te.

20151107001024.960|20151107001028.640|888|:-s - A mim ? - Não disseste para sermos livres ?

20151107001030.320|20151107001035.680|888|:-& Já não precisas de ir trabalhar para o escritório.

20151107001035.880|20151107001040.920|888|:-& A Tina dona de casa acabou. A Tina morreu.

20151107001042.840|20151107001045.520|888|Agora sou a Tina do morro da macaca.

20151107001046.240|20151107001048.920|888|(Tina canta) É a Tina Furacão

20151107001049.160|20151107001052.880|888|:-& - Não estás nada bem. - Estou maravilhosa.

20151107001054.200|20151107001057.920|888|:-s - Estás a beber ? - Desde o meio-dia.

20151107001059.960|20151107001105.120|888|:) A vida de favelada é demais. Comprei torresmos.

20151107001106.880|20151107001109.440|888|:) - Rapaz ! :-& - Estás bêbada !

20151107001110.000|20151107001112.280|888|(Tina) :-& E tu estás chato !

20151107001113.560|20151107001116.880|888|:-& - Acabou a festa. - Está só a começar.

20151107001117.160|20151107001121.640|888|:-& - Onde vais ? - Vou sair à noite !

20151107001122.080|20151107001127.840|888|:-& Não tens a menor condição... Tina ! Tina !

20151107001133.400|20151107001137.440|888|(Domingas) :-& - Olá. - Tens um vestido novo ?

20151107001137.640|20151107001139.960|888|:-& - Gostaste ? - É parecido contigo.

20151107001142.120|20151107001145.640|888|Horroroso, foleiro, ridículo.

20151107001146.600|20151107001148.160|888|:-& Vai para o inferno.

20151107001150.520|20151107001154.600|888|:-/ - Como ? Queres apanhar ? :-& - Larga-me !

20151107001154.840|20151107001156.840|888|:-/ - Dá-me o dinheiro ! :-& - Não tenho dinheiro.

20151107001157.040|20151107001200.320|888|:-/ - Como compraste o vestido ? :-& - A Indira deu-mo.

20151107001200.520|20151107001203.720|888|:-/ - Porquê ? :-& - Porque tem pena de mim.

20151107001203.960|20151107001206.760|888|:) - Pena de ti ? :-& - Ela vê como me tratas.

20151107001206.960|20151107001212.480|888|:-& - Um dia denunciam-te. :-/ - Antes parto-te os dentes.

20151107001214.520|20151107001220.320|888|:-/ És uma ingrata. Devias agradecer por não te ter deixado.

20151107001220.960|20151107001224.600|888|:-/ Há tantas mulheres jeitosas e fico contigo.

20151107001225.160|20151107001228.760|888|(Juca) :-/ Esse cinto não combina com essa roupa.

20151107001228.960|20151107001230.520|888|:-/ O dinheiro está aqui ?

20151107001230.760|20151107001234.440|888|(Domingas) :( Para ! Não tenho dinheiro.

20151107001234.640|20151107001236.920|888|:( - Por favor... :-/ - O dinheiro ?

20151107001237.280|20151107001240.680|888|:( O dinheiro é para pagar as contas da casa.

20151107001241.600|20151107001244.960|888|:-/ Sou o homem da casa. O dinheiro é meu.

20151107001245.080|20151107001250.240|888|:( - Então devias ganhar dinheiro. :-/ - O que disseste ?

20151107001250.520|20151107001252.640|888|:-/ Repete !

20151107001257.400|20151107001300.440|888|:-/ Da próxima vez, mando-te para o hospital !

20151107001301.000|20151107001305.280|888|(Juca) :-/ Quando eu voltar, quero tudo arrumado !

20151107001306.800|20151107001310.920|888|:( Dá-me o meu dinheiro. Por favor, Juca.

20151107001311.440|20151107001315.480|888|(# música instrumental - triste) (Domingas chora)

20151107001315.640|20151107001328.880|888|(#) (---)

20151107001329.560|20151107001333.560|888|(# música ritmada - funk)

20151107001333.760|20151107001346.800|888|(Merlô canta) O que é isto ? Esta mulher é muito quente

20151107001347.040|20151107001351.440|888|(---) Ela roda e enrosca-se E faz de mim um varão de dança

20151107001351.640|20151107001354.840|888|(---) O que é isto ? Esta mulher é muito quente

20151107001355.040|20151107001359.000|888|(---) Ela roda e enrosca-se E faz de mim um varão de dança

20151107001359.240|20151107001402.040|888|(---) Ela é toda boa Gosta de provocar

20151107001402.280|20151107001405.600|888|(---) Adora fantasia Ela quer amarrar-me

20151107001405.800|20151107001409.440|888|(---) Não vai ao ginásio Diz que sou o seu treinador

20151107001409.680|20151107001412.480|888|(---) Que saúde Já me sinto mal

20151107001412.720|20151107001416.000|888|(---) Já te agachaste ? E já saltaste ?

20151107001416.200|20151107001417.640|888|(---) O que falta ?

20151107001417.760|20151107001421.560|888|(Alisson e Ninfa cantam) Espera que eu vou mostrar

20151107001421.720|20151107001424.400|888|(---) Fica quieto Não te movimentes

20151107001424.640|20151107001426.720|888|(---) Vou dançar no varão

20151107001426.920|20151107001430.120|888|(---) O que é isto ? Esta mulher é muito quente

20151107001430.360|20151107001434.640|888|(---) Ela roda e enrosca-se E faz de mim um varão de dança

20151107001436.320|20151107001440.120|888|(Rui) :-s Tina ! Tina, estás maluca ?

20151107001440.520|20151107001444.680|888|(Rui) :-& - Vamos embora ! (Tina) :) - Estou a adorar !

20151107001444.960|20151107001448.480|888|:-& - Que vergonha. :-/ - Deixa-me em paz !

20151107001448.880|20151107001452.040|888|:-& Estás louca ? Estás a expor-te !

20151107001452.640|20151107001457.400|888|:-& Quero que falem de mim na favela !

20151107001457.800|20151107001501.160|888|:-& - Vamos para casa ! :-& - Aquela barraca ?

20151107001502.200|20151107001505.720|888|:-& - Vamos embora ! :-& - Vem para o varão !

20151107001506.400|20151107001510.360|888|:-& - Chega ! - A barraca está uma confusão !

20151107001510.600|20151107001515.560|888|:-& Cheira mal e está nojenta ! Mas não há problema !

20151107001515.760|20151107001518.280|888|:-& Vamos ser felizes !

20151107001518.600|20151107001522.560|888|(# música ritmada - funk)

20151107001523.080|20151107001527.040|888|(Rui) :-s Tina ! Tina !

20151107001527.960|20151107001530.960|888|:) Aqui acontece sempre o inesperado !

20151107001531.200|20151107001532.840|888|:) É o melhor da cidade.

20151107001533.120|20151107001536.360|888|:) Temos camarão e champanhe do bom.

20151107001536.760|20151107001538.720|888|:) - És o Juca ? :) - Sim.

20151107001538.960|20151107001542.000|888|:) - O que fazes ? :) - Sou empresário.

20151107001542.200|20151107001547.760|888|:) Todos me conhecem. Nunca vi o teu sorriso na macaca.

20151107001548.360|20151107001550.920|888|:) - Sou de Madureira. - Que beleza.

20151107001553.160|20151107001557.800|888|:) Estou a pensar abrir um negócio em Madureira.

20151107001558.840|20151107001604.040|888|(Domingas) :-& É nesta mulher que gastas o meu dinheiro ?

20151107001604.440|20151107001608.400|888|(Juca) :-/ Estás louca ? Vai para casa !

20151107001609.000|20151107001614.760|888|(---) :-O Larguem-me ! Não me toquem !

20151107001617.920|20151107001621.760|888|(# música alegre - samba)

20151107001621.960|20151107001629.560|888|(Zé Maria) Aqueci o molho.

20151107001630.600|20151107001634.360|888|- O molho fica à parte. :) - Nossa Senhora.

20151107001634.560|20151107001638.440|888|Tem muitas ervas e vinho tinto.

20151107001639.200|20151107001643.080|888|:) - Trouxe pão fresco. (Djanira) :) - Surpresa !

20151107001643.320|20151107001647.320|888|(Djanira) :) - Vê quem veio ! :) - Tudo bem, Tóia ?

20151107001649.680|20151107001653.320|888|(Djanira) :) Ainda bem que trouxeste o pão fresco.

20151107001653.560|20151107001656.560|888|Tínhamos de fazer a sanduíche com pão velho.

20151107001657.600|20151107001701.880|888|:) Que beleza. Vai ficar muito bom.

20151107001702.120|20151107001704.400|888|Está lindo. Desculpa.

20151107001706.240|20151107001711.720|888|Antes do pernil que eu trouxe, quero dar-te esta prenda.

20151107001712.280|20151107001715.680|888|- É de coração, para ti. :-s - Para mim ?

20151107001716.560|20151107001720.200|888|:) Queria ter-te dado da outra vez que estive aqui.

20151107001720.400|20151107001722.560|888|(Zé Maria) :) Vê se gostas.

20151107001724.400|20151107001727.520|888|:) - É bonito. :) - Põe ao pescoço.

20151107001730.120|20151107001733.680|888|:) - Obrigada. (Zé Maria) :) - De nada.

20151107001734.800|20151107001737.880|888|Só penso que arrisca a vida ao vir aqui.

20151107001738.160|20151107001741.360|888|- Porquê ? - Está a ser perseguido.

20151107001741.800|20151107001744.760|888|A minha família vale o risco.

20151107001746.880|20151107001752.080|888|Da última vez que estive aqui, fui-me embora chateado.

The RTP-1 broadcasts Telejornal, its main evening news show, at 20:00, with excellent teletext captions. We now record this show on a regular schedule, tracked by the new schedule script.

Video compression

We currently transfer the uncompressed files to the Hoffman2 high-performance computing cluster at UCLA, which processes the files and sends them to the NewsScape search engines and archival servers. However, in some cases it may be necessary or desirable to perform the compression locally, either with software codecs or with the hardware codec built into the Raspberry Pi.

Video compression is done with HandBrake or ffmpeg. Get HandBrake with NEON support:

sudo apt-get update

sudo apt-get upgrade

sudo apt-get install gdebi-core

wget http://steinerdatenbank.de/software/ghb_0.10.5-1_armhf.deb

sudo gdebi ghb_0.10.5-1_armhf.deb

It's supposed to encode SD video faster than real time.

The current script uses ffmpeg, with HandBrake as a fallback if ffmpeg fails. The process is scripted:

bash -xv ts2mp4-single-01.sh 2015-11-25_0000_PT_SIC_A_Regra_do_Jogo.mpg ; date

On a Raspberry Pi 3B, such as odin, software compression takes around twice the length of the recording. This is fast enough to be a realistic option. On a Raspberry Pi 2B, such as redhen3rpi, software compression takes 4 hours and 39 minutes to processes a one-hour 981MB mpeg file; this is typically too slow to be useful.

The Raspberry Pi also has a hardware compression chip. To support it, build a custom version of ffmpeg:

./configure --enable-gpl --enable-libx264 --enable-nonfree --enable-mmal --enable-omx --enable-omx-rpi --enable-libfdk-aac

make -j4

sudo make install

ldconfig

You can then use the ts2mp4-single-02.sh script as above and encode a one-hour video in about 20 minutes. The quality is decent, but not as good as software-encoded files.

Search engine access

The files are now available to the UCLA campus and the Red Hen community of students and researchers. In the NewsScape search engine, the timestamped teletext is used as a search index and a navigational framework for the news videos: