Slick capture

Solved by Turner & Steen. We have now shown that a Raspberry Pi 2B with an HDHomeRun on its local network can capture a broadcast, extract its closed captions, and compress the .mpg file to the appropriate .mp4 file. Transferring the derived files to the archive using rsync etc. is trivial.

Capture of broadcasts is performed at UCLA using powerful Linux servers and HDHomeRun tuners. Since 2011, Red Hen has established a number of small, remote capture stations, using a Mac and an Elgato EyeTV Hybrid tuner. For a description of this design, see a How to set up a Red Hen capture station. For pictures of such an installation (in Cleveland), see redhen1. But that installation requires augmentation of the Mac operating system with many macport packages and some configuration at a distance.

Has the future arrived in the form of credit card computers running Linux, such as Raspberry Pi? These computers are very small and very inexpensive. It looks as if we should be able to purchase one, configure it with the right user accounts, load it with all the right Red Hen software, and then ship it to someone with instructions to give it power, ethernet, an external hard disk for storage, a tuner, and a TV broadcast feed. The machine would then start up and do its job with little further configuration (e.g., selection of broadcasts to record) and little local routine maintenance. It might even be that the complete system would be an RPi ($35), a 16 GB microSD card ($15),a 32 GB USB Flash Drive ($15), and an HDHomeRun Dual ($70 on ebay, $120 elsewhere). Add $15 for anything else nice to have but not necessary (e.g., a case, a wifi adaptor). The result is a complete system for $150-$200, not counting power, ethernet connection, or RTF cable carrying the TV broadcast.

Task

Design a capture station based on a credit-card Linux computer, to become the new standard for a miniature international capture station, replacing the Mac-based design. Can it record a broadcast (the transport stream), extract the closed captions, and compress the original file to a suitable .mp4 file? Let's assume for this task that if we can manually get it to achieve this performance, then we can script the performance as a separate task. Let's also assume for this task that scripting the movement of the output files around the Red Hen network is a minor task, for later, because it is something we have already done for all the remote capture stations.

Would you like to accomplish all or part of this task?

If so, write to

and we will try to connect you with a mentor.

Update July 2015. This page will now be used as the basis for establishing a Portugal capture station.

Some initial information

  1. Getting started with Raspberry Pi:

      1. 2016-01-07: UPDATE about ssh: As of the November 2016 release, Raspbian has the SSH server disabled by default. You will have to enable it manually. This is done using raspi-config: Enter sudo raspi-config in the terminal, first select Interfacing options, then navigate to ssh, press Enter and select Enable or disable ssh server. For headless setup, SSH can be enabled by placing a file named 'ssh', without any extension, onto the boot partition of the SD card.

      2. Update 2016-07-04:

        • If you format your SD card with SDformatter.app and use PiFiller to put the raspbian jessie image on it, and insert the card into a Raspberry Pi 3, then you can connect to the command line on the RPi 3 by connecting an ethernet cable from a Mac to the RPi, giving electrical power to the RPi, opening Terminal on the Mac. From the Mac command line, issue

        • ssh pi@raspberrypi.local

        • The password is raspberry.

        • Once you have connected to the command line on the RPi, configure everything, beginning by changing the password for user pi.

  1. Debian Linux on Raspberry Pi: https://wiki.debian.org/RaspberryPi

Turner Development Raspberry Pi 2 B

  1. Turner configured the following development machine:

    1. Equipment

      1. Raspberry Pi 2 B, 1GB RAM, 900MHz Quad Core CPU. http://www.amazon.com/gp/product/B00WULV84Y?psc=1&redirect=true&ref_=oh_aui_detailpage_o00_s01

      2. CanaKit Raspberry Pi Micro USB Power Supply / Adapter / Charger

      3. SanDisk Ultra 32GB UHS-I/Class 10 Micro SDHC Memory Card Up to 48MB/s With Adapter- SDSDQUAN-032G-G4A [Newest Version]

      4. Just because they came with the Raspberry Pi 2 B as freebies: Clear Transparent Case and 802.11n Wifi Adapter (RT5370 CHIP). These seem to be unnecessary for our project.

      5. Western Digitial My Book for Mac 1 TB USB 2.0 Desktop External Hard Drive, Model Number WDBAAG0010HCH-NESN. Note that it comes with its own independent power supply: 12 Volts, 1.5 Amps, 18 Watts. Such a large external drive cannot be powered through the RPi USB2 ports. (A clean hard disk formatted for Linux or Windows would have worked better.)

      6. HDHomeRun Dual (US version). This is the networked TV tuner.

      7. Pix:

    1. Formatting the microSD card and installing the operating system on it (using a MacBook Air):

      1. Insert SanDisk microSD into the adapter it came with. Check that the lock switch is off. Insert the adapter into the SDcard slot on MacBook Air. Format the microSD card using the SD Association's Formatting Tool from https://www.sdcard.org/downloads/formatter_4/eula_mac/

      2. Various possibilities for obtaining and installing the operating system

        1. Update 2016-10-16. We no longer make disk images of existing capture stations. Instead, download the latest Raspbian (currently Debian Buster) from https://www.raspberrypi.org/downloads/raspbian/ and install it on the formatted microSD card with something like Pi Filler from http://ivanx.com/raspberrypi/. Doing so will make your Raspberry Pi's system up to date. We then build up the additional Red Hen Software from /nest. For information about /nest, see https://sites.google.com/site/distributedlittleredhen/home/the-cognitive-core-research-topics-in-red-hen/the-barnyard/completed-barnyard-projects/re-design-the-process-for-installing-a-capture-station-convert-cwru-to-the-new-design . UPDATE 2015-06-23: If someone solves the task Scripted Slick Capture, then we could make an image of an SD card already containing not only the system but also the relevant Red Hen software and make that image available via internet. Accordingly, the local capture site could purchase all the equipment and receive the image to install on the SD card, meaning that there would be no need to ship anything physically from Red Hen to the remote capturing site.]

        2. Download NOOBS (offline and network install version) from https://www.raspberrypi.org/downloads/ . NOOBS is an easy operating system installer which includes Raspian [Debian Linux optimized for RPi] and a selection of operating systems which are then downloaded from the internet for installation. Follow instructions in the INSTRUCTIONS-readme.txt inside the downloaded and decompressed NOOBS folder.

        3. Update 2017-08-08. We now use dd on a Mac in preference to PiFiller. See instructions at http://www.runeaudio.com/documentation/quick-start/sd-card-setup-mac/. 2015-05-29. This is the method selected: Download Raspian from https://www.raspberrypi.org/downloads/. Turner downloaded 2015-05-05-raspbian-wheezy.zip. Decompress the .zip file into the .img file. If it is an .xz file, decompress with unxz. Download PiFiller and PiFinder from http://ivanx.com/raspberrypi/ and follow the instructions for PiFiller on that page. That is, just click on the PiFiller.app and do what it says.

        4. For later, here's a way to make an image of the microSD card: PiCopier (http://ivanx.com/raspberrypi/) offers to make a snapshot of an SD card—a kind of PiFiller in reverse.

    2. Connecting to the RPi: PiFinder (http://ivanx.com/raspberrypi/) offers to determine the IP address of your RPi on your network, so you can connect to it via a terminal window, and also to find the MAC address of the RPi, so it can be added to the router as a DHCP reservation, with the result that the RPi will be assigned the same IP address every time and you won't need to run PiFinder again.

    1. Configure raspbian with

      1. $ sudo raspi-config

      2. See http://elinux.org/RPi_raspi-config . Expand filesystem. Use Internationalization Options to change locale & timezone: e.g., for installation of redhen3rpi in Cleveland, Ohio, deselect en_GB.UTF-8 UTF-8 and select en_US.UTF8 UTF-8; set timezone to US ET. Use Advanced Options to set hostname (e.g. redhen3rpi). Reboot.

      3. Update. (First update existing distribution as below, then edit /etc/apt/sources.list to change to the distribution you want, e.g. deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi, then update by going selectively through the list again. )

        1. $ sudo apt-get update

        2. $ sudo apt-get upgrade

        3. $ sudo shutdown -r now

        4. [now log back in]

        5. $ sudo apt-get install ca-certificates

        6. $ sudo apt-get distupgrade

        7. $ sudo apt-get install git-core

        8. $ sudo apt-get install rpi-update

        9. $ sudo rpi-update

        10. $ sudo apt-get install eject

        11. $ sudo shutdown -r now

        12. [now log back in]

        13. Change default editor to vi

        14. $ sudo update-alternatives --set editor /usr/bin/vim.tiny

        15. Generate RSA keys. ssh-keygen -t rsa. Put them wherever needed on other systems.

        16. Optional: add wifi adapter, just because we happen to have one. Ours is labeled 802.11n. Plug it into a USB port on RPi. Follow instructions at https://www.raspberrypi.org/documentation/configuration/wireless/wireless-cli.md . Ours works, as we see from the existence of an inet address:

          1. mark@redhen3rpi ~ $ ifconfig wlan0

          2. wlan0 Link encap:Ethernet HWaddr 00:13:ef:c0:05:28

          3. inet addr:10.0.1.131 Bcast:10.0.1.255 Mask:255.255.255.0

    2. Can we add an external hard drive? Yes. Here is how we added a USB Verbatim 8GB thumbdrive. First, insert the thumb drive into a USB port on the RPi.

      1. See http://superuser.com/questions/662614/raspberry-pi-how-to-format-hdd and http://www.itechlounge.net/2012/01/linux-partition-and-format-external-hard-drive-as-ext3-filesystem/

      2. mark@redhen3rpi ~ $ dmesg | tail -20

      3. [31247.412404] usb 1-1.4: new high-speed USB device number 82 using dwc_otg

      4. [31247.517453] usb 1-1.4: New USB device found, idVendor=18a5, idProduct=0302

      5. [31247.517480] usb 1-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3

      6. [31247.517498] usb 1-1.4: Product: STORE N GO

      7. [31247.517514] usb 1-1.4: Manufacturer: Verbatim

      8. [31247.517531] usb 1-1.4: SerialNumber: 070141FE0CC22009

      9. [31247.518728] usb-storage 1-1.4:1.0: USB Mass Storage device detected

      10. [31247.519245] scsi host11: usb-storage 1-1.4:1.0

      11. [31248.575384] scsi 11:0:0:0: Direct-Access Verbatim STORE N GO 5.00 PQ: 0 ANSI: 6\

      12. [31248.576714] sd 11:0:0:0: Attached scsi generic sg0 type 0

      13. [31249.372751] sd 11:0:0:0: [sda] 15646720 512-byte logical blocks: (8.01 GB/7.46 GiB)

      14. [31249.373725] sd 11:0:0:0: [sda] Write Protect is off

      15. [31249.373750] sd 11:0:0:0: [sda] Mode Sense: 23 00 00 00

      16. [31249.374709] sd 11:0:0:0: [sda] No Caching mode page found

      17. [31249.374729] sd 11:0:0:0: [sda] Assuming drive cache: write through

      18. [31249.399695] sda: sda1

      19. [31249.404068] sd 11:0:0:0: [sda] Attached SCSI removable disk

      20. [UPDATE 2016-08-21 to the following section for formatting and mounting the hard disk: Now we use gdisk instead of fdisk. See full information in the yellow section at the bottom]

      21. mark@redhen3rpi ~ $ sudo fdisk -l

      22. Device Boot Start End Blocks Id System

      23. /dev/sda1 2 15646719 7823359 b W95 FAT32

        1. mark@redhen3rpi ~ $ sudo fdisk /dev/sda

        2. Command (m for help): d

        3. Selected partition 1

        4. Command (m for help): n

        5. Partition type:

        6. p primary (0 primary, 0 extended, 4 free)

        7. e extended

        8. Select (default p): p

        9. Partition number (1-4, default 1):

        10. Using default value 1

        11. First sector (2048-15646719, default 2048):

        12. Using default value 2048

        13. Last sector, +sectors or +size{K,M,G} (2048-15646719, default 15646719):

        14. Using default value 15646719

        15. Command (m for help): w

        16. The partition table has been altered!

        17. Calling ioctl() to re-read partition table.

        18. Syncing disks.

        19. [Now let's check that it's actually connected and with its partition:]

        20. mark@redhen3rpi ~ $ sudo fdisk -l

      24. Disk /dev/sda: 8011 MB, 8011120640 bytes

      25. 8 heads, 40 sectors/track, 48896 cylinders, total 15646720 sectors

      26. Units = sectors of 1 * 512 = 512 bytes

      27. Sector size (logical/physical): 512 bytes / 512 bytes

      28. I/O size (minimum/optimal): 512 bytes / 512 bytes

      29. Disk identifier: 0x00000000

      30. Device Boot Start End Blocks Id System

      31. /dev/sda1 2048 15646719 7822336 83 Linux

      32. [Now let's format it for Linux as ext4:]

      33. mark@redhen3rpi ~ $ sudo mkfs -t ext4 /dev/sda1

      34. mke2fs 1.42.5 (29-Jul-2012)

      35. Filesystem label=

      36. OS type: Linux

      37. Block size=4096 (log=2)

      38. Fragment size=4096 (log=2)

      39. Stride=0 blocks, Stripe width=0 blocks

      40. 489600 inodes, 1955584 blocks

      41. 97779 blocks (5.00%) reserved for the super user

      42. First data block=0

      43. Maximum filesystem blocks=2004877312

      44. 60 block groups

      45. 32768 blocks per group, 32768 fragments per group

      46. 8160 inodes per group

      47. Superblock backups stored on blocks:

      48. 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

      49. Allocating group tables: done

      50. Writing inode tables: done

      51. Creating journal (32768 blocks): done

      52. Writing superblocks and filesystem accounting information: done

      53. [Now let's mount it as V8usbstick]

      54. mark@redhen3rpi ~ $ cd /mnt

      55. mark@redhen3rpi /mnt $ sudo mkdir V8usbstick

      56. [sudo] password for mark:

      57. mark@redhen3rpi /mnt $ sudo mount /dev/sda1 /mnt/V8usbstick

      58. mark@redhen3rpi /mnt $ cd V8usbstick

      59. [Now let's check that we can use V8usbstick]

      60. mark@redhen3rpi /mnt/V8usbstick $ sudo touch hello

      61. mark@redhen3rpi /mnt/V8usbstick $ ls

      62. hello lost+found

      63. [OK, as we see, that worked:]

      64. mark@redhen3rpi /mnt/V8usbstick $ df

      65. Filesystem 1K-blocks Used Available Use% Mounted on

      66. /dev/sda1 7568312 17196 7143616 1% /mnt/V8usbstick

      67. [OK, now let's unmount & eject it:]

      68. mark@redhen3rpi ~ $ sudo umount /dev/sda1

      69. mark@redhen3rpi ~ $ sudo eject /dev/sda

    1. Can we add a large external hard drive? Yes. Here is how we added a USB2 Western Digital 1TB external hard disk. It is importat that the WD hard disk have its own independent power supply: the RPi cannot power such a drive through its USB ports.

      1. First, connect the hard drive to a USB port on the RPi.

      2. mark@redhen3rpi ~ $ dmesg | tail -30

        1. [ 2397.017950] usb 1-1.5: new high-speed USB device number 5 using dwc_otg

        2. [ 2397.119128] usb 1-1.5: New USB device found, idVendor=1058, idProduct=1110

        3. [ 2397.119154] usb 1-1.5: New USB device strings: Mfr=1, Product=2, SerialNumber=3

        4. [ 2397.119172] usb 1-1.5: Product: My Book 1110

        5. [ 2397.119189] usb 1-1.5: Manufacturer: Western Digital

        6. [ 2397.119206] usb 1-1.5: SerialNumber: 5743415A4130323333333034

        7. [ 2397.120394] usb-storage 1-1.5:1.0: USB Mass Storage device detected

        8. [ 2397.122856] scsi host0: usb-storage 1-1.5:1.0

        9. [ 2398.120384] scsi 0:0:0:0: Direct-Access WD My Book 1110 2018 PQ: 0 ANSI: 4

        10. [ 2398.124489] scsi 0:0:0:1: CD-ROM WD Virtual CD 1110 2018 PQ: 0 ANSI: 4

        11. [ 2398.124976] sd 0:0:0:0: [sda] 1952151552 512-byte logical blocks: (999 GB/930 GiB)

        12. [ 2398.128851] sd 0:0:0:0: [sda] Write Protect is off

        13. [ 2398.128900] sd 0:0:0:0: [sda] Mode Sense: 2b 00 10 08

        14. [ 2398.132293] sd 0:0:0:0: [sda] No Caching mode page found

        15. [ 2398.132332] sd 0:0:0:0: [sda] Assuming drive cache: write through

        16. [ 2398.139498] scsi 0:0:0:2: Enclosure WD SES Device 2018 PQ: 0 ANSI: 4

        17. [ 2398.158885] sd 0:0:0:0: Attached scsi generic sg0 type 0

        18. [ 2398.159276] scsi 0:0:0:1: Attached scsi generic sg1 type 5

        19. [ 2398.164808] sda: sda1

        20. [ 2398.170516] scsi 0:0:0:2: Attached scsi generic sg2 type 13

        21. [ 2398.176030] sr 0:0:0:1: [sr0] scsi3-mmc drive: 51x/51x caddy

        22. [ 2398.176114] cdrom: Uniform CD-ROM driver Revision: 3.20

        23. [ 2398.178107] sd 0:0:0:0: [sda] Attached SCSI disk

        24. [ 2398.179302] sr 0:0:0:1: Attached scsi CD-ROM sr0

        25. [Note that this drive notoriously contains a very small partition with a virtual CD for "WD Smartware" that is impervious to reformatting. We'll just ignore it. For the future, better to have a clean hard drive formatted for Linux or Windows, and then to reformat it for raspbian, as below.]

        26. mark@redhen3rpi ~ $ sudo fdisk -l

        27. Disk /dev/sda: 999.5 GB, 999501594624 bytes

        28. 255 heads, 63 sectors/track, 121515 cylinders, total 1952151552 sectors

        29. Units = sectors of 1 * 512 = 512 bytes

        30. Sector size (logical/physical): 512 bytes / 512 bytes

        31. I/O size (minimum/optimal): 512 bytes / 512 bytes

        32. Disk identifier: 0x00000000

        33. Device Boot Start End Blocks Id System

        34. /dev/sda1 2 1952151551 976075775 b W95 FAT32

        35. mark@redhen3rpi ~ $ sudo fdisk /dev/sda

        36. Command (m for help): d

        37. Selected partition 1

        38. Command (m for help): n

        39. Partition type:

        40. p primary (0 primary, 0 extended, 4 free)

        41. e extended

        42. Select (default p): p

        43. Partition number (1-4, default 1):

        44. Using default value 1

        45. First sector (2048-1952151551, default 2048):

        46. Using default value 2048

        47. Last sector, +sectors or +size{K,M,G} (2048-1952151551, default 1952151551):

        48. Using default value 1952151551

        49. Command (m for help): w

        50. The partition table has been altered!

        51. Calling ioctl() to re-read partition table.

        52. Syncing disks.

        53. mark@redhen3rpi /dev $ sudo fdisk -l

        54. Disk /dev/sda: 999.5 GB, 999501594624 bytes

        55. 208 heads, 36 sectors/track, 260704 cylinders, total 1952151552 sectors

        56. Units = sectors of 1 * 512 = 512 bytes

        57. Sector size (logical/physical): 512 bytes / 512 bytes

        58. I/O size (minimum/optimal): 512 bytes / 512 bytes

        59. Disk identifier: 0x00000000

        60. Device Boot Start End Blocks Id System

        61. /dev/sda1 2048 1952151551 976074752 83 Linux

        62. mark@redhen3rpi ~ $ sudo mkfs -t ext4 /dev/sda1

        63. mke2fs 1.42.5 (29-Jul-2012)

        64. Filesystem label=

        65. OS type: Linux

        66. Block size=4096 (log=2)

        67. Fragment size=4096 (log=2)

        68. Stride=0 blocks, Stripe width=0 blocks

        69. 61005824 inodes, 244018688 blocks

        70. 12200934 blocks (5.00%) reserved for the super user

        71. First data block=0

        72. Maximum filesystem blocks=0

        73. 7447 block groups

        74. 32768 blocks per group, 32768 fragments per group

        75. 8192 inodes per group

        76. Superblock backups stored on blocks:

        77. 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,

        78. 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,

        79. 102400000, 214990848

        80. Allocating group tables: done

        81. Writing inode tables: done

        82. Creating journal (32768 blocks): done

        83. Writing superblocks and filesystem accounting information: done

        84. mark@redhen3rpi ~ $ cd /mnt

        85. mark@redhen3rpi /mnt $ sudo mkdir HD1

        86. mark@redhen3rpi /mnt $ sudo mount /dev/sda1 /mnt/HD1

        87. mark@redhen3rpi /mnt $ cd HD1

        88. [Now let's check that we can use HD1]

        89. mark@redhen3rpi /mnt/HD1 $ sudo touch hello

        90. mark@redhen3rpi /mnt/HD1 $ ls

        91. hello lost+found

        92. [OK, as we see, that worked:]

        93. mark@redhen3rpi /mnt/HD1 $ df

        94. Filesystem 1K-blocks Used Available Use% Mounted on

        95. /dev/sda1 960628088 73364 911734604 1% /mnt/HD1

        96. We want all heavy work done on HD1 rather than on the microSD card. So make a symlink

        97. $ ln -s /mnt/HD1 HD1

        98. so that the directory HD1 shows up in the home directory of the user. Users should work on HD1 exclusively.

        99. For the future: consider the possibility of moving everything over to HD1 so as to avoid the corruption that can be induced by writing the microSD card: https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=44177

      3. Moved the RPi to a university network. It now has a static IP address. Remounted the external hard drive HD1 with sudo mount /dev/sda1 /mnt/HD1.

      4. Now there are two Red Hen devices in one drawer in Crawford Hall, Case Western Reserve University:

          1. On the left, redhen1, a macmini, with a USB extender connecting to an Elgato EyeTV Hybrid off-screen. On the right, redhen3rpi, a Raspberry Pi 2 B, with a USB connection to its external hard disk in the upper right. redhen3rpi and an HDHomeRun (in the drawer below) are on the same network.

      1. [UPDATE 2015-06-20: Debian Jessie is now available for direct installation, so this step should be simpler.] Interlude: Francis Steen ran the script needed to upgrade Debian Wheezy to Debian Jessie, installed hdhomerun-config, uuid, gpac, ffmpeg, and established the usual tunnel (with RedHen scripts both /usr/local/bin/tunneld and backup /usr/local/bin/tunnel) between redhen3rpi and the rest of the Red Hen network. A few notes:

      2. $ sudo apt-get update

      3. $ sudo apt-get install wajig

      4. $ sudo apt-get upgrade

      5. $ sudo apt-get dist-upgrade

      6. $ sudo vi /etc/apt/sources.list

      7. Should have:

      8. deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi

      9. $ sudo apt-get upgrade

      10. $ sudo shutdown -r now

      11. $ sudo apt-get install ca-certificates

      12. $ sudo apt-get dist-upgrade

      13. $ sudo apt-get install git-core

      14. $ sudo apt-get install rpi-update

      15. $ sudo rpi-update

      16. $ sudo shutdown -r now

      17. $ sudo apt-get install eject

      18. $ sudo apt-get install uuid

    1. [UPDATE 2015-06-20. ffmpeg can be installed pre-compiled from deb-multimedia. The

    2. /etc/apt/sources.list file, which specifies sources for updates, now has the lines:

    3. # RPI Sources.list 2015-06-20

    4. # Stable official Debian

    5. deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi

    6. # Uncomment line below then 'apt-get update' to enable 'apt-get source'

    7. #deb-src http://archive.raspbian.org/raspbian/ wheezy main contrib non-free rpi

    8. # Debian jessie security team

    9. deb http://security.debian.org jessie/updates main

    10. # Debian Multimedia (http://www.deb-multimedia.org/)

    11. deb http://www.deb-multimedia.org jessie main

    12. #deb http://www.deb-multimedia.org jessie-backport main

    13. #deb http://www.deb-multimedia.org stretch main

    14. #deb http://www.deb-multimedia.org sid main]

    15. Is RPi powerful enough to compress a captured broadcast (.ts, .mpg) to .mp4? Yes. Francis Steen has deployed the method used at UCLA (two-pass ffmpeg) on redhen3rpi on a file captured on redhen1, with good results.

        1. Note for future: handbrake does not seem to be part of Debian Jessie. Installing handbrake on Debian Jessie: http://linuxg.net/how-to-install-handbrake-0-9-9-on-debian-jessie-debian-sid-and-debian-wheezy/ .

      1. Is RPi powerful enough to extract closed-captions? Yes. First, install ccextractor by getting the source:

      2. ==============

      3. csa@redhen3rpi ~/software $ lynx http://ccextractor.sourceforge.net/download-ccextractor.html

      4. csa@redhen3rpi ~/software $ ls

      5. ccextractor.src.0.77.zip

      6. csa@redhen3rpi ~/software $ ls -l

      7. total 1264

      8. -rw-r--r-- 1 csa csa 1292599 Jun 20 19:27 ccextractor.src.0.77.zip

      9. csa@redhen3rpi ~/software $ unzip *p

      10. Archive: ccextractor.src.0.77.zip

      11. creating: ccextractor.0.77/

      12. [etc. etc. etc.]

      13. csa@redhen3rpi ~/software $ ls -l

      14. total 1268

      15. drwxr-xr-x 9 csa csa 4096 Jun 20 18:03 ccextractor.0.77

      16. -rw-r--r-- 1 csa csa 1292599 Jun 20 19:27 ccextractor.src.0.77.zip

      17. csa@redhen3rpi ~/software $ cd *77

      18. csa@redhen3rpi ~/software/ccextractor.0.77 $ ls

      19. Dictionary docs linux mac OpenBSD src windows

      20. csa@redhen3rpi ~/software/ccextractor.0.77 $ cd linux

      21. csa@redhen3rpi ~/software/ccextractor.0.77/linux $ ls -l

      22. total 12

      23. -rwxr-xr-x 1 csa csa 245 Jun 20 14:29 build

      24. -rwxr-xr-x 1 csa csa 248 Jun 20 18:02 builddebug

      25. -rw-r--r-- 1 csa csa 3448 Jun 20 14:29 Makefile

      26. csa@redhen3rpi ~/software/ccextractor.0.77/linux $ ./build

      27. csa@redhen3rpi ~/software/ccextractor.0.77/linux $ ls -l

      28. total 1620

      29. -rwxr-xr-x 1 csa csa 245 Jun 20 14:29 build

      30. -rwxr-xr-x 1 csa csa 248 Jun 20 18:02 builddebug

      31. -rwxr-xr-x 1 csa csa 1642552 Jun 20 19:30 ccextractor

      32. -rw-r--r-- 1 csa csa 3448 Jun 20 14:29 Makefile

      33. csa@redhen3rpi ~/software/ccextractor.0.77/linux $ mv ccextractor ccextractor-0.77

      34. csa@redhen3rpi ~/software/ccextractor.0.77/linux $ sudo cp cc* /usr/local/bin

      35. csa@redhen3rpi ~/software/ccextractor.0.77/linux $ cd /usr/local/bin

      36. csa@redhen3rpi /usr/local/bin $ ls -l

      37. total 1620

      38. -rwxr-xr-x 1 root staff 1642552 Jun 20 19:31 ccextractor-0.77

      39. drwxr-sr-x 6 root staff 4096 Jun 19 18:45 hdhomerun_config_gui

      40. drwxrwsr-x 2 root indiecity 4096 May 6 19:17 indiecity

      41. drwxr-sr-x 2 root staff 4096 Jun 19 20:06 libhdhomerun

      42. csa@redhen3rpi /usr/local/bin $ sudo ln -s ccextractor-0.77 ccextractor

      43. ==============

      44. Steen tweaked and ran cc-extract-rpi on an mpeg file. It works, except for one minor remaining issue: the timestamps are inaccurate. Perhaps a setting in CCextractor needs to be changed.

      45. Can RPi capture a broadcast? Yes.

        1. Establish an HDHomeRun. That is, plug an HDHomeRun into the same internet subnet to which the RPi is connected. [UPDATE 2015-06-20: hdhomerun_config is present in the Debian repository, so can be installed directly instead of compiled from source.] From http://www.silicondust.com/support/downloads/ , follow the instructions for Linux Downloads, i.e., using a text browser like lynx (sudo apt-get install lynx), download libhdhomerun (source) and HDHomeRun Config GTK (source). These are .tgz files. Extract them with

        2. $ tar -xvzf hdhomerun_config_gui_20150615.tgz

        3. and

        4. $ tar -xvzf libhdhomerun_20150615.tgz

        5. From the hdhomerun_config_gui directory, run:

        6. $ sudo ./configure

        7. $ sudo make

        8. $ sudo make install

        9. Then go looking for the HDHomeRun on the subnet:

        10. $ ./hdhomerun_config discover

        11. And hope the response is something like: hdhomerun device 103B659E found at 129.22.115.124.

        12. Now you know the device number of your HDHomeRun (103B659E) and its IP address (129.22.115.124).

        13. Then find the model of your HDHomeRun:

        14. $ ./hdhomerun_config 103B659E get /sys/model

        15. And hope the response is something like: hdhomerun3_atsc

        16. Then find the firmware version you are currently running:

        17. $ ./hdhomerun_config 103B659E get /sys/version

        18. And hope the response is something like: 20130328

        19. Then find the latest firmware for your model on the silicondust.com site and download it. It will have a name like hdhomerun3_atsc_firmware_20150604.bin once you download it.

        20. Then upgrade your firmware:

        21. $ ./hdhomerun_config 103B659E upgrade ~/HD1/hdhomerun3_atsc_firmware_20150604.bin

        22. and hope you see something like:

        23. uploading firmware...upgrading firmware...rebooting...upgrade complete - now running firmware 20150604

      46. Now configure the HDHomeRun using hdhomerun_config. There are 2 tuners in the HDHomeRun on redhen3rpi, numbered 0 and 1.

          1. Set a variable for DEVICE and a variable for TUNER

          2. $ DEVICE=103B659E TUNER=0

          3. Get or set the channelmap:

          4. $ ./hdhomerun_config $DEVICE get /tuner0/channelmap

          5. And hope the response is what you want to see, e.g.: us-cable

          6. Scan tuner 0 for channels:

            1. $ sudo ./hdhomerun_config $DEVICE scan /tuner$TUNER scan0us-cable.log

            2. And hope it finds some, e.g.:

            3. SCANNING: 69000000 (us-irc:4, us-cable:4)

            4. LOCK: qam256 (ss=93 snq=69 seq=100)

            5. PROGRAM 1: 4.1 WQHS

            6. PROGRAM 2: 4.2 WMFD

            7. PROGRAM 3: 4.3 CNN

            8. PROGRAM 4: 4.4 CNN HLN

            9. PROGRAM 5: 4.5 FOX New

            10. PROGRAM 6: 4.6 MSNBC

            11. PROGRAM 7: 4.7 CNBC

            12. PROGRAM 8: 4.8 CSPAN

          7. So, let's tune to us-cable channel 4:

          8. $ sudo ./hdhomerun_config $DEVICE set /tuner$TUNER/channel auto:4

          9. And check how we are doing:

          10. $ sudo ./hdhomerun_config $DEVICE get /tuner$TUNER/status

          11. We get the response: ch=auto:4 lock=qam256 ss=94 snq=0 seq=0 bps=38813728 pps=0

          12. Let's select program 3 (4.3); that's CNN, as we learned above:

          13. $ ./hdhomerun_config $DEVICE set /tuner$TUNER/program 3

          14. Now, let's start capturing CNN for a while

          15. $ ./hdhomerun_config $DEVICE save /tuner$TUNER ~/HD1/TestCapture2015-06-19.ts

          16. And we see (note that we stop the recording with control-C):

          17. .................................................n....t........^C

          18. -- Video statistics --

          19. 233303 packets received, 0 overflow errors, 1 network errors, 6 transport errors, 0 sequence errors

          20. Similar tests show that we can capture MSNBC, Fox, etc.

          21. And when we move the captured file to a Mac and look at it in VLC, we see the subtitles:

Fox News with subtitles

Update 2016-08-21: Using gdisk instead of fdisk:

csa@rusalka:~ $ sudo just install gdisk

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following NEW packages will be installed:

gdisk

0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.

Need to get 193 kB of archives.

After this operation, 799 kB of additional disk space will be used.

Get:1 http://mirror.ox.ac.uk/sites/archive.raspbian.org/archive/raspbian/ jessie/main gdisk armhf 0.8.10-2 [193 kB]

Fetched 193 kB in 0s (317 kB/s)

Selecting previously unselected package gdisk.

(Reading database ... 89584 files and directories currently installed.)

Preparing to unpack .../gdisk_0.8.10-2_armhf.deb ...

Unpacking gdisk (0.8.10-2) ...

Processing triggers for man-db (2.7.0.2-5) ...

Setting up gdisk (0.8.10-2) ...

csa@rusalka:~ $ sudo gdisk /dev/sda

GPT fdisk (gdisk) version 0.8.10

Partition table scan:

MBR: MBR only

BSD: not present

APM: not present

GPT: not present

***************************************************************

Found invalid GPT and valid MBR; converting MBR to GPT format

in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by

typing 'q' if you don't want to convert your MBR partitions

to GPT format!

***************************************************************

Warning! Secondary partition table overlaps the last partition by

33 blocks!

You will need to delete this partition or resize it in another utility.

Command (? for help): ?

b back up GPT data to a file

c change a partition's name

d delete a partition

i show detailed information on a partition

l list known partition types

n add a new partition

o create a new empty GUID partition table (GPT)

p print the partition table

q quit without saving changes

r recovery and transformation options (experts only)

s sort partitions

t change a partition's type code

v verify disk

w write table to disk and exit

x extra functionality (experts only)

? print this menu

Command (? for help): x

Expert command (? for help): ?

a set attributes

c change partition GUID

d display the sector alignment value

e relocate backup data structures to the end of the disk

g change disk GUID

h recompute CHS values in protective/hybrid MBR

i show detailed information on a partition

l set the sector alignment value

m return to main menu

n create a new protective MBR

o print protective MBR data

p print the partition table

q quit without saving changes

r recovery and transformation options (experts only)

s resize partition table

t transpose two partition table entries

u replicate partition table on new device

v verify disk

w write table to disk and exit

z zap (destroy) GPT data structures and exit

? print this menu

Expert command (? for help): z

About to wipe out GPT on /dev/sda. Proceed? (Y/N): y

GPT data structures destroyed! You may now partition the disk using fdisk or

other utilities.

Blank out MBR? (Y/N): y

csa@rusalka:~ $ sudo gdisk /dev/sda

GPT fdisk (gdisk) version 0.8.10

Partition table scan:

MBR: not present

BSD: not present

APM: not present

GPT: not present

Creating new GPT entries.

Command (? for help): n

Partition number (1-128, default 1):

First sector (34-3906963422, default = 2048) or {+-}size{KMGTP}:

Last sector (2048-3906963422, default = 3906963422) or {+-}size{KMGTP}:

Current type is 'Linux filesystem'

Hex code or GUID (L to show codes, Enter = 8300):

Changed type of partition to 'Linux filesystem'

Command (? for help): i

Using 1

Partition GUID code: 0FC63DAF-8483-4772-8E79-3D69D8477DE4 (Linux filesystem)

Partition unique GUID: 7C1DEEB3-DFDE-4502-ADFE-1DF9323DADCE

First sector: 2048 (at 1024.0 KiB)

Last sector: 3906963422 (at 1.8 TiB)

Partition size: 3906961375 sectors (1.8 TiB)

Attribute flags: 0000000000000000

Partition name: 'Linux filesystem'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING

PARTITIONS!!

Do you want to proceed? (Y/N): y

OK; writing new GUID partition table (GPT) to /dev/sda.

The operation has completed successfully.

csa@rusalka:~ $ mkfs.xfs /dev/sda1

bash: mkfs.xfs: command not found

csa@rusalka:~ $ sudo just install xfsprogs

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following extra packages will be installed:

libreadline5

Suggested packages:

xfsdump attr quota

The following NEW packages will be installed:

libreadline5 xfsprogs

0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.

Need to get 746 kB of archives.

After this operation, 3,453 kB of additional disk space will be used.

Do you want to continue? [Y/n]

Get:1 http://mirror.ox.ac.uk/sites/archive.raspbian.org/archive/raspbian/ jessie/main libreadline5 armhf 5.2+dfsg-2 [131 kB]

Get:2 http://mirror.ox.ac.uk/sites/archive.raspbian.org/archive/raspbian/ jessie/main xfsprogs armhf 3.2.1 [614 kB]

Fetched 746 kB in 1s (746 kB/s)

Selecting previously unselected package libreadline5:armhf.

(Reading database ... 89605 files and directories currently installed.)

Preparing to unpack .../libreadline5_5.2+dfsg-2_armhf.deb ...

Unpacking libreadline5:armhf (5.2+dfsg-2) ...

Selecting previously unselected package xfsprogs.

Preparing to unpack .../xfsprogs_3.2.1_armhf.deb ...

Unpacking xfsprogs (3.2.1) ...

Processing triggers for man-db (2.7.0.2-5) ...

Setting up libreadline5:armhf (5.2+dfsg-2) ...

Setting up xfsprogs (3.2.1) ...

Processing triggers for libc-bin (2.19-18+deb8u4) ...

csa@rusalka:~ $ sudo mkfs -t ext4 /dev/sda1

csa@rusalka:~ $ sudo mkfs.xfs /dev/sda1

mkfs.xfs: /dev/sda1 appears to contain an existing filesystem (ntfs).

mkfs.xfs: Use the -f option to force overwrite.

csa@rusalka:~ $ sudo mkfs.xfs /dev/sda1 -f

meta-data=/dev/sda1 isize=256 agcount=4, agsize=122092543 blks

= sectsz=512 attr=2, projid32bit=1

= crc=0 finobt=0

data = bsize=4096 blocks=488370171, imaxpct=5

= sunit=0 swidth=0 blks

naming =version 2 bsize=4096 ascii-ci=0 ftype=0

log =internal log bsize=4096 blocks=238461, version=2

= sectsz=512 sunit=0 blks, lazy-count=1

realtime =none extsz=4096 blocks=0, rtextents=0

csa@rusalka:~ $ sudo mount /dev/sda1 /mnt/HD1

csa@rusalka:~ $ mount /mnt/HD1

csa@rusalka:~ $ df -h

Filesystem Size Used Avail Use% Mounted on

/dev/root 29G 2.5G 25G 9% /

devtmpfs 459M 0 459M 0% /dev

tmpfs 463M 0 463M 0% /dev/shm

tmpfs 463M 6.4M 457M 2% /run

tmpfs 5.0M 4.0K 5.0M 1% /run/lock

tmpfs 463M 0 463M 0% /sys/fs/cgroup

/dev/mmcblk0p1 63M 21M 43M 33% /boot

tmpfs 93M 0 93M 0% /run/user/1000

tmpfs 93M 0 93M 0% /run/user/1001

/dev/sda1 1.9T 33M 1.9T 1% /mnt/HD1


Use blkid to get the PARTUUID. Edit /etc/fstab to include the hard drive. Use e2label to label the hard disk. Then issue mount -av to see that it is mounted. Here's an example from a different installation, where the partition is not /dev/sda1 but instead /dev/scd1:

$ sudo e2label /dev/sdc1 HDbig

$ sudo e2label /dev/sdc1

HDbig

$ mount -av

/proc : already mounted

/boot : already mounted

/ : ignored

/ : ignored

/mnt/HD1 : already mounted

/mnt/HDbig : already mounted