Slick capture

Solved by Turner & Steen. We have now shown that a Raspberry Pi 2B with an HDHomeRun on its local network can capture a broadcast, extract its closed captions, and compress the .mpg file to the appropriate .mp4 file. Transferring the derived files to the archive using rsync etc. is trivial.

Capture of broadcasts is performed at UCLA using powerful Linux servers and HDHomeRun tuners. Since 2011, Red Hen has established a number of small, remote capture stations, using a Mac and an Elgato EyeTV Hybrid tuner. For a description of this design, see a How to set up a Red Hen capture station. For pictures of such an installation (in Cleveland), see redhen1. But that installation requires augmentation of the Mac operating system with many macport packages and some configuration at a distance.

Has the future arrived in the form of credit card computers running Linux, such as Raspberry Pi? These computers are very small and very inexpensive. It looks as if we should be able to purchase one, configure it with the right user accounts, load it with all the right Red Hen software, and then ship it to someone with instructions to give it power, ethernet, an external hard disk for storage, a tuner, and a TV broadcast feed. The machine would then start up and do its job with little further configuration (e.g., selection of broadcasts to record) and little local routine maintenance. It might even be that the complete system would be an RPi ($35), a 16 GB microSD card ($15),a 32 GB USB Flash Drive ($15), and an HDHomeRun Dual ($70 on ebay, $120 elsewhere). Add $15 for anything else nice to have but not necessary (e.g., a case, a wifi adaptor). The result is a complete system for $150-$200, not counting power, ethernet connection, or RTF cable carrying the TV broadcast.

Task

Design a capture station based on a credit-card Linux computer, to become the new standard for a miniature international capture station, replacing the Mac-based design. Can it record a broadcast (the transport stream), extract the closed captions, and compress the original file to a suitable .mp4 file? Let's assume for this task that if we can manually get it to achieve this performance, then we can script the performance as a separate task. Let's also assume for this task that scripting the movement of the output files around the Red Hen network is a minor task, for later, because it is something we have already done for all the remote capture stations.

Would you like to accomplish all or part of this task?

If so, write to

and we will try to connect you with a mentor.

Update July 2015. This page will now be used as the basis for establishing a Portugal capture station.

Some initial information

Getting started with Raspberry Pi:
1. 1. 2016-01-07: UPDATE about ssh: As of the November 2016 release, Raspbian has the SSH server disabled by default. You will have to enable it manually. This is done using raspi-config: Enter sudo raspi-config in the terminal, first select Interfacing options, then navigate to ssh, press Enter and select Enable or disable ssh server. For headless setup, SSH can be enabled by placing a file named 'ssh', without any extension, onto the boot partition of the SD card.
  2. Update 2016-07-04:
    - If you format your SD card with SDformatter.app and use PiFiller to put the raspbian jessie image on it, and insert the card into a Raspberry Pi 3, then you can connect to the command line on the RPi 3 by connecting an ethernet cable from a Mac to the RPi, giving electrical power to the RPi, opening Terminal on the Mac. From the Mac command line, issue
    - ssh pi@raspberrypi.local
    - The password is raspberry.
    - Once you have connected to the command line on the RPi, configure everything, beginning by changing the password for user pi.

- - 1. http://www.howtogeek.com/138281/the-htg-guide-to-getting-started-with-raspberry-pi/

1. How to configure Raspberry Pi for Remote Shell, Desktop, and File Transfer

Debian Linux on Raspberry Pi: https://wiki.debian.org/RaspberryPi

1. MicroSD card for Raspberry Pi 2 B: http://elinux.org/RPi_SD_cards
2. (A mere frill: Wifi. How to add wifi to the Raspberry Pi: http://raspberrypihq.com/how-to-add-wifi-to-the-raspberry-pi/ . The instructions on that site work for a private network, but to connect to Wifi with WPA2 Enterprise (e.g., at a university), see http://netbeez.net/2014/10/14/connect-your-raspberry-pi-to-wireless-enterprise-environments-with-wpa-supplicant/ . Given Pi Finder (see below), this is not necessary at all, and certainly not after configuration and installation as capture station, but it might be convenient in some environments during setup. Possible wifi USB adapters:
3. http://elinux.org/RPi_USB_Wi-Fi_Adapters
  1. http://thepihut.com/products/usb-wifi-adapter-for-the-raspberry-pi
  2. http://www.amazon.com/gp/product/B003MTTJOY/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=B003MTTJOY&linkCode=as2&tag=rapihq-20&linkId=7QAXDGG72H4EWYFB )
4. Developer machine: The ODROID-C2 is a candidate if you need a Linux server for development; it's similar to the RPi 3B but has a faster CPU, more RAM, and hardware decoding of h264, so better for playing videos.

Turner Development Raspberry Pi 2 B

Turner configured the following development machine:
1. Equipment
  1. Raspberry Pi 2 B, 1GB RAM, 900MHz Quad Core CPU. http://www.amazon.com/gp/product/B00WULV84Y?psc=1&redirect=true&ref_=oh_aui_detailpage_o00_s01
  2. CanaKit Raspberry Pi Micro USB Power Supply / Adapter / Charger
  3. SanDisk Ultra 32GB UHS-I/Class 10 Micro SDHC Memory Card Up to 48MB/s With Adapter- SDSDQUAN-032G-G4A [Newest Version]
  4. Just because they came with the Raspberry Pi 2 B as freebies: Clear Transparent Case and 802.11n Wifi Adapter (RT5370 CHIP). These seem to be unnecessary for our project.
  5. Western Digitial My Book for Mac 1 TB USB 2.0 Desktop External Hard Drive, Model Number WDBAAG0010HCH-NESN. Note that it comes with its own independent power supply: 12 Volts, 1.5 Amps, 18 Watts. Such a large external drive cannot be powered through the RPi USB2 ports. (A clean hard disk formatted for Linux or Windows would have worked better.)
  6. HDHomeRun Dual (US version). This is the networked TV tuner.
  7. Pix:

1. Formatting the microSD card and installing the operating system on it (using a MacBook Air):
  1. Insert SanDisk microSD into the adapter it came with. Check that the lock switch is off. Insert the adapter into the SDcard slot on MacBook Air. Format the microSD card using the SD Association's Formatting Tool from https://www.sdcard.org/downloads/formatter_4/eula_mac/
  2. Various possibilities for obtaining and installing the operating system
    1. Update 2016-10-16. We no longer make disk images of existing capture stations. Instead, download the latest Raspbian (currently Debian Buster) from https://www.raspberrypi.org/downloads/raspbian/ and install it on the formatted microSD card with something like Pi Filler from http://ivanx.com/raspberrypi/. Doing so will make your Raspberry Pi's system up to date. We then build up the additional Red Hen Software from /nest. For information about /nest, see https://sites.google.com/site/distributedlittleredhen/home/the-cognitive-core-research-topics-in-red-hen/the-barnyard/completed-barnyard-projects/re-design-the-process-for-installing-a-capture-station-convert-cwru-to-the-new-design . UPDATE 2015-06-23: ~~If someone solves the task~~ ~~Scripted Slick Capture~~, then we could make an image of an SD card already containing not only the system but also the relevant Red Hen software and make that image available via internet. Accordingly, the local capture site could purchase all the equipment and receive the image to install on the SD card, meaning that there would be no need to ship anything physically from Red Hen to the remote capturing site.]
    2. Download NOOBS (offline and network install version) from https://www.raspberrypi.org/downloads/ . NOOBS is an easy operating system installer which includes Raspian [Debian Linux optimized for RPi] and a selection of operating systems which are then downloaded from the internet for installation. Follow instructions in the INSTRUCTIONS-readme.txt inside the downloaded and decompressed NOOBS folder.
    3. Update 2017-08-08. We now use dd on a Mac in preference to PiFiller. See instructions at http://www.runeaudio.com/documentation/quick-start/sd-card-setup-mac/. 2015-05-29. This is the method selected: Download Raspian from https://www.raspberrypi.org/downloads/. Turner downloaded 2015-05-05-raspbian-wheezy.zip. Decompress the .zip file into the .img file. If it is an .xz file, decompress with unxz. ~~Download PiFiller and PiFinder from~~ ~~http://ivanx.com/raspberrypi/~~ ~~and follow the instructions for PiFiller on that page. That is, just click on the PiFiller.app and do what it says.~~
    4. ~~For later, here's a way to make an image of the microSD card: PiCopier (http://ivanx.com/raspberrypi/) offers to make a snapshot of an SD card—a kind of PiFiller in reverse.~~
2. Connecting to the RPi: PiFinder (http://ivanx.com/raspberrypi/) offers to determine the IP address of your RPi on your network, so you can connect to it via a terminal window, and also to find the MAC address of the RPi, so it can be added to the router as a DHCP reservation, with the result that the RPi will be assigned the same IP address every time and you won't need to run PiFinder again.

1. Configuring users. User management in Raspbian is done on the command line. The default user is pi with the password raspberry. You can add users and change each user's password. See https://www.raspberrypi.org/documentation/linux/usage/users.md

1. Configure raspbian with
  1. $ sudo raspi-config
  2. See http://elinux.org/RPi_raspi-config . Expand filesystem. Use Internationalization Options to change locale & timezone: e.g., for installation of redhen3rpi in Cleveland, Ohio, deselect en_GB.UTF-8 UTF-8 and select en_US.UTF8 UTF-8; set timezone to US ET. Use Advanced Options to set hostname (e.g. redhen3rpi). Reboot.
  3. Update. (First update existing distribution as below, then edit /etc/apt/sources.list to change to the distribution you want, e.g. deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi, then update by going selectively through the list again. )
    1. $ sudo apt-get update
    2. $ sudo apt-get upgrade
    3. $ sudo shutdown -r now
    4. [now log back in]
    5. $ sudo apt-get install ca-certificates
    6. $ sudo apt-get distupgrade
    7. $ sudo apt-get install git-core
    8. $ sudo apt-get install rpi-update
    9. $ sudo rpi-update
    10. $ sudo apt-get install eject
    11. $ sudo shutdown -r now
    12. [now log back in]
    13. Change default editor to vi
    14. $ sudo update-alternatives --set editor /usr/bin/vim.tiny
    15. Generate RSA keys. ssh-keygen -t rsa. Put them wherever needed on other systems.
    16. Optional: add wifi adapter, just because we happen to have one. Ours is labeled 802.11n. Plug it into a USB port on RPi. Follow instructions at https://www.raspberrypi.org/documentation/configuration/wireless/wireless-cli.md . Ours works, as we see from the existence of an inet address:
      1. mark@redhen3rpi ~ $ ifconfig wlan0
      2. wlan0 Link encap:Ethernet HWaddr 00:13:ef:c0:05:28
      3. inet addr:10.0.1.131 Bcast:10.0.1.255 Mask:255.255.255.0
2. Can we add an external hard drive? Yes. Here is how we added a USB Verbatim 8GB thumbdrive. First, insert the thumb drive into a USB port on the RPi.

- 1. See http://superuser.com/questions/662614/raspberry-pi-how-to-format-hdd and http://www.itechlounge.net/2012/01/linux-partition-and-format-external-hard-drive-as-ext3-filesystem/
  2. mark@redhen3rpi ~ $ dmesg | tail -20
  3. [31247.412404] usb 1-1.4: new high-speed USB device number 82 using dwc_otg
  4. [31247.517453] usb 1-1.4: New USB device found, idVendor=18a5, idProduct=0302
  5. [31247.517480] usb 1-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
  6. [31247.517498] usb 1-1.4: Product: STORE N GO
  7. [31247.517514] usb 1-1.4: Manufacturer: Verbatim
  8. [31247.517531] usb 1-1.4: SerialNumber: 070141FE0CC22009
  9. [31247.518728] usb-storage 1-1.4:1.0: USB Mass Storage device detected
  10. [31247.519245] scsi host11: usb-storage 1-1.4:1.0
  11. [31248.575384] scsi 11:0:0:0: Direct-Access Verbatim STORE N GO 5.00 PQ: 0 ANSI: 6\
  12. [31248.576714] sd 11:0:0:0: Attached scsi generic sg0 type 0
  13. [31249.372751] sd 11:0:0:0: [sda] 15646720 512-byte logical blocks: (8.01 GB/7.46 GiB)
  14. [31249.373725] sd 11:0:0:0: [sda] Write Protect is off
  15. [31249.373750] sd 11:0:0:0: [sda] Mode Sense: 23 00 00 00
  16. [31249.374709] sd 11:0:0:0: [sda] No Caching mode page found
  17. [31249.374729] sd 11:0:0:0: [sda] Assuming drive cache: write through
  18. [31249.399695] sda: sda1
  19. [31249.404068] sd 11:0:0:0: [sda] Attached SCSI removable disk
  20. [UPDATE 2016-08-21 to the following section for formatting and mounting the hard disk: Now we use gdisk instead of fdisk. See full information in the yellow section at the bottom]
  21. mark@redhen3rpi ~ $ sudo fdisk -l
  22. Device Boot Start End Blocks Id System
  23. /dev/sda1 2 15646719 7823359 b W95 FAT32
    1. mark@redhen3rpi ~ $ sudo fdisk /dev/sda
    2. Command (m for help): d
    3. Selected partition 1
    4. Command (m for help): n
    5. Partition type:
    6. p primary (0 primary, 0 extended, 4 free)
    7. e extended
    8. Select (default p): p
    9. Partition number (1-4, default 1):
    10. Using default value 1
    11. First sector (2048-15646719, default 2048):
    12. Using default value 2048
    13. Last sector, +sectors or +size{K,M,G} (2048-15646719, default 15646719):
    14. Using default value 15646719
    15. Command (m for help): w
    16. The partition table has been altered!
    17. Calling ioctl() to re-read partition table.
    18. Syncing disks.
    19. [Now let's check that it's actually connected and with its partition:]
    20. mark@redhen3rpi ~ $ sudo fdisk -l
  24. Disk /dev/sda: 8011 MB, 8011120640 bytes
  25. 8 heads, 40 sectors/track, 48896 cylinders, total 15646720 sectors
  26. Units = sectors of 1 * 512 = 512 bytes
  27. Sector size (logical/physical): 512 bytes / 512 bytes
  28. I/O size (minimum/optimal): 512 bytes / 512 bytes
  29. Disk identifier: 0x00000000
  30. Device Boot Start End Blocks Id System
  31. /dev/sda1 2048 15646719 7822336 83 Linux
  32. [Now let's format it for Linux as ext4:]
  33. mark@redhen3rpi ~ $ sudo mkfs -t ext4 /dev/sda1
  34. mke2fs 1.42.5 (29-Jul-2012)
  35. Filesystem label=
  36. OS type: Linux
  37. Block size=4096 (log=2)
  38. Fragment size=4096 (log=2)
  39. Stride=0 blocks, Stripe width=0 blocks
  40. 489600 inodes, 1955584 blocks
  41. 97779 blocks (5.00%) reserved for the super user
  42. First data block=0
  43. Maximum filesystem blocks=2004877312
  44. 60 block groups
  45. 32768 blocks per group, 32768 fragments per group
  46. 8160 inodes per group
  47. Superblock backups stored on blocks:
  48. 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
  49. Allocating group tables: done
  50. Writing inode tables: done
  51. Creating journal (32768 blocks): done
  52. Writing superblocks and filesystem accounting information: done
  53. [Now let's mount it as V8usbstick]
  54. mark@redhen3rpi ~ $ cd /mnt
  55. mark@redhen3rpi /mnt $ sudo mkdir V8usbstick
  56. [sudo] password for mark:
  57. mark@redhen3rpi /mnt $ sudo mount /dev/sda1 /mnt/V8usbstick
  58. mark@redhen3rpi /mnt $ cd V8usbstick
  59. [Now let's check that we can use V8usbstick]
  60. mark@redhen3rpi /mnt/V8usbstick $ sudo touch hello
  61. mark@redhen3rpi /mnt/V8usbstick $ ls
  62. hello lost+found
  63. [OK, as we see, that worked:]
  64. mark@redhen3rpi /mnt/V8usbstick $ df
  65. Filesystem 1K-blocks Used Available Use% Mounted on
  66. /dev/sda1 7568312 17196 7143616 1% /mnt/V8usbstick
  67. [OK, now let's unmount & eject it:]
  68. mark@redhen3rpi ~ $ sudo umount /dev/sda1
  69. mark@redhen3rpi ~ $ sudo eject /dev/sda

1. Can we add a large external hard drive? Yes. Here is how we added a USB2 Western Digital 1TB external hard disk. It is importat that the WD hard disk have its own independent power supply: the RPi cannot power such a drive through its USB ports.

- 1. First, connect the hard drive to a USB port on the RPi.
  2. mark@redhen3rpi ~ $ dmesg | tail -30
    1. [ 2397.017950] usb 1-1.5: new high-speed USB device number 5 using dwc_otg
    2. [ 2397.119128] usb 1-1.5: New USB device found, idVendor=1058, idProduct=1110
    3. [ 2397.119154] usb 1-1.5: New USB device strings: Mfr=1, Product=2, SerialNumber=3
    4. [ 2397.119172] usb 1-1.5: Product: My Book 1110
    5. [ 2397.119189] usb 1-1.5: Manufacturer: Western Digital
    6. [ 2397.119206] usb 1-1.5: SerialNumber: 5743415A4130323333333034
    7. [ 2397.120394] usb-storage 1-1.5:1.0: USB Mass Storage device detected
    8. [ 2397.122856] scsi host0: usb-storage 1-1.5:1.0
    9. [ 2398.120384] scsi 0:0:0:0: Direct-Access WD My Book 1110 2018 PQ: 0 ANSI: 4
    10. [ 2398.124489] scsi 0:0:0:1: CD-ROM WD Virtual CD 1110 2018 PQ: 0 ANSI: 4
    11. [ 2398.124976] sd 0:0:0:0: [sda] 1952151552 512-byte logical blocks: (999 GB/930 GiB)
    12. [ 2398.128851] sd 0:0:0:0: [sda] Write Protect is off
    13. [ 2398.128900] sd 0:0:0:0: [sda] Mode Sense: 2b 00 10 08
    14. [ 2398.132293] sd 0:0:0:0: [sda] No Caching mode page found
    15. [ 2398.132332] sd 0:0:0:0: [sda] Assuming drive cache: write through
    16. [ 2398.139498] scsi 0:0:0:2: Enclosure WD SES Device 2018 PQ: 0 ANSI: 4
    17. [ 2398.158885] sd 0:0:0:0: Attached scsi generic sg0 type 0
    18. [ 2398.159276] scsi 0:0:0:1: Attached scsi generic sg1 type 5
    19. [ 2398.164808] sda: sda1
    20. [ 2398.170516] scsi 0:0:0:2: Attached scsi generic sg2 type 13
    21. [ 2398.176030] sr 0:0:0:1: [sr0] scsi3-mmc drive: 51x/51x caddy
    22. [ 2398.176114] cdrom: Uniform CD-ROM driver Revision: 3.20
    23. [ 2398.178107] sd 0:0:0:0: [sda] Attached SCSI disk
    24. [ 2398.179302] sr 0:0:0:1: Attached scsi CD-ROM sr0
    25. [Note that this drive notoriously contains a very small partition with a virtual CD for "WD Smartware" that is impervious to reformatting. We'll just ignore it. For the future, better to have a clean hard drive formatted for Linux or Windows, and then to reformat it for raspbian, as below.]
    26. mark@redhen3rpi ~ $ sudo fdisk -l
    27. Disk /dev/sda: 999.5 GB, 999501594624 bytes
    28. 255 heads, 63 sectors/track, 121515 cylinders, total 1952151552 sectors
    29. Units = sectors of 1 * 512 = 512 bytes
    30. Sector size (logical/physical): 512 bytes / 512 bytes
    31. I/O size (minimum/optimal): 512 bytes / 512 bytes
    32. Disk identifier: 0x00000000
    33. Device Boot Start End Blocks Id System
    34. /dev/sda1 2 1952151551 976075775 b W95 FAT32
    35. mark@redhen3rpi ~ $ sudo fdisk /dev/sda
    36. Command (m for help): d
    37. Selected partition 1
    38. Command (m for help): n
    39. Partition type:
    40. p primary (0 primary, 0 extended, 4 free)
    41. e extended
    42. Select (default p): p
    43. Partition number (1-4, default 1):
    44. Using default value 1
    45. First sector (2048-1952151551, default 2048):
    46. Using default value 2048
    47. Last sector, +sectors or +size{K,M,G} (2048-1952151551, default 1952151551):
    48. Using default value 1952151551
    49. Command (m for help): w
    50. The partition table has been altered!
    51. Calling ioctl() to re-read partition table.
    52. Syncing disks.
    53. mark@redhen3rpi /dev $ sudo fdisk -l
    54. Disk /dev/sda: 999.5 GB, 999501594624 bytes
    55. 208 heads, 36 sectors/track, 260704 cylinders, total 1952151552 sectors
    56. Units = sectors of 1 * 512 = 512 bytes
    57. Sector size (logical/physical): 512 bytes / 512 bytes
    58. I/O size (minimum/optimal): 512 bytes / 512 bytes
    59. Disk identifier: 0x00000000
    60. Device Boot Start End Blocks Id System
    61. /dev/sda1 2048 1952151551 976074752 83 Linux
    62. mark@redhen3rpi ~ $ sudo mkfs -t ext4 /dev/sda1
    63. mke2fs 1.42.5 (29-Jul-2012)
    64. Filesystem label=
    65. OS type: Linux
    66. Block size=4096 (log=2)
    67. Fragment size=4096 (log=2)
    68. Stride=0 blocks, Stripe width=0 blocks
    69. 61005824 inodes, 244018688 blocks
    70. 12200934 blocks (5.00%) reserved for the super user
    71. First data block=0
    72. Maximum filesystem blocks=0
    73. 7447 block groups
    74. 32768 blocks per group, 32768 fragments per group
    75. 8192 inodes per group
    76. Superblock backups stored on blocks:
    77. 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
    78. 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
    79. 102400000, 214990848
    80. Allocating group tables: done
    81. Writing inode tables: done
    82. Creating journal (32768 blocks): done
    83. Writing superblocks and filesystem accounting information: done
    84. mark@redhen3rpi ~ $ cd /mnt
    85. mark@redhen3rpi /mnt $ sudo mkdir HD1
    86. mark@redhen3rpi /mnt $ sudo mount /dev/sda1 /mnt/HD1
    87. mark@redhen3rpi /mnt $ cd HD1
    88. [Now let's check that we can use HD1]
    89. mark@redhen3rpi /mnt/HD1 $ sudo touch hello
    90. mark@redhen3rpi /mnt/HD1 $ ls
    91. hello lost+found
    92. [OK, as we see, that worked:]
    93. mark@redhen3rpi /mnt/HD1 $ df
    94. Filesystem 1K-blocks Used Available Use% Mounted on
    95. /dev/sda1 960628088 73364 911734604 1% /mnt/HD1
    96. We want all heavy work done on HD1 rather than on the microSD card. So make a symlink
    97. $ ln -s /mnt/HD1 HD1
    98. so that the directory HD1 shows up in the home directory of the user. Users should work on HD1 exclusively.
    99. For the future: consider the possibility of moving everything over to HD1 so as to avoid the corruption that can be induced by writing the microSD card: https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=44177
  3. Moved the RPi to a university network. It now has a static IP address. Remounted the external hard drive HD1 with sudo mount /dev/sda1 /mnt/HD1.
  4. Now there are two Red Hen devices in one drawer in Crawford Hall, Case Western Reserve University:

- - - 1. On the left, redhen1, a macmini, with a USB extender connecting to an Elgato EyeTV Hybrid off-screen. On the right, redhen3rpi, a Raspberry Pi 2 B, with a USB connection to its external hard disk in the upper right. redhen3rpi and an HDHomeRun (in the drawer below) are on the same network.

- 1. [UPDATE 2015-06-20: Debian Jessie is now available for direct installation, so this step should be simpler.] Interlude: Francis Steen ran the script needed to upgrade Debian Wheezy to Debian Jessie, installed hdhomerun-config, uuid, gpac, ffmpeg, and established the usual tunnel (with RedHen scripts both /usr/local/bin/tunneld and backup /usr/local/bin/tunnel) between redhen3rpi and the rest of the Red Hen network. A few notes:
  2. $ sudo apt-get update
  3. $ sudo apt-get install wajig
  4. $ sudo apt-get upgrade
  5. $ sudo apt-get dist-upgrade
  6. $ sudo vi /etc/apt/sources.list
  7. Should have:
  8. deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi
  9. $ sudo apt-get upgrade
  10. $ sudo shutdown -r now
  11. $ sudo apt-get install ca-certificates
  12. $ sudo apt-get dist-upgrade
  13. $ sudo apt-get install git-core
  14. $ sudo apt-get install rpi-update
  15. $ sudo rpi-update
  16. $ sudo shutdown -r now
  17. $ sudo apt-get install eject
  18. $ sudo apt-get install uuid

1. [UPDATE 2015-06-20. ffmpeg can be installed pre-compiled from deb-multimedia. The
2. /etc/apt/sources.list file, which specifies sources for updates, now has the lines:
3. # RPI Sources.list 2015-06-20
4. # Stable official Debian
5. deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi
6. # Uncomment line below then 'apt-get update' to enable 'apt-get source'
7. #deb-src http://archive.raspbian.org/raspbian/ wheezy main contrib non-free rpi
8. # Debian jessie security team
9. deb http://security.debian.org jessie/updates main
10. # Debian Multimedia (http://www.deb-multimedia.org/)
11. deb http://www.deb-multimedia.org jessie main
12. #deb http://www.deb-multimedia.org jessie-backport main
13. #deb http://www.deb-multimedia.org stretch main
14. #deb http://www.deb-multimedia.org sid main]
15. Is RPi powerful enough to compress a captured broadcast (.ts, .mpg) to .mp4? Yes. Francis Steen has deployed the method used at UCLA (two-pass ffmpeg) on redhen3rpi on a file captured on redhen1, with good results.
  1. 1. Note for future: handbrake does not seem to be part of Debian Jessie. Installing handbrake on Debian Jessie: http://linuxg.net/how-to-install-handbrake-0-9-9-on-debian-jessie-debian-sid-and-debian-wheezy/ .

- 1. Is RPi powerful enough to extract closed-captions? Yes. First, install ccextractor by getting the source:
  2. ==============
  3. csa@redhen3rpi ~/software $ lynx http://ccextractor.sourceforge.net/download-ccextractor.html
  4. csa@redhen3rpi ~/software $ ls
  5. ccextractor.src.0.77.zip
  6. csa@redhen3rpi ~/software $ ls -l
  7. total 1264
  8. -rw-r--r-- 1 csa csa 1292599 Jun 20 19:27 ccextractor.src.0.77.zip
  9. csa@redhen3rpi ~/software $ unzip *p
  10. Archive: ccextractor.src.0.77.zip
  11. creating: ccextractor.0.77/
  12. [etc. etc. etc.]
  13. csa@redhen3rpi ~/software $ ls -l
  14. total 1268
  15. drwxr-xr-x 9 csa csa 4096 Jun 20 18:03 ccextractor.0.77
  16. -rw-r--r-- 1 csa csa 1292599 Jun 20 19:27 ccextractor.src.0.77.zip
  17. csa@redhen3rpi ~/software $ cd *77
  18. csa@redhen3rpi ~/software/ccextractor.0.77 $ ls
  19. Dictionary docs linux mac OpenBSD src windows
  20. csa@redhen3rpi ~/software/ccextractor.0.77 $ cd linux
  21. csa@redhen3rpi ~/software/ccextractor.0.77/linux $ ls -l
  22. total 12
  23. -rwxr-xr-x 1 csa csa 245 Jun 20 14:29 build
  24. -rwxr-xr-x 1 csa csa 248 Jun 20 18:02 builddebug
  25. -rw-r--r-- 1 csa csa 3448 Jun 20 14:29 Makefile
  26. csa@redhen3rpi ~/software/ccextractor.0.77/linux $ ./build
  27. csa@redhen3rpi ~/software/ccextractor.0.77/linux $ ls -l
  28. total 1620
  29. -rwxr-xr-x 1 csa csa 245 Jun 20 14:29 build
  30. -rwxr-xr-x 1 csa csa 248 Jun 20 18:02 builddebug
  31. -rwxr-xr-x 1 csa csa 1642552 Jun 20 19:30 ccextractor
  32. -rw-r--r-- 1 csa csa 3448 Jun 20 14:29 Makefile
  33. csa@redhen3rpi ~/software/ccextractor.0.77/linux $ mv ccextractor ccextractor-0.77
  34. csa@redhen3rpi ~/software/ccextractor.0.77/linux $ sudo cp cc* /usr/local/bin
  35. csa@redhen3rpi ~/software/ccextractor.0.77/linux $ cd /usr/local/bin
  36. csa@redhen3rpi /usr/local/bin $ ls -l
  37. total 1620
  38. -rwxr-xr-x 1 root staff 1642552 Jun 20 19:31 ccextractor-0.77
  39. drwxr-sr-x 6 root staff 4096 Jun 19 18:45 hdhomerun_config_gui
  40. drwxrwsr-x 2 root indiecity 4096 May 6 19:17 indiecity
  41. drwxr-sr-x 2 root staff 4096 Jun 19 20:06 libhdhomerun
  42. csa@redhen3rpi /usr/local/bin $ sudo ln -s ccextractor-0.77 ccextractor
  43. ==============
  44. Steen tweaked and ran cc-extract-rpi on an mpeg file. It works, except for one minor remaining issue: the timestamps are inaccurate. Perhaps a setting in CCextractor needs to be changed.
  45. Can RPi capture a broadcast? Yes.
    1. Establish an HDHomeRun. That is, plug an HDHomeRun into the same internet subnet to which the RPi is connected. [UPDATE 2015-06-20: hdhomerun_config is present in the Debian repository, so can be installed directly instead of compiled from source.] From http://www.silicondust.com/support/downloads/ , follow the instructions for Linux Downloads, i.e., using a text browser like lynx (sudo apt-get install lynx), download libhdhomerun (source) and HDHomeRun Config GTK (source). These are .tgz files. Extract them with
    2. $ tar -xvzf hdhomerun_config_gui_20150615.tgz
    3. and
    4. $ tar -xvzf libhdhomerun_20150615.tgz
    5. From the hdhomerun_config_gui directory, run:
    6. $ sudo ./configure
    7. $ sudo make
    8. $ sudo make install
    9. Then go looking for the HDHomeRun on the subnet:
    10. $ ./hdhomerun_config discover
    11. And hope the response is something like: hdhomerun device 103B659E found at 129.22.115.124.
    12. Now you know the device number of your HDHomeRun (103B659E) and its IP address (129.22.115.124).
    13. Then find the model of your HDHomeRun:
    14. $ ./hdhomerun_config 103B659E get /sys/model
    15. And hope the response is something like: hdhomerun3_atsc
    16. Then find the firmware version you are currently running:
    17. $ ./hdhomerun_config 103B659E get /sys/version
    18. And hope the response is something like: 20130328
    19. Then find the latest firmware for your model on the silicondust.com site and download it. It will have a name like hdhomerun3_atsc_firmware_20150604.bin once you download it.
    20. Then upgrade your firmware:
    21. $ ./hdhomerun_config 103B659E upgrade ~/HD1/hdhomerun3_atsc_firmware_20150604.bin
    22. and hope you see something like:
    23. uploading firmware...upgrading firmware...rebooting...upgrade complete - now running firmware 20150604
  46. Now configure the HDHomeRun using hdhomerun_config. There are 2 tuners in the HDHomeRun on redhen3rpi, numbered 0 and 1.
    1. 1. Set a variable for DEVICE and a variable for TUNER
      2. $ DEVICE=103B659E TUNER=0
      3. Get or set the channelmap:
      4. $ ./hdhomerun_config $DEVICE get /tuner0/channelmap
      5. And hope the response is what you want to see, e.g.: us-cable
      6. Scan tuner 0 for channels:
        $ sudo ./hdhomerun_config $DEVICE scan /tuner$TUNER scan0us-cable.log
        And hope it finds some, e.g.:
        SCANNING: 69000000 (us-irc:4, us-cable:4)
        LOCK: qam256 (ss=93 snq=69 seq=100)
        PROGRAM 1: 4.1 WQHS
        PROGRAM 2: 4.2 WMFD
        PROGRAM 3: 4.3 CNN
        PROGRAM 4: 4.4 CNN HLN
        PROGRAM 5: 4.5 FOX New
        PROGRAM 6: 4.6 MSNBC
        PROGRAM 7: 4.7 CNBC
        PROGRAM 8: 4.8 CSPAN
      7. So, let's tune to us-cable channel 4:
      8. $ sudo ./hdhomerun_config $DEVICE set /tuner$TUNER/channel auto:4
      9. And check how we are doing:
      10. $ sudo ./hdhomerun_config $DEVICE get /tuner$TUNER/status
      11. We get the response: ch=auto:4 lock=qam256 ss=94 snq=0 seq=0 bps=38813728 pps=0
      12. Let's select program 3 (4.3); that's CNN, as we learned above:
      13. $ ./hdhomerun_config $DEVICE set /tuner$TUNER/program 3
      14. Now, let's start capturing CNN for a while
      15. $ ./hdhomerun_config $DEVICE save /tuner$TUNER ~/HD1/TestCapture2015-06-19.ts
      16. And we see (note that we stop the recording with control-C):
      17. .................................................n....t........^C
      18. -- Video statistics --
      19. 233303 packets received, 0 overflow errors, 1 network errors, 6 transport errors, 0 sequence errors
      20. Similar tests show that we can capture MSNBC, Fox, etc.
      21. And when we move the captured file to a Mac and look at it in VLC, we see the subtitles:

Update 2016-08-21: Using gdisk instead of fdisk:

csa@rusalka:~ $ sudo just install gdisk

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following NEW packages will be installed:

gdisk

0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.

Need to get 193 kB of archives.

After this operation, 799 kB of additional disk space will be used.

Get:1 http://mirror.ox.ac.uk/sites/archive.raspbian.org/archive/raspbian/ jessie/main gdisk armhf 0.8.10-2 [193 kB]

Fetched 193 kB in 0s (317 kB/s)

Selecting previously unselected package gdisk.

(Reading database ... 89584 files and directories currently installed.)

Preparing to unpack .../gdisk_0.8.10-2_armhf.deb ...

Unpacking gdisk (0.8.10-2) ...

Processing triggers for man-db (2.7.0.2-5) ...

Setting up gdisk (0.8.10-2) ...

csa@rusalka:~ $ sudo gdisk /dev/sda

GPT fdisk (gdisk) version 0.8.10

Partition table scan:

MBR: MBR only

BSD: not present

APM: not present

GPT: not present

***************************************************************

Found invalid GPT and valid MBR; converting MBR to GPT format

in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by

typing 'q' if you don't want to convert your MBR partitions

to GPT format!

***************************************************************

Warning! Secondary partition table overlaps the last partition by

33 blocks!

You will need to delete this partition or resize it in another utility.

Command (? for help): ?

b back up GPT data to a file

c change a partition's name

d delete a partition

i show detailed information on a partition

l list known partition types

n add a new partition

o create a new empty GUID partition table (GPT)

p print the partition table

q quit without saving changes

r recovery and transformation options (experts only)

s sort partitions

t change a partition's type code

v verify disk

w write table to disk and exit

x extra functionality (experts only)

? print this menu

Command (? for help): x

Expert command (? for help): ?

a set attributes

c change partition GUID

d display the sector alignment value

e relocate backup data structures to the end of the disk

g change disk GUID

h recompute CHS values in protective/hybrid MBR

i show detailed information on a partition

l set the sector alignment value

m return to main menu

n create a new protective MBR

o print protective MBR data

p print the partition table

q quit without saving changes

r recovery and transformation options (experts only)

s resize partition table

t transpose two partition table entries

u replicate partition table on new device

v verify disk

w write table to disk and exit

z zap (destroy) GPT data structures and exit

? print this menu

Expert command (? for help): z

About to wipe out GPT on /dev/sda. Proceed? (Y/N): y

GPT data structures destroyed! You may now partition the disk using fdisk or

other utilities.

Blank out MBR? (Y/N): y

csa@rusalka:~ $ sudo gdisk /dev/sda

GPT fdisk (gdisk) version 0.8.10

Partition table scan:

MBR: not present

BSD: not present

APM: not present

GPT: not present

Creating new GPT entries.

Command (? for help): n

Partition number (1-128, default 1):

First sector (34-3906963422, default = 2048) or {+-}size{KMGTP}:

Last sector (2048-3906963422, default = 3906963422) or {+-}size{KMGTP}:

Current type is 'Linux filesystem'

Hex code or GUID (L to show codes, Enter = 8300):

Changed type of partition to 'Linux filesystem'

Command (? for help): i

Using 1

Partition GUID code: 0FC63DAF-8483-4772-8E79-3D69D8477DE4 (Linux filesystem)

Partition unique GUID: 7C1DEEB3-DFDE-4502-ADFE-1DF9323DADCE

First sector: 2048 (at 1024.0 KiB)

Last sector: 3906963422 (at 1.8 TiB)

Partition size: 3906961375 sectors (1.8 TiB)

Attribute flags: 0000000000000000

Partition name: 'Linux filesystem'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING

PARTITIONS!!

Do you want to proceed? (Y/N): y

OK; writing new GUID partition table (GPT) to /dev/sda.

The operation has completed successfully.

~~csa@rusalka:~ $ mkfs.xfs /dev/sda1~~

~~bash: mkfs.xfs: command not found~~

csa@rusalka:~ $ sudo just install xfsprogs

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following extra packages will be installed:

libreadline5

Suggested packages:

xfsdump attr quota

The following NEW packages will be installed:

libreadline5 xfsprogs

0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.

Need to get 746 kB of archives.

After this operation, 3,453 kB of additional disk space will be used.

Do you want to continue? [Y/n]

Get:1 http://mirror.ox.ac.uk/sites/archive.raspbian.org/archive/raspbian/ jessie/main libreadline5 armhf 5.2+dfsg-2 [131 kB]

Get:2 http://mirror.ox.ac.uk/sites/archive.raspbian.org/archive/raspbian/ jessie/main xfsprogs armhf 3.2.1 [614 kB]

Fetched 746 kB in 1s (746 kB/s)

Selecting previously unselected package libreadline5:armhf.

(Reading database ... 89605 files and directories currently installed.)

Preparing to unpack .../libreadline5_5.2+dfsg-2_armhf.deb ...

Unpacking libreadline5:armhf (5.2+dfsg-2) ...

Selecting previously unselected package xfsprogs.

Preparing to unpack .../xfsprogs_3.2.1_armhf.deb ...

Unpacking xfsprogs (3.2.1) ...

Processing triggers for man-db (2.7.0.2-5) ...

Setting up libreadline5:armhf (5.2+dfsg-2) ...

Setting up xfsprogs (3.2.1) ...

Processing triggers for libc-bin (2.19-18+deb8u4) ...

csa@rusalka:~ $ sudo mkfs -t ext4 /dev/sda1

~~csa@rusalka:~ $ sudo mkfs.xfs /dev/sda1~~

~~mkfs.xfs: /dev/sda1 appears to contain an existing filesystem (ntfs).~~

~~mkfs.xfs: Use the -f option to force overwrite.~~

~~csa@rusalka:~ $ sudo mkfs.xfs /dev/sda1 -f~~

~~meta-data=/dev/sda1 isize=256 agcount=4, agsize=122092543 blks~~

~~= sectsz=512 attr=2, projid32bit=1~~

~~= crc=0 finobt=0~~

~~data = bsize=4096 blocks=488370171, imaxpct=5~~

~~= sunit=0 swidth=0 blks~~

~~naming =version 2 bsize=4096 ascii-ci=0 ftype=0~~

~~log =internal log bsize=4096 blocks=238461, version=2~~

~~= sectsz=512 sunit=0 blks, lazy-count=1~~

~~realtime =none extsz=4096 blocks=0, rtextents=0~~

csa@rusalka:~ $ sudo mount /dev/sda1 /mnt/HD1

~~csa@rusalka:~ $ mount /mnt/HD1~~

csa@rusalka:~ $ df -h

Filesystem Size Used Avail Use% Mounted on

/dev/root 29G 2.5G 25G 9% /

devtmpfs 459M 0 459M 0% /dev

tmpfs 463M 0 463M 0% /dev/shm

tmpfs 463M 6.4M 457M 2% /run

tmpfs 5.0M 4.0K 5.0M 1% /run/lock

tmpfs 463M 0 463M 0% /sys/fs/cgroup

/dev/mmcblk0p1 63M 21M 43M 33% /boot

tmpfs 93M 0 93M 0% /run/user/1000

tmpfs 93M 0 93M 0% /run/user/1001

/dev/sda1 1.9T 33M 1.9T 1% /mnt/HD1

Use blkid to get the PARTUUID. Edit /etc/fstab to include the hard drive. Use e2label to label the hard disk. Then issue mount -av to see that it is mounted. Here's an example from a different installation, where the partition is not /dev/sda1 but instead /dev/scd1:

$ sudo e2label /dev/sdc1 HDbig

$ sudo e2label /dev/sdc1

HDbig

$ mount -av

/proc : already mounted

/boot : already mounted

/ : ignored

/mnt/HD1 : already mounted

/mnt/HDbig : already mounted