Mails from Francis - Chapter 2

schedule and channel commands

So, where were we? Ah yes, at the end of Chapter1 we talked about cron (crontab) and mentioned the schedule command,

the skill to learn is cron, the unix task scheduler. See the content of crontab -l - when we issue crontab -l (Enter), we get for example:

and we see this line there: 31 07 * * * schedule TVP1 "Teleexpress" 1 777 "National afternoon news" , and we have already explained that in means that every day at 7.31 a command will be executed and this command is: schedule TVP1 "Teleexpress" 1 777 "National afternoon news" . So this is the schedule command and this command tells the RPI (the little Raspberry PI robot that does all the work for us) to record from TVP1 (Polish channel 1) a program called Teleexpress from tuner 1 (hdhomerun has two tuners), and to record the subtitles from teletext page 777, and "National afternoon news" is just an optional description of what we are recording. And how does it do it? It looks for a file containing the tv schedule (called xmltv file), reads from this file, learns the start time and the duration of the program and then writes in crontab (above) all the lines you can see below it. And, as we have already explained at the end of Chapter 1, each of those lines tells our RPI robot to execute the channel commands at specific dates and times. And the channel command makes the RPI record tv programs. Now, the schedule command is based on a script Francis wrote.

bash script (1)

A script is a computer program or, more specifically, a text that (in our case) is interpreted by something called Bash (I'd like to think of it as 'Man Bash', and this is what I call Francis privately, to myself only; and I have other names for him as well- I'll tell you later). So this Bash is a 'command line interpreter' (or a shell) and it's a Unix shell. I am this clever because I know how to Wiki things. I am a Wikid wizard, I am. So, if you'd like to see this schedule script (and who wouldn't) write cat /usr/local/bin/schedule and Enter.

The above is just the beginning - it goes on for pages and pages. We'll return to bash scripting in the next parts, let's just say for now that the first line of the script (#!/bin/bash) is called THE SHEBANG and, I am sure you'll agree with me, it explains a lot. Let's move on to the next email.

manual recording and scheduling

It's useful to test, so try composing a channel command for a real show and give it on the command line. The file will come in on /mnt/HD1/spool (shortcut key s), and the txt file in the tv tree (shortcut 'day') -- these shortcuts should also work for user jacek. The script channel triggers a sequence of scripts that extracts the text and does some post-processing. We've seen text extraction is not giving us clean Polish text, so we need samples we can contribute to the CCExtractor team for debugging.

MHM, I see, where can I find a command line interpreter for the above? I may write one myself one day- it'll be called Francis Man Bash Shebang Shell. But in fact, when we look at the first black screen picture on this page, we already know what channel command should look like and Francis is suggesting I try it not in crontab but simply writing it in the black box console and pressing Enter so it would record a program. And then he is telling me where the text and the mpg file will land. Is there really a CCExtractor team out there? I can imagine them now, white lab coats, doors secured with eye-scanners, walls lined with gauges and flashing controls. Anyway, I salute them now with thanks for their excellent work.

Manual scheduling works fine as long as the program is broadcast at a fixed time and for a fixed duration. We need the schedule only for the cases where the program moves around, which is more common than not. To experiment with the schedule M gave you, could you copy it to this location and file name: /mnt/HD1/xmltv/2016/2016-09/2016-09-16.xmltv

Could I copy it to this location and file name? I could of course if only I knew how. There are several methods for it. Probably the easiest one is to use winscp (see the winscp section). Francis and Mark would know about 112 ways of doing it in the console - I only know one (or two?). For example, go: nano /mnt/HD1/xmltv/2016/2016-09/2016-09-16.xmltv (this will open an empty file of this name) , then in Windows open 'the file from M' (any file you want to copy into RPI), sctrl A, ctrl C, then click back into the console and right-click - it should copy the text from (Windows, Mac) clipboard into nano (editor). And then just save it on RPI - ctrl x, then press y, Enter. Complicated, confusing don't I know it. But try it, please. On to the next mail.

bash script (2)

In /usr/local/bin/scheduler-DVB-06.sh I made two single-character changes to achieve what you want. First old version:

# Skip the 19:30 broadcast of TVP-1 Wiadomości (or not)

if [ "$SHOW" = "Wiadomości" ] ; then

if [ "$WKD" -lt "0" ]

then if [[ $STIM == *19:30* ]] ; then continue ; fi

fi

fi

Set weekday to less than 8 (meaning all) and skip if time is NOT 19:30:

# Skip all but the 19:30 broadcast of TVP-1 Wiadomości

if [ "$SHOW" = "Wiadomości" ] ; then

if [ "$WKD" -lt "8" ]

then if [[ $STIM != *19:30* ]] ; then continue ; fi

fi

fi

Francis did it 'to achieve 'what I want', which was first not to record a show at a certain hour on TVP1 channel and then (whimsically) I changed it to: 'now I want to record only this show'. So he changed a script that schedules programs (the same one we talked at the beginning of this section, usually called 'schedule' but this particular script we were using at the time was called scheduler-DVB-06.sh and it was in /usr/local/bin directory (where many vital scripts are located) so its full name (with location, called 'path') is /usr/local/bin/scheduler-DVB-06.sh. And in translation (switching on the Francis Man Bash Shebang Shell now) the lines mean: if the name of the show is Wiadomości then if the day of the week number is less than 8 (it always is) then if the show begins at 19.30 then don't schedule it (continue, get out of the loop). And then (upon my whim and nagging), he changed it to: if it doesn't begin at 19.30 then don't schedule. So he replaced == with != (replaced "equal to" with "different from"). All very well, but did he HAVE to tell me all this? Well this is Francis for you, he believes people can learn (even dummies, like myself). And he explained that he left the redundant if [ "$WKD" -lt "8" ] line to retain flexibility (in case we wanted to add, say, only from Monday to Thursday and then it would be enough to change 8 to 5 in it, etc.).

xmltv files

Make sure we have a pl schedule for today: mv 2016-09-25-pl.xmltv 2016-09-26-pl.xmltv.

Run the command as given in crontab: scheduler-DVB-06.sh TVP1 "Wiadomości" 1 777 "National evening news"

the mv command moves (renames) a file. In our case a file called 2016-09-25-pl.xmltv was renamed to 2016-09-26-pl.xmltv. And those *.xmltv files contain the tv schedule in a certain format.

"Run the command as given in crontab" (we talked about crontab in section 1). If you want to see this command it in crontab- go: crontab -l. And, moving on-

closed captions, ccexractor

If you run this command:ffprobe 2016-09-27_0600_PL_Polsat_Trudne_sprawy.mpg

You'll get this information about the video:

Input #0, mpegts, from '2016-09-27_0600_PL_Polsat_Trudne_sprawy.mpg':

Duration: 00:59:55.50, start: 26695.745611, bitrate: 3188 kb/s

Program 3

Stream #0:0[0x6a](pol): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006) (hearing impaired)

Stream #0:1[0x66]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, bt470bg), 720x576 [SAR 16:11 DAR 20:11], 25 fps, 50 tbr, 90k tbn, 50 tbc

Stream #0:2[0x67](pol): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16p, 192 kb/s

Stream #0:3[0x68](pol): Audio: eac3 (EAC3 / 0x33434145), 48000 Hz, stereo, fltp, 256 kb/s

Stream #0:4[0x6c](aux): Audio: eac3 (EAC3 / 0x33434145), 48000 Hz, stereo, fltp, 128 kb/s (visual impaired)

Stream #0:5[0x69](pol,pol): Subtitle: dvb_teletext ([6][0][0][0] / 0x0006), 492x250

You'll notice streams 0:0 and 0:5 mention subtitles -- the first dvb_subtitle and the second dvb_teletext. I believe the dvb standard is small images that require optical character recognition. This is supported by CCExtractor, though I don't know for a fact it is supported for Polish. When you get Vila, I would use it to build (compile) CCExtractor with DVB support. I haven't experimented with it.

I know this may sound like an empty threat to you, but when Francis says he will build (compile) CCExtractor with DVB support he will, and there is no stopping him. And I already have Vila (my second RPI) and it's already doing some crucial work. Yes- it captures schedule (xmltv files) for other capture stations (at the moment Dola and Rusalka) - see also the webgrab++ Appendix.

bash script (3) - the pipe (|), escaping characters

Dear Jacek, You uncovered a bug in the script, so that's good. In the scripts, some characters have a functional meaning -- for instance, / and |.

If you want them to have their literal meaning, you have to "escape" them, which is to say, place a backslash before them: \| means, "Interpret the pipe symbol literally -- just print the pipe symbol, don't take it to mean "pipe the output of the previous function to a new function". So I had "escaped" > and < on line 107 in scheduler-DVB-06.sh, and it wasn't necessary. See if it works now. Make sure you use simulation mode -- it requires including the teletext page number, as below. Francis

$ scheduler-DVB-06.sh 6 "Trudne sprawy" 1 778 "test" s

Of course I did not 'uncover a bug' - I just run to Francis crying 'this does not work, I don't know how to!, why?, why?, I don't understand, no, no!, HELP me, help me, please etc. - in a never ending loop). And he not only helps, finds a bug, kills it, explains patiently but credits me with discovering it. This man, Francis, I can tell you- is a saint. There is no other way to put it. So this pipe is one of those characters you never knew existed, here: |||||| |||| | | |||| | | | (you'll find it over backslash- I could play with typing pipes all day). Other characters of this kind are # and $.

# is used for comments in scripts - so if this were a script (a program) this line would be ignored. and $ means value (still does). So if there is a variable like, say, SHOW, and you want to print its value on the screen, you write: echo "$SHOW" (also see the webgrab++ section where it is explained at length with Ferdinand de Saussure and his cat as examples). And sometimes, Francis says, we need 'to escape' a character and we put backslash in front of it. I have this Raspberry PI here. Let me demonstrate: