Mails from Francis - Chapter 4
reboot, sudo, tunnels
Dear Jacek,
I've lost the tunnel, and it's taking time to reestablish. Could you reboot dola? As csa, issue "sudo reboot". It comes back quickly.
I'm making the Czech schedule work, but it needs some more changes.
Best wishes, Francis
We already talked of sudo a little in the webgrab++ appendix. Adding sudo before a command (if your user is on the list of sudoers) is like a (barking mad) KIAI!, karate spirit shout. You can do anything with sudo, so please be careful with it (Jacek). So, are we ready and brave enough to reboot? Let's try it on vila (dola might be recording, better not bother it).
See - when I just went reboot (politely) it said: no, sorry, go away I have better things to do, etc. And then I got red in the face, inflated my lungs and: SUUUDOOOO reboot (arghrghrgh) - it started smoking and then exploded. Not really- it just said: I lost connection to the server (in a pop-up window). I wonder why? Oh, yes- likely because it has just rebooted and I have to open a new session in putty. Reeeeeesult! And when Francis talks about 'losing a tunnel', he speaks of tunneling through my firewall (metaphors and blends hanging in the air). How can one tunnel through a wall of fire? They will get scorched, won't they? And when he tunnels he does unspeakable things to RPIs around the world. I call it "the long invisible hand of Francis". And in the next mail, Francis is giving us some important life tips.
what red hens do
There's quite a bit of work and new research you could do if you want to run with this. For instance, you could collect, deploy, and develop natural language processing tools for Polish, building on what we've already done with English (and some cautious attempts with French and German). Or you could look at multimodal constructions -- Red Hen is actively developing tools that are language independent, focusing on visual analysis, but also on audio qualities. You clearly have a knack for this kind of thing and get a kick out of it; like you, Mark and I have a blast seeing what we can do with simple tools.
Yes, I do very much want to deploy, develop and look at - all of those - and as much as I can get with my, har har, "knack for this kind of things" - cruel irony there, see?
The first focus is automation -- I wrote the schedule script, for instance, purely to avoid having to worry about manually maintaining the schedule. Automation makes us productive and at the same time frees us up. The second is to deploy existing tools and leverage in a systematic way work others have already accomplished. Deploying these tools systematically on a large dataset is often not done, and I imagine there is corpus linguistic analyses and tools developed for Polish that could be applied to a rapidly growing Polish corpus.
So this is what Red Hens do - proud to be part of it. A dummy Red Hen but still a Red Hen. Me. Cpt. Dola. And the next mail is from Mark (who was always copied on our lenghty exchanges)
Jacek,
You are invaluable: linguist, treasurer of the Polish Cognitive Linguistics Association, host and organizer for my visit, and now a blazing fast operator of a capture station that is going to give us Polish and Czech! I invite you to list yourself at https://sites.google.com/site/distributedlittleredhen/home/profiles-of-red-hen-participants .
I have given you permission to edit that page. My confidence is high, and I have accordingly declared this task “solved”: see
m
And Mark calls me 'a blazing fast operator' and this is only because he shares a cruel sense of humour with Francis. Either that, or they are both incredibly kind. You decide.
a station is born
Let's have another of the very early mails from Mark:
the real raspberry- GUI, mailing
I am often copied (can spy on) Francis and Mark's mails to other capture station captains. Here's one of them:
Dear Anna,
Please do the following after generating the rsa codes. In Terminal, issue this command:
cp -p ~/.ssh/id_rsa.pub ~/Desktop/Anna.pub
Then e-mail me the file Anna.pub on your desktop.
Best wishes,
Francis
And to do what Francis asks, after running the Terminal command, Anna would probably use the VNC (tightvnc, etc.) viewer that lets you access an RPI's the way we should- through graphical interface. And it's lovely - like an old Mac (I never had):
So this is how a capture station begins. Mark and Francis see it, tunnel into it (get scorched by the firewall) and set things up for you.
In my case, I found that tightvnc works only if I issue vncserver in the console (black box, terminal, putty, the Matrix interface) first. Otherwise I just get a grey box.
And from there you can do (almost) anything you can do in Windows or MacOS. For example, open a web browser, go to www.google.com, log in to your Google account and check your gmail:))):
And when you are there, you can just send a mail (with attachments) to, say, Mark and Francis (you'll never guess where I am now, etc.). Let's do it. Yes, it went through (I expect to get an earful from Francis for using processor-heavy apps on a matchbox size PC, but it's too late now). And, in fact, you don't need to use VNC - just plug a hdmi cable (to your TV) a keyboard and a mouse and you have a fully functioning PC (Mac) that you can hide in the palm of your hand and that costs about as much as two packets of cigarettes. And comes with educational software and consumes almost no energy and can do all the things your expensive laptop can do, and more. How can one not love the idea of it! And it's probably one of the best documented open source projects on the net. If it were a place- I would say: bury me there.
alternatives to using GUI- winscp and terminal
So, Anna can use the GUI (graphical user interface) but is there another way? Yes there is. And the answer is - WinSCP (see also WinSCP Appendix), And in winscp I could go to /home/csa/Desktop, for example (works like the old Norton Commander or Windows Explorer- just point and click) and drag the file Francis wants to my Windows desktop. And then attach it to the mail (without overheating and burning down the RPI). That's all very well, but is there yet ANOTHER way Francis can get a mail with the attachment he wants? Yes, there is. Instead of cp -p ~/.ssh/id_rsa.pub ~/Desktop/Anna.pub, which would copy the file into RPI GUI desktop, we can just print the file in the console by issuing: (sudo- if necessary) cat ~/.ssh/id_rsa.pub and then mark the file in the console with the mouse (no right-clicking, just mark it- it puts the text into the clipboard). And then, when you compose you mail to Francis, just right click/paste (or ctrl V) and that's it. Francis will have the id_rsa.pub file (key) he needs for secure connection without password (more about it in Chapter 5).
manual recording quality check
But is it all always sweetness and light? No, it is not. There definitely are dark days:
Dear Jacek,
2016-10-03_1700_CZ_ČT1_Události-HD.log has an impressive number of transport errors; I removed the files.
Corrupt files gum up the works on hoffman2, so I might add a routine to detect them. In the meantime, let's remove them whenever we see them.
Francis
csa@dola:/tv/2016/2016-10/2016-10-03 $ cat 2016-10-03_1700_CZ_ČT1_Události*g
.ttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt, etc,
Impressive? Just terrible. Shameful. Who said only the British use understatement? Look at what Francis is saying. My station 'recorded' a file in which there is (almost) nothing but errors, and Francis calls it 'impressive'. How delicate. How kind. I have a shortcut for those transport errors - I call them t-errors. And I hate them.
However, ever since i put the hdhomerun device into an old tea-tin box (Faraday cage) and improved the roof-top antenna- all is (mostly) well (see also the TV antenna Appendix). We already talked about accessing those video.logs files in Chaptes 1 and 3 but a quick reminder: issue day, Enter. And you are now in today's recordings directory (day 1 - for yesterday, etc.). Then cat *.log and you will see all the log files and all the t-errors, etc. Or you may just want to see one specific log- then go, for example, cat 2016-11-14_1800_CZ_ČT1_Události-HD.log (use ls first to see the contents and copy the long names by marking them with a mouse and then right-clicking to get them where you cursor is at the moment).
In the above, as we can see, after I entered day - it reacted with indignation and said ? no: such and such directory. Don't be alarmed - the directory is created when the recording starts. And then I issued day 1 and ls to see yesterday's recordings, and finally cat *.log to see all those errors (t-errors). It's certainly advisable (look who is talking - me, cpt. Dummy, the t-error importer) to check the quality often in the early days of the station (to make sure everything works as it should).
automated recording quality check
On the other hand, and I don't want to discourage you from checking the quality manually, I have to warn you now: FRANCIS WAS HERE- it's really difficult to find a place in the world without graffiti saying exactly that and it always looks like this:
And we know what Francis does - he writes scripts to make things run automatically. And this is exactly what he did recently (November 2016).
Hi Mark and Jacek,
I've added a check of the video quality to the recording workflow, in the script check-mpg-single.sh and for testing in videotest.
The script acts on the HDHomeRun recording log. It divides the number of packets by the sum of different errors.
If the file is damaged, it sends an e-mail to recipients listed in /usr/local/bin/failure-email -- typically the local user csa, redhenlab.gmail, and for dola, Jacek.
That said, dola does not yet know how to send e-mails out; we might look into seeing if that can be arranged. In the meantime, it just generates a local e-mail.
Redalpha does know how to send out e-mails, so Mark and I will be notified of video recording failures.
The failures are ranked or graded according to the proportion of errors relative to the whole file -- from Perfect to Junk, as below.
An e-mail is triggered if the quality falls below Poor, or a score of 100. This requires manual intervention -- the file may be too damaged to be worth keeping.
If the quality sinks below 10, the file is summarily deleted, to avoid clogging up hoffman2 -- it will invariably fail repair, image extraction, and compression, and hog a processing node until I intervene.
We will likely want to tweak this as we get more experience; this is the first time we're using automated quality checking with clear consequences. So this doesn't free us from manual monitoring, but does remove and warn us about the most egregious failures.
Best wishes,
Francis
If you want to see Francis's script for checking the video quality, click HERE. Let's just look at a fragment:
# Calculate a ratio
VideoQuality="$( echo "scale=0 ; $PACKETS / ( $OVERFLOW + $NETWORK + $TRANSPORT + $SEQUENCE )" | bc )"
# Add quality labels
if [ "$VideoQuality" -ge 1000000 ] ; then QualityLabel=Perfect
elif [ "$VideoQuality" -ge 100000 ] ; then QualityLabel=Excellent
elif [ "$VideoQuality" -ge 10000 ] ; then QualityLabel=Good
elif [ "$VideoQuality" -ge 1000 ] ; then QualityLabel=OK
elif [ "$VideoQuality" -ge 100 ] ; then QualityLabel=Poor
elif [ "$VideoQuality" -ge 50 ] ; then QualityLabel=Damaged
elif [ "$VideoQuality" -ge 10 ] ; then QualityLabel=Bad
elif [ "$VideoQuality" -lt 10 ] ; then QualityLabel=Junk
fi
I just looked into one of my yesterday's logs ( cat 2016-11-14_1800_CZ_ČT1_Události-HD.log) and it has: 1013429 packets received, 0 overflow errors, 0 network errors, 65 transport errors, 11 sequence errors. Now, where is my pocket calculator? 1013429/(65+11) = ~ 13334 (you can use your RPI to calculate it, just go: echo $[1013429/$[65+11]] ). So - by Francis's standards the quality of this recording is slightly above GOOD but far from excellent. A question we'd all like to ask Francis is: "Sir, what if it is a very short recording, say 50 packets, with just 1 error? It'll be labeled as 'damaged'. Is it fair?" Well-Is anything?
To continue hitchhiking through the galaxy, go to Chapter 5.
