Mails from Francis - Chapter 4

manual recording quality check

But is it all always sweetness and light? No, it is not. There definitely are dark days:

Dear Jacek,

2016-10-03_1700_CZ_ČT1_Události-HD.log has an impressive number of transport errors; I removed the files.

Corrupt files gum up the works on hoffman2, so I might add a routine to detect them. In the meantime, let's remove them whenever we see them.

Francis

csa@dola:/tv/2016/2016-10/2016-10-03 $ cat 2016-10-03_1700_CZ_ČT1_Události*g

.ttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt, etc,

Impressive? Just terrible. Shameful. Who said only the British use understatement? Look at what Francis is saying. My station 'recorded' a file in which there is (almost) nothing but errors, and Francis calls it 'impressive'. How delicate. How kind. I have a shortcut for those transport errors - I call them t-errors. And I hate them.

However, ever since i put the hdhomerun device into an old tea-tin box (Faraday cage) and improved the roof-top antenna- all is (mostly) well (see also the TV antenna Appendix). We already talked about accessing those video.logs files in Chaptes 1 and 3 but a quick reminder: issue day, Enter. And you are now in today's recordings directory (day 1 - for yesterday, etc.). Then cat *.log and you will see all the log files and all the t-errors, etc. Or you may just want to see one specific log- then go, for example, cat 2016-11-14_1800_CZ_ČT1_Události-HD.log (use ls first to see the contents and copy the long names by marking them with a mouse and then right-clicking to get them where you cursor is at the moment).

In the above, as we can see, after I entered day - it reacted with indignation and said ? no: such and such directory. Don't be alarmed - the directory is created when the recording starts. And then I issued day 1 and ls to see yesterday's recordings, and finally cat *.log to see all those errors (t-errors). It's certainly advisable (look who is talking - me, cpt. Dummy, the t-error importer) to check the quality often in the early days of the station (to make sure everything works as it should).

automated recording quality check

On the other hand, and I don't want to discourage you from checking the quality manually, I have to warn you now: FRANCIS WAS HERE- it's really difficult to find a place in the world without graffiti saying exactly that and it always looks like this:

And we know what Francis does - he writes scripts to make things run automatically. And this is exactly what he did recently (November 2016).

Hi Mark and Jacek,

I've added a check of the video quality to the recording workflow, in the script check-mpg-single.sh and for testing in videotest.

The script acts on the HDHomeRun recording log. It divides the number of packets by the sum of different errors.

If the file is damaged, it sends an e-mail to recipients listed in /usr/local/bin/failure-email -- typically the local user csa, redhenlab.gmail, and for dola, Jacek.

That said, dola does not yet know how to send e-mails out; we might look into seeing if that can be arranged. In the meantime, it just generates a local e-mail.

Redalpha does know how to send out e-mails, so Mark and I will be notified of video recording failures.

The failures are ranked or graded according to the proportion of errors relative to the whole file -- from Perfect to Junk, as below.

An e-mail is triggered if the quality falls below Poor, or a score of 100. This requires manual intervention -- the file may be too damaged to be worth keeping.

If the quality sinks below 10, the file is summarily deleted, to avoid clogging up hoffman2 -- it will invariably fail repair, image extraction, and compression, and hog a processing node until I intervene.

We will likely want to tweak this as we get more experience; this is the first time we're using automated quality checking with clear consequences. So this doesn't free us from manual monitoring, but does remove and warn us about the most egregious failures.

Best wishes,

Francis

If you want to see Francis's script for checking the video quality, click HERE. Let's just look at a fragment:

# Calculate a ratio

VideoQuality="$( echo "scale=0 ; $PACKETS / ( $OVERFLOW + $NETWORK + $TRANSPORT + $SEQUENCE )" | bc )"

# Add quality labels

if [ "$VideoQuality" -ge 1000000 ] ; then QualityLabel=Perfect

elif [ "$VideoQuality" -ge 100000 ] ; then QualityLabel=Excellent

elif [ "$VideoQuality" -ge 10000 ] ; then QualityLabel=Good

elif [ "$VideoQuality" -ge 1000 ] ; then QualityLabel=OK

elif [ "$VideoQuality" -ge 100 ] ; then QualityLabel=Poor

elif [ "$VideoQuality" -ge 50 ] ; then QualityLabel=Damaged

elif [ "$VideoQuality" -ge 10 ] ; then QualityLabel=Bad

elif [ "$VideoQuality" -lt 10 ] ; then QualityLabel=Junk

fi

I just looked into one of my yesterday's logs ( cat 2016-11-14_1800_CZ_ČT1_Události-HD.log) and it has: 1013429 packets received, 0 overflow errors, 0 network errors, 65 transport errors, 11 sequence errors. Now, where is my pocket calculator? 1013429/(65+11) = ~ 13334 (you can use your RPI to calculate it, just go: echo $[1013429/$[65+11]] ). So - by Francis's standards the quality of this recording is slightly above GOOD but far from excellent. A question we'd all like to ask Francis is: "Sir, what if it is a very short recording, say 50 packets, with just 1 error? It'll be labeled as 'damaged'. Is it fair?" Well-Is anything?

To continue hitchhiking through the galaxy, go to Chapter 5.