Brazil Capture Station

Task

Can you help establish a Red Hen capture station in Brazil? 
If so, write to 
and we will try to connect you with a mentor.

Related links

Linux resources

Support for the Brazilian ISDB-T television standard, which is similar to the standard used in Japan, has been added to the Linux kernel's DVB subsystem by Mauro Carvalho Chehab, Anshul Maheswari added ISDB subtitle support to CCExtractor in 2015, sponsored by a team of Brazilian scholars and Red Hen.

More information

The USB capture device PixelView Play TV USB SBTVD Full Seg (HDTV for Brazil), available from ProLink, works well in a Linux capture machine. While the PixelView is well-supported in Linux (using the dvb-usb-dib0700 kernel module, cf. CONFIG_DVB_USB_DIB0700; see instructions), we still need to verify that we can get it working. 

A promising candidate for a Linux capture station  the Raspberry Pi 3. For details, see How to set up a Red Hen capture station.

Note that the Raspberry Pi 3 has a power supply limit for the USB ports that appears to be 600mA (unverified). If the device consumption is higher than the limit, the device won't work. We do not yet know the power requirements of the PixelView PlayTV USB SBTVD. Other USB capture devices have power requirements as low as 250mA. It's possible we need a powered USB hub for the PixelView.

For CCExtractor, we need version 0.78 or later; see How to install CCExtractor.

Configuration

Add FQDN esfinge.virtua.com.br to /etc/hostname and /etc/mailname and /etc/hosts.
apt-get install netselect-apt wajig
cd /usr/bin ; ln -s wajig just
Run netselect-apt to identify the best local repository and add it to /etc/apt/sources.list. 
Add stable-backport and deb-multimedia repositories. 
just install dvb-apps screen dvbsnoop tightvncserver strace vlc mplayer

Kernel module

The driver for the PixelView PlayTV USB SBTVD capture device is the kernel module dvb-usb-dib0700. The firmware is available through firmware-linux-nonfree. The module didn't load on boot, so I issued:

       modprobe dvb-usb-dib0700

and got this response (the middle line is from before we installed the firmware):

        [   14.196753] dvb-usb: found a 'Prolink Pixelview SBTVD' in cold state, will try to load a firmware
        [   14.221158] usb 5-1: firmware: failed to load dvb-usb-dib0700-1.20.fw (-2)
        [   14.223825] usbcore: registered new interface driver dvb_usb_dib0700

On a fresh reboot, we see the device load the firmware and become active:

$ grep dvb 2016-01-15_dmesg-esfinge-02

[   14.214329] dvb-usb: found a 'Prolink Pixelview SBTVD' in cold state, will try to load a firmware

[   14.295585] usb 5-1: firmware: direct-loading firmware dvb-usb-dib0700-1.20.fw

[   15.000490] dvb-usb: found a 'Prolink Pixelview SBTVD' in warm state.

[   15.000663] dvb-usb: will pass the complete MPEG2 transport stream to the software demuxer.

[   15.542172] dvb-usb: schedule remote query interval to 50 msecs.

[   15.542180] dvb-usb: Prolink Pixelview SBTVD successfully initialized and connected.

[   15.544161] usbcore: registered new interface driver dvb_usb_dib0700


Configure the channels

Get the Brazilian ISDB-T frequency table and save it as FrequencyTable. Issue
        scan FrequencyTable >channels.conf 
On esfinge, we're picking up these channels:

RCI_Movel:473142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:529:530:32024

RCI_HD:473142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:711:731:32000

[bde0]:515142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:97:99:48608

[bdf8]:515142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:0:0:48632

[ebc0]:521142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:501:511:60352

[ebd8]:521142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:0:0:60376

[bd00]:533142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:513:514:48384

[bd18]:533142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:0:0:48408

[bdc0]:551142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:233:234:48576

[bdd9]:551142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:0:0:48601

[bce0]:563142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:273:274:48352

[bcf8]:563142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:0:0:48376

[bd20]:623142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:273:274:48416

[bd38]:623142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:0:0:48440

NGT Digital:659142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:110:111:48992

NGT 1SEG:659142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:210:211:49017

[ee20]:665142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:33:34:60960

[ee38]:665142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:0:0:60984

[ea60]:671142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:258:259:60000

[ea78]:671142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:0:0:60024

[ea7e]:671142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:0:0:60030

[ea7f]:671142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:0:0:60031

[e858]:707142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:0:0:59480

[e840]:707142857:INVERSION_AUTO:BANDWIDTH_6_MHZ:FEC_AUTO:FEC_AUTO:QAM_AUTO:
TRANSMISSION_MODE_AUTO:GUARD_INTERVAL_AUTO:HIERARCHY_NONE:273:274:59456


The logical device for the adapter is /dev/dvb/adapter0/dvr0:
            crw-rw----+ 1 root video 212, 1 Jan 15 23:45 /dev/dvb/adapter0/dvr0
Give user csa access (we may not need all of these):
            adduser csa video audio plugdev dip cdrom netdev
            chmod g+rw /dev/dvb/adapter0/dvr0
            
This is now working -- user csa has access to the adapter.

Make a recording

The channel script now works, so you can use
            channel 4, 38min, "Jornal Hoje", 1, 48352, "Morning news and special reports"
Record to file straight with gnutv:
            CH=NGT_Digital
            gnutv -out file $CH.mpg -timeout 100 -channels channels.conf $CH
            Using frontend "DiBcom 8000 ISDB-T", type DVB-T
            status S VYL | signal 4da2 | snr 0064 | ber 00000140 | unc 000017df | FE_HAS_LOCK
Try all the channels:
     for CH in `cat channels.conf | cut -d: -f1` ; do
          gnutv -out file $(date +%F_%H%M)_BR_"$CH"_test.mpg -timeout 100 -channels channels.conf $CH
     done
The bd20 (Globo) channel works!
Record half an hour:
      CH=bd20
      gnutv -out file $(date +%F_%H%M)_BR_"$CH"_test.mpg -timeout 1800 -channels channels.conf $CH

Here are the various streams according to ffprobe -- note the program number, "Program 48352":

Input #0, mpegts, from 'globo1.mpg':
  Duration: 00:01:59.73, start: 31897.135756, bitrate: 15775 kb/s
  Program 48352 
    Stream #0:0[0x111]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
    Stream #0:1[0x112](por): Audio: aac_latm (HE-AAC) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
    Stream #0:2[0x113](eng): Audio: aac_latm (HE-AAC) ([17][0][0][0] / 0x0011), 48000 Hz, 5.1, fltp
    Stream #0:3[0x116]: Unknown: none ([6][0][0][0] / 0x0006)
    Stream #0:4[0x384]: Unknown: none ([11][0][0][0] / 0x000B)
    Stream #0:5[0x1f4]: Unknown: none ([5][0][0][0] / 0x0005)
    Stream #0:6[0x114](por): Audio: aac_latm (HE-AAC) ([17][0][0][0] / 0x0011), 48000 Hz, stereo, fltp
    Stream #0:7[0x115](por): Audio: aac_latm (HE-AAC) ([17][0][0][0] / 0x0011), 48000 Hz, 5.1, fltp

The claim that stream 113 is English audio is likely not correct, but it's worth checking. The stream mapping for Globo should be "FFMAP="-map 0:0 -map 0:1" HBMAP="-a 1".

Extracting the text


We use ccextractor-0.82 (with ISDB timestamp fixes) to extract the text, first in debug mode:

csa@esfinge:~$ ccextractor-0.82 -debug -ts -noru -out=ttxt -utf8 -o globo1.ccx.out globo1.mpg

CCExtractor 0.82, Carlos Fernandez Sanz, Volker Quetschke.
Teletext portions taken from Petr Kutalek's telxcc
--------------------------------------------------------------------------
Input: globo1.mpg
[Extract: 1] [Stream mode: Transport]
[Program : Auto ] [Hauppage mode: No] [Use MythTV code: Auto]
[Timing mode: Auto] [Debug: Yes] [Buffer input: No]
[Use pic_order_cnt_lsb for H.264: No] [Print CC decoder traces: No]
[Target format: (null)] [Encoding: UTF-8] [Delay: 0] [Trim lines: No]
[Add font color data: Yes] [Add font typesetting: Yes]
[Convert case: No] [Video-edit join: No]
[Extraction start time: not set (from start)]
[Extraction end time: not set (to end)]
[Live stream: No] [Clock frequency: 90000]
[Teletext page: Autodetect]
[Start credits text: None]

-----------------------------------------------------------------
Opening file: globo1.mpg
Analyzing data in general mode

This TS file has more than one program. These are the program numbers found:
48352
48376

Program map section (PMT)
  H.264 video stream [0x1b]  -  PID: 273
  Unknown stream [0x11]  -  PID: 274
  Unknown stream [0x11]  -  PID: 275
*****ISDB subtitles detected
  MPEG-2 private data stream [0x06]  -  PID: 278
  ISO/IEC 13818-6 type B stream [0x0b]  -  PID: 900
  MPEG-2 private table sections stream [0x05]  -  PID: 500
  Unknown stream [0x11]  -  PID: 276
  Unknown stream [0x11]  -  PID: 277

So the ISDB subtitles are detected; note the two numbers above -- 48352 and 48376.

It turns out it's the first program number that contains the text (as ffprobe tells us above):

csa@esfinge:~$ ccextractor-0.82 -debug -ts -pn 48352 -noru -out=ttxt -utf8 -o globo1.ccx.out globo1.mpg

When we cat the output or read it in nano, the characters don't display correctly, but vi shows they are in fact correct:

20161114152002.000|20161114152012.699|CC1|>> [ZILEIDE SILVA] BOA TARDE.

20161114152002.000|20161114152012.699|CC1|UMA VISÃO ÚNICA EM 68 ANOS. A

20161114152002.000|20161114152012.699|CC1|>> [ZILEIDE SILVA] BOA TARDE.

20161114152012.699|20161114152015.764|CC1|MAIOR SUPER LUA DAS ÚLTIMAS

20161114152015.764|20161114152016.458|CC1|DÉCADAS ILUMINA O CÉU PELO MUNDO

20161114152016.458|20161114152019.292|CC1|E PODERÁ SER VISTA HOJE À NOITE

20161114152019.292|20161114152019.870|CC1|AQUI NO BRASIL. O CHÃO VOLTA A

20161114152019.870|20161114152021.663|CC1|TREMER NA NOVA ZELÂNDIA DEPOIS

20161114152021.663|20161114152023.398|CC1|DO TERREMOTO QUE MATOU DUAS

20161114152023.398|20161114152025.191|CC1|PESSOAS NO FIM DE SEMANA. A ONU

20161114152025.191|20161114152026.984|CC1|PREVÊ UM CALOR RECORDE PARA ESSE

20161114152026.984|20161114152027.909|CC1|ANO NO PLANETA. O PRESIDENTE

20161114152027.909|20161114152030.222|CC1|ELEITO DONALD TRUMP ANUNCIA DOIS

20161114152030.222|20161114152031.784|CC1|NOMES DA FUTURA EQUIPE DE

20161114152031.784|20161114152032.709|CC1|GOVERNO. E AQUI NO BRASIL,

20161114152032.709|20161114152035.081|CC1|ECONOMISTAS APOSTAM NUMA QUEDA

20161114152035.081|20161114152036.816|CC1|MAIS LENTA DO JUROS DEPOIS DA

20161114152036.816|20161114152039.649|CC1|ELEIÇÃO DE TRUMP. DISPARA O

20161114152039.649|20161114152040.922|CC1|NÚMERO DE PESSOAS QUE NÃO PAGAM

20161114152040.922|20161114152041.963|CC1|A CONTA DE LUZ E TEM A ENERGIA

20161114152041.963|20161114152045.549|CC1|CORTADA MAS A GENTE EXPLICA QUE

20161114152045.549|20161114152046.821|CC1|DÁ PARA PARCELAR A FATURA. E

20161114152046.821|20161114152048.787|CC1|MESMO COM A CRISE, O MERCADO DE

20161114152048.787|20161114152049.481|CC1|CARTÃO DE CRÉDITO NÃO PARA DE

20161114152049.481|20161114152051.274|CC1|CRESCER. O PERIGO ESTÁ NOS

20161114152051.274|20161114152055.033|CC1|JUROS. O MAIOR AVIÃO DO MUNDO

20161114152055.033|20161114152056.074|CC1|POUSA NO BRASIL E ATRAI UMA

20161114152056.074|20161114152057.636|CC1|MULTIDÃO DE CURIOSOS. E O JOGO

20161114152057.636|20161114152058.792|CC1|COMUM ENTRE JOVENS E

20161114152058.792|20161114152102.725|CC1|ADOLESCENTES GANHA ESPAÇO NO

20161114152102.725|20161114152110.012|CC1|TREINAMENTO DE EXECUTIVOS. O

20161114152110.012|20161114152111.979|CC1|JORNAL HOJE ESTÁ COMEÇANDO.



Comments