Red Hen corpus data format

Introduction

The Red Hen data format has certain drawbacks for the use as a corpus that are to be remedied by the Red Hen corpus data format. It follows a relatively standard and easy-to-process pattern called "vertical format" that is used as input format by many corpus managers, e.g. CQP (the backend to CQPweb) or manatee (the backend to the SketchEngine and NoSektchEngine).

Related

• Overview of research (with dataset description)
• Red Hen data format
• Edge Search Engine Documentation (lists searchable tags)
• How to use the Edge2 search engine
• Current state of text tagging

Basic Concepts

There are two levels of representation in the Red Hen corpus data format:

1. Token-level annotation, i.e. every word/punctuation mark has annotation, e.g. Part-of-Speech, lemma, etc. These are called p[ositional]-attributes in CQP.
2. Annotation potentially spanning multiple words or not directly related to individual words, such as texts, sentences, pauses, gestures, etc. These are called s[tructural]-attributes in CQP.

Thus, in the following example, the lines containing the actual text have the p-attribute word in the first column, the p-attribute Part-of-Speech (in the Lancaster CLAWS5 tagset) in the second and the p-attribute lemma (i.e. the base form of the word) in the third column. The columns are not labeled in the file itself. The s-attributes in this snippet are text and s (s-unit, "a sentence-like division of a text", TEI). They can have multiple attributes (things like id, title, author, publisher) themselves.

<text id="file0001" title="Cat stories" author="Tom Cat" publisher="Feline Press">

<s id="1">

The    AT0    the

cat    NN1    cat

sat    VVD    sit

on     PRP    on

the    AT0    the

mat    NN1    mat

.      PUN    .

</s>

<s id="2">

Its    DPS    its

name   NN1    name

is     VBZ    be

Pi     NP0    pi

.      PUN    .

</s>

</text>

File-level metadata

The attributes within the <text> s-attribute correspond to the header information in the Red Hen data format. However, since every attribute can only have one value, fields with multiple values are distributed over multiple attributes (see the example of VID below).