Red Hen Google Summer of Code 2021

redhenlab@gmail.com

See Guidelines for Red Hen Developers and Guidelines for Red Hen Mentors

How to Apply

The great distinction between a course and a research opportunity is that in a course, the professor lays it out, gives assignments, start to finish. In research, the senior researchers are expected to do all of that themselves. They cogitate on the possibilities, use their library and networking skills to locate and review the state of the art, make judicious decisions about investment of time and other resources, and chart a path. Usually, the path they choose turns out to be a dead-end, but in research, success even some of the time is a great mark of distinction. Remember that research is about doing something that has not ever been done before. The junior researcher, or student learning to do research, is not expected to do everything that a senior researcher does, but is expected, first, to work continuously to learn how to improve by studying senior researchers, and, second, to explore general research opportunities picked out by senior researchers, review the literature, get a strong sense of the state of the art, and think about how it could be built upon. The junior researcher, having been directed to an area or areas by the senior researchers, is expected to find and read the research articles, explore the possibilities, and propose a tractable piece of work to undertake. The senior researchers then mentor now and then. The senior researchers are especially valuable for their experience, which usually gives them a much sharper sense of which path is more likely to be fruitful, but nonetheless, the senior researchers are sometimes surprised by what the junior researchers manage to hit upon. Junior researchers are largely self-organizing, self-starting, self-inspiring. A proposal from a student of the form, “I am highly motivated and know about X, Y, and Z and would love to do something related to topic W. Where do I start?” is not a research proposal. It looks like a request for a course, or a series of private, 1-on-1 tutorials. It is not appropriate for Red Hen Lab.

The Great Range of Projects that You Might Design

Red Hen Lab will consider any mature pre-proposal related to the study of multimodal communication. A pre-proposal is not a collaboration between a mentor and a student; rather, the mentor begins to pay attention once a reasonably mature and detailed outline for a pre-proposal is submitted. A mature pre-proposal is one that completes all the Template sections in a thorough and detailed manner. Red Hen lists a few project ideas below; more are listed in the Barnyard of Potential Possible Projects. But you are not limited to these lists. Do not write to ask whether you may propose something not on this list. The answer is, of course! We look forward to your mature and detailed pre-proposals.

Once you have a mature and detailed idea, and a pretty good sketch for the template above, you may send them to Red Hen to learn whether a mentor is interested in your idea and sketch, and then to receive some initial feedback and direction for finishing your pre-proposal. Red Hen mentors are extremely busy and influential people, and typically do not have time to respond to messages that do not include a mature and detailed idea and a pretty good sketch of the template above. Use the Template above to sketch your pre-proposal, print it to pdf, and send it to redhenlab@gmail.com. If a mentor is already listed for a specific project, send it also to that mentor.

The ability to generate a meaningful pre-proposal is a requirement for joining the team; if you require more hand-holding to get going, Red Hen Lab is probably not the right organization for you this year. Red Hen wants to work with you at a high level, and this requires initiative on your part and the ability to orient in a complex environment. It is important that you read the guidelines of the project ideas, and you have a general idea of the project before writing your pre-proposal.

When Red Hen receives your pre-proposal, Red Hen will assess it and attempt to locate a suitable mentor; if Red Hen succeeds, she will get back to you and provide feedback to allow you to develop a fully-fledged proposal to submit to GSoC 2021. Note that your final proposal must be submitted directly to Google, not to redhenlab@gmail.com

Red Hen is excited to be working with skilled students on advanced projects and looks forward to your pre-proposals.

Know Red Hen Before You Apply

Red Hen Lab is an international cooperative of major researchers in multimodal communication, with mentors spread around the globe. Together, the Red Hen cooperative has crafted this Ideas page, which offers some information about the Red Hen dataset of multimodal communication (see some sample data here and here) and a long list of tasks.

To succeed in your collaboration with Red Hen, the first step is to orient yourself carefully in the relevant material. The Red Hen Lab website that you are currently visiting is voluminous. Please explore it carefully. There are many extensive introductions and tutorials on aspects of Red Hen research. Make sure you have at least an overarching concept of our mission, the nature of our research, our data, and the range of the example tasks Red Hen has provided to guide your imagination. Having contemplated the Red Hen research program on multimodal communication, come up with a task that is suitable for Red Hen and that you might like to embrace or propose. Many illustrative tasks are sketched below. Orient in this landscape, and decide where you want to go.

The second step is to formulate a pre-proposal sketch of 1-3 pages that outlines your project idea. In your proposal, you should spell out in detail what kind of data you need for your input and the broad steps of your process through the summer, including the basic tools you propose to use. Give careful consideration to your input requirements; in some cases, Red Hen will be able to provide annotations for the feature you need, but in other cases successful applicants will craft their own metadata, or work with us to recruit help to generate it. Please use the Latex template to write your pre-proposal, and send us the pdf format.

Red Hen emphasizes: Red Hen has programs and processes—see, e.g., her Τέχνη Public Site, Red Hen Lab's Learning Environment—for tutoring high-school and college students. But Red Hen Google Summer of Code does not operate at that level. Red Hen GSoC seeks mature students who can think about the entire arc of a project: how to get data, how to make datasets, how to create code that produces an advance in the analysis of multimodal communication, how to put that code into production in a Red Hen pipeline. Red Hen is looking for the 1% of students who can think through the arc of a project that produces something that does not yet exist. Red Hen does not hand-hold through the process, but she can supply elite and superb mentoring that consists of occasional recommendations and guidance to the dedicated and innovative student.

Requirements for Commitment

In all but exceptional cases, recognized as such in advance, your project must be put into production by the end of Google Summer of Code or you will not be passed or paid. Most projects will create a pipeline or contribute to an existing pipeline in the Red Hen central operations. This can mean, e.g., scripting (typically in bash) an automated process for reading input files from Red Hen's data repository, submitting jobs to the CWRU HPC using the Slurm workload manager, running your code, and finally formatting the output to match Red Hen's Data Format. Consider these requirements as opportunities for developing all-round skills and for being proud of having written code that is not only merged but in regular production! Explore the current Red Hen Lab pipelines and think about how your project would work with them.

Tips for working with your mentors

Note that your project must be implemented inside a Singularity container (see instructions). This makes it portable between Red Hen's high-performance computing clusters. Red Hen has no interest in toy, proof-of-concept systems that run on your laptop or in your user account on a server. Red Hen is dedicated exclusively to pipelines and applications that run on servers anywhere and are portable. Please study Guidelines for Red Hen Developers, and master the section on building Singularity containers. You are required to maintain a github account and a blog.

In almost all cases, you will do your work on CWRU HPC, although of course you might first develop code on your device and then transfer it to CWRU HPC. On CWRU HPC, do not try to sudo; do not try to install software. Check for installed software on CWRU HPC using the command

module

e.g.,

module spider singularity

module load gcc

module load python

On CWRU HPC, do not install software into your user account; instead, if it is not already installed on CWRU HPC, install it inside a Singularity container so that it is portable. Red Hen expects that Singularity will be used in 95% of cases. Why Singularity? Here are 4 answers; note especially #2 and #4:

What is so special about Singularity?

While Singularity is a container solution (like many others), Singularity differs in its primary design goals and architecture:

• 1. Reproducible software stacks: These must be easily verifiable via checksum or cryptographic signature in such a manner that does not change formats (e.g. splatting a tarball out to disk). By default Singularity uses a container image file which can be checksummed, signed, and thus easily verified and/or validated.

  2. Mobility of compute: Singularity must be able to transfer (and store) containers in a manner that works with standard data mobility tools (rsync, scp, gridftp, http, NFS, etc..) and maintain software and data controls compliancy (e.g. HIPPA, nuclear, export, classified, etc..)

  3. Compatibility with complicated architectures: The runtime must be immediately compatible with existing HPC, scientific, compute farm and even enterprise architectures any of which maybe running legacy kernel versions (including RHEL6 vintage systems) which do not support advanced namespace features (e.g. the user namespace)

  4. Security model: Unlike many other container systems designed to support trusted users running trusted containers, we must support the opposite model of untrusted users running untrusted containers. This changes the security paradigm considerably and increases the breadth of use cases we can support.

A few further tips for rare, outlier cases:

1. In rare cases, if you feel that some software should be installed by CWRU HPC rather than inside your Singularity container, write to us with an argument and an explanation, and we will consider it.

   1. In rare cases, if you feel that Red Hen should install some software to be shared on gallina but not otherwise available to the CWRU HPC community, explain what you have in mind, and we will consider it.

Remember to study the blogs of other students for tips, and document on your own blogs anything you think would help other students.

More Tips for Working with your Mentors

• 1. Rely on your network and take the lead in building it. It is easy to think of GSoC as a coding job, but the sociology of the operation is at least as important. This is your chance to work on the inside of a high-level global collaboratory and see how such a network thrives. Take the lead in developing your community, your network, your resources.

  2. For everyday learning and help with coding, mentors are only a last resort. Students are paid; mentors are not. The mentors are volunteers who see value in giving some of their time to helping the student and the project, but they are extremely busy people, with many responsibilities. They are angels, but appear only when it's actually necessary. GSoC is not a coding collaboration between you and your mentor. Most of the help you will need should come from the other students in your cohort, from your own research on how to pick up skills, from your connecting with people in your network. The mentor is available for high-level guidance on the goal, strategy, and timeline of the project. But if you encounter routine difficulties, your first request for help should not go to your mentor.

  3. All students in Red Hen Lab GSoC will, during the community bonding period, be put through a common package of setup tasks. Instructions will be provided. Students in a year's cohort always find it useful to establish some communication channel, such as Slack, through which they can mentor each other as needed through the accomplishment of the initial setup. After everyone has completed setup, the Org Admins will schedule a meet-and-greet group videoconference. Thereafter, each student will work principally on his or her self-guided project but should continue to rely on the network of students throughout the GSoC period.

  4. Red Hen requires that you document and explain everything in your blog and github, right down to the commands and code and steps needed to accomplish anything in your project. Think of it this way: suppose that, once you have completed GSoC by installing and demonstrating your working pipeline, fully in production, a later student comes along and wants to build on your work. Of course, we would ask you to mentor; and we hope that you will stay active in Red Hen and keep your project and similar projects going. Many Red Hen mentors were once Red Hen students. Your blog and your github must supply everything in a clear way for that student to hit the ground running (except security credentials in Red Hen). What would that student need to understand, know, re-use, imitate? Put that all in your blog and github. Red Hen needs a full post from you at least weekly. Many people—in Red Hen, universities, tech companies, etc.—will be looking at your blog.

  5. Work closely from the beginning with your mentor on installing the production system. Red Hen is not interested in toy or proof-of-concept efforts. Be sure to work with your mentor on a plan for actually installing and testing your production system before the final evaluation.

Background Information

Red Hen Lab participated in Google Summer of Code in 2015, 2016, 2017, 2018, 2019, 2020, and 2021, working with brilliant students and expert mentors from all over the world. Each year, Red Hen has mentored students in developing and deploying cutting-edge techniques of multimodal data mining, search, and visualization, with an emphasis on automatic speech recognition, tagging for natural language, co-speech gesture, paralinguistic elements, facial detection and recognition, and a great variety of behavioral forms used in human communication. With significant contributions from Google Summer of Code students from all over the world, Red Hen has constructed tagging pipelines for text, audio, and video elements. These pipelines are undergoing continuous development, improvement, and extension. Red Hens have excellent access to high-performance computing clusters at UCLA, Case Western Reserve University, and FAU Erlangen; for massive jobs Red Hen Lab has an open invitation to apply for time on NSF's XSEDE network.

Red Hen's largest dataset is the NewsScape Library of International Television News, a collection of more than 600,000 television news programs, initiated by UCLA's Department of Communication, developed in collaboration with Red Hens from around the world, and curated by the UCLA Library, with processing pipelines at UCLA, Case Western Reserve University, and FAU Erlangen in Germany. Red Hen develops and tests tools on this dataset that can be used on a great variety of data—texts, photographs, audio and audiovisual recordings. Red Hen also acquires big data of many kinds in addition to television news, such as photographs of Medieval art, and is open to the acquisition of data needed for particular projects. Red Hen creates tools that are useful for generating a semantic understanding of big data collections of multimodal data, opening them up for scientific study, search, and visualization. See Overview of Research for a description of Red Hen datasets.

In 2015, Red Hen's principal focus was audio analysis; see the Google Summer of Code 2015 Ideas page. Red Hen students created a modular series of audio signal processing tools, including forced alignment, speaker diarization, gender detection, and speaker recognition (see the 2015 reports, extended 2015 collaborations, and github repository). This audio pipeline is currently running on Case Western Reserve University's high-performance computing cluster, which gives Red Hen the computational power to process the hundreds of thousands of recordings in the Red Hen dataset. With the help of GSoC students and a host of other participants, the organization continues to enhance and extend the functionality of this pipeline. Red Hen is always open to new proposals for high-level audio analysis.

In 2016, Red Hen's principal focus was deep learning techniques in computer vision; see the Google Summer of Code 2016 Ideas page and Red Hen Lab page on the Google Summer of Code 2016 site. Talented Red Hen students, assisted by Red Hen mentors, developed an integrated workflow for locating, characterizing, and identifying elements of co-speech gestures, including facial expressions, in Red Hen's massive datasets, this time examining not only television news but also ancient statues; see the Red Hen Reports from Google Summer of Code 2016 and code repository. This computer vision pipeline is also deployed on CWRU's HPC in Cleveland, Ohio, and was demonstrated at Red Hen's 2017 International Conference on Multimodal Communication. Red Hen is planning a number of future conferences and training institutes. Red Hen GSoC students from previous years typically continue to work with Red Hen to improve the speed, accuracy, and scope of these modules, including recent advances in pose estimation.

In 2017, Red Hen invited proposals from students for components for a unified multimodal processing pipeline, whose purpose is to extract information about human communicative behavior from text, audio, and video. Students developed audio signal analysis tools, extended the Deep Speech project with Audio-Visual Speech Recognition, engineered a large-scale speaker recognition system, made progress on laughter detection, and developed Multimodal Emotion Detection in videos. Focusing on text input, students developed techniques for show segmentation, neural network models for studying news framing, and controversy and sentiment detection and analysis tools (see Google Summer of Code 2017 Reports). Rapid development in convolutional and recurrent neural networks is opening up the field of multimodal analysis to a slew of new communicative phenomena, and Red Hen is in the vanguard.

In 2018, Red Hen GSoC students created Chinese and Arabic ASR (speech-to-text) pipelines, a fabulous rapid annotator, a multi-language translation system, and multiple computer vision projects. The Chinese pipeline was implemented as a Singularity container on the Case HPC, built with a recipe on Singularity Hub, and put into production ingesting daily news recordings from our new Center for Cognitive Science at Hunan Normal University in Hunan Province in China, directed by Red Hen Lab Co-Director Mark Turner. It represents the model Red Hen expects projects in 2019 to follow.

In 2019, Red Hen Lab GSoC students made significant contributions to add speech to text and OCR to Arabic, Bengali, Chinese, German, Hindi, Russian, and Urdu. We built a new global recording monitoring system, developed a show-splitting system for ingesting digitized news shows, and made significant improvements to the Rapid Annotator. For an overview with links to the code repositories, see Red Hen Lab's GSoC 2019 Projects.

In 2020, Red Hen focused on 8 separate themes, all available on the report from that year.

For 2021, see details below.

In large part thanks to Google Summer of Code, Red Hen Lab has been able to create a global open-source community devoted to computational approaches to parsing, understanding, and modeling human multimodal communication. With continued support from Google, Red Hen will continue to bring top students from around the world into the open-source community.

The profiles of mentors not included in the portrait gallery are linked to their name below.

More guidelines for project ideas

Your project should be in the general area of multimodal communication, whether it involves tagging, parsing, analyzing, searching, or visualizing. Red Hen is particularly interested in proposals that make a contribution to integrative cross-modal feature detection tasks. These are tasks that exploit two or even three different modalities, such as text and audio or audio and video, to achieve higher-level semantic interpretations or greater accuracy. You could work on one or more of these modalities. Red Hen invites you to develop your own proposals in this broad and exciting field.

Red Hen studies all aspects of human multimodal communication, such as the relation between verbal constructions and facial expressions, gestures, and auditory expressions. Examples of concrete proposals are listed below, but Red Hen wants to hear your ideas! What do you want to do? What is possible? You might focus on a very specific type of gesture, or facial expression, or sound pattern, or linguistic construction; you might train a classifier using machine learning, and use that classifier to identify the population of this feature in a large dataset. Red Hen aims to annotate her entire dataset, so your application should include methods of locating as well as characterizing the feature or behavior you are targeting. Contact Red Hen for access to existing lists of features and sample clips. Red Hen will work with you to generate the training set you need, but note that your project proposal might need to include time for developing the training set.

Red Hen develops a multi-level set of tools as part of an integrated research workflow, and invites proposals at all levels. Red Hen is excited to be working with the Media Ecology Project to extend the Semantic Annotation Tool, making it more precise in tracking moving objects. The "Red Hen Rapid Annotator" is also ready for improvements. Red Hen is open to proposals that focus on a particular communicative behavior, examining a range of communicative strategies utilized within that particular topic. See for instance the ideas "Tools for Transformation" and "Multimodal rhetoric of climate change". Several new deep learning projects are on the menu, from "Hindi ASR" to "Gesture Detection and Recognition". On the search engine front, Red Hen also has several candidates: the "Development of a Query Interface for Parsed Data" to "Multimodal CQPweb". Red Hen welcomes visualization proposals; see for instance the "Semantic Art from Big Data" idea below.

Red Hen is now capturing television in China, Egypt, and India and is happy to provide shared datasets and joint mentoring with our partners CCExtractor, who provides the vital tools for text extraction in several television standards, for on-screen text detection and extraction..

When you plan your proposal, bear in mind that your project should result in a production pipeline. For Red Hen, that means it finds its place within the integrated research workflow. The application will typically be required to be located within a Singularity module that is installed on Red Hen's high-performance computing clusters, fully tested, with clear instructions, and fully deployed to process a massive dataset. The architecture of your project should be designed so that it is clear and understandable for coders who come after you, and fully documented, so that you and others can continue to make incremental improvements. Your module should be accompanied by a python application programming interface or API that specifies the input and output, to facilitate the construction of the development of a unified multimodal processing pipeline for extracting information from text, audio, and video. Red Hen prefers projects that use C/C++ and python and run on Linux. For some of the ideas listed, but by no means all, it's useful to have prior experience with deep learning tools.

Your project should be scaled to the appropriate level of ambition, so that at the end of the summer you have a working product. Be realistic and honest with yourself about what you think you will be able to accomplish in the course of the summer. Provide a detailed list of the steps you believe are needed, the tools you propose to use, and a weekly schedule of milestones. Chose a task you care about, in an area where you want to grow. The most important thing is that you are passionate about what you are going to work on with us. Red Hen looks forward to welcoming you to the team!

Ideas for Projects

Red Hen strongly emphasizes that a student should not browse the following ideas without first having read the text above them on this page. Red Hen remains interested in proposals for any of the activities listed throughout this website (http://redhenlab.org).

See especially the

Barnyard of Possible Specific Projects

Red Hen is uninterested in a preproposal that merely picks out one of the following ideas and expresses an interest. Red Hen looks instead for an intellectual engagement with the project of developing open-source code that will be put into production in our working pipelines to further the data science of multimodal communication. What is your full idea? Why is it worthy? Why are you interested in it? What is the arc of its execution? What data will you acquire, and where? How will you succeed?

Please read the instructions on how to apply carefully before applying for any project. Failing to follow the guidelines of the application will result in your (pre)proposal's not being considered for GSoC2021.

1. IPTV capture

Red Hen looks to expand her capture to include IPTV. Resources such as BBC iPlayer and Channel4 in the UK, RTVE in Spain, or ARD and ZDF Mediathek in Germany need to be channeled into Red Hen's standard scheduling and capture pipelines, including closed-captions or subtitles that are included in the broadcast stream. Red Hen routinely uses youtube-dl for such captures. You are asked to (1) extend youtube-dl's capabilities; see, e.g. https://github.com/ytdl-org/youtube-dl/issues/16779#issuecomment-781608403; and (2) create robust automated ingestion of IPTV broadcast into the Red Hen format. See http://redhenlab.org for specifics on Red Hen data format.

Mentors: Francis Steen, Jacek Wózny, Melanie Bell, and Javier Valenzuela.

2. Develop a system for manual joint annotation

While tools like OpenPose can help annotate joint key points in images or videos where the people are fully observable, there are many cases where these tools perform poorly due to occlusion. The problem of pose estimation with partial observation requires data with manual joint annotation. Red Hen is interested in developing a system that can help with this process.

Ideally, this tool should allow human annotators to select certain frames from a video and click on certain positions of each frame to annotate the joint key points. Some possible key points are Left Ankle, Left Knee, Left Hip, Left Wrist, Left Elbow, Left Shoulder, Left Ear, Left Eye, Right Ankle, Right Knee, Right Hip, Right Wrist, Right Elbow, Right Shoulder, Right Ear, Right Eye, Nose, Top Head, Neck. The annotator should also have the option to create new key point names. To improve the accuracy of this system, there should be both an option to enter the exact pixel position for a key point and an option to select and drag existing key points on the frame to adjust their positions.

To make the annotation process faster, when the annotator moves to a subsequent frame of an already annotated frame, there should be an option to display the previous positions of the key points so that they can be dragged to the correct positions in the new frame. Alternatively, estimated positions of these joint key points can be calculated and displayed using interpolation with the key points of previous or following annotated frames. If possible, the tool can also run OpenPose (or another tool that can estimate joint key points) on a frame and display the results so that the annotator can just drag the labels to the correct positions to finish the annotation of that frame.

This functionality needs to be integrated into Red Hen Rapid Annotator. Follow this link; view the talk; and see below. One possible starting point is the VGG Image Annotator. Languages needed include JavaScript and Python.

3. Design and develop a faceted search engine for Red Hen data

Red Hen Lab's data is under continual development, with a large number of data types and new types arriving at a steady clip. Our current search engines are functional, but they also have shortcomings. We invite proposals for the design and implementation of a faceted search engine, meaning a search engine capable of searching and displaying a wide array of different types of data.

For this project, the successful candidate will need to develop ideas centered around UX or user experience, in addition to implementing these ideas after gaining approval from Red Hen's mentors. The code of our current search engines will be made available and can be utilized and built on to speed up the implementation process.

Your task is to design and code a prototype that can search and query a particular annotated dataset that Red Hen Lab controls. It consists of 3,000 hours of audiovisual recordings, with full OpenPose annotations.

4. Create a Red Hen Open Dataset for gestures with performance baselines

Red Hen Lab has an extensive dataset of gestures in a talk show, expertly annotated by top gesture scholars. It comprises roughly 1,000 gestures over 150 minutes of recordings. The task is to organize the data, systematically characterize the annotations, and develop baseline performance benchmarks for machine learning. The annotations are made in ELAN and can be exported as xml or csv files. The videos are on Youtube and in Red Hen datasets. The annotations were performed by gesture scholars and should be systematized for computer science researchers.

Each major category in the systematized characterization of the annotations should be utilized as a training dataset for machine learning. The successful candidate will have the skills required to develop appropriate multi-layer neural networks and use deep learning protocols to train one or more classifiers for each category. These will serve as baseline performance measures for users of the Open Dataset.

5. Develop a system for using OpenPose to tag video, based on existing ELAN annotations

See the description under #2, above. This is a subcategory of #2, focused exclusively on the use of OpenPose. There are two approaches that we want to explore in this project: (1) A rule-based approach in the vein of Spudnig, https://github.com/jorrip/SPUDNIG. Here the challenge is the varied nature of TV data [multiple people on screen, different camera angles, changes in scene, occlusions, etc. . . . ]. (2) A machine-learning system, taking the Red Hen dataset described in #2 and #3 above, into account as a resource for training. This dataset consists of recordings from the Ellen DeGeneres talk show. The goal is to train different models for different camera angles evidenced in the dataset. This project would explore the usefulness of systems such as http://ai.googleblog.com/2021/01/recognizing-pose-similarity-in-images.html. This system seeks to recognize 3D similarities.

6. Gesture temporal detection pipeline for news videos. Must use the Latex Template.

Mentored by Mahnaz Parian <mahnaz.parian-scherb@unibas.ch> and Heiko Schuldt's team

Red Hen invites proposals to build a gesture temporal detection pipeline. For gesture detection, a good starting point is OpenPose, and a useful extension is hand keypoint detection. Our dataset is around 600,000 hours of television news recordings in multiple languages, so the challenge is to obtain good recall rates with this particular content.

For the GSoC gesture project, Red Hen has the following goals:

• • Build a system inside a Singularity container for deployment on high-performance computing clusters (see instructions)

  • Reliably detect the presence or absence of hand gestures

A good command of python and deep learning libraries (Tensorflow/caffe/Keras) is necessary. Please see here for more information regarding proposals.

7. Red Hen Rapid Annotator

Mentored by Peter Uhrig and Vaibhav Gupta

This task is aimed at extending the Red Hen Rapid Annotator, which was re-implemented from scratch as a Python/Flask application during GSoC 2018 and improved in GSoC 2019 and GSoC2020. Still, there are some bugs and feature requests.

Please familiarize yourself with the project and play around with it.

A good command of Python and HTML5/Javascript are necessary for this project.

8. System Integration of Existing Tools Into a New Modular Multimodal Pipeline

Red Hen is integrating multiple separate processing pipelines into a single new multimodal pipeline. Orchestrating the processing of hundreds of thousands of videos on a high-performance computing cluster along multiple dimensions is a challenging design task. The winning design for this task will be flexible, but at the same time make efficient use of CPU cycles and file accesses, so that it can scale. Pipelines to be integrated include:

• 1. Shot detection

  2. Commercial detection

  3. Speaker recognition

  4. Frame annotation (for English)

  5. Text and Story segmentation

  6. Sentiment Analysis

  7. Emotion detection

  8. Gesture detection

This infrastructure task requires familiarity with Linux, bash scripting, and a range of programming languages such as Java, Python, and Perl, used in the different modules. The output of all modules will need to be added as additional columns to our vertical format. Here is documentation of the vertical format from a different project: http://fedora.clarin-d.uni-saarland.de/teaching/Comparing_Corpora_Tutorials/Tutorial_VRT.html .

Write to us if you need an example of the kind of files we work with.

9. Integration of Gesture Retrieval into vitrivr. Must use the Latex Template

Mentored by Mahnaz Parian <mahnaz.parian-scherb@unibas.ch> and Luca Rossetto.

Gestures are a common component of daily communications where can carry some of the weight of spoken language. Query by gesture can be used in different contexts to search for gestures that accompanied the spoken words. This is done in collaboration with vitrivr, a multimodal retrieval system, on the basis of the newscape video collections and the semantic annotations.

For this project, we have the following goals:

• Integrate the gesture feature extraction into Cineast

• Adjust the vitrivr UI to accommodate necessary filters and query modes

• Test the setup on Newscape Dataset.

A good command of python and deep learning libraries (Tensorflow/caffe/Keras) plus a very good knowledge of Java and typescript is necessary. Please see here for more information regarding proposals.

10. CQPweb plugins (and plugin structure)

Red Hen uses an open-source software called CQPweb to facilitate linguistic research. However, CQPweb is not yet fully equipped to handle audio and video data, so it needs modifications for our purposes. Your task is to create plugins for audio analysis using the EMU webApp and better query options (e.g. the ability to search by sounds using IPA symbols), and additional downloaders for ELAN and Praat files. Where CQPweb's plugin structure cannot cater to our needs, you will submit merge requests to the CQPweb codebase. Proficiency in PHP, JavaScript and HTML is required.

Mentors: Peter Uhrig, Javier Valenzuela, and others

11. CQP and CQPweb date ranges

This project requires an understanding of C/C++ as well as PHP and MySQL. Your task is to implement a date range query feature (e.g. "Give me all the instances of x uttered between 5 September 2016 and 3 March 2018") in both CQP (the backend of CWB) and the CQPweb frontend. Please familiarize yourself with the codebase of CWB before applying for this project!

Mentors: Peter Uhrig, Javier Valenzuela, and others

12. Development of a Query Interface for Parsed Data

Mentored by Peter Uhrig's team

This infrastructure task is to create a new and improved version of a graphical user interface for graph-based search on dependency-annotated data. The new version should have all functionality provided by the prototype plus a set of new features. The back-end is already in place.

Develop current functionality:

• • add nodes to the query graph

  • offer choice of dependency relation, PoS/word class based on the configuration in the database (the database is already there)

  • allow for use of a hierarchy of dependencies (if supported by the grammatical model)

  • allow for word/lemma search

  • allow one node to be a "collo-item" (i.e. collocate or collexeme in a collostructional analysis)

  • color nodes based on a finite list of colors

  • paginate results

  • export xls of collo-items

  • create a JSON object that represents the query to pass it on to the back-end

Develop new functionality:

• • allow for removal of nodes

  • allow for query graphs that are not trees

  • allow for specification of the order of the elements

  • pagination of search results should be possible even if several browser windows or tabs are open.

  • configurable export to csv for use with R

  • compatibility with all major Web Browsers (Edge, Firefox, Chrome, Safari)

  • parse of example sentence can be used as the basis of a query ("query by example")

Steps:

• 1. Visit http://www.treebank.info and play around with the interface (user: gsoc2018, password: redhen) [taz is a German corpus, the other two are English]

  2. In consultation with Red Hen, decide on a suitable JavaScript Framework, possibly combined with Python/Flask.

Contact Peter Uhrig <peter.uhrig@fau.de> to discuss details or to ask for clarification on any point.

13. AI Recognizers of Frame Blends

Mentored by Wenyue Xi and Mark Turner. The purpose of this project is to extend the highly-successful work done by Wenyue Xi during Google Summer of Code 2020. Mark Turner was her mentor. Wenyue and Mark will mentor this project. Study Wenyue Xi's blog and github page at Red Hen Lab GSoC 2020 Projects.

Contact turner@case.edu to discuss details or ask for clarification.

Red Hen already has a frame tagging system for English that exploits FrameNet; for details, see Tagging for Conceptual Frames. Red Hen Lab works closely with Framenet Brasil, another Google Summer of Code organization, and is eager to involve other languages in her tagging of conceptual frames. Conceptual blending of frames is a major area of research in cognitive science and cognitive linguistics. Can we develop a system that locates them in language and images? Wenyue Xi's Frame Blend Nomination System does just that. Study http://redhenlab.org to familiarize yourself with the Red Hen data holdings and other existing tools before submitting a pre-proposal for this project.

14. Detecting Joint Meaning Construal by Language and Gesture

(This idea will be developed in a co-mentorship project with the Red Hen Lab. Applicants may choose whether they will apply to FrameNet Brasil or Red Hen. However, if the student gets accepted, mentors from both labs will be involved in the mentorship.)

Mentors: Francis Steen (Red Hen) | Fred Belcavello (FN-Br | UFJF) | Mark Turner (Red Hen) | Tiago Torrent (FN-Br | UFJF)

General Context:

Both FrameNet Brasil and Red Hen have been investigating how meaning is construed in multimodal communication. While Red Hen has been focusing more on the relation between speech and co-speech gestures, FN-Br has been looking into how frames are evoked by different modalities, especially audio and video. In both cases, however, research interest revolves around how different modalities interact for meaning production.

The Idea:

For this idea, we expect projects focused on identifying joint meaning construal patterns. Recent work has defined a non-exhaustive list of construal dimensions, which could be used for inspiration. Also, Red Hen has a collection of multimodal corpora already annotated for the kind of co-speech gesture that accompanies speech. Good examples of these kinds are air quotes gestures, which can accompany very different types of speech with very different functions.

Why this Idea is Innovative:

Although research in multimodal communication has advanced greatly in the past decade, practitioners in the field still fall short in ways of analyzing how meaning is construed from the interaction between modalities in large amounts of data. A successful implementation of this idea would then allow for human in the loop solutions for annotating patterns of joint meaning construal in multimodal communication.

15. Machine detection of film edits

Students in film school study textbook types of video cuts. See, for example, The Cutting Edge. Red Hen seeks proposals for code that would automatically tag data for such standard film cuts. For inspiration, see https://filmcolors.org/ and http://mediaecology.dartmouth.edu/wp/ and http://www.bmva.org/bmvc/2001/papers/111/accepted_111.pdf. See also http://www.cinemetrics.lv/.

16. Project Idea: Developing the Visual Recognition of Aztec Hieroglyphs

This project involves collaborating with Aztec-language specialist Stephanie Wood, Ph.D., at the University of Oregon, who is deciphering the hieroglyphic writing system for Nahuatl that was in use prior to contact with Europeans in 1519 and for some decades afterwards. There are many manuscripts ("codices") from the sixteenth century that employ these glyphs, but there is no comprehensive database of glyphs.

Dr. Wood is still in the process of building the database of images and their added metadata and analysis. The idea for this project is to create a decipherment aid by subjecting the images in her database (currently about 1200 images, but continuously growing in number) to visual recognition software and then create a module or companion website whereby users could upload hieroglyphs, one at a time, from a manuscript they are studying, whereby their glyphs could then be compared against the database of identified glyphs, and finally, the decipherment tool would show possible matches.

Another, more complicated goal takes this process farther. Some glyphs are compounds of atomic glyphs, and there is a hope that visual recognition would also be able to help identify not just atomic glyphs but also the various parts of a compound glyph that might be submitted by a user. For example, there is a compound hieroglyph for the place name Coaixtlahuacan consisting of three atomic glyphs: coa(tl), snake; ix(tli), eyes; and, plains, ixtlahua(tl). Perhaps visual recognition could also help isolate the atomic parts, which are currently isolated and digitally cut out by hand, using Photoshop.

Some of the challenges of mechanizing the visual recognition of atomic parts: eyes, or ix(tli), will sometimes appear upside-down or right-side-up, they can appear singly or in multiples. Snakes, or coa(tl) are sometimes stretched out straight like the one being shared or sometimes undulating, and their colors will vary. Sometimes plains, or ixtlahua(tl), will be all one color, such as purple, or have alternating orange and purple segments. Sometimes animals appear in full body and sometimes just as heads. Human hands can be just hands or arms with hands. The human "shoulder" is a full arm with a protruding bone at the site of a would-be shoulder. Hieroglyphs of water can be pools, drips, sprinkles, rivers, etc.; they are almost always turquoise blue, but they can even be red. Our analytical input will be required to mitigate such challenges. Additional challenges will come when we have glyphs from more than one manuscript, where the scribes have very different styles, where the size of the glyphs will change, where the color palette may be different, and more.

Besides developing this visual recognition decipherment tool to add onto the Drupal database as a module--or perhaps a companion website, if that is easier--the end goal would also be to publish an article about the digital humanities challenges of processing cultural heritage material in these ways. We will also go after additional funding to enhance the user's options for locating, identifying, and deciphering glyphs.