Speech Gestures in Art

Introduction

Images of speech gestures in art.

Some papers have already done OpenPose on art collections, e.g. https://biblhertz.github.io/atlas/ or https://litlab.stanford.edu/LiteraryLabPamphlet16.pdf Make sure you refer to past papers that do computer vision on art (e.g. for iconography) - take a look at the ECCV VISART workshops.

See especially https://www.researchgate.net/publication/263525615_Nonverbal_Communication_in_Medieval_Illustrations_Revisited_by_Computer_Vision_and_Art_History.

Pretrained models have already been applied for gesture in art -- e.g. https://biblhertz.github.io/atlas/ uses CMU's OpenPose, or https://litlab.stanford.edu/LiteraryLabPamphlet16.pdf. How would you do it differently? Hand gestures remain an open problem.

Contacts:

• • Prof. Dr. Line Engh, Medieval History, University of Oslo, Norway. She is a Medieval scholar with a focus on multimodal communication. She led a Red Hen GSoC project on the automated parsing of Roman portraits and assembled the dataset below.

  • Prof. Dr. Peter Bell, Digital Humanities / Art History, FAU Erlangen-Nürnberg, Germany. He is an art historian with focus on medieval and early modern art, specifically on christian iconography and computer vision. He has worked on the Annunciation and has applied OpenPose to a large collection of depictions of this scene with remarkable success, extracting systematic patterns of configurations, including eye-gaze and more. Works with Leo Impett at Lausanne and Rome. We want to analyze pose, interaction and surroundings, to understand iconography and narration better and are open to collaboration with scholars from computer vision or deep learning.

  • Leonardo Impett, Image and Visualization Lab and Digital Humanities Institute, EFPL (Swiss Federal Institute of Technology in Lausanne)

Related scrolls

• Dataset for Medieval Art

• Christian Iconography: The Émile Mâle pipeline

  • Christian Iconography (external site)

Art collections

Web Scrapers are not needed, well-formatted datasets are available, e.g. the Rijksmuseum dataset, the Fototeca Zeri data (http://data.fondazionezeri.unibo.it/query/) or the Bibliotheca Hertziana dataset (http://foto.biblhertz.it). All of these have different kinds of images (the Zeri and Hertziana are mostly black and white, Rijksmuseum is mostly colour) and different kinds of metadata.

Here are sites for additional images:

Hand-raising gestures

Teaching / dictating gestures:

St. Catherine of Alexandria (foreground) is counting on her fingers, indicating the act of teaching, dictation, making a point; she's telling the pagan philosophers about the truth of things, they variously react with gestures of disbelief (3rd left). Also the enthroned emperor (background) is making a speech gesture: a classical orator's gesture. He is the one who has invited the scholars to refute Catherine's Christian teaching. They fail miserably, converted to Christianity by Catherine's superior teaching. In the small panel (right) Catherine is still addressing the philosophers with a speech gesture as they are about to be executed by burning for their failure to rhetorically defeat Catherine and for their Christian conversion.

St. Catherine of Alexandria disputing with scholars, Masolino, in Castiglione Chapel, S. Clemente Rome (1425-31)

The angel (attribute of S. Matthew) dictating / inspiring S. Matthew as he writes his gospel makes the gesture of counting, presumably telling him what to write.

S. Matteo e l'angelo, Caravaggio, S. Luigi dei Francesi, Contarelli chapel Rome (1602)

Detail:

Resurrected Christ appears to two female disciples (Mary and Martha?). S. Sabina door panels, Rome, early 5th cent. Photos Bill Storage and Laura Maish (http://rome101.com/Topics/Christian/Sabina/ retrieved 18.3.16)

Resurrected Christ appears to male disciples. S. Sabina door panels, Rome, early 5th cent. Photos Bill Storage and Laura Maish (http://rome101.com/Topics/Christian/Sabina/ retrieved 18.3.16)

Christ predicting Peter's betrayal (Mt 26.33-35, Mk 14.29-31, Lk 22.33-34, Jn 13.36-38: "This very night, before the rooster crows, you will disown me three times.") S. Sabina door panels, Rome, early 5th cent. Photos Bill Storage and Laura Maish (http://rome101.com/Topics/Christian/Sabina/ retrieved 18.3.16)

Christ standing in judgement before Caiphas (seated). S. Sabina door panels, Rome, early 5th cent. Photos Bill Storage and Laura Maish (http://rome101.com/Topics/Christian/Sabina/ retrieved 18.3.16)

Note that while the three previous S. Sabina panels may display blessing gestures on the part of Christ (although the prediction of Peter's betrayal renders a blessing gesture unlikely), this this scene is likely linked to Christ's stern rebuke to Caiphas: "I tell you, hereafter you will see the Son of man seated at the right hand of Power, and coming on the clouds of heaven" (Matt 26:64).

Adlocutio gesture

Gesture indicating a formal speech, given usually by a general or emperor in ancient Roman art. Characteristic of the formula is the outstretched as well as the contrapposto pose with the weight shifted to one leg. Demonstrates authority and power.

Emperor Augustus, Prima Porta, Vatican Museums Rome (1st century CE)

Another example:

Emperor Marcus Aurelius, equestrian statue, bronze, Capitoline Museums Rome (late 2nd century)

Imperial adventus:

Imperial adventus. Decius - AR antoninianus of Decius struck in Rome 250 AD. radiate cuirassed bust right from behind IMP C M Q TRAIANVS DECIVS AVG Trajan Decius on horse riding left, raising hand and holding scepter (https://en.wikipedia.org/wiki/Adventus_(ceremony retrieved 18.3.16)

Adventus of Christ:

Apse mosaics SS. Cosma e Damiano, Rome, early 6th cent. (Source Wikipedia)

Christ is approaching, with right arm extended and hand open, in a (faint) blessing gesture. Saints Peter (r.) and Paul (l.) (presenting Saints Cosma and Damiano and, far right, St. Theodorus and, far left, donator pope Felix IV) in gestures of greeting / acclamation. Note that while the whole scene centres on Christ's appearance, all gazes are fixed on viewers in the nave below. Note also that Christ is descending not just towards us, but towards the altar where the Eucharistic meal is taken.

Apse mosaics, S. Cecilia, Rome, early 9th century. (http://personal.stthomas.edu/plgavrilyuk/PLGAVRILYUK/Art/Lamb/Lamb%20Rome.htm retrieved 18.3.16)

Apse mosaics S. Prassede, Rome early 9th century. (http://www.istantidibellezza.it/mosaici-absidali-a-roma.html retrieved 18.3.16)

Apse Mosaic S. Pudenziana. Christ enthroned with gesture of blessing, surrounded by apostles with acclaiming gestures (http://rome101.com/Topics/Christian/Sabina/ retrieved 18-3-16)

Gestures of blessing:

The Swearing of a Oath on a Relic in Sachsenspiegel, Heidelberger Bilderhandschrift (13th century)