Facial Expressions of Comprehension (FEC)

Cigdem Turan, Karl David Neergaard, Kin Man Lam

Research output: Journal article publicationJournal articleAcademic researchpeer-review

Abstract

While the relationship between facial expressions and emotion has been a productive area of inquiry, research is only recently exploring whether a link exists between facial expressions and cognitive processes. Using findings from psychology and neuroscience to guide predictions of affectation during a cognitive task, this article aimed to study facial dynamics as a mean to understand comprehension. We present a new multimodal facial expression database, named Facial Expressions of Comprehension (FEC), consisting of the videos recorded during a computer-mediated task in which each trial consisted of reading, answering, and feedback to general knowledge true and false statements. To identify the level of engagement with the corresponding stimuli, we present a new methodology using animation units (AnUs) from the Kinect v2 device to explore the changes in facial configuration caused by an event: Event-Related Intensities (ERIs). To identify dynamic facial configurations, we used ERIs in statistical analyses with generalized additive models. To identify differential facial dynamics linked to knowing vs. guessing and true vs. false responses, we employed an SVM classifier with facial appearance information extracted using LPQ-TOP. Results of ERIs in sentence comprehension show that facial dynamics are promising to help understand affective and cognitive states of the mind.

Original languageEnglish
Article number8907450
Pages (from-to)335-346
Number of pages12
JournalIEEE Transactions on Affective Computing
Volume13
Issue number1
DOIs
Publication statusPublished - Jan 2022

Keywords

  • Facial behavior understanding
  • facial expression database
  • sentence comprehension

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction

Cite this