Identifying User-Specific Facial Affects from Spontaneous Expressions with Minimal Annotation

Michael Xuelin Huang, Grace Ngai, Kien A. Hua, Stephen C.F. Chan, Hong Va Leong

Research output: Journal article publicationJournal articleAcademic researchpeer-review

6 Citations (Scopus)


This paper presents Personalized Affect Detection with Minimal Annotation (PADMA), a user-dependent approach for identifying affective states from spontaneous facial expressions without the need for expert annotation. The conventional approach relies on the use of key frames in recorded affect sequences and requires an expert observer to identify and annotate the frames. It is susceptible to user variability and accommodating individual differences is difficult. The alternative is a user-dependent approach, but it would be prohibitively expensive to collect and annotate data for each user. PADMA uses a novel Association-based Multiple Instance Learning (AMIL) method, which learns a personal facial affect model through expression frequency analysis, and does not need expert input or frame-based annotation. PADMA involves a training/calibration phase in which the user watches short video segments and reports the affect that best describes his/her overall feeling throughout the segment. The most indicative facial gestures are identified and extracted from the facial response video, and the association between gesture and affect labels is determined by the distribution of the gesture over all reported affects. Hence both the geometric deformation and distribution of key facial gestures are specially adapted for each user. We show results that demonstrate the feasibility, effectiveness and extensibility of our approach.
Original languageEnglish
Pages (from-to)360-373
Number of pages14
JournalIEEE Transactions on Affective Computing
Issue number4
Publication statusPublished - 1 Oct 2016


  • Facial affect detection
  • user-dependent model
  • weakly supervised learning

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction


Dive into the research topics of 'Identifying User-Specific Facial Affects from Spontaneous Expressions with Minimal Annotation'. Together they form a unique fingerprint.

Cite this