Realistic human action recognition with multimodal feature selection and fusion

Qiuxia Wu, Zhiyong Wang, Feiqi Deng, Zheru Chi, David Dagan Feng

Research output: Journal article publicationJournal articleAcademic researchpeer-review

63 Citations (Scopus)


Although promising results have been achieved for human action recognition under well-controlled conditions, it is very challenging to recognize human actions in realistic scenarios due to increased difficulties such as dynamic backgrounds. In this paper, we propose to take multimodal (i.e., audiovisual) characteristics of realistic human action videos into account in human action recognition for the first time, since, in realistic scenarios, audio signals accompanying an action generally provide a cue to the nature of the action, such as phone ringing to answering the phone. In order to cope with diverse audio cues of an action in realistic scenarios, we propose to identify effective features from a large number of audio features with the generalized multiple kernel learning algorithm. The widely used space-time interest point descriptors are utilized as visual features, and a support vector machine is employed for both audio- and video-based classifications. At the final stage, fuzzy integral is utilized to fuse recognition results of both audio and visual modalities. Experimental results on the challenging Hollywood-2 Human Action data set demonstrate that the proposed approach is able to achieve better recognition performance improvement than that of integrating scene context. It is also discovered how audio context influences realistic action recognition from our comprehensive experiments.
Original languageEnglish
Pages (from-to)875-885
Number of pages11
JournalIEEE Transactions on Systems, Man, and Cybernetics Part A:Systems and Humans
Issue number4
Publication statusPublished - 5 Apr 2013


  • Fuzzy integral
  • Multimodal fusion
  • Multiple kernel learning (MKL)
  • Realistic human action recognition

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Human-Computer Interaction
  • Computer Science Applications
  • Electrical and Electronic Engineering


Dive into the research topics of 'Realistic human action recognition with multimodal feature selection and fusion'. Together they form a unique fingerprint.

Cite this