User profiling based on nonlinguistic audio data

Jiaxing Shen, Oren Lederman, Jiannong Cao, Shaojie Tang, Alex Lsandy Pentland

Research output: Chapter in book / Conference proceedingConference article published in proceeding or bookAcademic researchpeer-review


User profiling refers to inferring people's attributes of interest (AoIs) like gender and occupation, which enables various applications ranging from personalized services to collective analyses. Massive nonlinguistic audio data brings a novel opportunity for user profiling due to the prevalence of studying spontaneous face-to-face communication. In this poster, we are the first to build a user profiling system to infer gender and personality based on nonlinguistic audio. Instead of linguistic or acoustic features which are unable to extract, we focus on conversational features that could reflect AoIs. We firstly develop an adaptive voice activity detection algorithm that could address individual differences in voice and false-positive voice activities caused by people nearby. Secondly, we propose a gender-assisted multi-task learning method to combat dynamics in human behavior by integrating gender differences and the correlation of personality traits. The experimental evaluation of 100 people in 273 meetings indicates the superiority of the proposed method in gender identification and personality recognition respectively.

Original languageEnglish
Title of host publicationProceedings - 2021 IEEE 37th International Conference on Data Engineering, ICDE 2021
PublisherIEEE Computer Society
Number of pages6
ISBN (Electronic)9781728191843
Publication statusPublished - Apr 2021
Event37th IEEE International Conference on Data Engineering, ICDE 2021 - Virtual, Chania, Greece
Duration: 19 Apr 202122 Apr 2021

Publication series

NameProceedings - International Conference on Data Engineering
ISSN (Print)1084-4627


Conference37th IEEE International Conference on Data Engineering, ICDE 2021
CityVirtual, Chania


  • Gender identification
  • Multi-task learning
  • Nonlinguistic audio
  • Personality recognition
  • User profiling

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Information Systems

Cite this