Emo-MG Framework: LSTM-based Multi-modal Emotion Detection through Electroencephalography Signals and Micro Gestures

Human-computer interaction has seen growing interest in emotion detection. To gain deeper insights into the physiological indicators of emotions, researchers have delved into utilizing electroencephalography (EEG) and micro-gestures (MGs). This study assesses the efficacy of EEG and MG features in emotion detection by recruiting 15 participants to gather EEG and MG data in response to diverse figure-based emotional stimuli. To incorporate these features, this article introduces Emo-MG, a multimodal interface that integrates EEG and MG features and employs a long short-term memory (LSTM) model to predict emotional states within the valence-arousal-dominance (VAD) space. This study presents an in-depth analysis of feature importance and correlation results based on EEG and MG features for feature selection in emotion detection tasks. Through accuracy and F1-score metrics, Emo-MG achieves outstanding performance in emotion detection by comparing it to baseline and deep learning models, validating the efficacy of integrating EEG and MG features.