Exploring Audio-Visual Fusion Methods in Foundation Model-Based Deception Detection

The deception detection task aims to identify if a speaker is speaking truth or lie. It is a challenging problem due to limited training data and hence this poses a restriction for representation learning models to learn better features. Audio-visual multi-modal detection has gained significant attention for its superior performance compared to single-modality approaches. In practical scenarios, multi-modal integration can be challenging due to the distinct characteristics of each modality, making the fusion process difficult. In this paper, we employ the Querying Transformer (Q-former) to temporally align audio and visual features from foundation models and explore various methods to fuse these two modalities. Comprehensive experiments on the DOLOS dataset show that our proposed fusion technique outperforms systems based on individual audio and visual modalities. The experiments also indicate that early fusion using layer-bylayer alignment between the two modalities in the foundation model is not required. Instead, integrating over all layers of the two modalities yields the best performance.