Learning Shared Semantic Space with Correlation Alignment for Cross-Modal Event Retrieval

Zhenguo Yang, Zehang Lin, Peipei Kang, Jianming Lv, Qing Li, Wenyin Liu

Research output: Journal article publicationJournal articleAcademic researchpeer-review

3 Citations (Scopus)

Abstract

In this article, we propose to learn shared semantic space with correlation alignment (S3CA) for multimodal data representations, which aligns nonlinear correlations of multimodal data distributions in deep neural networks designed for heterogeneous data. In the context of cross-modal (event) retrieval, we design a neural network with convolutional layers and fully connected layers to extract features for images, including images on Flickr-like social media. Simultaneously, we exploit a fully connected neural network to extract semantic features for text documents, including news articles from news media. In particular, nonlinear correlations of layer activations in the two neural networks are aligned with correlation alignment during the joint training of the networks. Furthermore, we project the multimodal data into a shared semantic space for cross-modal (event) retrieval, where the distances between heterogeneous data samples can be measured directly. In addition, we contribute a Wiki-Flickr Event dataset, where the multimodal data samples are not describing each other in pairs like the existing paired datasets, but all of them are describing semantic events. Extensive experiments conducted on both paired and unpaired datasets manifest the effectiveness of S3CA, outperforming the state-of-the-art methods.

Original languageEnglish
Article number9
Pages (from-to)1-22
Number of pages22
JournalACM Transactions on Multimedia Computing, Communications and Applications
Volume16
Issue number1
DOIs
Publication statusPublished - Mar 2020

Keywords

  • Cross-modal retrieval
  • deep learning
  • heterogeneous data
  • Wiki-Flickr Event dataset

ASJC Scopus subject areas

  • Hardware and Architecture
  • Computer Networks and Communications

Cite this