Learning from Deep Stereoscopic Attention for Simulator Sickness Prediction

Minghan Du, Hui Cui, Yuan Wang, Henry Duh

Research output: Journal article publicationJournal articleAcademic researchpeer-review

Abstract

Simulator sickness induced by 360 stereoscopic video contents is a prolonged challenging issue in Virtual Reality (VR) system. Current machine learning models for simulator sickness prediction ignore the underlying interdependencies and correlations across multiple visual features which may lead to simulator sickness. We propose a model for sickness prediction by automatic learning and adaptive integrating multi-level mappings from stereoscopic video features to simulator sickness scores. Firstly, saliency, optical flow and disparity features are extracted from videos to reflect the factors causing simulator sickness, including human attention area, motion velocity and depth information. Then, these features are embedded and fed into a 3-dimensional convolutional neural network (3D CNN) to extract the underlying multi-level knowledge which includes low-level and higher-order visual concepts, and global image descriptor. Finally, an attentional mechanism is exploited to adaptively fuse multi-level information with attentional weights for sickness score estimation. The proposed model is trained by an end-to-end approach and validated over a public dataset. Comparison results with state-of-the-art models and ablation studies demonstrated improved performance in terms of Root Mean Square Error (RMSE) and Pearson Linear Correlation Coefficient.

Original languageEnglish
Pages (from-to)1
Number of pages1
JournalIEEE Transactions on Visualization and Computer Graphics
Early online date28 Sept 2021
DOIs
Publication statusE-pub ahead of print - 28 Sept 2021
Externally publishedYes

Keywords

  • 3D CNN I49 [Image Processing and Computer Vision]: Applications
  • attention mechanism
  • Feature extraction
  • H51 [Information Interfaces and Presentation]: Multimedia Information Systems
  • Optical flow
  • Predictive models
  • simulator sickness
  • Solid modeling
  • Stereo image processing
  • Stereoscopic video
  • Three-dimensional displays
  • virtual reality
  • Visualization

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Computer Vision and Pattern Recognition
  • Computer Graphics and Computer-Aided Design

Fingerprint

Dive into the research topics of 'Learning from Deep Stereoscopic Attention for Simulator Sickness Prediction'. Together they form a unique fingerprint.

Cite this