Deep Color Guided Coarse-to-Fine Convolutional Network Cascade for Depth Image Super-Resolution

Yang Wen, Bin Sheng, Ping Li, Weiyao Lin, David Dagan Feng

Research output: Journal article publicationJournal articleAcademic researchpeer-review

43 Citations (Scopus)

Abstract

Depth image super-resolution is a significant yet challenging task. In this paper, we introduce a novel deep color guided coarse-to-fine convolutional neural network (CNN) framework to address this problem. First, we present a data-driven filter method to approximate the ideal filter for depth image super-resolution instead of hand-designed filters. Based on large data samples, the filter learned is more accurate and stable for upsampling depth image. Second, we introduce a coarse-to-fine CNN to learn different sizes of filter kernels. In the coarse stage, larger filter kernels are learned by the CNN to achieve crude high-resolution depth image. As to the fine stage, the crude high-resolution depth image is used as the input so that smaller filter kernels are learned to gain more accurate results. Benefit from this network, we can progressively recover the high frequency details. Third, we construct a color guidance strategy that fuses color difference and spatial distance for depth image upsampling. We revise the interpolated high-resolution depth image according to the corresponding pixels in high-resolution color maps. Guided by color information, the depth of high-resolution image obtained can alleviate texture copying artifacts and preserve edge details effectively. Quantitative and qualitative experimental results demonstrate our state-of-the-art performance for depth map super-resolution.

Original languageEnglish
Article number8485321
Pages (from-to)994-1006
Number of pages13
JournalIEEE Transactions on Image Processing
Volume28
Issue number2
DOIs
Publication statusPublished - Feb 2019
Externally publishedYes

Keywords

  • coarse-to-fine convolutional neural network
  • color guidance
  • Depth super-resolution
  • filter kernel learning

ASJC Scopus subject areas

  • Software
  • Computer Graphics and Computer-Aided Design

Cite this