TY - GEN
T1 - Progressive multi-scale reconstruction for guided depth map super-resolution via deep residual gate fusion network
AU - Wen, Yang
AU - Wang, Jihong
AU - Li, Zhen
AU - Sheng, Bin
AU - Li, Ping
AU - Chi, Xiaoyu
AU - Mao, Lijuan
N1 - Funding Information:
Acknowledgement. This work was supported in part by the National Natural Science Foundation of China under Grants 62077037 and 61872241, in part by Shanghai Municipal Science and Technology Major Project under Grant 2021SHZDZX0102, in part by the Science and Technology Commission of Shanghai Municipality under Grants 18410750700 and 17411952600, in part by Shanghai Lin-Gang Area Smart Manufacturing Special Project under Grant ZN2018020202-3, and in part by Project of Shanghai Municipal Health Commission(2018ZHYL0230).
Publisher Copyright:
© 2021, Springer Nature Switzerland AG.
PY - 2021/9
Y1 - 2021/9
N2 - Depth maps obtained by consumer depth sensors are often accompanied by two challenging problems: low spatial resolution and insufficient quality, which greatly limit the potential applications of depth images. To overcome these shortcomings, some depth map super-resolution (DSR) methods tend to extrapolate a high-resolution depth map from a low-resolution depth map with the additional guidance of the corresponding high-resolution intensity image. However, these methods are still prone to texture copying and boundary discontinuities due to improper guidance. In this paper, we propose a deep residual gate fusion network (DRGFN) for guided depth map super-resolution with progressive multi-scale reconstruction. To alleviate the misalignment between color images and depth maps, DRGFN applies a color-guided gate fusion module to acquire content-adaptive attention for better fusing the color and depth features. To focus on restoring details such as boundaries, DRGFN applies a residual attention module to highlight the different importance of different channels. Furthermore, DRGFN applies a multi-scale fusion reconstruction module to make use of multi-scale information for better image reconstruction. Quantitative and qualitative experiments on several benchmarks fully show that DRGFN obtains the state-of-the-art performance for depth map super-resolution.
AB - Depth maps obtained by consumer depth sensors are often accompanied by two challenging problems: low spatial resolution and insufficient quality, which greatly limit the potential applications of depth images. To overcome these shortcomings, some depth map super-resolution (DSR) methods tend to extrapolate a high-resolution depth map from a low-resolution depth map with the additional guidance of the corresponding high-resolution intensity image. However, these methods are still prone to texture copying and boundary discontinuities due to improper guidance. In this paper, we propose a deep residual gate fusion network (DRGFN) for guided depth map super-resolution with progressive multi-scale reconstruction. To alleviate the misalignment between color images and depth maps, DRGFN applies a color-guided gate fusion module to acquire content-adaptive attention for better fusing the color and depth features. To focus on restoring details such as boundaries, DRGFN applies a residual attention module to highlight the different importance of different channels. Furthermore, DRGFN applies a multi-scale fusion reconstruction module to make use of multi-scale information for better image reconstruction. Quantitative and qualitative experiments on several benchmarks fully show that DRGFN obtains the state-of-the-art performance for depth map super-resolution.
KW - Attention mechanism
KW - Color image guidance
KW - Depth map super-resolution
KW - Gate fusion network
KW - Multi-scale
UR - http://www.scopus.com/inward/record.url?scp=85118337978&partnerID=8YFLogxK
U2 - 10.1007/978-3-030-89029-2_5
DO - 10.1007/978-3-030-89029-2_5
M3 - Conference article published in proceeding or book
AN - SCOPUS:85118337978
SN - 9783030890285
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 67
EP - 79
BT - Advances in Computer Graphics - 38th Computer Graphics International Conference, CGI 2021, Proceedings
A2 - Magnenat-Thalmann, Nadia
A2 - Magnenat-Thalmann, Nadia
A2 - Interrante, Victoria
A2 - Thalmann, Daniel
A2 - Papagiannakis, George
A2 - Sheng, Bin
A2 - Kim, Jinman
A2 - Gavrilova, Marina
PB - Springer Science and Business Media Deutschland GmbH
T2 - 38th Computer Graphics International Conference, CGI 2021
Y2 - 6 September 2021 through 10 September 2021
ER -