GR-PSN: Learning to Estimate Surface Normal and Reconstruct Photometric Stereo Images

Yakun Ju, Boxin Shi, Yang Chen, Huiyu Zhou, Junyu Dong, Kin Man Lam

Research output: Journal article publicationJournal articleAcademic researchpeer-review

8 Citations (Scopus)

Abstract

In this paper, we propose a novel method, namely GR-PSN, which learns surface normals from photometric stereo images and generates the photometric images under distant illumination from different lighting directions and surface materials. The framework is composed of two subnetworks, named GeometryNet and ReconstructNet, which are cascaded to perform shape reconstruction and image rendering in an end-to-end manner. ReconstructNet introduces additional supervision for surface-normal recovery, forming a closed-loop structure with GeometryNet. We also encode lighting and surface reflectance in ReconstructNet, to achieve arbitrary rendering. In training, we set up a parallel framework to simultaneously learn two arbitrary materials for an object, providing an additional transform loss. Therefore, our method is trained based on the supervision by three different loss functions, namely the surface-normal loss, reconstruction loss, and transform loss. We alternately input the predicted surface-normal map and the ground-truth into ReconstructNet, to achieve stable training for ReconstructNet. Experiments show that our method can accurately recover the surface normals of an object with an arbitrary number of inputs, and can re-render images of the object with arbitrary surface materials. Extensive experimental results show that our proposed method outperforms those methods based on a single surface recovery network and shows realistic rendering results on 100 different materials. Our code can be found in <uri>https://github.com/Kelvin-Ju/GR-PSN</uri>.

Original languageEnglish
Pages (from-to)1-16
Number of pages16
JournalIEEE Transactions on Visualization and Computer Graphics
DOIs
Publication statusAccepted/In press - Nov 2023

Keywords

  • 3D reconstruction
  • deep neural networks
  • Image reconstruction
  • Lighting
  • photometric image reconstruction
  • photometric stereo
  • Reflectivity
  • Rendering (computer graphics)
  • surface normal estimate
  • Surface reconstruction
  • Three-dimensional displays
  • Training

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Computer Vision and Pattern Recognition
  • Computer Graphics and Computer-Aided Design

Fingerprint

Dive into the research topics of 'GR-PSN: Learning to Estimate Surface Normal and Reconstruct Photometric Stereo Images'. Together they form a unique fingerprint.

Cite this