Joint learning of frequency and spatial domains for dense image prediction

Shaocheng Jia, Wei Yao

Research output: Journal article publicationJournal articleAcademic researchpeer-review

6 Citations (Scopus)


Current artificial neural networks mainly conduct the learning process in the spatial domain but neglect the frequency domain learning. However, the learning course performed in the frequency domain can be more efficient than that in the spatial domain. In this paper, we fully explore frequency domain learning and propose a joint learning paradigm of frequency and spatial domains. This paradigm can take full advantage of the combined preponderances of frequency learning and spatial learning; specifically, frequency and spatial domain learning can effectively capture intrinsic global and local information, respectively. To achieve this, an innovative but effective linear learning block is proposed to conduct the learning process directly in the frequency domain. Together with the prevailing spatial learning operation, i.e., convolution, a powerful and scalable joint learning framework is further proposed. Exhaustive experiments on the diverse Benchmark datasets — KITTI, Make3D, and Cityscapes demonstrate the effectiveness and superiority of the proposed joint learning paradigm in dense image prediction tasks, including self-supervised depth estimation, ego-motion estimation, and semantic segmentation. In particular, the proposed model can achieve performance competitive to those of state-of-the-art methods in all three tasks, even without pretraining. Moreover, the proposed model reduces the number of parameters by over 78% for self-supervised depth estimation on the KITTI dataset while retaining the time complexity on par with other state-of-the-art methods; this provides a great chance to develop real-world applications. We hope that the proposed method can encourage more research in cross-domain learning. The codes are publicly available at

Original languageEnglish
Pages (from-to)14-28
Number of pages15
JournalISPRS Journal of Photogrammetry and Remote Sensing
Publication statusPublished - Jan 2023


  • Depth estimation
  • Frequency learning
  • Joint learning
  • Semantic image segmentation
  • Spatial learning

ASJC Scopus subject areas

  • Atomic and Molecular Physics, and Optics
  • Engineering (miscellaneous)
  • Computer Science Applications
  • Computers in Earth Sciences


Dive into the research topics of 'Joint learning of frequency and spatial domains for dense image prediction'. Together they form a unique fingerprint.

Cite this