Eye-tracking based relevance feedback for iterative face image retrieval

Mengli Sun, Jiajun Wang, Zheru Chi

Research output: Chapter in book / Conference proceedingConference article published in proceeding or bookAcademic researchpeer-review

2 Citations (Scopus)


The performance of computer-vision based face image retrieval system declines significantly when large illumination, pose, and facial expression variations are presented. To tackle such a problem, we propose a closed loop face image retrieval system with implicit eye-tracking based feedback. It combines the state-of-the-art computer vision method Face++ with the powerful cognitive ability of human. In this system, the Face++ provides initial retrieving results corresponding to a target sample face image whose top ranked 36 images are then displayed on the screen for collecting eye-tracking data of the users. Upon mining the user's cognition results from the eye-tracking data with a deep neural network and feeding them back to the system, the system begins its new round retrieving process. Experimental results from 10 volunteers in a face database containing 1,500 images of 50 celebrities show that the performance of our system becomes better and better over iterations and finally our system achieve an average precision of higher than 0.918 and an average recall rate of higher than 0.897 upon convergence.

Original languageEnglish
Title of host publicationTenth International Conference on Graphics and Image Processing, ICGIP 2018
EditorsHui Yu, Yifei Pu, Zhigeng Pan, Chunming Li
ISBN (Electronic)9781510628281
Publication statusPublished - 12 Dec 2018
Event10th International Conference on Graphics and Image Processing, ICGIP 2018 - Chengdu, China
Duration: 12 Dec 201814 Dec 2018

Publication series

NameProceedings of SPIE - The International Society for Optical Engineering
ISSN (Print)0277-786X
ISSN (Electronic)1996-756X


Conference10th International Conference on Graphics and Image Processing, ICGIP 2018


  • Computer vision
  • Eye-tracking
  • Face image retrieval
  • Relevance feedback

ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Condensed Matter Physics
  • Computer Science Applications
  • Applied Mathematics
  • Electrical and Electronic Engineering


Dive into the research topics of 'Eye-tracking based relevance feedback for iterative face image retrieval'. Together they form a unique fingerprint.

Cite this