Eye-tracking based relevance feedback for iterative face image retrieval

The performance of computer-vision based face image retrieval system declines significantly when large illumination, pose, and facial expression variations are presented. To tackle such a problem, we propose a closed loop face image retrieval system with implicit eye-tracking based feedback. It combines the state-of-the-art computer vision method Face++ with the powerful cognitive ability of human. In this system, the Face++ provides initial retrieving results corresponding to a target sample face image whose top ranked 36 images are then displayed on the screen for collecting eye-tracking data of the users. Upon mining the user's cognition results from the eye-tracking data with a deep neural network and feeding them back to the system, the system begins its new round retrieving process. Experimental results from 10 volunteers in a face database containing 1,500 images of 50 celebrities show that the performance of our system becomes better and better over iterations and finally our system achieve an average precision of higher than 0.918 and an average recall rate of higher than 0.897 upon convergence.