Ev'ry little movement has a meaning of its own: Using past mouse movements to predict the next interaction

Tiffany C.K. Kwok, Eugene Yujun Fu, Erin You Wu, Michael Xuelin Huang, Grace Ngai, Hong Va Leong

Research output: Chapter in book / Conference proceedingConference article published in proceeding or bookAcademic researchpeer-review

4 Citations (Scopus)


User experience could be enhanced if the computer could understand human interaction intention. For instance, it could react to intercept and prevent interaction errors. This paper presents an approach to predicting users' intention in interaction tasks based on past mouse movements. We adopt a long short-term memory (LSTM) model to predict the users' intention via their next mouse click interaction, upon being trained with past mouse interaction behaviors. To evaluate, we consider two scenarios in daily computer usage: a more structured crowdsourcing annotation task and a more free-form, open-ended web search task. Our results indicate that we could predict the next interaction event with reasonable accuracy. We also conducted a pilot study to investigate the possibility of applying our model for nonintentional mouse click detection. We believe that our findings would be beneficial towards the development of better intelligent agents.
Original languageEnglish
Title of host publicationIUI 2018 - Proceedings of the 23rd International Conference on Intelligent User Interfaces
PublisherAssociation for Computing Machinery
Number of pages5
ISBN (Electronic)9781450349451
Publication statusPublished - 5 Mar 2018
Event23rd ACM International Conference on Intelligent User Interfaces, IUI 2018 - Tokyo, Japan
Duration: 7 Mar 201811 Mar 2018


Conference23rd ACM International Conference on Intelligent User Interfaces, IUI 2018


  • Human-computer interaction
  • Mouse interaction
  • Non-intentional mouse click detection
  • User intention

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction

Cite this