Your Mouse Reveals Your Next Activity: Towards Predicting User Intention from Mouse Interaction

Eugene Yujun Fu, Tiffany C.K. Kwok, Erin You Wu, Hong Va Leong, Grace Ngai, Stephen C.F. Chan

Research output: Chapter in book / Conference proceedingConference article published in proceeding or bookAcademic researchpeer-review

3 Citations (Scopus)


This paper presents an investigation into user intention prediction in two common web-based tasks: Crowdsourcing annotation and web search, based on human-mouse interaction information. User experience is gaining importance within the research area of human-centered computing, and is particularly useful for complex, multi-step tasks. To enhance user experience, the computer should be intelligent enough to be able to predict the user intention. For instance, an intelligent agent might be able to anticipate when the user is about to press a button, and helpfully enlarge or highlight it in advance. In this paper, we propose two prediction models on user intention: A classical model that considers only historical mouse activity sequence, and a multimodal model that utilizes mouse interaction signals as well as features extracted from mouse trajectory and clicking events. We evaluate our models and find that they achieve reasonable accuracy. Our preliminary results indicate that we can dynamically learn a multimodal model that can effectively predict a user's next activity from historical activity sequence and mouse interaction signals.
Original languageEnglish
Title of host publicationProceedings - 2017 IEEE 41st Annual Computer Software and Applications Conference, COMPSAC 2017
PublisherIEEE Computer Society
Number of pages6
ISBN (Electronic)9781538603673
Publication statusPublished - 7 Sep 2017
Event41st IEEE Annual Computer Software and Applications Conference, COMPSAC 2017 - Torino, Italy
Duration: 4 Jul 20178 Jul 2017


Conference41st IEEE Annual Computer Software and Applications Conference, COMPSAC 2017


  • User intention; mouse interaction; prediction; multimodal model

ASJC Scopus subject areas

  • Software
  • Computer Science Applications

Cite this