TwinkleTwinkle: Interacting with Your Smart Devices by Eye Blink

Haiming Cheng, Wei Lou, Yanni Yang, Yi Pu Chen, Xinyu Zhang

Research output: Journal article publicationJournal articleAcademic researchpeer-review

6 Citations (Scopus)

Abstract

Recent years have witnessed the rapid boom of mobile devices interweaving with changes the epidemic has made to people's lives. Though a tremendous amount of novel human-device interaction techniques have been put forward to facilitate various audiences and scenarios, limitations and inconveniences still occur to people having difficulty speaking or using their fingers/hands/arms or wearing masks/glasses/gloves. To fill the gap of such interaction contexts beyond using hands, voice, face, or mouth, in this work, we take the first step to propose a novel Human-Computer Interaction (HCI) system, TwinkleTwinkle, which senses and recognizes eye blink patterns in a contact-free and training-free manner leveraging ultrasound signals on commercial devices. TwinkleTwinkle first applies a phase difference based approach to depicting candidate eye blink motion profiles without removing any noises, followed by modeling intrinsic characteristics of blink motions through adaptive constraints to separate tiny patterns from interferences in conditions where blink habits and involuntary movements vary between individuals. We propose a vote-based approach to get final patterns designed to map with number combinations either self-defined or based on carriers like ASCII code and Morse code to make interaction seamlessly embedded with normal and well-known language systems. We implement TwinkleTwinkle on smartphones with all methods realized in the time domain and conduct extensive evaluations in various settings. Results show that TwinkleTwinkle achieves about 91% accuracy in recognizing 23 blink patterns among different people.

Original languageEnglish
Article number3596238
Pages (from-to)1-30
JournalProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Volume7
Issue number2
DOIs
Publication statusPublished - 12 Jun 2023

Keywords

  • acoustic sensing
  • eye blink
  • HCI

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Hardware and Architecture
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'TwinkleTwinkle: Interacting with Your Smart Devices by Eye Blink'. Together they form a unique fingerprint.

Cite this