Handwriting-Assistant: Reconstructing Continuous Strokes with Millimeter-level Accuracy via Attachable Inertial Sensors

Yanling Bu, Lei Xie, Yafeng Yin, Chuyu Wang, Jingyi Ning, Jiannong Cao, Sanglu Lu

Research output: Journal article publicationJournal articleAcademic researchpeer-review


Pen-based handwriting has become one of the major human-computer interaction methods. Traditional approaches either require writing on the specific supporting device like the touch screen, or limit the way of using the pen to pure rotation or translation. In this paper, we propose Handwriting-Assistant, to capture the free handwriting of ordinary pens on regular planes with mm-level accuracy. By attaching the inertial measurement unit (IMU) to the pen tail, we can infer the handwriting on the notebook, blackboard or other planes. Particularly, we build a generalized writing model to correlate the rotation and translation of IMU with the tip displacement comprehensively, thereby we can infer the tip trace accurately. Further, to display the effective handwriting during the continuous writing process, we leverage the principal component analysis (PCA) based method to detect the candidate writing plane, and then exploit the distance variation of each segment relative to the plane to distinguish on-plane strokes. Moreover, our solution can apply to other rigid bodies, enabling smart devices embedded with IMUs to act as handwriting tools. Experiment results show that our approach can capture the handwriting with high accuracy, e.g., the average tracking error is 1.84mm for letters with the size of about 2cmx1cm, and the average character recognition rate of recovered single letters achieves 98.2% accuracy of the ground-truth recorded by touch screen.
Original languageEnglish
Article number146
Pages (from-to)1-25
Number of pages25
JournalProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Issue number4
Publication statusPublished - Dec 2021


  • Human-centered computing
  • Ubiquitous and mobile computing
  • Handwriting Reconstruction
  • Inertial Sensor
  • Millimeter-level Tracking


Dive into the research topics of 'Handwriting-Assistant: Reconstructing Continuous Strokes with Millimeter-level Accuracy via Attachable Inertial Sensors'. Together they form a unique fingerprint.

Cite this