Tracking Stuffed Toy for Naturally Mapped Interactive Play via a Soft-Pose Estimator

Zackary P.T. Sin, Peter Q. Chen, Peter H.F. Ng, Hong Va Leong

Research output: Journal article publicationJournal articleAcademic researchpeer-review


Have you ever picked up a stuffed toy and pretended to play with it in your childhood? We are motivated by the novel use of stuffed toys in enhancing extended reality interaction. A key goal of extended reality is to induce the feeling of presence in its users. Naturally mapped control interface has been shown to enhance presence. The literature also indicates that a high degree of freedom tracking is important to extended reality. Based on these observations, we show that a free-form naturally mapped control interface is well-motivated via a theoretical contextualization. We explore the possibility of building such a controller in the form of stuffed toys. To realize stuffed toys as controllers, a novel soft-pose estimator empowered by cage-based deformation is proposed. It is shown to be effective in tracking the poses and deformations of real soft objects even by training with synthetic data only. Three gameplay prototypes are developed to demonstrate that interactive play can be enabled by the soft-pose estimator. They also form the basis for two user studies that validate the success of tracking stuffed toys with the soft-pose estimator for interactive play.

Original languageEnglish
Article number255
Pages (from-to)1-25
JournalProceedings of the ACM on Human-Computer Interaction
Publication statusPublished - 29 Oct 2022


  • interactive XR play
  • naturally mapping control interface
  • novel controls for games and play
  • soft-pose estimator
  • tracking stuffed toy

ASJC Scopus subject areas

  • Social Sciences (miscellaneous)
  • Human-Computer Interaction
  • Computer Networks and Communications


Dive into the research topics of 'Tracking Stuffed Toy for Naturally Mapped Interactive Play via a Soft-Pose Estimator'. Together they form a unique fingerprint.

Cite this