A mobile app for examining oral-facial movements in healthy and stroke patients: Its development and potential applications

Background
Deficits to oral-facial function, a common sequel of stroke, may have huge impacts on the
speech, communication, and swallowing abilities of the patients. Clinical examination of oralfacial examinations, such as facial symmetry and range-of-movements of the jaw, lips and
tongue, are usually conducted by clinicians via subjective judgment, which may not be reliable
and consistent across clinicians. With the advancement in image processing and motion
tracking of mobile technology, mobile apps that capture oral-facial movements of stroke
patients may provide objective and reliable information to clinicians in assisting diagnosis. The
aim of the current study is to develop a mobile app for assessing oral-motor functions of
Chinese adults.
Methods
ORal-motor Assessment and Rehabilitation mobile App (ORAR App), a mobile app dedicated
to assisting clinicians in assessing and training of oral-motor functions of post-stroke patients,
was developed with support from the Innovation and Technology Fund for Better Living of the
Innovation and Technology Commission of the Hong Kong SAR Government. ORAR captures
the oral-motor performance of users via combination of 3 AI models that identify different
oral-facial and speech features. The AI models includes: 1) MediaPipe Facial Landmarker - a
pre-trained computer vision model developed by Google AI Edge that recognizes the location
of various keypoints of the face, including nose, lips, and jaw; 2) YOLO tongue detection model
- a transfer learning computer vision model originally developed by Ultralytics and fine-tuned
by PI and her team with Asian/Hong Kong population that recognize the pose of the tongue
(IP: PAT-1949-HK-NP); 3) spectrogram based speech analysis model - a custom model based
on tensorflow framework originally developed by Google Brain and extended by PI’ team with
the dataset from Asian/Hong Kong population.
Results
ORAR App was used to examine the oral-motor functions of 140 neurologically healthy adults
and 60 adult stroke patients in Hong Kong. Their performance was analyzed by the app. A
dataset containing the performance of healthy and stroke individuals was obtained.
Comparisons in performance between healthy and stroke participants among patients with
different severity levels were conducted.
Outcomes and implications - including reference to healthcare management
The clinical applications of ORAR App, including its potential in assisting clinicians in
assessing and managing patients with stroke and related oral-motor dysfunctions, are worth
exploring