Jointly Modelling Transcriptions and Phonemes with Optimal Features to Detect Dementia from Spontaneous Cantonese

Dementia is a severe cognitive impairment that affects the health of older adults. This paper analyzes diverse speech-based features extracted from spoken languages and selects the most discriminative ones for dementia detection. We propose a two-step feature selection method to handle the circumstance where the feature dimension is far larger than the number of training samples. Recently, the performance of dementia detection has been significantly improved by utilizing Transformer-based models that automatically capture the linguistic properties of spoken languages. We combine features extracted from BERT with selected speech-based features to enhance dementia detection performance. We propose a novel strategy to model the transcriptions and their phonemes using BERT and phoneme-BERT. The proposed method is evaluated on a Cantonese dataset called CU-Marvel, which contains 185 healthy older adults, 98 older adults having minor neurocognitive disorders (minor NCD), and 26 older adults suffering from major NCD. Experimental results show that simultaneously fine-tuning the BERT and phoneme-BERT can leverage information from the recognized phonemes and make the detection performance robust to automatic speech recognition errors. Simultaneous fine-tuning of the BERT and phoneme-BERT models results in a 6% improvement in F1 scores, compared to fine-tuning the BERT model alone.