Researchers at Hong Kong Polytechnic University have created a new framework for assessing the personalities of large language models (LLMs) such as ChatGPT, addressing a fundamental challenge in human-computer interaction.
Led by Dr Lik-Hang Lee from the university’s Department of Industrial and Systems Engineering, the team has developed the Language Model Linguistic Personality Assessment (LMLPA), which evaluates AI personalities based solely on their linguistic outputs rather than traditional human-centric questionnaires.
Current methods for measuring AI personalities rely on psychological tests designed for humans, which assume emotional experiences that AI systems don’t possess. These approaches also suffer from sensitivity to the order of multiple-choice options, producing inconsistent results.
The LMLPA overcomes these limitations through two key innovations. One was an adapted questionnaire focusing on language patterns rather than emotions or actions. The questionnaire uses open-ended questions, resulting in textual answers. The second innovation was the use of an AI rater that converts text responses into numerical personality scores. The study also used ratings by humans to validate the scores.