TY - GEN
T1 - Do Large Language Models Understand Conversational Implicature – A Case Study with a Chinese Sitcom
AU - Yue, Shisen
AU - Song, Siyuan
AU - Cheng, Xinyuan
AU - Hu, Hai
PY - 2024/11/29
Y1 - 2024/11/29
N2 - Understanding the non-literal meaning of an utterance is critical for large language models (LLMs) to become human-like social communicators. In this work, we introduce SwordsmanImp, the first Chinese multi-turn-dialogue-based dataset aimed at conversational implicature, sourced from dialogues in the Chinese sitcom My Own Swordsman. It includes 200 carefully handcrafted questions, all annotated on which Gricean maxims have been violated. We test eight close-source and open-source LLMs under two tasks: a multiple-choice question task and an implicature explanation task. Our results show that GPT-4 attains human-level accuracy (94%) on multiple-choice questions. CausalLM demonstrates a 78.5% accuracy following GPT-4. Other models, including GPT3.5 and several open-source models, demonstrate a lower accuracy ranging from 20% to 60% on multiple-choice questions. Human raters were asked to rate the explanation of the implicatures generated by LLMs on their reasonability, logic and fluency. While all models generate largely fluent and self-consistent text, their explanations score low on reasonability except for GPT-4, suggesting that most LLMs cannot produce satisfactory explanations of the implicatures in the conversation. Moreover, we find LLMs’ performance does not vary significantly by Gricean maxims, suggesting that LLMs do not seem to process implicatures derived from different maxims differently. Our data and code are available at https://github.com/sjtu-compling/llm-pragmatics.
AB - Understanding the non-literal meaning of an utterance is critical for large language models (LLMs) to become human-like social communicators. In this work, we introduce SwordsmanImp, the first Chinese multi-turn-dialogue-based dataset aimed at conversational implicature, sourced from dialogues in the Chinese sitcom My Own Swordsman. It includes 200 carefully handcrafted questions, all annotated on which Gricean maxims have been violated. We test eight close-source and open-source LLMs under two tasks: a multiple-choice question task and an implicature explanation task. Our results show that GPT-4 attains human-level accuracy (94%) on multiple-choice questions. CausalLM demonstrates a 78.5% accuracy following GPT-4. Other models, including GPT3.5 and several open-source models, demonstrate a lower accuracy ranging from 20% to 60% on multiple-choice questions. Human raters were asked to rate the explanation of the implicatures generated by LLMs on their reasonability, logic and fluency. While all models generate largely fluent and self-consistent text, their explanations score low on reasonability except for GPT-4, suggesting that most LLMs cannot produce satisfactory explanations of the implicatures in the conversation. Moreover, we find LLMs’ performance does not vary significantly by Gricean maxims, suggesting that LLMs do not seem to process implicatures derived from different maxims differently. Our data and code are available at https://github.com/sjtu-compling/llm-pragmatics.
KW - pragmatic reasoning
KW - large language models
KW - the cooperative principle
U2 - 10.1007/978-981-97-8367-0_24
DO - 10.1007/978-981-97-8367-0_24
M3 - Conference article published in proceeding or book
SN - 9789819783663
T3 - Lecture Notes in Computer Science
SP - 402
EP - 418
BT - Chinese Computational Linguistics: 23rd China National Conference, CCL 2024, Taiyuan, China, July 25–28, 2024, Proceedings
A2 - Sun, Maosong
A2 - Liang, Jiye
A2 - Han, Xianpei
A2 - Liu, Zhiyuan
A2 - He, Yulan
A2 - Rao, Gaoqi
A2 - Chen, Yubo
A2 - Tian, Zhiliang
PB - Springer
T2 - The 24th China National Conference on Computational Linguistics
Y2 - 25 July 2024 through 28 July 2024
ER -