Skip to main navigation Skip to search Skip to main content

A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinese

  • Yikang Liu
  • , Yeting Shen
  • , Hongao Zhu
  • , Lilong Xu
  • , Zhiheng Qian
  • , Siyuan Song
  • , Kejia Zhang
  • , Jialong Tang
  • , Pei Zhang
  • , Baosong Yang
  • , Rui Wang (Corresponding Author)
  • , Hai Hu (Corresponding Author)

Research output: Journal article publication › Journal article › Academic research › peer-review

Abstract

We present ZhoBLiMP, the largest linguistic minimal pair benchmark for Chinese, with over 100 paradigms, ranging from topicalization to the Ba construction. We then train from scratch a suite of Chinese language models (LMs) with different tokenizers, parameter sizes, and token volumes, to study the learning curves of LMs on Chinese. To mitigate the biases introduced by unequal lengths of the sentences in a minimal pair, we propose a new metric named sub-linear length normalized log-probabilities (SLLN-LP). Using SLLN-LP as the metric, our results show that Anaphor, Quantifiers, and Ellipsis in Chinese are difficult for LMs even up to 32B parameters, and that SLLN-LP successfully mitigates biases in ZhoBLiMP, JBLiMP and BLiMP. We conclude that future evaluations should be more carefully designed to consider the intricate relations between linking functions, LMs, and targeted minimal pairs.
Original languageEnglish
Pages (from-to)755-771
Number of pages17
JournalTransactions of the Association for Computational Linguistics
Volume14
DOIs
Publication statusPublished - 4 May 2026
Externally publishedYes

Fingerprint

Dive into the research topics of 'A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinese'. Together they form a unique fingerprint.

Cite this