TY - JOUR
T1 - A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinese
AU - Liu, Yikang
AU - Shen, Yeting
AU - Zhu, Hongao
AU - Xu, Lilong
AU - Qian, Zhiheng
AU - Song, Siyuan
AU - Zhang, Kejia
AU - Tang, Jialong
AU - Zhang, Pei
AU - Yang, Baosong
AU - Wang, Rui
AU - Hu, Hai
N1 - Publisher Copyright:
© 2026 Association for Computational Linguistics. This is an open-access article distributed under the terms of the https://creativecommons.org/licenses/by/4.0/, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. For a full description of the license, please visit https://creativecommons.org/licenses/by/4.0/legalcode.
PY - 2026/5/4
Y1 - 2026/5/4
N2 - We present ZhoBLiMP, the largest linguistic minimal pair benchmark for Chinese, with over 100 paradigms, ranging from topicalization to the Ba construction. We then train from scratch a suite of Chinese language models (LMs) with different tokenizers, parameter sizes, and token volumes, to study the learning curves of LMs on Chinese. To mitigate the biases introduced by unequal lengths of the sentences in a minimal pair, we propose a new metric named sub-linear length normalized log-probabilities (SLLN-LP). Using SLLN-LP as the metric, our results show that Anaphor, Quantifiers, and Ellipsis in Chinese are difficult for LMs even up to 32B parameters, and that SLLN-LP successfully mitigates biases in ZhoBLiMP, JBLiMP and BLiMP. We conclude that future evaluations should be more carefully designed to consider the intricate relations between linking functions, LMs, and targeted minimal pairs.
AB - We present ZhoBLiMP, the largest linguistic minimal pair benchmark for Chinese, with over 100 paradigms, ranging from topicalization to the Ba construction. We then train from scratch a suite of Chinese language models (LMs) with different tokenizers, parameter sizes, and token volumes, to study the learning curves of LMs on Chinese. To mitigate the biases introduced by unequal lengths of the sentences in a minimal pair, we propose a new metric named sub-linear length normalized log-probabilities (SLLN-LP). Using SLLN-LP as the metric, our results show that Anaphor, Quantifiers, and Ellipsis in Chinese are difficult for LMs even up to 32B parameters, and that SLLN-LP successfully mitigates biases in ZhoBLiMP, JBLiMP and BLiMP. We conclude that future evaluations should be more carefully designed to consider the intricate relations between linking functions, LMs, and targeted minimal pairs.
UR - https://www.scopus.com/pages/publications/105042059172
U2 - 10.1162/TACL.a.648
DO - 10.1162/TACL.a.648
M3 - Journal article
SN - 2307-387X
VL - 14
SP - 755
EP - 771
JO - Transactions of the Association for Computational Linguistics
JF - Transactions of the Association for Computational Linguistics
ER -