Addressing Token Uniformity in Transformers via Singular Value Transformation

Hanqi Yan, Lin Gui, Wenjie Li, Yulan He

Research output: Journal article publicationConference articleAcademic researchpeer-review

1 Citation (Scopus)


Token uniformity is commonly observed in transformer-based models, in which different tokens share a large proportion of similar information after going through stacked multiple self-attention layers in a transformer. In this paper, we propose to use the distribution of singular values of outputs of each transformer layer to characterise the phenomenon of token uniformity and empirically illustrate that a less skewed singular value distribution can alleviate the 'token uniformity' problem. Base on our observations, we define several desirable properties of singular value distributions and propose a novel transformation function for updating the singular values. We show that apart from alleviating token uniformity, the transformation function should preserve the local neighbourhood structure in the original embedding space. Our proposed singular value transformation function is applied to a range of transformer-based language models such as BERT, ALBERT, RoBERTa and DistilBERT, and improved performance is observed in semantic textual similarity evaluation and a range of GLUE tasks. Our source code is available at

Original languageEnglish
Pages (from-to)2181-2191
Number of pages11
JournalProceedings of Machine Learning Research
Publication statusPublished - 2022
Event38th Conference on Uncertainty in Artificial Intelligence, UAI 2022 - Eindhoven, Netherlands
Duration: 1 Aug 20225 Aug 2022

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software
  • Control and Systems Engineering
  • Statistics and Probability


Dive into the research topics of 'Addressing Token Uniformity in Transformers via Singular Value Transformation'. Together they form a unique fingerprint.

Cite this