TY - GEN
T1 - Denoising Concept Vectors with Sparse Autoencoders for Improved Language Model Steering
AU - Zhao, Haiyan
AU - Wu, Xuansheng
AU - Yang, Fan
AU - Shen, Bo
AU - Liu, Ninghao
AU - Du, Mengnan
N1 - Publisher Copyright:
©2026 Association for Computational Linguistics.
PY - 2026/3
Y1 - 2026/3
N2 - Linear concept vectors effectively steer LLMs, but existing methods suffer from noisy features in diverse datasets that undermine steering robustness. We propose Sparse Autoencoder-Denoised Concept Vectors (SDCV), which selectively keep the most discriminative SAE latents while reconstructing hidden representations. Our key insight is that concept-relevant signals can be explicitly separated from dataset noise by scaling up activations of top-k latents that best differentiate positive and negative samples. Applied to linear probing and difference-in-mean, SDCV consistently improves steering success rates by 4-16% across six challenging concepts, while maintaining topic relevance.
AB - Linear concept vectors effectively steer LLMs, but existing methods suffer from noisy features in diverse datasets that undermine steering robustness. We propose Sparse Autoencoder-Denoised Concept Vectors (SDCV), which selectively keep the most discriminative SAE latents while reconstructing hidden representations. Our key insight is that concept-relevant signals can be explicitly separated from dataset noise by scaling up activations of top-k latents that best differentiate positive and negative samples. Applied to linear probing and difference-in-mean, SDCV consistently improves steering success rates by 4-16% across six challenging concepts, while maintaining topic relevance.
UR - https://www.scopus.com/pages/publications/105039135975
U2 - 10.18653/v1/2026.findings-eacl.40
DO - 10.18653/v1/2026.findings-eacl.40
M3 - Conference article published in proceeding or book
AN - SCOPUS:105039135975
T3 - 19th Conference of the European Chapter of the Association for Computational Linguistics, Findings of EACL 2026
SP - 797
EP - 808
BT - 19th Conference of the European Chapter of the Association for Computational Linguistics, Findings of EACL 2026
PB - Association for Computational Linguistics (ACL)
T2 - 19th Conference of the European Chapter of the Association for Computational Linguistics, Findings of EACL 2026
Y2 - 24 March 2026 through 29 March 2026
ER -