Toward a Unified Perspective on Parameter-Efficient Fine Tuning for Speaker Verification

Fine-tuning pre-trained Transformer models (PTMs) for speech tasks in a parameter-efficient fine-tuning (PEFT) manner can optimize memory usage while leveraging rich representations from large-scale unlabeled data. Although PEFT is effective, the interconnections between various PEFT methods are notfully understood. This paper analyzes state-of-the-art PEFT methods and introduces a unified frameworkto clarify their interrelationships. Specifically, we employ a dynamic prompts tuning strategy that selects optimal prompts from a predefined pool, ensuring each prompt is fine-tuned by its closely matched speaker. The goal is to cluster the prompts in the pool according to speaker traits, improving speaker predictionin the downstream classifier while preserving the flexibility of the pre-trained Transformers. Additionally, we integrate the mixture-of-experts (MoE) adapter into the Transformer encoders, enabling the fine-tunedPTM to select the most relevant output for extracting task-specific information. Furthermore, we improve existing PEFT techniques by incorporating spectral information from pre-trained weight matrices into the LoRA-based fine-tuning process. Extensive experiments on VoxCeleb, CN-Celeb, and CU-MARVEL demonstrate that the proposed method offers a memory- and computation-efficient solution for fine-tuning pre-trained Transformers.