TY - GEN
T1 - Machine Unlearning of Pre-trained Large Language Models
AU - Yao, Jin
AU - Chien, Eli
AU - Du, Minxin
AU - Niu, Xinyao
AU - Wang, Tianhao
AU - Cheng, Zezhou
AU - Yue, Xiang
N1 - Publisher Copyright:
© 2024 Association for Computational Linguistics.
PY - 2024/8
Y1 - 2024/8
N2 - This study investigates the concept of the 'right to be forgotten' within the context of large language models (LLMs). We explore machine unlearning as a pivotal solution, with a focus on pre-trained models-a notably under-researched area. Our research delineates a comprehensive framework for machine unlearning in pretrained LLMs, encompassing a critical analysis of seven diverse unlearning methods. Through rigorous evaluation using curated datasets from arXiv, books, and GitHub, we establish a robust benchmark for unlearning performance, demonstrating that these methods are over 105 times more computationally efficient than retraining. Our results show that integrating gradient ascent with gradient descent on in-distribution data improves hyperparameter robustness. We also provide detailed guidelines for efficient hyperparameter tuning in the unlearning process. Our findings advance the discourse on ethical AI practices, offering substantive insights into the mechanics of machine unlearning for pretrained LLMs and underscoring the potential for responsible AI development.
AB - This study investigates the concept of the 'right to be forgotten' within the context of large language models (LLMs). We explore machine unlearning as a pivotal solution, with a focus on pre-trained models-a notably under-researched area. Our research delineates a comprehensive framework for machine unlearning in pretrained LLMs, encompassing a critical analysis of seven diverse unlearning methods. Through rigorous evaluation using curated datasets from arXiv, books, and GitHub, we establish a robust benchmark for unlearning performance, demonstrating that these methods are over 105 times more computationally efficient than retraining. Our results show that integrating gradient ascent with gradient descent on in-distribution data improves hyperparameter robustness. We also provide detailed guidelines for efficient hyperparameter tuning in the unlearning process. Our findings advance the discourse on ethical AI practices, offering substantive insights into the mechanics of machine unlearning for pretrained LLMs and underscoring the potential for responsible AI development.
UR - http://www.scopus.com/inward/record.url?scp=85204470303&partnerID=8YFLogxK
M3 - Conference article published in proceeding or book
AN - SCOPUS:85204470303
T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics
SP - 8403
EP - 8419
BT - Long Papers
A2 - Ku, Lun-Wei
A2 - Martins, Andre F. T.
A2 - Srikumar, Vivek
PB - Association for Computational Linguistics (ACL)
T2 - 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024
Y2 - 11 August 2024 through 16 August 2024
ER -