TY - GEN
T1 - Guiding symbolic natural language grammar induction via transformer-based sequence probabilities
AU - Goertzel, Ben
AU - Suárez-Madrigal, Andrés
AU - Yu, Gino
N1 - Publisher Copyright:
© Springer Nature Switzerland AG 2020.
Copyright:
Copyright 2020 Elsevier B.V., All rights reserved.
PY - 2020/7
Y1 - 2020/7
N2 - A novel approach to automated learning of syntactic rules governing natural languages is proposed, based on using probabilities assigned to sentences (and potentially longer word sequences) by transformer neural network language models to guide symbolic learning processes like clustering and rule induction. This method exploits the learned linguistic knowledge in transformers, without any reference to their inner representations; hence, the technique is readily adaptable to the continuous appearance of more powerful language models. We show a proof-of-concept example of our proposed technique, using it to guide unsupervised symbolic link-grammar induction methods drawn from our prior research.
AB - A novel approach to automated learning of syntactic rules governing natural languages is proposed, based on using probabilities assigned to sentences (and potentially longer word sequences) by transformer neural network language models to guide symbolic learning processes like clustering and rule induction. This method exploits the learned linguistic knowledge in transformers, without any reference to their inner representations; hence, the technique is readily adaptable to the continuous appearance of more powerful language models. We show a proof-of-concept example of our proposed technique, using it to guide unsupervised symbolic link-grammar induction methods drawn from our prior research.
KW - BERT
KW - Transformers
KW - Unsupervised grammar induction
UR - http://www.scopus.com/inward/record.url?scp=85088502212&partnerID=8YFLogxK
U2 - 10.1007/978-3-030-52152-3_16
DO - 10.1007/978-3-030-52152-3_16
M3 - Conference article published in proceeding or book
AN - SCOPUS:85088502212
SN - 9783030521516
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 153
EP - 163
BT - Artificial General Intelligence - 13th International Conference, AGI 2020, Proceedings
A2 - Goertzel, Ben
A2 - Potapov, Alexey
A2 - Panov, Aleksandr I.
A2 - Yampolskiy, Roman
PB - Springer Nature Switzerland AG
T2 - 13th International Conference on Artificial General Intelligence, AGI 2020
Y2 - 16 September 2020 through 19 September 2020
ER -