Information-oriented evaluation metric for dialogue response generation systems

Peiqi Liu, Sheng Hua Zhong, Zhong Ming, Yan Liu

Research output: Chapter in book / Conference proceedingConference article published in proceeding or bookAcademic researchpeer-review

1 Citation (Scopus)

Abstract

Dialogue response generation system is one of the hot topics in natural language processing, but it is still a long way to go before it can generate human-like dialogues. A good evaluation method will help narrow the gap between the machine and human in dialogue generation. Unfortunately, current evaluation methods cannot measure whether the dialogue response generation system is able to produce high-quality, knowledge-related, and informative dialogues. Aiming to identify and measure the existence of information in dialogues, we propose a novel automatic evaluation metric. By learning from the knowledge representation method in knowledge base, we define the heuristic rules to extract the information triples from dialogue pairs. And we design an information matching method to measure the probability of the existence of information in a dialogue. In experiments, our proposed metric demonstrates its effectiveness in dialogue selection and model evaluation on the Reddit dataset (English) and the Weibo dataset (Chinese).

Original languageEnglish
Title of host publicationProceedings - 2018 IEEE 30th International Conference on Tools with Artificial Intelligence, ICTAI 2018
PublisherIEEE Computer Society
Pages780-785
Number of pages6
ISBN (Electronic)9781538674499
DOIs
Publication statusPublished - 13 Dec 2018
Event30th International Conference on Tools with Artificial Intelligence, ICTAI 2018 - Volos, Greece
Duration: 5 Nov 20187 Nov 2018

Publication series

NameProceedings - International Conference on Tools with Artificial Intelligence, ICTAI
Volume2018-November
ISSN (Print)1082-3409

Conference

Conference30th International Conference on Tools with Artificial Intelligence, ICTAI 2018
CountryGreece
CityVolos
Period5/11/187/11/18

Keywords

  • Dialogue response generation system
  • Information oriented evaluation metric
  • Knowledge base

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence
  • Computer Science Applications

Cite this