PLATO-K: Internal and External Knowledge Enhanced Dialogue Generation
Siqi Bao Huang He11 1 The DuSinc and Diamante are two companion papers of PLATO-K. Please refer to the original papers for more details. Jun Xu11 1 The DuSinc and Diamante are two companion papers of PLATO-K. Please refer to the original papers for more details. Hua Lu11 1 The DuSinc and Diamante are two companion papers of PLATO-K. Please refer to the original papers for more details. Fan Wang Hua Wu22 2 Compared to PLATO-XL, PLATO-K also makes some modifications on the model design and training process, including the model scale (11B -> 22B parameters), computation amount (150B -> 200B tokens), position encoding (naive relative -> rotary position encoding), etc. According to previous studies (Kaplan et al. 2020; Hoffmann et al. 2022; Su et al. 2021) and our preliminary experiments, these modifications also bring minor improvements. Thanks: Equal contribution. Thanks: Corresponding authors. Affiliation: Han Zhou Wenquan Wu Zheng-Yu Niu Haifeng Wang Affiliation: Baidu Inc., China Affiliation: {baosiqi, hehuang, xujun03, luhua05, wang.fan,
Abstract
Recently, the practical deployment of open-domain dialogue systems has been plagued by the knowledge issue of information deficiency and factual inaccuracy. To this end, we introduce PLATO-K based on two-stage dialogic learning to strengthen internal knowledge memorization and external knowledge exploitation. In the first stage, PLATO-K learns through massive dialogue corpora and memorizes essential knowledge into model parameters. In the second stage, PLATO-K mimics human beings to search for external information and to leverage the knowledge in response generation. Extensive experiments reveal that the knowledge issue is alleviated significantly in PLATO-K with such comprehensive internal and external knowledge enhancement. Compared to the existing state-of-the-art Chinese dialogue model, the overall engagingness of PLATO-K is improved remarkably by 36.2% and 49.2% on chit-chat and knowledge-intensive conversations.
原文 arXiv:2211.00910;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2211.00910v1