TY - JOUR AU - AB - The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, Haifeng Wang Baidu Inc., Beijing, China {sunyu02, wangshuohuan, tianhao, wu hua, wanghaifeng}@baidu.com Abstract sentence proximity enables the models to learn structure- aware representations. And semantic similarity at the doc- Recently pre-trained models have achieved state-of-the-art ument level or discourse relations among sentences allow results in various language understanding tasks. Current pre- the models to learn semantic-aware representations. In or- training procedures usually focus on training the model with der to discover all valuable information in training corpora, several simple tasks to grasp the co-occurrence of words or be it lexical, syntactic or semantic representations, we pro- sentences. However, besides co-occurring information, there pose a continual pre-training framework named ERNIE 2.0 exists other valuable lexical, syntactic and semantic infor- which could incrementally build and train a large variety of mation in training corpora, such as named entities, semantic pre-training tasks through continual multi-task learning. closeness and discourse relations. In order to extract the lexi- cal, syntactic and semantic information from training corpora, Our ERNIE framework supports the introduction of vari- we propose a continual pre-training framework named ERNIE ous customized TI - ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding JF - Proceedings of the AAAI Conference on Artificial Intelligence DO - 10.1609/aaai.v34i05.6428 DA - 2020-04-03 UR - https://www.deepdyve.com/lp/unpaywall/ernie-2-0-a-continual-pre-training-framework-for-language-dZuDAYqTj6 DP - DeepDyve ER -