About This Document
- sl:arxiv_author :
- sl:arxiv_firstAuthor : Xiaozhi Wang
- sl:arxiv_num : 1911.06136
- sl:arxiv_published : 2019-11-13T05:21:45Z
- sl:arxiv_summary : Pre-trained language representation models (PLMs) cannot well capture factual
knowledge from text. In contrast, knowledge embedding (KE) methods can
effectively represent the relational facts in knowledge graphs (KGs) with
informative entity embeddings, but conventional KE models do not utilize the
rich text data. In this paper, we propose a unified model for Knowledge
Embedding and Pre-trained LanguagE Representation (KEPLER), which can not only
better integrate factual knowledge into PLMs but also effectively learn KE
through the abundant information in text. In KEPLER, we encode textual
descriptions of entities with a PLM as their embeddings, and then jointly
optimize the KE and language modeling objectives. Experimental results show
that KEPLER achieves state-of-the-art performance on various NLP tasks, and
also works remarkably well as an inductive KE model on the link prediction
task. Furthermore, for pre-training KEPLER and evaluating the KE performance,
we construct Wikidata5M, a large-scale KG dataset with aligned entity
descriptions, and benchmark state-of-the-art KE methods on it. It shall serve
as a new KE benchmark and facilitate the research on large KG, inductive KE,
and KG with text. The dataset can be obtained from
https://deepgraphlearning.github.io/project/wikidata5m.@en
- sl:arxiv_title : KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation@en
- sl:arxiv_updated : 2020-02-19T07:46:52Z
- sl:bookmarkOf : https://arxiv.org/abs/1911.06136
- sl:creationDate : 2020-11-03
- sl:creationTime : 2020-11-03T16:41:30Z
- sl:relatedDoc : http://www.semanlink.net/doc/2020/10/representation_learning_of_know
Documents with similar tags (experimental)