Au sujet de ce document
- sl:arxiv_author :
- sl:arxiv_firstAuthor : Michihiro Yasunaga
- sl:arxiv_num : 2210.09338
- sl:arxiv_published : 2022-10-17T18:02:52Z
- sl:arxiv_summary : Pretraining a language model (LM) on text has been shown to help various
downstream NLP tasks. Recent works show that a knowledge graph (KG) can
complement text data, offering structured background knowledge that provides a
useful scaffold for reasoning. However, these works are not pretrained to learn
a deep fusion of the two modalities at scale, limiting the potential to acquire
fully joint representations of text and KG. Here we propose DRAGON (Deep
Bidirectional Language-Knowledge Graph Pretraining), a self-supervised approach
to pretraining a deeply joint language-knowledge foundation model from text and
KG at scale. Specifically, our model takes pairs of text segments and relevant
KG subgraphs as input and bidirectionally fuses information from both
modalities. We pretrain this model by unifying two self-supervised reasoning
tasks, masked language modeling and KG link prediction. DRAGON outperforms
existing LM and LM+KG models on diverse downstream tasks including question
answering across general and biomedical domains, with +5% absolute gain on
average. In particular, DRAGON achieves notable performance on complex
reasoning about language and knowledge (+10% on questions involving long
contexts or multi-step reasoning) and low-resource QA (+8% on OBQA and
RiddleSense), and new state-of-the-art results on various BioNLP tasks. Our
code and trained models are available at
https://github.com/michiyasunaga/dragon.@en
- sl:arxiv_title : Deep Bidirectional Language-Knowledge Graph Pretraining@en
- sl:arxiv_updated : 2022-10-19T01:56:31Z
- sl:bookmarkOf : https://arxiv.org/abs/2210.09338
- sl:creationDate : 2022-10-23
- sl:creationTime : 2022-10-23T16:22:03Z
Documents with similar tags (experimental)
Préférences...