About This Document
- sl:arxiv_author :
- sl:arxiv_firstAuthor : Ruize Wang
- sl:arxiv_num : 2002.01808
- sl:arxiv_published : 2020-02-05T14:30:49Z
- sl:arxiv_summary : We study the problem of injecting knowledge into large pre-trained models
like BERT and RoBERTa. Existing methods typically update the original
parameters of pre-trained models when injecting knowledge. However, when
multiple kinds of knowledge are injected, the historically injected knowledge
would be flushed away. To address this, we propose K-Adapter, a framework that
retains the original parameters of the pre-trained model fixed and supports the
development of versatile knowledge-infused model. Taking RoBERTa as the
backbone model, K-Adapter has a neural adapter for each kind of infused
knowledge, like a plug-in connected to RoBERTa. There is no information flow
between different adapters, thus multiple adapters can be efficiently trained
in a distributed way. As a case study, we inject two kinds of knowledge in this
work, including (1) factual knowledge obtained from automatically aligned
text-triplets on Wikipedia and Wikidata and (2) linguistic knowledge obtained
via dependency parsing. Results on three knowledge-driven tasks, including
relation classification, entity typing, and question answering, demonstrate
that each adapter improves the performance and the combination of both adapters
brings further improvements. Further analysis indicates that K-Adapter captures
versatile knowledge than RoBERTa.@en
- sl:arxiv_title : K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters@en
- sl:arxiv_updated : 2020-12-28T06:07:06Z
- sl:bookmarkOf : https://arxiv.org/abs/2002.01808
- sl:creationDate : 2023-01-12
- sl:creationTime : 2023-01-12T16:20:46Z
Documents with similar tags (experimental)
Preferences...