Semanlink - Language Models + Knowledge

Parents :

Language Models + Knowledge

Amis :

Knowledge Graphs and NLP

Descendants

5 Documents (Liste longue)

[2002.01808] K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

Tags:

2023-01-12 A propos

Akari Asai sur Twitter : "Can we solely rely on LLMs’ memories (eg replace search w ChatGPT)? Probably not... Our analysis shows how retrieval is complementary to LLMs’ parametric knowledge..."

Tags:

2022-12-21 A propos

cs224n Lecture 13: Integrating Knowledge in Language Models

Tags:

2022-07-08 A propos

Language Models are Open Knowledge Graphs... but are hard to mine - Towards Data Science

Tags:

2021-01-17 A propos

[1909.04120] Span Selection Pre-training for Question Answering

Tags:

> a **new pre-training task inspired by reading
comprehension** and an **effort to avoid encoding general knowledge in the transformer network itself**

Current transformer architectures store general knowledge -> large models, long pre-training time. Better to offload the requirement of general knowledge to a sparsely activated network.

"Span selection" as an additional auxiliary task: the query is a sentence drawn from a corpus
with a term replaced with a special token: [BLANK]. The term replaced by the blank is the answer term. The passage is
relevant as determined by a BM25 search, and answer-bearing (containing the answer
term). Unlike BERT’s cloze task, where the answer must be drawn from the model itself, the answer is found in a passage
using language understanding.

> **We hope to progress to a model of general purpose language modeling that uses an indexed long
term memory to retrieve world knowledge, rather than holding it in the densely activated transformer encoder layers.**

2019-09-18 A propos

Propriétés

sl:creationDate : 2020-07-11
sl:creationTime : 2020-07-11T15:10:34Z
rdf:type : sl:Tag
skos:prefLabel : Language Models + Knowledge