Semanlink - NLP@ENS

- the first one retrieves documents from a large source of knowledge (the retriever)
- the second one processes the support documents to solve the task (the reader).

> First the retriever selects support passages in a large knowledge
source. Then these passages are processed by the reader, along with the question, to generate an
answer

Inspired by knowledge distillation: the reader model is the teacher and the retriever is the student.

> More precisely, we use a sequence-to-sequence model as the reader, and use
the attention activations over the input documents as synthetic labels to train the retriever. 
> (**train the retriever by learning to approximate the attention score of the reader**)

Refers to:

- [REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)
- [Dehghani: Neural Ranking Models with Weak Supervision](doc:?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1704.08803)

2020-12-11 A propos

From Random Grammars to Learning Language - Département de Physique de l'Ecole Normale supérieure

Tags:

2020-09-17 A propos

[1802.07044] The Description Length of Deep Learning Models

Tags:

2019-10-11 A propos

Propriétés

sl:creationDate : 2021-04-30
sl:creationTime : 2021-04-30T23:07:57Z
rdf:type : sl:Tag
skos:prefLabel : NLP@ENS