Au sujet de ce document
- sl:arxiv_author :
- sl:arxiv_firstAuthor : Patrick Lewis
- sl:arxiv_num : 2005.11401
- sl:arxiv_published : 2020-05-22T21:34:34Z
- sl:arxiv_summary : Large pre-trained language models have been shown to store factual knowledge
in their parameters, and achieve state-of-the-art results when fine-tuned on
downstream NLP tasks. However, their ability to access and precisely manipulate
knowledge is still limited, and hence on knowledge-intensive tasks, their
performance lags behind task-specific architectures. Additionally, providing
provenance for their decisions and updating their world knowledge remain open
research problems. Pre-trained models with a differentiable access mechanism to
explicit non-parametric memory can overcome this issue, but have so far been
only investigated for extractive downstream tasks. We explore a general-purpose
fine-tuning recipe for retrieval-augmented generation (RAG) -- models which
combine pre-trained parametric and non-parametric memory for language
generation. We introduce RAG models where the parametric memory is a
pre-trained seq2seq model and the non-parametric memory is a dense vector index
of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG
formulations, one which conditions on the same retrieved passages across the
whole generated sequence, the other can use different passages per token. We
fine-tune and evaluate our models on a wide range of knowledge-intensive NLP
tasks and set the state-of-the-art on three open domain QA tasks, outperforming
parametric seq2seq models and task-specific retrieve-and-extract architectures.
For language generation tasks, we find that RAG models generate more specific,
diverse and factual language than a state-of-the-art parametric-only seq2seq
baseline.@en
- sl:arxiv_title : Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks@en
- sl:arxiv_updated : 2021-04-12T15:42:18Z
- sl:bookmarkOf : https://arxiv.org/abs/2005.11401
- sl:creationDate : 2022-01-19
- sl:creationTime : 2022-01-19T16:56:31Z
Documents with similar tags (experimental)
Préférences...