Au sujet de ce document
- sl:arxiv_author :
- sl:arxiv_firstAuthor : Ori Ram
- sl:arxiv_num : 2112.07708
- sl:arxiv_published : 2021-12-14T19:18:08Z
- sl:arxiv_summary : Dense retrievers for open-domain question answering (ODQA) have been shown to
achieve impressive performance by training on large datasets of
question-passage pairs. In this work we ask whether this dependence on labeled
data can be reduced via unsupervised pretraining that is geared towards ODQA.
We show this is in fact possible, via a novel pretraining scheme designed for
retrieval. Our \"recurring span retrieval\" approach uses recurring spans across
passages in a document to create pseudo examples for contrastive learning. Our
pretraining scheme directly controls for term overlap across pseudo queries and
relevant passages, thus allowing to model both lexical and semantic relations
between them. The resulting model, named Spider, performs surprisingly well
without any labeled training examples on a wide range of ODQA datasets.
Specifically, it significantly outperforms all other pretrained baselines in a
zero-shot setting, and is competitive with BM25, a strong sparse baseline.
Moreover, a hybrid retriever over Spider and BM25 improves over both, and is
often competitive with DPR models, which are trained on tens of thousands of
examples. Last, notable gains are observed when using Spider as an
initialization for supervised training.@en
- sl:arxiv_title : Learning to Retrieve Passages without Supervision@en
- sl:arxiv_updated : 2022-05-17T07:54:42Z
- sl:bookmarkOf : https://arxiv.org/abs/2112.07708
- sl:creationDate : 2022-08-28
- sl:creationTime : 2022-08-28T10:35:36Z
Documents with similar tags (experimental)
Préférences...